1.2 KiB
1.2 KiB
Prometheus / Grafana Monitoring
The monitoring profile starts:
- Prometheus 3.14.0
- Grafana 13.2.0
- node-exporter 1.12.1 on the Master
Prometheus also scrapes the CPU/GPU subagent node-exporters and NVIDIA DCGM exporter using targets from the Master .env. NeuroForge /metrics is authenticated with the metrics bearer token; an init container writes that token into a Docker volume instead of embedding it in the Prometheus config.
Provisioned dashboard panels include:
- Master up, memories, synapses
- online CPU/GPU workers
- durable jobs by status
- inflight/capacity by worker
- linked/isolated/multi-linked memories
- graph degree/components/largest component
- HNSW and Disk-PQ population
- HTTP request rate and P95 latency
- Master heap
- Master/CPU/GPU host CPU and memory
- NVIDIA GPU utilization, framebuffer and power (when DCGM exporter is enabled)
The bundled Prometheus alert rules cover Master down, missing CPU/GPU workers, failed jobs, stuck apply_wait, large queue, and a persistently highly-isolated graph.
Validation:
./validate.sh
This renders the Prometheus template with the targets from the Master .env and runs the pinned Prometheus image's own promtool check config.