30 lines
1.2 KiB
Markdown
30 lines
1.2 KiB
Markdown
# Prometheus / Grafana Monitoring
|
|
|
|
The `monitoring` profile starts:
|
|
- Prometheus 3.14.0
|
|
- Grafana 13.2.0
|
|
- node-exporter 1.12.1 on the Master
|
|
|
|
Prometheus also scrapes the CPU/GPU subagent node-exporters and NVIDIA DCGM exporter using targets from the Master `.env`. NeuroForge `/metrics` is authenticated with the metrics bearer token; an init container writes that token into a Docker volume instead of embedding it in the Prometheus config.
|
|
|
|
Provisioned dashboard panels include:
|
|
- Master up, memories, synapses
|
|
- online CPU/GPU workers
|
|
- durable jobs by status
|
|
- inflight/capacity by worker
|
|
- linked/isolated/multi-linked memories
|
|
- graph degree/components/largest component
|
|
- HNSW and Disk-PQ population
|
|
- HTTP request rate and P95 latency
|
|
- Master heap
|
|
- Master/CPU/GPU host CPU and memory
|
|
- NVIDIA GPU utilization, framebuffer and power (when DCGM exporter is enabled)
|
|
|
|
The bundled Prometheus alert rules cover Master down, missing CPU/GPU workers, failed jobs, stuck `apply_wait`, large queue, and a persistently highly-isolated graph.
|
|
|
|
Validation:
|
|
```bash
|
|
./validate.sh
|
|
```
|
|
This renders the Prometheus template with the targets from the Master `.env` and runs the pinned Prometheus image's own `promtool check config`.
|