diff --git a/client/internal/metrics/infra/README.md b/client/internal/metrics/infra/README.md index 5a93dbd87..7941a30cf 100644 --- a/client/internal/metrics/infra/README.md +++ b/client/internal/metrics/infra/README.md @@ -78,6 +78,25 @@ Tags: - `os`: Operating system (linux, darwin, windows, android, ios, etc.) - `arch`: CPU architecture (amd64, arm64, etc.) +### Sync Phase Timing + +Measurement: `netbird_sync_phase` + +Breaks down where time goes inside a single sync, so the total `netbird_sync` duration can be attributed to the sub-step that dominates. + +| Field | Description | +|-------|-------------| +| `duration_seconds` | Time spent in one sub-phase of sync processing | + +Tags: +- `phase`: the sub-phase — `netbird_config`, `checks`, `persist`, `dns_server`, `routes_classify`, `routes_apply`, `filtering`, `dns_forwarder`, `forward_rules`, `offline_peers`, `removed_peers`, `modified_peers`, `added_peers`, `lazy_exclude` +- `deployment_type`: "cloud" | "selfhosted" | "unknown" +- `version`: NetBird version string +- `os`: Operating system (linux, darwin, windows, android, ios, etc.) +- `arch`: CPU architecture (amd64, arm64, etc.) + +**Note:** this is wall-time per phase — it includes both CPU work and time spent waiting on locks. A slow phase points to *where* the time goes, not *why*; pair it with lock-wait metrics to tell contention apart from real work. + ### Login Duration Measurement: `netbird_login` @@ -191,4 +210,52 @@ docker compose exec influxdb influx query \ # Check ingest server health curl http://localhost:8087/health -``` \ No newline at end of file +``` + +## Analyzing a Debug Bundle + +Metrics collection is always on, so every debug bundle ships a `metrics.txt` in InfluxDB line protocol — a timestamped time series of all recorded events (sync durations, sync phases, connection stages, login). You can replay it into the local stack and graph it, without a running client. + +The bundle's `metrics.txt` is a rolling window (capped at 5 days / ~20k samples, see [Buffer Limits](#buffer-limits)). For a connection incident the relevant window is short (connection setup is seconds), so a bundle captured during the issue is enough. + +### 1. Start the stack + +```bash +# From this directory (client/internal/metrics/infra) +INFLUXDB_ADMIN_TOKEN=admin123 INFLUXDB_ADMIN_PASSWORD=admin123 GRAFANA_ADMIN_PASSWORD=admin123 \ + docker compose up -d +``` + +(`admin123` are throwaway local credentials — fine for offline analysis.) + +### 2. Clear any previous data + +So you only see this bundle: + +```bash +docker exec influxdb influx delete --org netbird --bucket metrics --token admin123 \ + --start 1970-01-01T00:00:00Z --stop 2100-01-01T00:00:00Z +``` + +### 3. Import the bundle's metrics.txt + +InfluxDB is not exposed on the host, so import inside the container: + +```bash +docker cp /path/to/bundle/metrics.txt influxdb:/tmp/m.txt +docker exec influxdb influx write --org netbird --bucket metrics --precision ns \ + --token admin123 --file /tmp/m.txt +``` + +Re-importing the same file is idempotent (same measurement+tags+timestamp overwrites). + +### 4. View the dashboards + +Grafana on http://localhost:3001 (login `admin` / `admin123`), datasource pre-provisioned: + +- **Where sync time goes:** http://localhost:3001/d/netbird-sync-phases/netbird-sync-phases-where-time-goes +- **General client metrics:** http://localhost:3001/d/netbird-influxdb-metrics + +**Set the time range** to cover the bundle's timestamps (e.g. "Last 7 days" or an absolute range matching when the bundle was taken) — with the default short range the panels look empty. + +Bundles are distinguishable by the `version` tag; add a tag at import time (e.g. `sed 's/^netbird_\([a-z_]*\),/netbird_\1,bundle=mycase,/' metrics.txt`) if you want to compare several side by side. \ No newline at end of file