fix(metrics): enhance documentation clarity and structure for metrics recommendations

docs/METRICS_RECOMMENDATIONS.md

@@ -3,6 +3,7 @@
 This document captures the current state of Newt metrics, prioritized fixes, and a pragmatic roadmap for near-term improvements.
 
 1) Current setup (summary)
+
    - Export: Prometheus exposition (default), optional OTLP (gRPC)
    - Existing instruments:
      - Sites: newt_site_registrations_total, newt_site_online (0/1), newt_site_last_heartbeat_seconds
@@ -12,6 +13,7 @@ This document captures the current state of Newt metrics, prioritized fixes, and
      - Go runtime: GC, heap, goroutines via runtime instrumentation
 
 2) Main issues addressed now
+
    - Attribute filter (allow-list) extended to include site_id and region in addition to existing keys (tunnel_id, transport, protocol, direction, result, reason, error_type, version, commit).
    - site_id and region propagation: site_id is now attached as a metric label across newt_*; region is added as a metric label when set. Both remain resource attributes for consistency with OTEL.
    - Label semantics clarified:
@@ -21,11 +23,13 @@ This document captures the current state of Newt metrics, prioritized fixes, and
    - Robustness improvements: removed duplicate clear logic on reconnect; avoided empty site_id by reading NEWT_SITE_ID/NEWT_ID and OTEL_RESOURCE_ATTRIBUTES.
 
 3) Remaining gaps and deviations
+
    - Some call sites still need initiator label on reconnect outcomes (client vs server). This is planned.
    - WebSocket and Proxy metrics (connect latency, messages, active connections, buffer/drops, async backlog) are planned additions.
    - Config apply duration and cert rotation counters are planned.
 
 4) Roadmap (phased)
+
    - Phase 1 (done in this iteration)
      - Fix attribute filter (site_id, region)
      - Propagate site_id (and optional region) across metrics
@@ -38,11 +42,13 @@ This document captures the current state of Newt metrics, prioritized fixes, and
      - Config & PKI: newt_config_apply_seconds{phase,result}; newt_cert_rotation_total{result}
 
 5) Operational guidance
+
    - Do not double scrape: scrape either Newt (/metrics) or the Collector’s Prometheus exporter (not both) to avoid double-counting cumulative counters.
    - For high cardinality tunnel_id, consider relabeling or dropping per-tunnel series in Prometheus to control cardinality.
    - OTLP troubleshooting: enable TLS via OTEL_EXPORTER_OTLP_CERTIFICATE, use OTEL_EXPORTER_OTLP_HEADERS for auth; verify endpoint reachability.
 
 6) Example alerts/recording rules (suggestions)
+
    - Reconnect spikes:
      - increase(newt_tunnel_reconnects_total[5m]) by (site_id)
    - Sustained connection errors:
@@ -55,12 +61,12 @@ This document captures the current state of Newt metrics, prioritized fixes, and
      - histogram_quantile(0.95, sum(rate(newt_websocket_connect_latency_seconds_bucket[5m])) by (le,site_id))
 
 7) Collector configuration
+
    - Direct scrape variant requires no attribute promotion since site_id is already a metric label.
    - Transform/promote variant remains optional for environments that rely on resource-to-label promotion.
 
 8) Testing
+
 - curl :2112/metrics | grep ^newt_
 - Verify presence of site_id across series; region appears when set.
 - Ensure disallowed attributes are filtered; allowed (site_id) retained.
-
-