--- title: "Metrics and Observability" description: "Understand metrics, traces, logs, and profiling support across Pangolin components" --- import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx"; Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments. **Newt** provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional `pprof`. **Gerbil** supports metrics through either a Prometheus backend or an OTLP backend. The **Pangolin Kubernetes Controller** exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint. This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the [community metrics guide](/self-host/community-guides/metrics). All currently documented component metrics are listed in the [Full Metric Reference](#full-metric-reference). Compare metrics, traces, logs, and profiling support across Pangolin components. Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof. Metrics-only observability using either a Prometheus or OTLP backend. Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof. ## Observability Capability Matrix | Component | Metrics | Traces | Logs | Profiling | | --- | --- | --- | --- | --- | | `newt` | Prometheus, OTLP | OTLP | stdout | pprof optional | | `gerbil` | Prometheus or OTLP | — | stdout | — | | `pangolin-kube-controller` | Prometheus scrape | — | stdout | pprof optional | ## Supported Signals | Signal | What it means | | --- | --- | | Prometheus scrape metrics | Pull-based metrics served over HTTP on `/metrics` | | OTLP metrics | Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend | | OTLP traces | Distributed traces sent to an OTel Collector or trace backend | | Application logs | stdout, file logs, audit logs, or platform logs | | Profiling | Debug endpoints such as `pprof` | ## Collection Patterns Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo. ```yaml title="prometheus.yml (fragment)" scrape_configs: - job_name: newt static_configs: - targets: ["newt:2112"] - job_name: gerbil metrics_path: /metrics static_configs: - targets: ["gerbil:3003"] - job_name: pangolin-kube-controller static_configs: - targets: ["pangolin-kube-controller:9090"] ``` ```yaml title="otel-collector.yaml" receivers: otlp: protocols: grpc: http: prometheus: config: scrape_configs: - job_name: pangolin-kube-controller static_configs: - targets: ["pangolin-kube-controller:9090"] processors: batch: {} exporters: prometheusremotewrite: endpoint: https://mimir.example.com/api/v1/push otlp/tempo: endpoint: tempo:4317 tls: insecure: true service: pipelines: metrics: receivers: [otlp, prometheus] processors: [batch] exporters: [prometheusremotewrite] traces: receivers: [otlp] processors: [batch] exporters: [otlp/tempo] ``` Today the traces pipeline is relevant only for `newt`. `pangolin-kube-controller` exposes additional OTel instruments, but they are exported on the same Prometheus scrape endpoint rather than pushed through OTLP. ```yaml title="servicemonitor.yaml" apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: pangolin-kube-controller spec: selector: matchLabels: app: pangolin-kube-controller endpoints: - port: http-metrics path: /metrics interval: 30s ``` ```yaml title="helm-values.yaml" global: metrics: enabled: true service: enabled: true serviceMonitor: enabled: true controller: monitoring: serviceMonitor: enabled: true ``` Helm values for Newt are documented on the [Kubernetes Newt configuration page](/self-host/manual/kubernetes/newt/configuration). Controller and Gerbil ServiceMonitor values are documented on the [Kubernetes Pangolin configuration page](/self-host/manual/kubernetes/pangolin/configuration). ```yaml title="promtail.yaml" server: http_listen_port: 9080 positions: filename: /tmp/positions.yaml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: pangolin static_configs: - targets: [localhost] labels: job: pangolin __path__: /var/log/containers/*pangolin*.log ``` Use Grafana Alloy, Promtail, Fluent Bit, or your preferred SIEM forwarder to collect container and application logs. ## Newt Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in [Newt metrics](#newt-metrics). ```text NEWT_METRICS_PROMETHEUS_ENABLED=true NEWT_METRICS_OTLP_ENABLED=true NEWT_ADMIN_ADDR=:2112 OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317 OTEL_EXPORTER_OTLP_INSECURE=true OTEL_METRIC_EXPORT_INTERVAL=15s NEWT_PPROF_ENABLED=false ``` ```text newt \ --metrics=true \ --otlp=true \ --metrics-admin-addr=:2112 \ --pprof=false \ --endpoint=https://pangolin.example.com \ --id=saz281jfa8z37zg \ --secret=your-secret ``` ```yaml global: metrics: enabled: true adminAddr: ":2112" service: enabled: true serviceMonitor: enabled: true pprofEnabled: false ``` - Prometheus scrape endpoint: `/metrics` on `NEWT_ADMIN_ADDR` - Health endpoint: `/healthz` on the same admin server - OTLP metrics: enabled with `NEWT_METRICS_OTLP_ENABLED=true` or `--otlp=true` - OTLP traces: initialized when OTLP export is enabled - Profiling: `/debug/pprof/*` when `NEWT_PPROF_ENABLED=true` The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains `METRICS`, but the implementation also initializes OTLP tracing when OTLP is enabled. Newt's admin server defaults to a loopback bind address. In containers or Kubernetes, set `NEWT_ADMIN_ADDR=:2112` or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it. ```yaml title="prometheus.yml (fragment)" scrape_configs: - job_name: newt static_configs: - targets: ["newt:2112"] ``` ```yaml title="otel-collector.yaml" receivers: otlp: protocols: grpc: http: processors: batch: {} exporters: prometheusremotewrite: endpoint: https://mimir.example.com/api/v1/push otlp/tempo: endpoint: tempo:4317 tls: insecure: true service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [prometheusremotewrite] traces: receivers: [otlp] processors: [batch] exporters: [otlp/tempo] ``` ## Gerbil Gerbil supports **metrics only**. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in [Gerbil metrics](#gerbil-metrics). ```text METRICS_ENABLED=true METRICS_BACKEND=prometheus METRICS_PATH=/metrics LISTEN=:3003 # OTel mode OTEL_METRICS_PROTOCOL=grpc OTEL_METRICS_ENDPOINT=otel-collector:4317 OTEL_METRICS_INSECURE=true OTEL_METRICS_EXPORT_INTERVAL=60s OTEL_METRICS_TIMEOUT=10s ``` ```text ./gerbil \ --metrics-enabled \ --metrics-backend=prometheus \ --metrics-path=/metrics \ --config=/etc/gerbil/config.json ./gerbil \ --metrics-enabled \ --metrics-backend=otel \ --otel-metrics-protocol=grpc \ --otel-metrics-endpoint=otel-collector:4317 \ --otel-metrics-insecure \ --otel-metrics-export-interval=10s \ --otel-metrics-timeout=10s \ --config=/etc/gerbil/config.json ``` - Prometheus metrics endpoint: `METRICS_PATH`, default `/metrics` - Metrics are served on Gerbil's configured HTTP listen address - The Docker Compose metrics example commonly scrapes `gerbil:3003` - Health endpoint: `/healthz` - OTLP metrics: enabled when `METRICS_BACKEND=otel` - Traces: not supported yet - Profiling: not supported yet ```yaml title="prometheus.yml (fragment)" scrape_configs: - job_name: gerbil metrics_path: /metrics static_configs: - targets: ["gerbil:3003"] ``` Replace `gerbil:3003` with the actual Gerbil HTTP listen address in your deployment. ```yaml title="otel-collector.yaml" receivers: otlp: protocols: grpc: http: processors: batch: {} exporters: prometheusremotewrite: endpoint: https://mimir.example.com/api/v1/push service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [prometheusremotewrite] ``` ## Pangolin Kubernetes Controller The controller exposes a Prometheus-compatible `/metrics` endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in [Controller metrics](#pangolin-kubernetes-controller-metrics). `pprof` is available only when explicitly enabled via `ENABLE_PPROF=true`. ```text METRICS_ADDR=:9090 DISABLE_LIVEZ=false ENABLE_PPROF=false ``` ```yaml controller: service: enabled: true port: 9090 portName: metrics monitoring: serviceMonitor: enabled: true podMonitor: enabled: false prometheusRule: enabled: false ``` Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified. - `/metrics` on `METRICS_ADDR` - `/healthz` and `/readyz` for readiness - `/livez` and `/health/live` for liveness unless `DISABLE_LIVEZ=true` - `/debug/pprof/*` when `ENABLE_PPROF=true` ```yaml title="service-and-servicemonitor.yaml" apiVersion: v1 kind: Service metadata: name: pangolin-kube-controller labels: app: pangolin-kube-controller spec: selector: app: pangolin-kube-controller ports: - name: http-metrics port: 9090 targetPort: 9090 --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: pangolin-kube-controller spec: selector: matchLabels: app: pangolin-kube-controller endpoints: - port: http-metrics path: /metrics interval: 30s ``` ## Alerting Examples ```promql increase(newt_connection_errors_total[5m]) > 10 ``` ```promql histogram_quantile( 0.95, sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le) ) > 1 ``` ```promql increase(newt_tunnel_reconnects_total[10m]) > 20 ``` ```promql max_over_time(gerbil_wg_interface_up[5m]) by (ifname) < 1 ``` ```promql sum(gerbil_wg_peers_total) > 0 and sum(gerbil_wg_peer_connected) == 0 ``` ```promql min_over_time(pangolin_kube_controller_ready[5m]) == 0 ``` ```promql increase(pangolin_kube_controller_reconcile_errors_total[10m]) > 5 ``` Traefik and metrics collection with Prometheus and Grafana. Verified Newt chart values for metrics, Services, and ServiceMonitor resources. Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources. ## Versions | Component | Signal | Since version | | --- | --- | --- | | Newt | Prometheus scrape metrics | `v1.6.0` | | Newt | OTLP metrics | `v1.6.0` | | Newt | OTLP traces | `v1.6.0` | | Newt | pprof | `v1.10.4` | | Gerbil | Prometheus scrape metrics | `v1.4.0` | | Gerbil | OTLP metrics | `v1.4.0` | | Pangolin Kubernetes Controller | Prometheus scrape metrics | `v0.1.0-alpha.1` | ## Full Metric Reference The full reference below is grouped by component. - Newt: Prometheus metrics, OTLP metrics, OTLP traces - Gerbil: Prometheus or OTLP metrics - Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics Metric names, labels, and defaults can change between component releases. ### Newt metrics OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples. Counts Pangolin registration attempts keyed by result. **Unit:** 1 **Labels:** `result` (`success`|`failure`), `site_id` **Emission path:** `telemetry.IncSiteRegistration` **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1` 0/1 heartbeat for the active site. **Unit:** 1 **Labels:** `site_id` **Emission path:** `state.TelemetryView` (callback) **Example:** `newt_site_online{site_id="self"} 1` Seconds since last Pangolin heartbeat. **Unit:** seconds **Labels:** `site_id` **Emission path:** `TouchHeartbeat` (callback) **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2` Constant 1 with build metadata labels. **Unit:** 1 **Labels:** `version`, `commit` **Emission path:** Build info registration **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1` Process boot indicator (increments once per process start). **Unit:** 1 **Labels:** — **Emission path:** `RegisterBuildInfo` **Example:** `newt_restart_count_total 1` Certificate rotation events keyed by result. **Unit:** 1 **Labels:** `result` **Emission path:** `IncCertRotation` **Example:** `newt_cert_rotation_total{result="success"} 1` Config reload attempts keyed by result. **Unit:** 1 **Labels:** `result` **Emission path:** `telemetry.IncConfigReload` **Example:** `newt_config_reloads_total{result="success"} 1` Duration per config-apply phase keyed by `phase` and `result`. **Unit:** seconds **Labels:** `phase`, `result` **Emission path:** `telemetry.ObserveConfigApply` **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3` Active sessions per tunnel (or collapsed). **Unit:** 1 **Labels:** `site_id`, `tunnel_id` **Emission path:** `RegisterStateView` **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2` Traffic per tunnel, direction, and protocol. **Unit:** bytes **Labels:** `tunnel_id`, `direction` (`ingress`|`egress`), `protocol` (`tcp`|`udp`) **Emission path:** Proxy manager **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192` RTT samples per tunnel/transport. **Unit:** seconds **Labels:** `tunnel_id`, `transport` **Emission path:** Health checks **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4` Reconnect attempts keyed by initiator & reason. **Unit:** 1 **Labels:** `tunnel_id`, `initiator` (`client`|`server`), `reason` **Emission path:** `telemetry.IncReconnect` **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3` Auth/WebSocket connection attempts keyed by transport & result. **Unit:** 1 **Labels:** `transport`, `result` **Emission path:** `telemetry.IncConnAttempt` **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2` Connection errors keyed by transport and type. **Unit:** 1 **Labels:** `transport`, `error_type` **Emission path:** `telemetry.IncConnError` **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1` Dial latency for Pangolin WebSocket. **Unit:** seconds **Labels:** `result`, `transport` **Emission path:** `ObserveWSConnectLatency` **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1` WebSocket disconnects keyed by reason. **Unit:** 1 **Labels:** `reason`, `tunnel_id` **Emission path:** `IncWSDisconnect` **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2` Ping/Pong failures observed by keepalive. **Unit:** 1 **Labels:** `reason` (e.g., `ping_write`, `pong_timeout`) **Emission path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")` **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1` Duration of established WS sessions keyed by result. **Unit:** seconds **Labels:** `result` (`success`|`error`) **Emission path:** `telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")` **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3` Current WS connection state (0/1). **Unit:** 1 **Labels:** — **Emission path:** `telemetry.SetWSConnectionState(true|false)` **Example:** `newt_websocket_connected 1` WebSocket reconnect attempts keyed by reason. **Unit:** 1 **Labels:** `reason` **Emission path:** `telemetry.IncWSReconnect(ctx, "ping_write")` **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1` In/out WS messages keyed by direction & type. **Unit:** 1 **Labels:** `direction` (`in`|`out`), `msg_type` (`ping`|`pong`|`text`|...) **Emission path:** `IncWSMessage` **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4` Active TCP/UDP proxy connections per tunnel/protocol. **Unit:** 1 **Labels:** `protocol`, `tunnel_id` **Emission path:** Proxy callback **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3` Proxy buffer pool size. **Unit:** bytes **Labels:** `protocol`, `tunnel_id` **Emission path:** Proxy callback **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240` Unflushed async byte backlog. **Unit:** bytes **Labels:** `protocol`, `tunnel_id` **Emission path:** Proxy callback **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096` Proxy write drops keyed by protocol/tunnel. **Unit:** 1 **Labels:** `protocol`, `tunnel_id` **Emission path:** `IncProxyDrops` **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2` Proxy accept events keyed by result/reason. **Unit:** 1 **Labels:** `tunnel_id`, `protocol`, `result`, `reason` **Emission path:** `telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")` **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1` Lifecycle events (opened/closed) per connection. **Unit:** 1 **Labels:** `tunnel_id`, `protocol`, `event` (`opened`|`closed`) **Emission path:** `telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)` **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1` Duration of completed proxy connections. **Unit:** seconds **Labels:** `tunnel_id`, `protocol`, `result` **Emission path:** `telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)` **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3` Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab. Counts Pangolin registration attempts keyed by result. **Labels:** `result`, `site_id` • **Unit:** 1 • **Path:** `telemetry.IncSiteRegistration` **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1` 0/1 heartbeat for the active site. **Labels:** `site_id` • **Unit:** 1 • **Path:** `state.TelemetryView` **Example:** `newt_site_online{site_id="self"} 1` Seconds since last Pangolin heartbeat. **Labels:** `site_id` • **Unit:** seconds • **Path:** `TouchHeartbeat` **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2` Constant 1 with build metadata labels. **Labels:** `version`, `commit` • **Unit:** 1 • **Path:** Build info registration **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1` Process boot indicator (increments once). **Labels:** — • **Unit:** 1 • **Path:** `RegisterBuildInfo` **Example:** `newt_restart_count_total 1` Certificate rotation events keyed by result. **Labels:** `result` • **Unit:** 1 • **Path:** `IncCertRotation` **Example:** `newt_cert_rotation_total{result="success"} 1` Config reload attempts keyed by result. **Labels:** `result` • **Unit:** 1 • **Path:** `telemetry.IncConfigReload` **Example:** `newt_config_reloads_total{result="success"} 1` Duration per config-apply phase & result. **Labels:** `phase`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveConfigApply` **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3` Active sessions per tunnel (or collapsed). **Labels:** `site_id`, `tunnel_id` • **Unit:** 1 • **Path:** `RegisterStateView` **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2` Traffic per tunnel/direction/protocol. **Labels:** `tunnel_id`, `direction`, `protocol` • **Unit:** bytes • **Path:** Proxy manager **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192` RTT samples per tunnel/transport. **Labels:** `tunnel_id`, `transport` • **Unit:** seconds • **Path:** Health checks **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4` Reconnect attempts by initiator & reason. **Labels:** `tunnel_id`, `initiator`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncReconnect` **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3` Auth/WebSocket attempts by transport & result. **Labels:** `transport`, `result` • **Unit:** 1 • **Path:** `telemetry.IncConnAttempt` **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2` Connection errors by transport and type. **Labels:** `transport`, `error_type` • **Unit:** 1 • **Path:** `telemetry.IncConnError` **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1` Dial latency for Pangolin WebSocket. **Labels:** `result`, `transport` • **Unit:** seconds • **Path:** `ObserveWSConnectLatency` **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1` WS disconnects by reason. **Labels:** `reason`, `tunnel_id` • **Unit:** 1 • **Path:** `IncWSDisconnect` **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2` Keepalive Ping/Pong failures. **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")` **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1` Duration of established WebSocket sessions by result. **Labels:** `result` • **Unit:** seconds • **Path:** `telemetry.ObserveWSSessionDuration(...)` **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3` Current WS connection status (0/1). **Labels:** — • **Unit:** 1 • **Path:** `telemetry.SetWSConnectionState(true|false)` **Example:** `newt_websocket_connected 1` Reconnect attempts by reason. **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSReconnect(ctx, "ping_write")` **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1` In/out WS messages by direction & type. **Labels:** `direction`, `msg_type` • **Unit:** 1 • **Path:** `IncWSMessage` **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4` Active TCP/UDP proxy connections per tunnel/protocol. **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** Proxy callback **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3` Proxy buffer pool size. **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240` Unflushed async byte backlog. **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096` Proxy write drops per protocol/tunnel. **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** `IncProxyDrops` **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2` Proxy accept events by result/reason. **Labels:** `tunnel_id`, `protocol`, `result`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncProxyAccept(...)` **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1` Connection lifecycle events (opened/closed). **Labels:** `tunnel_id`, `protocol`, `event` • **Unit:** 1 • **Path:** `telemetry.IncProxyConnectionEvent(...)` **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1` Duration of completed proxy connections. **Labels:** `tunnel_id`, `protocol`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveProxyConnectionDuration(...)` **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3` ### Gerbil metrics OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: `prometheus`, `otel`, or `none`. In `otel` mode, Gerbil pushes metrics to an OTLP collector. The `/metrics` endpoint is not exposed in this mode. Operational state of a WireGuard interface. **Unit:** 1 **Labels:** `ifname`, `instance` **Emission path:** `metrics.RecordInterfaceUp` **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1` Number of configured peers per interface. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordPeersTotal` **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5` Current peer connection state. **Unit:** 1 **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordPeerConnected` **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1` Number of allowed IPs configured per peer. **Unit:** 1 **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordAllowedIPsCount` **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2` Key rotation events. **Unit:** 1 **Labels:** `ifname`, `reason` **Emission path:** `metrics.RecordKeyRotation` **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1` WireGuard handshake attempts keyed by result. **Unit:** 1 **Labels:** `ifname`, `peer`, `result` **Emission path:** `metrics.RecordHandshake` **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1` Distribution of WireGuard handshake latencies. **Unit:** seconds **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordHandshakeLatency` **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3` Observed peer round-trip time. **Unit:** seconds **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordPeerRTT` **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4` Bytes received from a WireGuard peer. **Unit:** bytes **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordBytesReceived` **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192` Bytes transmitted to a WireGuard peer. **Unit:** bytes **Labels:** `ifname`, `peer` **Emission path:** `metrics.RecordBytesTransmitted` **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384` Number of active UDP relay sessions. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordActiveSession` / `metrics.RecordSession` **Example:** `gerbil_active_sessions{ifname="wg0"} 3` UDP packets processed by relay workers. **Unit:** 1 **Labels:** `ifname`, `type`, `direction` **Emission path:** `metrics.RecordUDPPacket` **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42` Size distribution of packets forwarded through the relay. **Unit:** bytes **Labels:** `ifname`, `type` **Emission path:** `metrics.RecordUDPPacketSize` **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7` Hole punch messages processed by result. **Unit:** 1 **Labels:** `ifname`, `result` **Emission path:** `metrics.RecordHolePunchEvent` **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1` Active proxy mappings. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordProxyMapping` **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4` Sessions rebuilt from communication patterns. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordSessionRebuilt` **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1` Active communication patterns. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordCommPattern` **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2` Items removed by cleanup routines. **Unit:** 1 **Labels:** `ifname`, `component` **Emission path:** `metrics.RecordProxyCleanupRemoved` **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5` Proxy connection errors. **Unit:** 1 **Labels:** `ifname`, `error_type` **Emission path:** `metrics.RecordProxyConnectionError` **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1` Initial proxy mappings loaded. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordProxyInitialMappings` **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8` Proxy mapping updates. **Unit:** 1 **Labels:** `ifname` **Emission path:** `metrics.RecordProxyMappingUpdate` **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2` Duration of idle cleanup cycles. **Unit:** seconds **Labels:** `ifname`, `component` **Emission path:** `metrics.RecordProxyIdleCleanupDuration` **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1` Active SNI proxy connections. **Unit:** 1 **Labels:** — **Emission path:** `metrics.RecordActiveProxyConnection` **Example:** `gerbil_active_proxy_connections 2` Route lookups keyed by result. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordProxyRouteLookup` **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6` TLS handshake duration for the SNI proxy. **Unit:** seconds **Labels:** — **Emission path:** `metrics.RecordProxyTLSHandshake` **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2` Bytes sent or received by the SNI proxy. **Unit:** bytes **Labels:** `direction` **Emission path:** `metrics.RecordProxyBytesTransmitted` **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384` Connections processed by the SNI proxy. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordSNIConnection` **Example:** `gerbil_sni_connections_total{result="success"} 3` Lifetime distribution of proxied TLS connections. **Unit:** seconds **Labels:** — **Emission path:** `metrics.RecordSNIConnectionDuration` **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4` Active SNI tunnels. **Unit:** 1 **Labels:** — **Emission path:** `metrics.RecordSNIActiveConnection` **Example:** `gerbil_sni_active_connections 2` SNI route cache hits and misses. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordSNIRouteCacheHit` **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10` SNI route API requests. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordSNIRouteAPIRequest` **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5` Route API call latency. **Unit:** seconds **Labels:** — **Emission path:** `metrics.RecordSNIRouteAPILatency` **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4` Routes using local overrides. **Unit:** 1 **Labels:** `hit` **Emission path:** `metrics.RecordSNILocalOverride` **Example:** `gerbil_sni_local_override_total{hit="true"} 1` PROXY protocol events. **Unit:** 1 **Labels:** `event` **Emission path:** `metrics.RecordSNITrustedProxyEvent` **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2` PROXY protocol parse failures. **Unit:** 1 **Labels:** — **Emission path:** `metrics.RecordSNIProxyProtocolParseError` **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1` Bytes proxied through SNI tunnels. **Unit:** bytes **Labels:** `direction` **Emission path:** `metrics.RecordSNIDataBytes` **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096` SNI tunnel terminations keyed by reason. **Unit:** 1 **Labels:** `reason` **Emission path:** `metrics.RecordSNITunnelTermination` **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1` HTTP requests to the management API. **Unit:** 1 **Labels:** `endpoint`, `method`, `status_code` **Emission path:** `metrics.RecordHTTPRequest` **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1` HTTP request handling time. **Unit:** seconds **Labels:** `endpoint`, `method` **Emission path:** `metrics.RecordHTTPRequestDuration` **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3` Peer lifecycle operations. **Unit:** 1 **Labels:** `operation`, `result` **Emission path:** `metrics.RecordPeerOperation` **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1` Proxy mapping update API calls. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordProxyMappingUpdateRequest` **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1` Destination update API calls. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordDestinationsUpdateRequest` **Example:** `gerbil_destinations_update_requests_total{result="success"} 1` Remote configuration fetch attempts. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordRemoteConfigFetch` **Example:** `gerbil_remote_config_fetches_total{result="success"} 1` Bandwidth report transmissions. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordBandwidthReport` **Example:** `gerbil_bandwidth_reports_total{result="success"} 1` Bytes per peer tracked by bandwidth calculation. **Unit:** bytes **Labels:** `peer`, `direction` **Emission path:** `metrics.RecordPeerBandwidthBytes` **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192` Netlink events processed. **Unit:** 1 **Labels:** `event_type` **Emission path:** `metrics.RecordNetlinkEvent` **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1` Netlink or kernel errors. **Unit:** 1 **Labels:** `component`, `error_type` **Emission path:** `metrics.RecordNetlinkError` **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1` Duration of reconciliation or sync loops. **Unit:** seconds **Labels:** `component` **Emission path:** `metrics.RecordSyncDuration` **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2` Current length of internal work queues. **Unit:** 1 **Labels:** `queue` **Emission path:** `metrics.RecordWorkqueueDepth` **Example:** `gerbil_workqueue_depth{queue="relay"} 3` Kernel module load attempts. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordKernelModuleLoad` **Example:** `gerbil_kernel_module_loads_total{result="success"} 1` Firewall rules applied. **Unit:** 1 **Labels:** `result`, `chain` **Emission path:** `metrics.RecordFirewallRuleApplied` **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1` Configuration reloads. **Unit:** 1 **Labels:** `result` **Emission path:** `metrics.RecordConfigReload` **Example:** `gerbil_config_reloads_total{result="success"} 1` Process restart count. **Unit:** 1 **Labels:** — **Emission path:** `metrics.RecordRestart` **Example:** `gerbil_restart_total 1` Authentication or peer validation failures. **Unit:** 1 **Labels:** `peer`, `reason` **Emission path:** `metrics.RecordAuthFailure` **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1` Access-control denied events. **Unit:** 1 **Labels:** `ifname`, `peer`, `policy` **Emission path:** `metrics.RecordACLDenied` **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1` Days until certificate expiry. **Unit:** days **Labels:** `cert_name`, `ifname` **Emission path:** `metrics.RecordCertificateExpiry` **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42` Memory spikes detected by severity. **Unit:** 1 **Labels:** `severity` **Emission path:** `metrics.RecordMemorySpike` **Example:** `gerbil_memory_spike_total{severity="warning"} 1` Heap profile files generated. **Unit:** 1 **Labels:** — **Emission path:** `metrics.RecordHeapProfileWritten` **Example:** `gerbil_heap_profiles_written_total 1` Prometheus-style series for the same Gerbil metrics. Gerbil exposes these only when `METRICS_BACKEND=prometheus`. In Prometheus mode, Gerbil registers a native Prometheus client and exposes the configured metrics path, default `/metrics`. In OTel mode, `/metrics` is not exposed. Operational state of a WireGuard interface. **Labels:** `ifname`, `instance` • **Unit:** 1 • **Path:** `metrics.RecordInterfaceUp` **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1` Number of configured peers per interface. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordPeersTotal` **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5` Current peer connection state. **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordPeerConnected` **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1` Number of allowed IPs configured per peer. **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordAllowedIPsCount` **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2` Key rotation events. **Labels:** `ifname`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordKeyRotation` **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1` WireGuard handshake attempts keyed by result. **Labels:** `ifname`, `peer`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHandshake` **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1` Distribution of WireGuard handshake latencies. **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordHandshakeLatency` **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3` Observed peer round-trip time. **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordPeerRTT` **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4` Bytes received from a WireGuard peer. **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesReceived` **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192` Bytes transmitted to a WireGuard peer. **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesTransmitted` **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384` Active UDP relay sessions. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordActiveSession` / `metrics.RecordSession` **Example:** `gerbil_active_sessions{ifname="wg0"} 3` UDP packets processed by relay workers. **Labels:** `ifname`, `type`, `direction` • **Unit:** 1 • **Path:** `metrics.RecordUDPPacket` **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42` Size distribution of packets forwarded through the relay. **Labels:** `ifname`, `type` • **Unit:** bytes • **Path:** `metrics.RecordUDPPacketSize` **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7` Hole punch messages processed by result. **Labels:** `ifname`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHolePunchEvent` **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1` Active proxy mappings. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMapping` **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4` Sessions rebuilt from communication patterns. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordSessionRebuilt` **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1` Active communication patterns. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordCommPattern` **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2` Items removed by cleanup routines. **Labels:** `ifname`, `component` • **Unit:** 1 • **Path:** `metrics.RecordProxyCleanupRemoved` **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5` Proxy connection errors. **Labels:** `ifname`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordProxyConnectionError` **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1` Initial proxy mappings loaded. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyInitialMappings` **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8` Proxy mapping updates. **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdate` **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2` Duration of idle cleanup cycles. **Labels:** `ifname`, `component` • **Unit:** seconds • **Path:** `metrics.RecordProxyIdleCleanupDuration` **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1` Active SNI proxy connections. **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordActiveProxyConnection` **Example:** `gerbil_active_proxy_connections 2` Route lookups keyed by result. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyRouteLookup` **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6` TLS handshake duration for the SNI proxy. **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordProxyTLSHandshake` **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2` Bytes sent or received by the SNI proxy. **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordProxyBytesTransmitted` **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384` Connections processed by the SNI proxy. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIConnection` **Example:** `gerbil_sni_connections_total{result="success"} 3` Lifetime distribution of proxied TLS connections. **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIConnectionDuration` **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4` Active SNI tunnels. **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIActiveConnection` **Example:** `gerbil_sni_active_connections 2` SNI route cache hits and misses. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteCacheHit` **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10` SNI route API requests. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteAPIRequest` **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5` Route API call latency. **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIRouteAPILatency` **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4` Routes using local overrides. **Labels:** `hit` • **Unit:** 1 • **Path:** `metrics.RecordSNILocalOverride` **Example:** `gerbil_sni_local_override_total{hit="true"} 1` PROXY protocol events. **Labels:** `event` • **Unit:** 1 • **Path:** `metrics.RecordSNITrustedProxyEvent` **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2` PROXY protocol parse failures. **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIProxyProtocolParseError` **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1` Bytes proxied through SNI tunnels. **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordSNIDataBytes` **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096` SNI tunnel terminations keyed by reason. **Labels:** `reason` • **Unit:** 1 • **Path:** `metrics.RecordSNITunnelTermination` **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1` HTTP requests to the management API. **Labels:** `endpoint`, `method`, `status_code` • **Unit:** 1 • **Path:** `metrics.RecordHTTPRequest` **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1` HTTP request handling time. **Labels:** `endpoint`, `method` • **Unit:** seconds • **Path:** `metrics.RecordHTTPRequestDuration` **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3` Peer lifecycle operations. **Labels:** `operation`, `result` • **Unit:** 1 • **Path:** `metrics.RecordPeerOperation` **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1` Proxy mapping update API calls. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdateRequest` **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1` Destination update API calls. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordDestinationsUpdateRequest` **Example:** `gerbil_destinations_update_requests_total{result="success"} 1` Remote configuration fetch attempts. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordRemoteConfigFetch` **Example:** `gerbil_remote_config_fetches_total{result="success"} 1` Bandwidth report transmissions. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordBandwidthReport` **Example:** `gerbil_bandwidth_reports_total{result="success"} 1` Bytes per peer tracked by bandwidth calculation. **Labels:** `peer`, `direction` • **Unit:** bytes • **Path:** `metrics.RecordPeerBandwidthBytes` **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192` Netlink events processed. **Labels:** `event_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkEvent` **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1` Netlink or kernel errors. **Labels:** `component`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkError` **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1` Duration of reconciliation or sync loops. **Labels:** `component` • **Unit:** seconds • **Path:** `metrics.RecordSyncDuration` **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2` Current length of internal work queues. **Labels:** `queue` • **Unit:** 1 • **Path:** `metrics.RecordWorkqueueDepth` **Example:** `gerbil_workqueue_depth{queue="relay"} 3` Kernel module load attempts. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordKernelModuleLoad` **Example:** `gerbil_kernel_module_loads_total{result="success"} 1` Firewall rules applied. **Labels:** `result`, `chain` • **Unit:** 1 • **Path:** `metrics.RecordFirewallRuleApplied` **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1` Configuration reloads. **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordConfigReload` **Example:** `gerbil_config_reloads_total{result="success"} 1` Process restart count. **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordRestart` **Example:** `gerbil_restart_total 1` Authentication or peer validation failures. **Labels:** `peer`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordAuthFailure` **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1` Access-control denied events. **Labels:** `ifname`, `peer`, `policy` • **Unit:** 1 • **Path:** `metrics.RecordACLDenied` **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1` Days until certificate expiry. **Labels:** `cert_name`, `ifname` • **Unit:** days • **Path:** `metrics.RecordCertificateExpiry` **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42` Memory spikes detected by severity. **Labels:** `severity` • **Unit:** 1 • **Path:** `metrics.RecordMemorySpike` **Example:** `gerbil_memory_spike_total{severity="warning"} 1` Heap profile files generated. **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordHeapProfileWritten` **Example:** `gerbil_heap_profiles_written_total 1` ### Pangolin Kubernetes Controller metrics Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller. The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same `/metrics` endpoint. The OTel-backed series use the `pangolin_controller_*` prefix. Duration of each reconcile phase. **Unit:** seconds **Labels:** `phase`, `result` **Label values:** `phase`: `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp` `result`: `success` | `error` **Emission path:** OTel reconcile phase instrumentation **Example:** `pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3` Number of active reconcile routines by phase. **Unit:** 1 **Labels:** `phase` **Label values:** `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp` **Emission path:** Parallel reconcile instrumentation **Example:** `pangolin_controller_active_reconcile_routines{phase="routers"} 1` Controller loop iterations by outcome. **Unit:** 1 **Labels:** `outcome` **Label values:** `success` | `nochange` | `error` **Emission path:** Controller loop instrumentation **Example:** `pangolin_controller_loop_iterations_total{outcome="success"} 10` Duration of remote fetch cycle HTTP requests. **Unit:** seconds **Labels:** `status_code`, `status_class` **Label values:** `status_code`: `200` | `304` | `401` | `403` | `404` | `5xx` `status_class`: `2xx` | `3xx` | `4xx` | `5xx` **Emission path:** Remote config fetch instrumentation **Example:** `pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4` Duration of configuration parsing. **Unit:** seconds **Labels:** `section` **Label values:** `full` **Emission path:** Config parse instrumentation **Example:** `pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2` Duration of Kubernetes API requests. **Unit:** seconds **Labels:** `verb`, `resource_kind`, `result`, `forced` **Label values:** `verb`: `get` | `create` | `patch` | `update` | `delete` | `list` `resource_kind`: `IngressRoute` | `Middleware` | `TraefikService` | `ServersTransport` | `ServersTransportTCP` | `Service` | `EndpointSlice` `result`: `success` | `error` | `conflict` `forced`: `true` | `false` **Emission path:** Kubernetes API request instrumentation **Example:** `pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3` Total Kubernetes API requests. **Unit:** 1 **Labels:** `verb`, `resource_kind`, `result`, `forced` **Emission path:** Kubernetes API request instrumentation **Example:** `pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1` Retry attempts in the SSA apply loop. **Unit:** 1 **Labels:** `reason`, `operation`, `resource_kind` **Label values:** `reason`: `conflict` | `transient` | `timeout` `operation`: `get` | `create` | `patch` | `delete` | `apply` **Emission path:** SSA apply retry instrumentation **Example:** `pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1` Duration of garbage collection runs. **Unit:** seconds **Labels:** `result` **Label values:** `success` | `fail` | `dryrun` **Emission path:** GC instrumentation **Example:** `pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1` Prometheus-native metrics exposed by the Pangolin Kubernetes Controller. These use the `pangolin_kube_controller_*` prefix. The metrics endpoint is exposed at `GET /metrics` on `METRICS_ADDR`, default `:9090`. The same HTTP server also exposes `/healthz`, `/readyz`, and optionally `/debug/pprof/` when pprof is enabled. Duration of a full successful reconcile loop. **Labels:** — **Unit:** seconds **Path:** `Collector.ReconcileDuration` **Example:** `pangolin_kube_controller_reconcile_seconds_bucket{le="0.5"} 3` Total errors during reconcile steps. **Labels:** — **Unit:** 1 **Path:** `Collector.ReconcileErrors` **Example:** `pangolin_kube_controller_reconcile_errors_total 2` Number of consecutive reconcile or fetch errors. **Labels:** — **Unit:** 1 **Path:** `Collector.ConsecutiveErrors` **Example:** `pangolin_kube_controller_consecutive_errors 0` Unix timestamp of the last successful fetch. **Labels:** — **Unit:** seconds since Unix epoch **Path:** `Collector.LastFetchSuccess` **Example:** `pangolin_kube_controller_last_fetch_success_timestamp_seconds 1767225600` Controller readiness state. **Labels:** — **Unit:** 1 **Value:** `1` = ready, `0` = not ready **Path:** `Collector.Ready` **Example:** `pangolin_kube_controller_ready 1` Applied objects by kind and action. **Labels:** `kind`, `action` **Label values:** `kind`: `IngressRoute` | `Middleware` | `TraefikService` | other managed Traefik CRDs `action`: `create` | `patch` **Unit:** 1 **Path:** `Collector.AppliedObjects` **Example:** `pangolin_kube_controller_objects_applied_total{kind="IngressRoute",action="patch"} 1` Deleted objects by kind. **Labels:** `kind` **Unit:** 1 **Path:** `Collector.DeletedObjects` **Example:** `pangolin_kube_controller_objects_deleted_total{kind="Middleware"} 1` Desired objects count by kind from the last processed config. **Labels:** `kind` **Unit:** 1 **Path:** `Collector.DesiredObjects` **Example:** `pangolin_kube_controller_desired_objects_count{kind="IngressRoute"} 12` Garbage-collected objects by kind and reason. **Labels:** `kind`, `reason` **Label values:** `reason`: `immediate` | `grace` **Unit:** 1 **Path:** `Collector.GCDeletedTotal` **Example:** `pangolin_kube_controller_gc_deleted_total{kind="IngressRoute",reason="grace"} 1` Garbage collection runs by result. **Labels:** `result` **Label values:** `start` | `success` | `fail` | `dryrun` **Unit:** 1 **Path:** `Collector.GCRunsTotal` **Example:** `pangolin_kube_controller_gc_runs_total{result="success"} 1` Grace deletion queue items dropped due to queue overflow. **Labels:** — **Unit:** 1 **Path:** `Collector.GraceQueueDropped` **Example:** `pangolin_kube_controller_grace_queue_dropped_total 0` Current depth of the grace deletion queue. **Labels:** — **Unit:** 1 **Path:** `Collector.GraceQueueDepth` **Example:** `pangolin_kube_controller_grace_queue_depth 3` Successful Traefik instance label resolutions. **Labels:** — **Unit:** 1 **Path:** `Collector.InstanceLabelDetectSuccess` **Example:** `pangolin_kube_controller_instance_label_detect_success_total 1` Failed Traefik instance label resolutions or verifications. **Labels:** — **Unit:** 1 **Path:** `Collector.InstanceLabelDetectFailures` **Example:** `pangolin_kube_controller_instance_label_detect_failure_total 1` Unix timestamp of the last instance label verification. **Labels:** — **Unit:** seconds since Unix epoch **Path:** `Collector.InstanceLabelLastCheck` **Example:** `pangolin_kube_controller_instance_label_last_check_timestamp_seconds 1767225600` Current leader election state of the controller. **Labels:** — **Unit:** 1 **Value:** `1` = leader, `0` = follower, `-1` = leader election disabled **Path:** `Collector.LeaderState` **Example:** `pangolin_kube_controller_leader_state 1` Go runtime metrics registered by the Prometheus Go collector. **Labels:** varies **Unit:** varies **Path:** `collectors.NewGoCollector()` **Example:** `go_goroutines 14` Process metrics registered by the Prometheus process collector. **Labels:** varies **Unit:** varies **Path:** `collectors.NewProcessCollector()` **Example:** `process_resident_memory_bytes 52428800` --- ## References * OpenTelemetry Documentation * Prometheus Documentation Have improvements or a missing metric? Open an issue or PR referencing this page.