---
title: "Metrics and Observability"
description: "Understand metrics, traces, logs, and profiling support across Pangolin components"
---

import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx";

<PangolinCloudTocCta />

Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments.

**Newt** provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional `pprof`. **Gerbil** supports metrics through either a Prometheus backend or an OTLP backend. The **Pangolin Kubernetes Controller** exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint.

<Note>
This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the [community metrics guide](/self-host/community-guides/metrics).

All currently documented component metrics are listed in the [Full Metric Reference](#full-metric-reference).
</Note>

<CardGroup cols={2}>
  <Card title="Capability Matrix" icon="chart-line" href="#observability-capability-matrix">
    Compare metrics, traces, logs, and profiling support across Pangolin components.
  </Card>
  <Card title="Newt" icon="route" href="#newt">
    Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof.
  </Card>
  <Card title="Gerbil" icon="network-wired" href="#gerbil">
    Metrics-only observability using either a Prometheus or OTLP backend.
  </Card>
  <Card title="Kubernetes Controller" icon="cubes" href="#pangolin-kubernetes-controller">
    Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof.
  </Card>
</CardGroup>

## Observability Capability Matrix

| Component | Metrics | Traces | Logs | Profiling |
| --- | --- | --- | --- | --- |
| `newt` | Prometheus, OTLP | OTLP | stdout | pprof optional |
| `gerbil` | Prometheus or OTLP | — | stdout | — |
| `pangolin-kube-controller` | Prometheus scrape | — | stdout | pprof optional |

## Supported Signals

| Signal | What it means |
| --- | --- |
| Prometheus scrape metrics | Pull-based metrics served over HTTP on `/metrics` |
| OTLP metrics | Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend |
| OTLP traces | Distributed traces sent to an OTel Collector or trace backend |
| Application logs | stdout, file logs, audit logs, or platform logs |
| Profiling | Debug endpoints such as `pprof` |

## Collection Patterns

Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo.

<Tabs>
  <Tab title="Prometheus scrape mode">

```yaml title="prometheus.yml (fragment)"
scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

  - job_name: pangolin-kube-controller
    static_configs:
      - targets: ["pangolin-kube-controller:9090"]
```

  </Tab>
  <Tab title="OTel Collector">

```yaml title="otel-collector.yaml"
receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: pangolin-kube-controller
          static_configs:
            - targets: ["pangolin-kube-controller:9090"]

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
```

<Tip>
Today the traces pipeline is relevant only for `newt`. `pangolin-kube-controller` exposes additional OTel instruments, but they are exported on the same Prometheus scrape endpoint rather than pushed through OTLP.
</Tip>

  </Tab>
  <Tab title="Kubernetes ServiceMonitor">

```yaml title="servicemonitor.yaml"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s
```

```yaml title="helm-values.yaml"
global:
  metrics:
    enabled: true
    service:
      enabled: true
    serviceMonitor:
      enabled: true

controller:
  monitoring:
    serviceMonitor:
      enabled: true
```

<Note>
Helm values for Newt are documented on the [Kubernetes Newt configuration page](/self-host/manual/kubernetes/newt/configuration). Controller and Gerbil ServiceMonitor values are documented on the [Kubernetes Pangolin configuration page](/self-host/manual/kubernetes/pangolin/configuration).
</Note>

  </Tab>
  <Tab title="Logs to Loki">

```yaml title="promtail.yaml"
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: pangolin
    static_configs:
      - targets: [localhost]
        labels:
          job: pangolin
          __path__: /var/log/containers/*pangolin*.log
```

Use Grafana Alloy, Promtail, Fluent Bit, or your preferred SIEM forwarder to collect container and application logs.

  </Tab>
</Tabs>

## Newt

Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in [Newt metrics](#newt-metrics).

<AccordionGroup>
<Accordion title="Configuration">
<Tabs>
  <Tab title="Environment Variables">
    ```text
    NEWT_METRICS_PROMETHEUS_ENABLED=true
    NEWT_METRICS_OTLP_ENABLED=true
    NEWT_ADMIN_ADDR=:2112
    OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    OTEL_EXPORTER_OTLP_INSECURE=true
    OTEL_METRIC_EXPORT_INTERVAL=15s
    NEWT_PPROF_ENABLED=false
    ```
  </Tab>
  <Tab title="CLI Args">
    ```text
    newt \
      --metrics=true \
      --otlp=true \
      --metrics-admin-addr=:2112 \
      --pprof=false \
      --endpoint=https://pangolin.example.com \
      --id=saz281jfa8z37zg \
      --secret=your-secret
    ```
  </Tab>
  <Tab title="Helm Values">
    ```yaml
    global:
      metrics:
        enabled: true
        adminAddr: ":2112"
        service:
          enabled: true
        serviceMonitor:
          enabled: true
        pprofEnabled: false
    ```
  </Tab>
</Tabs>
</Accordion>

<Accordion title="Signals and endpoints">

- Prometheus scrape endpoint: `/metrics` on `NEWT_ADMIN_ADDR`
- Health endpoint: `/healthz` on the same admin server
- OTLP metrics: enabled with `NEWT_METRICS_OTLP_ENABLED=true` or `--otlp=true`
- OTLP traces: initialized when OTLP export is enabled
- Profiling: `/debug/pprof/*` when `NEWT_PPROF_ENABLED=true`

<Note>
The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains `METRICS`, but the implementation also initializes OTLP tracing when OTLP is enabled.
</Note>

<Warning>
Newt's admin server defaults to a loopback bind address. In containers or Kubernetes, set `NEWT_ADMIN_ADDR=:2112` or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it.
</Warning>

</Accordion>

<Accordion title="Examples">
<Tabs>
  <Tab title="Direct Prometheus">
    ```yaml title="prometheus.yml (fragment)"
    scrape_configs:
      - job_name: newt
        static_configs:
          - targets: ["newt:2112"]
    ```
  </Tab>
  <Tab title="OTLP metrics and traces">
    ```yaml title="otel-collector.yaml"
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch: {}

    exporters:
      prometheusremotewrite:
        endpoint: https://mimir.example.com/api/v1/push
      otlp/tempo:
        endpoint: tempo:4317
        tls:
          insecure: true

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/tempo]
    ```
  </Tab>
</Tabs>
</Accordion>
</AccordionGroup>

## Gerbil

Gerbil supports **metrics only**. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in [Gerbil metrics](#gerbil-metrics).

<AccordionGroup>
<Accordion title="Configuration">
<Tabs>
  <Tab title="Environment Variables">
    ```text
    METRICS_ENABLED=true
    METRICS_BACKEND=prometheus
    METRICS_PATH=/metrics
    LISTEN=:3003

    # OTel mode
    OTEL_METRICS_PROTOCOL=grpc
    OTEL_METRICS_ENDPOINT=otel-collector:4317
    OTEL_METRICS_INSECURE=true
    OTEL_METRICS_EXPORT_INTERVAL=60s
    OTEL_METRICS_TIMEOUT=10s
    ```
  </Tab>
  <Tab title="CLI Args">
    ```text
    ./gerbil \
      --metrics-enabled \
      --metrics-backend=prometheus \
      --metrics-path=/metrics \
      --config=/etc/gerbil/config.json

    ./gerbil \
      --metrics-enabled \
      --metrics-backend=otel \
      --otel-metrics-protocol=grpc \
      --otel-metrics-endpoint=otel-collector:4317 \
      --otel-metrics-insecure \
      --otel-metrics-export-interval=10s \
      --otel-metrics-timeout=10s \
      --config=/etc/gerbil/config.json
    ```
  </Tab>
</Tabs>
</Accordion>

<Accordion title="Signals and endpoints">

- Prometheus metrics endpoint: `METRICS_PATH`, default `/metrics`
- Metrics are served on Gerbil's configured HTTP listen address
- The Docker Compose metrics example commonly scrapes `gerbil:3003`
- Health endpoint: `/healthz`
- OTLP metrics: enabled when `METRICS_BACKEND=otel`
- Traces: not supported yet
- Profiling: not supported yet

</Accordion>

<Accordion title="Examples">
<Tabs>
  <Tab title="Prometheus scrape">
    ```yaml title="prometheus.yml (fragment)"
    scrape_configs:
      - job_name: gerbil
        metrics_path: /metrics
        static_configs:
          - targets: ["gerbil:3003"]
    ```

    <Note>
    Replace `gerbil:3003` with the actual Gerbil HTTP listen address in your deployment.
    </Note>
  </Tab>
  <Tab title="OTLP metrics">
    ```yaml title="otel-collector.yaml"
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch: {}

    exporters:
      prometheusremotewrite:
        endpoint: https://mimir.example.com/api/v1/push

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]
    ```
  </Tab>
</Tabs>
</Accordion>
</AccordionGroup>

## Pangolin Kubernetes Controller

The controller exposes a Prometheus-compatible `/metrics` endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in [Controller metrics](#pangolin-kubernetes-controller-metrics).

<Note>
`pprof` is available only when explicitly enabled via `ENABLE_PPROF=true`.
</Note>

<AccordionGroup>
<Accordion title="Configuration">
<Tabs>
  <Tab title="Environment Variables">
    ```text
    METRICS_ADDR=:9090
    DISABLE_LIVEZ=false
    ENABLE_PPROF=false
    ```
  </Tab>
  <Tab title="Helm Values">
    ```yaml
    controller:
      service:
        enabled: true
        port: 9090
        portName: metrics
      monitoring:
        serviceMonitor:
          enabled: true
        podMonitor:
          enabled: false
        prometheusRule:
          enabled: false
    ```
  </Tab>
</Tabs>

<Warning>
Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified.
</Warning>
</Accordion>

<Accordion title="Endpoints">

- `/metrics` on `METRICS_ADDR`
- `/healthz` and `/readyz` for readiness
- `/livez` and `/health/live` for liveness unless `DISABLE_LIVEZ=true`
- `/debug/pprof/*` when `ENABLE_PPROF=true`

</Accordion>

<Accordion title="Representative metrics and Kubernetes examples">

```yaml title="service-and-servicemonitor.yaml"
apiVersion: v1
kind: Service
metadata:
  name: pangolin-kube-controller
  labels:
    app: pangolin-kube-controller
spec:
  selector:
    app: pangolin-kube-controller
  ports:
    - name: http-metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s
```

</Accordion>
</AccordionGroup>

## Alerting Examples

<Tabs>
  <Tab title="Newt">

```promql
increase(newt_connection_errors_total[5m]) > 10
```

```promql
histogram_quantile(
  0.95,
  sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le)
) > 1
```

```promql
increase(newt_tunnel_reconnects_total[10m]) > 20
```

  </Tab>
  <Tab title="Gerbil">

```promql
max_over_time(gerbil_wg_interface_up[5m]) by (ifname) < 1
```

```promql
sum(gerbil_wg_peers_total) > 0
and
sum(gerbil_wg_peer_connected) == 0
```

  </Tab>
  <Tab title="Controller">

```promql
min_over_time(pangolin_kube_controller_ready[5m]) == 0
```

```promql
increase(pangolin_kube_controller_reconcile_errors_total[10m]) > 5
```

  </Tab>
</Tabs>

<CardGroup cols={2}>
  <Card title="Community Metrics Guide" icon="chart-simple" href="/self-host/community-guides/metrics">
    Traefik and metrics collection with Prometheus and Grafana.
  </Card>
  <Card title="Newt Kubernetes Monitoring" icon="cubes" href="/self-host/manual/kubernetes/newt/configuration">
    Verified Newt chart values for metrics, Services, and ServiceMonitor resources.
  </Card>
  <Card title="Controller Monitoring Values" icon="cubes" href="/self-host/manual/kubernetes/pangolin/configuration">
    Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources.
  </Card>
</CardGroup>

## Versions

| Component | Signal | Since version |
| --- | --- | --- |
| Newt | Prometheus scrape metrics | `v1.6.0` |
| Newt | OTLP metrics | `v1.6.0` |
| Newt | OTLP traces | `v1.6.0` |
| Newt | pprof | `v1.10.4` |
| Gerbil | Prometheus scrape metrics | `v1.4.0` |
| Gerbil | OTLP metrics | `v1.4.0` |
| Pangolin Kubernetes Controller | Prometheus scrape metrics | `v0.1.0-alpha.1` |

## Full Metric Reference

The full reference below is grouped by component.

- Newt: Prometheus metrics, OTLP metrics, OTLP traces
- Gerbil: Prometheus or OTLP metrics
- Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics

<Info>
Metric names, labels, and defaults can change between component releases.
</Info>

### Newt metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="newt">
      OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.

      <Expandable title="Site & Build">
        <ResponseField name="newt_site_registrations_total" type="Counter">
          Counts Pangolin registration attempts keyed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result` (`success`|`failure`), `site_id`  
            **Emission path:** `telemetry.IncSiteRegistration`  
            **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_online" type="ObservableGauge">
          0/1 heartbeat for the active site.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `site_id`  
            **Emission path:** `state.TelemetryView` (callback)  
            **Example:** `newt_site_online{site_id="self"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_last_heartbeat_seconds" type="ObservableGauge">
          Seconds since last Pangolin heartbeat.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `site_id`  
            **Emission path:** `TouchHeartbeat` (callback)  
            **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_build_info" type="ObservableGauge">
          Constant 1 with build metadata labels.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `version`, `commit`  
            **Emission path:** Build info registration  
            **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_restart_count_total" type="Counter">
          Process boot indicator (increments once per process start).
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `RegisterBuildInfo`  
            **Example:** `newt_restart_count_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_cert_rotation_total" type="Counter">
          Certificate rotation events keyed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `IncCertRotation`  
            **Example:** `newt_cert_rotation_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_reloads_total" type="Counter">
          Config reload attempts keyed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `telemetry.IncConfigReload`  
            **Example:** `newt_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_apply_seconds" type="Histogram (s)">
          Duration per config-apply phase keyed by `phase` and `result`.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `phase`, `result`  
            **Emission path:** `telemetry.ObserveConfigApply`  
            **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Tunnel">
        <ResponseField name="newt_tunnel_sessions" type="ObservableGauge">
          Active sessions per tunnel (or collapsed).
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `site_id`, `tunnel_id`  
            **Emission path:** `RegisterStateView`  
            **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_bytes_total" type="Counter (bytes)">
          Traffic per tunnel, direction, and protocol.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `tunnel_id`, `direction` (`ingress`|`egress`), `protocol` (`tcp`|`udp`)  
            **Emission path:** Proxy manager  
            **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_latency_seconds" type="Histogram (s)">
          RTT samples per tunnel/transport.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `tunnel_id`, `transport`  
            **Emission path:** Health checks  
            **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_reconnects_total" type="Counter">
          Reconnect attempts keyed by initiator & reason.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `tunnel_id`, `initiator` (`client`|`server`), `reason`  
            **Emission path:** `telemetry.IncReconnect`  
            **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Connection & Auth">
        <ResponseField name="newt_connection_attempts_total" type="Counter">
          Auth/WebSocket connection attempts keyed by transport & result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `transport`, `result`  
            **Emission path:** `telemetry.IncConnAttempt`  
            **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_connection_errors_total" type="Counter">
          Connection errors keyed by transport and type.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `transport`, `error_type`  
            **Emission path:** `telemetry.IncConnError`  
            **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="WebSocket">
        <ResponseField name="newt_websocket_connect_latency_seconds" type="Histogram (s)">
          Dial latency for Pangolin WebSocket.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `result`, `transport`  
            **Emission path:** `ObserveWSConnectLatency`  
            **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_disconnects_total" type="Counter">
          WebSocket disconnects keyed by reason.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `reason`, `tunnel_id`  
            **Emission path:** `IncWSDisconnect`  
            **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_keepalive_failures_total" type="Counter">
          Ping/Pong failures observed by keepalive.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `reason` (e.g., `ping_write`, `pong_timeout`)  
            **Emission path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")`  
            **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_session_duration_seconds" type="Histogram (s)">
          Duration of established WS sessions keyed by result.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `result` (`success`|`error`)  
            **Emission path:** `telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")`  
            **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_connected" type="ObservableGauge">
          Current WS connection state (0/1).
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `telemetry.SetWSConnectionState(true|false)`  
            **Example:** `newt_websocket_connected 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_reconnects_total" type="Counter">
          WebSocket reconnect attempts keyed by reason.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `reason`  
            **Emission path:** `telemetry.IncWSReconnect(ctx, "ping_write")`  
            **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_messages_total" type="Counter">
          In/out WS messages keyed by direction & type.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `direction` (`in`|`out`), `msg_type` (`ping`|`pong`|`text`|...)  
            **Emission path:** `IncWSMessage`  
            **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Proxy">
        <ResponseField name="newt_proxy_active_connections" type="ObservableGauge">
          Active TCP/UDP proxy connections per tunnel/protocol.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `protocol`, `tunnel_id`  
            **Emission path:** Proxy callback  
            **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_buffer_bytes" type="ObservableGauge (bytes)">
          Proxy buffer pool size.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `protocol`, `tunnel_id`  
            **Emission path:** Proxy callback  
            **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_async_backlog_bytes" type="ObservableGauge (bytes)">
          Unflushed async byte backlog.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `protocol`, `tunnel_id`  
            **Emission path:** Proxy callback  
            **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_drops_total" type="Counter">
          Proxy write drops keyed by protocol/tunnel.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `protocol`, `tunnel_id`  
            **Emission path:** `IncProxyDrops`  
            **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_accept_total" type="Counter">
          Proxy accept events keyed by result/reason.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `tunnel_id`, `protocol`, `result`, `reason`  
            **Emission path:** `telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")`  
            **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connections_total" type="Counter">
          Lifecycle events (opened/closed) per connection.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `tunnel_id`, `protocol`, `event` (`opened`|`closed`)  
            **Emission path:** `telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)`  
            **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connection_duration_seconds" type="Histogram (s)">
          Duration of completed proxy connections.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `tunnel_id`, `protocol`, `result`  
            **Emission path:** `telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)`  
            **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="newt">
      Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab.

      <Expandable title="Site & Build">
        <ResponseField name="newt_site_registrations_total" type="counter">
          Counts Pangolin registration attempts keyed by result.
          <Expandable title="Details">
            **Labels:** `result`, `site_id` • **Unit:** 1 • **Path:** `telemetry.IncSiteRegistration`  
            **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_online" type="gauge">
          0/1 heartbeat for the active site.
          <Expandable title="Details">
            **Labels:** `site_id` • **Unit:** 1 • **Path:** `state.TelemetryView`  
            **Example:** `newt_site_online{site_id="self"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_last_heartbeat_seconds" type="gauge">
          Seconds since last Pangolin heartbeat.
          <Expandable title="Details">
            **Labels:** `site_id` • **Unit:** seconds • **Path:** `TouchHeartbeat`  
            **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_build_info" type="gauge">
          Constant 1 with build metadata labels.
          <Expandable title="Details">
            **Labels:** `version`, `commit` • **Unit:** 1 • **Path:** Build info registration  
            **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_restart_count_total" type="counter">
          Process boot indicator (increments once).
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `RegisterBuildInfo`  
            **Example:** `newt_restart_count_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_cert_rotation_total" type="counter">
          Certificate rotation events keyed by result.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `IncCertRotation`  
            **Example:** `newt_cert_rotation_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_reloads_total" type="counter">
          Config reload attempts keyed by result.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `telemetry.IncConfigReload`  
            **Example:** `newt_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_apply_seconds" type="histogram">
          Duration per config-apply phase & result.
          <Expandable title="Details">
            **Labels:** `phase`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveConfigApply`  
            **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Tunnel">
        <ResponseField name="newt_tunnel_sessions" type="gauge">
          Active sessions per tunnel (or collapsed).
          <Expandable title="Details">
            **Labels:** `site_id`, `tunnel_id` • **Unit:** 1 • **Path:** `RegisterStateView`  
            **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_bytes_total" type="counter">
          Traffic per tunnel/direction/protocol.
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `direction`, `protocol` • **Unit:** bytes • **Path:** Proxy manager  
            **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_latency_seconds" type="histogram">
          RTT samples per tunnel/transport.
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `transport` • **Unit:** seconds • **Path:** Health checks  
            **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_reconnects_total" type="counter">
          Reconnect attempts by initiator & reason.
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `initiator`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncReconnect`  
            **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Connection & Auth">
        <ResponseField name="newt_connection_attempts_total" type="counter">
          Auth/WebSocket attempts by transport & result.
          <Expandable title="Details">
            **Labels:** `transport`, `result` • **Unit:** 1 • **Path:** `telemetry.IncConnAttempt`  
            **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_connection_errors_total" type="counter">
          Connection errors by transport and type.
          <Expandable title="Details">
            **Labels:** `transport`, `error_type` • **Unit:** 1 • **Path:** `telemetry.IncConnError`  
            **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="WebSocket">
        <ResponseField name="newt_websocket_connect_latency_seconds" type="histogram">
          Dial latency for Pangolin WebSocket.
          <Expandable title="Details">
            **Labels:** `result`, `transport` • **Unit:** seconds • **Path:** `ObserveWSConnectLatency`  
            **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_disconnects_total" type="counter">
          WS disconnects by reason.
          <Expandable title="Details">
            **Labels:** `reason`, `tunnel_id` • **Unit:** 1 • **Path:** `IncWSDisconnect`  
            **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_keepalive_failures_total" type="counter">
          Keepalive Ping/Pong failures.
          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")`  
            **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_session_duration_seconds" type="histogram">
          Duration of established WebSocket sessions by result.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** seconds • **Path:** `telemetry.ObserveWSSessionDuration(...)`  
            **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_connected" type="gauge">
          Current WS connection status (0/1).
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `telemetry.SetWSConnectionState(true|false)`  
            **Example:** `newt_websocket_connected 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_reconnects_total" type="counter">
          Reconnect attempts by reason.
          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSReconnect(ctx, "ping_write")`  
            **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_messages_total" type="counter">
          In/out WS messages by direction & type.
          <Expandable title="Details">
            **Labels:** `direction`, `msg_type` • **Unit:** 1 • **Path:** `IncWSMessage`  
            **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Proxy">
        <ResponseField name="newt_proxy_active_connections" type="gauge">
          Active TCP/UDP proxy connections per tunnel/protocol.
          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** Proxy callback  
            **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_buffer_bytes" type="gauge">
          Proxy buffer pool size.
          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback  
            **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_async_backlog_bytes" type="gauge">
          Unflushed async byte backlog.
          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback  
            **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_drops_total" type="counter">
          Proxy write drops per protocol/tunnel.
          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** `IncProxyDrops`  
            **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_accept_total" type="counter">
          Proxy accept events by result/reason.
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `result`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncProxyAccept(...)`  
            **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connections_total" type="counter">
          Connection lifecycle events (opened/closed).
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `event` • **Unit:** 1 • **Path:** `telemetry.IncProxyConnectionEvent(...)`  
            **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connection_duration_seconds" type="histogram">
          Duration of completed proxy connections.
          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveProxyConnectionDuration(...)`  
            **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

### Gerbil metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="gerbil">
      OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: `prometheus`, `otel`, or `none`.

      <Note>
        In `otel` mode, Gerbil pushes metrics to an OTLP collector. The `/metrics` endpoint is not exposed in this mode.
      </Note>

      <Expandable title="WireGuard">
        <ResponseField name="gerbil_wg_interface_up" type="Int64Gauge">
          Operational state of a WireGuard interface.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `instance`  
            **Emission path:** `metrics.RecordInterfaceUp`  
            **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peers_total" type="UpDownCounter">
          Number of configured peers per interface.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordPeersTotal`  
            **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_connected" type="Int64Gauge">
          Current peer connection state.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordPeerConnected`  
            **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_allowed_ips_count" type="UpDownCounter">
          Number of allowed IPs configured per peer.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordAllowedIPsCount`  
            **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_key_rotation_total" type="Counter">
          Key rotation events.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `reason`  
            **Emission path:** `metrics.RecordKeyRotation`  
            **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshakes_total" type="Counter">
          WireGuard handshake attempts keyed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `peer`, `result`  
            **Emission path:** `metrics.RecordHandshake`  
            **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshake_latency_seconds" type="Histogram (s)">
          Distribution of WireGuard handshake latencies.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordHandshakeLatency`  
            **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_rtt_seconds" type="Histogram (s)">
          Observed peer round-trip time.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordPeerRTT`  
            **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_received_total" type="Counter (bytes)">
          Bytes received from a WireGuard peer.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordBytesReceived`  
            **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_transmitted_total" type="Counter (bytes)">
          Bytes transmitted to a WireGuard peer.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `ifname`, `peer`  
            **Emission path:** `metrics.RecordBytesTransmitted`  
            **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Relay">
        <ResponseField name="gerbil_active_sessions" type="UpDownCounter">
          Number of active UDP relay sessions.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordActiveSession` / `metrics.RecordSession`  
            **Example:** `gerbil_active_sessions{ifname="wg0"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packets_total" type="Counter">
          UDP packets processed by relay workers.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `type`, `direction`  
            **Emission path:** `metrics.RecordUDPPacket`  
            **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packet_size_bytes" type="Histogram (bytes)">
          Size distribution of packets forwarded through the relay.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `ifname`, `type`  
            **Emission path:** `metrics.RecordUDPPacketSize`  
            **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_hole_punch_events_total" type="Counter">
          Hole punch messages processed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `result`  
            **Emission path:** `metrics.RecordHolePunchEvent`  
            **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_active" type="UpDownCounter">
          Active proxy mappings.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordProxyMapping`  
            **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_session_rebuilt_total" type="Counter">
          Sessions rebuilt from communication patterns.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordSessionRebuilt`  
            **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_comm_pattern_active" type="UpDownCounter">
          Active communication patterns.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordCommPattern`  
            **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_cleanup_removed_total" type="Counter">
          Items removed by cleanup routines.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `component`  
            **Emission path:** `metrics.RecordProxyCleanupRemoved`  
            **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_connection_errors_total" type="Counter">
          Proxy connection errors.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `error_type`  
            **Emission path:** `metrics.RecordProxyConnectionError`  
            **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_initial_mappings" type="Int64Gauge">
          Initial proxy mappings loaded.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordProxyInitialMappings`  
            **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_updates_total" type="Counter">
          Proxy mapping updates.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`  
            **Emission path:** `metrics.RecordProxyMappingUpdate`  
            **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_idle_cleanup_duration_seconds" type="Histogram (s)">
          Duration of idle cleanup cycles.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `ifname`, `component`  
            **Emission path:** `metrics.RecordProxyIdleCleanupDuration`  
            **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="SNI Proxy">
        <ResponseField name="gerbil_active_proxy_connections" type="UpDownCounter">
          Active SNI proxy connections.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `metrics.RecordActiveProxyConnection`  
            **Example:** `gerbil_active_proxy_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_route_lookups_total" type="Counter">
          Route lookups keyed by result.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordProxyRouteLookup`  
            **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_tls_handshake_seconds" type="Histogram (s)">
          TLS handshake duration for the SNI proxy.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** —  
            **Emission path:** `metrics.RecordProxyTLSHandshake`  
            **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_bytes_transmitted_total" type="Counter (bytes)">
          Bytes sent or received by the SNI proxy.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `direction`  
            **Emission path:** `metrics.RecordProxyBytesTransmitted`  
            **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connections_total" type="Counter">
          Connections processed by the SNI proxy.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordSNIConnection`  
            **Example:** `gerbil_sni_connections_total{result="success"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connection_duration_seconds" type="Histogram (s)">
          Lifetime distribution of proxied TLS connections.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** —  
            **Emission path:** `metrics.RecordSNIConnectionDuration`  
            **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_active_connections" type="UpDownCounter">
          Active SNI tunnels.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `metrics.RecordSNIActiveConnection`  
            **Example:** `gerbil_sni_active_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_cache_hits_total" type="Counter">
          SNI route cache hits and misses.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordSNIRouteCacheHit`  
            **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_requests_total" type="Counter">
          SNI route API requests.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordSNIRouteAPIRequest`  
            **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_latency_seconds" type="Histogram (s)">
          Route API call latency.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** —  
            **Emission path:** `metrics.RecordSNIRouteAPILatency`  
            **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_local_override_total" type="Counter">
          Routes using local overrides.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `hit`  
            **Emission path:** `metrics.RecordSNILocalOverride`  
            **Example:** `gerbil_sni_local_override_total{hit="true"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_trusted_proxy_events_total" type="Counter">
          PROXY protocol events.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `event`  
            **Emission path:** `metrics.RecordSNITrustedProxyEvent`  
            **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_proxy_protocol_parse_errors_total" type="Counter">
          PROXY protocol parse failures.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `metrics.RecordSNIProxyProtocolParseError`  
            **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_data_bytes_total" type="Counter (bytes)">
          Bytes proxied through SNI tunnels.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `direction`  
            **Emission path:** `metrics.RecordSNIDataBytes`  
            **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_tunnel_terminations_total" type="Counter">
          SNI tunnel terminations keyed by reason.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `reason`  
            **Emission path:** `metrics.RecordSNITunnelTermination`  
            **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="HTTP API & Peer Management">
        <ResponseField name="gerbil_http_requests_total" type="Counter">
          HTTP requests to the management API.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `endpoint`, `method`, `status_code`  
            **Emission path:** `metrics.RecordHTTPRequest`  
            **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_http_request_duration_seconds" type="Histogram (s)">
          HTTP request handling time.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `endpoint`, `method`  
            **Emission path:** `metrics.RecordHTTPRequestDuration`  
            **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_operations_total" type="Counter">
          Peer lifecycle operations.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `operation`, `result`  
            **Emission path:** `metrics.RecordPeerOperation`  
            **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_update_requests_total" type="Counter">
          Proxy mapping update API calls.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordProxyMappingUpdateRequest`  
            **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_destinations_update_requests_total" type="Counter">
          Destination update API calls.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordDestinationsUpdateRequest`  
            **Example:** `gerbil_destinations_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Remote Config & Reporting">
        <ResponseField name="gerbil_remote_config_fetches_total" type="Counter">
          Remote configuration fetch attempts.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordRemoteConfigFetch`  
            **Example:** `gerbil_remote_config_fetches_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_bandwidth_reports_total" type="Counter">
          Bandwidth report transmissions.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordBandwidthReport`  
            **Example:** `gerbil_bandwidth_reports_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_bandwidth_bytes_total" type="Counter (bytes)">
          Bytes per peer tracked by bandwidth calculation.
          <Expandable title="Details">
            **Unit:** bytes  
            **Labels:** `peer`, `direction`  
            **Emission path:** `metrics.RecordPeerBandwidthBytes`  
            **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="System & Operations">
        <ResponseField name="gerbil_netlink_events_total" type="Counter">
          Netlink events processed.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `event_type`  
            **Emission path:** `metrics.RecordNetlinkEvent`  
            **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_netlink_errors_total" type="Counter">
          Netlink or kernel errors.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `component`, `error_type`  
            **Emission path:** `metrics.RecordNetlinkError`  
            **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sync_duration_seconds" type="Histogram (s)">
          Duration of reconciliation or sync loops.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `component`  
            **Emission path:** `metrics.RecordSyncDuration`  
            **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_workqueue_depth" type="UpDownCounter">
          Current length of internal work queues.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `queue`  
            **Emission path:** `metrics.RecordWorkqueueDepth`  
            **Example:** `gerbil_workqueue_depth{queue="relay"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_kernel_module_loads_total" type="Counter">
          Kernel module load attempts.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordKernelModuleLoad`  
            **Example:** `gerbil_kernel_module_loads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_firewall_rules_applied_total" type="Counter">
          Firewall rules applied.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`, `chain`  
            **Emission path:** `metrics.RecordFirewallRuleApplied`  
            **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_config_reloads_total" type="Counter">
          Configuration reloads.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `result`  
            **Emission path:** `metrics.RecordConfigReload`  
            **Example:** `gerbil_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_restart_total" type="Counter">
          Process restart count.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `metrics.RecordRestart`  
            **Example:** `gerbil_restart_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_auth_failures_total" type="Counter">
          Authentication or peer validation failures.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `peer`, `reason`  
            **Emission path:** `metrics.RecordAuthFailure`  
            **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_acl_denied_total" type="Counter">
          Access-control denied events.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `ifname`, `peer`, `policy`  
            **Emission path:** `metrics.RecordACLDenied`  
            **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_certificate_expiry_days" type="Float64Gauge">
          Days until certificate expiry.
          <Expandable title="Details">
            **Unit:** days  
            **Labels:** `cert_name`, `ifname`  
            **Emission path:** `metrics.RecordCertificateExpiry`  
            **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_memory_spike_total" type="Counter">
          Memory spikes detected by severity.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `severity`  
            **Emission path:** `metrics.RecordMemorySpike`  
            **Example:** `gerbil_memory_spike_total{severity="warning"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_heap_profiles_written_total" type="Counter">
          Heap profile files generated.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** —  
            **Emission path:** `metrics.RecordHeapProfileWritten`  
            **Example:** `gerbil_heap_profiles_written_total 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="gerbil">
      Prometheus-style series for the same Gerbil metrics. Gerbil exposes these only when `METRICS_BACKEND=prometheus`.

      <Note>
        In Prometheus mode, Gerbil registers a native Prometheus client and exposes the configured metrics path, default `/metrics`.
        In OTel mode, `/metrics` is not exposed.
      </Note>

      <Expandable title="WireGuard">
        <ResponseField name="gerbil_wg_interface_up" type="gauge">
          Operational state of a WireGuard interface.
          <Expandable title="Details">
            **Labels:** `ifname`, `instance` • **Unit:** 1 • **Path:** `metrics.RecordInterfaceUp`  
            **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peers_total" type="gauge">
          Number of configured peers per interface.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordPeersTotal`  
            **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_connected" type="gauge">
          Current peer connection state.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordPeerConnected`  
            **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_allowed_ips_count" type="gauge">
          Number of allowed IPs configured per peer.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordAllowedIPsCount`  
            **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_key_rotation_total" type="counter">
          Key rotation events.
          <Expandable title="Details">
            **Labels:** `ifname`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordKeyRotation`  
            **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshakes_total" type="counter">
          WireGuard handshake attempts keyed by result.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHandshake`  
            **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshake_latency_seconds" type="histogram">
          Distribution of WireGuard handshake latencies.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordHandshakeLatency`  
            **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_rtt_seconds" type="histogram">
          Observed peer round-trip time.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordPeerRTT`  
            **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_received_total" type="counter">
          Bytes received from a WireGuard peer.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesReceived`  
            **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_transmitted_total" type="counter">
          Bytes transmitted to a WireGuard peer.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesTransmitted`  
            **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Relay">
        <ResponseField name="gerbil_active_sessions" type="gauge">
          Active UDP relay sessions.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordActiveSession` / `metrics.RecordSession`  
            **Example:** `gerbil_active_sessions{ifname="wg0"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packets_total" type="counter">
          UDP packets processed by relay workers.
          <Expandable title="Details">
            **Labels:** `ifname`, `type`, `direction` • **Unit:** 1 • **Path:** `metrics.RecordUDPPacket`  
            **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packet_size_bytes" type="histogram">
          Size distribution of packets forwarded through the relay.
          <Expandable title="Details">
            **Labels:** `ifname`, `type` • **Unit:** bytes • **Path:** `metrics.RecordUDPPacketSize`  
            **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_hole_punch_events_total" type="counter">
          Hole punch messages processed by result.
          <Expandable title="Details">
            **Labels:** `ifname`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHolePunchEvent`  
            **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_active" type="gauge">
          Active proxy mappings.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMapping`  
            **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_session_rebuilt_total" type="counter">
          Sessions rebuilt from communication patterns.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordSessionRebuilt`  
            **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_comm_pattern_active" type="gauge">
          Active communication patterns.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordCommPattern`  
            **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_cleanup_removed_total" type="counter">
          Items removed by cleanup routines.
          <Expandable title="Details">
            **Labels:** `ifname`, `component` • **Unit:** 1 • **Path:** `metrics.RecordProxyCleanupRemoved`  
            **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_connection_errors_total" type="counter">
          Proxy connection errors.
          <Expandable title="Details">
            **Labels:** `ifname`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordProxyConnectionError`  
            **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_initial_mappings" type="gauge">
          Initial proxy mappings loaded.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyInitialMappings`  
            **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_updates_total" type="counter">
          Proxy mapping updates.
          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdate`  
            **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_idle_cleanup_duration_seconds" type="histogram">
          Duration of idle cleanup cycles.
          <Expandable title="Details">
            **Labels:** `ifname`, `component` • **Unit:** seconds • **Path:** `metrics.RecordProxyIdleCleanupDuration`  
            **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="SNI Proxy">
        <ResponseField name="gerbil_active_proxy_connections" type="gauge">
          Active SNI proxy connections.
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordActiveProxyConnection`  
            **Example:** `gerbil_active_proxy_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_route_lookups_total" type="counter">
          Route lookups keyed by result.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyRouteLookup`  
            **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_tls_handshake_seconds" type="histogram">
          TLS handshake duration for the SNI proxy.
          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordProxyTLSHandshake`  
            **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_bytes_transmitted_total" type="counter">
          Bytes sent or received by the SNI proxy.
          <Expandable title="Details">
            **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordProxyBytesTransmitted`  
            **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connections_total" type="counter">
          Connections processed by the SNI proxy.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIConnection`  
            **Example:** `gerbil_sni_connections_total{result="success"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connection_duration_seconds" type="histogram">
          Lifetime distribution of proxied TLS connections.
          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIConnectionDuration`  
            **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_active_connections" type="gauge">
          Active SNI tunnels.
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIActiveConnection`  
            **Example:** `gerbil_sni_active_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_cache_hits_total" type="counter">
          SNI route cache hits and misses.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteCacheHit`  
            **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_requests_total" type="counter">
          SNI route API requests.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteAPIRequest`  
            **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_latency_seconds" type="histogram">
          Route API call latency.
          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIRouteAPILatency`  
            **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_local_override_total" type="counter">
          Routes using local overrides.
          <Expandable title="Details">
            **Labels:** `hit` • **Unit:** 1 • **Path:** `metrics.RecordSNILocalOverride`  
            **Example:** `gerbil_sni_local_override_total{hit="true"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_trusted_proxy_events_total" type="counter">
          PROXY protocol events.
          <Expandable title="Details">
            **Labels:** `event` • **Unit:** 1 • **Path:** `metrics.RecordSNITrustedProxyEvent`  
            **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_proxy_protocol_parse_errors_total" type="counter">
          PROXY protocol parse failures.
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIProxyProtocolParseError`  
            **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_data_bytes_total" type="counter">
          Bytes proxied through SNI tunnels.
          <Expandable title="Details">
            **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordSNIDataBytes`  
            **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_tunnel_terminations_total" type="counter">
          SNI tunnel terminations keyed by reason.
          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `metrics.RecordSNITunnelTermination`  
            **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="HTTP API & Peer Management">
        <ResponseField name="gerbil_http_requests_total" type="counter">
          HTTP requests to the management API.
          <Expandable title="Details">
            **Labels:** `endpoint`, `method`, `status_code` • **Unit:** 1 • **Path:** `metrics.RecordHTTPRequest`  
            **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_http_request_duration_seconds" type="histogram">
          HTTP request handling time.
          <Expandable title="Details">
            **Labels:** `endpoint`, `method` • **Unit:** seconds • **Path:** `metrics.RecordHTTPRequestDuration`  
            **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_operations_total" type="counter">
          Peer lifecycle operations.
          <Expandable title="Details">
            **Labels:** `operation`, `result` • **Unit:** 1 • **Path:** `metrics.RecordPeerOperation`  
            **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_update_requests_total" type="counter">
          Proxy mapping update API calls.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdateRequest`  
            **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_destinations_update_requests_total" type="counter">
          Destination update API calls.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordDestinationsUpdateRequest`  
            **Example:** `gerbil_destinations_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Remote Config & Reporting">
        <ResponseField name="gerbil_remote_config_fetches_total" type="counter">
          Remote configuration fetch attempts.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordRemoteConfigFetch`  
            **Example:** `gerbil_remote_config_fetches_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_bandwidth_reports_total" type="counter">
          Bandwidth report transmissions.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordBandwidthReport`  
            **Example:** `gerbil_bandwidth_reports_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_bandwidth_bytes_total" type="counter">
          Bytes per peer tracked by bandwidth calculation.
          <Expandable title="Details">
            **Labels:** `peer`, `direction` • **Unit:** bytes • **Path:** `metrics.RecordPeerBandwidthBytes`  
            **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="System & Operations">
        <ResponseField name="gerbil_netlink_events_total" type="counter">
          Netlink events processed.
          <Expandable title="Details">
            **Labels:** `event_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkEvent`  
            **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_netlink_errors_total" type="counter">
          Netlink or kernel errors.
          <Expandable title="Details">
            **Labels:** `component`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkError`  
            **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sync_duration_seconds" type="histogram">
          Duration of reconciliation or sync loops.
          <Expandable title="Details">
            **Labels:** `component` • **Unit:** seconds • **Path:** `metrics.RecordSyncDuration`  
            **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_workqueue_depth" type="gauge">
          Current length of internal work queues.
          <Expandable title="Details">
            **Labels:** `queue` • **Unit:** 1 • **Path:** `metrics.RecordWorkqueueDepth`  
            **Example:** `gerbil_workqueue_depth{queue="relay"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_kernel_module_loads_total" type="counter">
          Kernel module load attempts.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordKernelModuleLoad`  
            **Example:** `gerbil_kernel_module_loads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_firewall_rules_applied_total" type="counter">
          Firewall rules applied.
          <Expandable title="Details">
            **Labels:** `result`, `chain` • **Unit:** 1 • **Path:** `metrics.RecordFirewallRuleApplied`  
            **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_config_reloads_total" type="counter">
          Configuration reloads.
          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordConfigReload`  
            **Example:** `gerbil_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_restart_total" type="counter">
          Process restart count.
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordRestart`  
            **Example:** `gerbil_restart_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_auth_failures_total" type="counter">
          Authentication or peer validation failures.
          <Expandable title="Details">
            **Labels:** `peer`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordAuthFailure`  
            **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_acl_denied_total" type="counter">
          Access-control denied events.
          <Expandable title="Details">
            **Labels:** `ifname`, `peer`, `policy` • **Unit:** 1 • **Path:** `metrics.RecordACLDenied`  
            **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_certificate_expiry_days" type="gauge">
          Days until certificate expiry.
          <Expandable title="Details">
            **Labels:** `cert_name`, `ifname` • **Unit:** days • **Path:** `metrics.RecordCertificateExpiry`  
            **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_memory_spike_total" type="counter">
          Memory spikes detected by severity.
          <Expandable title="Details">
            **Labels:** `severity` • **Unit:** 1 • **Path:** `metrics.RecordMemorySpike`  
            **Example:** `gerbil_memory_spike_total{severity="warning"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_heap_profiles_written_total" type="counter">
          Heap profile files generated.
          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordHeapProfileWritten`  
            **Example:** `gerbil_heap_profiles_written_total 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

### Pangolin Kubernetes Controller metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="pangolin-kube-controller">
      Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller.

      <Note>
        The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same `/metrics` endpoint.
        The OTel-backed series use the `pangolin_controller_*` prefix.
      </Note>

      <Expandable title="Reconcile">
        <ResponseField name="pangolin_controller_reconcile_phase_duration_seconds" type="Histogram (s)">
          Duration of each reconcile phase.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `phase`, `result`  
            **Label values:**  
            `phase`: `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp`  
            `result`: `success` | `error`  
            **Emission path:** OTel reconcile phase instrumentation  
            **Example:** `pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_active_reconcile_routines" type="UpDownCounter">
          Number of active reconcile routines by phase.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `phase`  
            **Label values:** `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp`  
            **Emission path:** Parallel reconcile instrumentation  
            **Example:** `pangolin_controller_active_reconcile_routines{phase="routers"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_loop_iterations_total" type="Counter">
          Controller loop iterations by outcome.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `outcome`  
            **Label values:** `success` | `nochange` | `error`  
            **Emission path:** Controller loop instrumentation  
            **Example:** `pangolin_controller_loop_iterations_total{outcome="success"} 10`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Fetch & Config">
        <ResponseField name="pangolin_controller_fetch_duration_seconds" type="Histogram (s)">
          Duration of remote fetch cycle HTTP requests.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `status_code`, `status_class`  
            **Label values:**  
            `status_code`: `200` | `304` | `401` | `403` | `404` | `5xx`  
            `status_class`: `2xx` | `3xx` | `4xx` | `5xx`  
            **Emission path:** Remote config fetch instrumentation  
            **Example:** `pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_config_parse_duration_seconds" type="Histogram (s)">
          Duration of configuration parsing.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `section`  
            **Label values:** `full`  
            **Emission path:** Config parse instrumentation  
            **Example:** `pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Kubernetes API">
        <ResponseField name="pangolin_controller_k8s_request_duration_seconds" type="Histogram (s)">
          Duration of Kubernetes API requests.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `verb`, `resource_kind`, `result`, `forced`  
            **Label values:**  
            `verb`: `get` | `create` | `patch` | `update` | `delete` | `list`  
            `resource_kind`: `IngressRoute` | `Middleware` | `TraefikService` | `ServersTransport` | `ServersTransportTCP` | `Service` | `EndpointSlice`  
            `result`: `success` | `error` | `conflict`  
            `forced`: `true` | `false`  
            **Emission path:** Kubernetes API request instrumentation  
            **Example:** `pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_k8s_requests_total" type="Counter">
          Total Kubernetes API requests.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `verb`, `resource_kind`, `result`, `forced`  
            **Emission path:** Kubernetes API request instrumentation  
            **Example:** `pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_retries_total" type="Counter">
          Retry attempts in the SSA apply loop.
          <Expandable title="Details">
            **Unit:** 1  
            **Labels:** `reason`, `operation`, `resource_kind`  
            **Label values:**  
            `reason`: `conflict` | `transient` | `timeout`  
            `operation`: `get` | `create` | `patch` | `delete` | `apply`  
            **Emission path:** SSA apply retry instrumentation  
            **Example:** `pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Garbage Collection">
        <ResponseField name="pangolin_controller_gc_run_duration_seconds" type="Histogram (s)">
          Duration of garbage collection runs.
          <Expandable title="Details">
            **Unit:** seconds  
            **Labels:** `result`  
            **Label values:** `success` | `fail` | `dryrun`  
            **Emission path:** GC instrumentation  
            **Example:** `pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="pangolin-kube-controller">
      Prometheus-native metrics exposed by the Pangolin Kubernetes Controller. These use the `pangolin_kube_controller_*` prefix.

      <Note>
        The metrics endpoint is exposed at `GET /metrics` on `METRICS_ADDR`, default `:9090`.
        The same HTTP server also exposes `/healthz`, `/readyz`, and optionally `/debug/pprof/` when pprof is enabled.
      </Note>

      <Expandable title="Reconcile">
        <ResponseField name="pangolin_kube_controller_reconcile_seconds" type="histogram">
          Duration of a full successful reconcile loop.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** seconds  
            **Path:** `Collector.ReconcileDuration`  
            **Example:** `pangolin_kube_controller_reconcile_seconds_bucket{le="0.5"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_reconcile_errors_total" type="counter">
          Total errors during reconcile steps.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.ReconcileErrors`  
            **Example:** `pangolin_kube_controller_reconcile_errors_total 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_consecutive_errors" type="gauge">
          Number of consecutive reconcile or fetch errors.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.ConsecutiveErrors`  
            **Example:** `pangolin_kube_controller_consecutive_errors 0`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_last_fetch_success_timestamp_seconds" type="gauge">
          Unix timestamp of the last successful fetch.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** seconds since Unix epoch  
            **Path:** `Collector.LastFetchSuccess`  
            **Example:** `pangolin_kube_controller_last_fetch_success_timestamp_seconds 1767225600`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_ready" type="gauge">
          Controller readiness state.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Value:** `1` = ready, `0` = not ready  
            **Path:** `Collector.Ready`  
            **Example:** `pangolin_kube_controller_ready 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Objects">
        <ResponseField name="pangolin_kube_controller_objects_applied_total" type="counter">
          Applied objects by kind and action.
          <Expandable title="Details">
            **Labels:** `kind`, `action`  
            **Label values:**  
            `kind`: `IngressRoute` | `Middleware` | `TraefikService` | other managed Traefik CRDs  
            `action`: `create` | `patch`  
            **Unit:** 1  
            **Path:** `Collector.AppliedObjects`  
            **Example:** `pangolin_kube_controller_objects_applied_total{kind="IngressRoute",action="patch"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_objects_deleted_total" type="counter">
          Deleted objects by kind.
          <Expandable title="Details">
            **Labels:** `kind`  
            **Unit:** 1  
            **Path:** `Collector.DeletedObjects`  
            **Example:** `pangolin_kube_controller_objects_deleted_total{kind="Middleware"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_desired_objects_count" type="gauge">
          Desired objects count by kind from the last processed config.
          <Expandable title="Details">
            **Labels:** `kind`  
            **Unit:** 1  
            **Path:** `Collector.DesiredObjects`  
            **Example:** `pangolin_kube_controller_desired_objects_count{kind="IngressRoute"} 12`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Garbage Collection">
        <ResponseField name="pangolin_kube_controller_gc_deleted_total" type="counter">
          Garbage-collected objects by kind and reason.
          <Expandable title="Details">
            **Labels:** `kind`, `reason`  
            **Label values:**  
            `reason`: `immediate` | `grace`  
            **Unit:** 1  
            **Path:** `Collector.GCDeletedTotal`  
            **Example:** `pangolin_kube_controller_gc_deleted_total{kind="IngressRoute",reason="grace"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_gc_runs_total" type="counter">
          Garbage collection runs by result.
          <Expandable title="Details">
            **Labels:** `result`  
            **Label values:** `start` | `success` | `fail` | `dryrun`  
            **Unit:** 1  
            **Path:** `Collector.GCRunsTotal`  
            **Example:** `pangolin_kube_controller_gc_runs_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_grace_queue_dropped_total" type="counter">
          Grace deletion queue items dropped due to queue overflow.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.GraceQueueDropped`  
            **Example:** `pangolin_kube_controller_grace_queue_dropped_total 0`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_grace_queue_depth" type="gauge">
          Current depth of the grace deletion queue.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.GraceQueueDepth`  
            **Example:** `pangolin_kube_controller_grace_queue_depth 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Instance Label">
        <ResponseField name="pangolin_kube_controller_instance_label_detect_success_total" type="counter">
          Successful Traefik instance label resolutions.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.InstanceLabelDetectSuccess`  
            **Example:** `pangolin_kube_controller_instance_label_detect_success_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_instance_label_detect_failure_total" type="counter">
          Failed Traefik instance label resolutions or verifications.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Path:** `Collector.InstanceLabelDetectFailures`  
            **Example:** `pangolin_kube_controller_instance_label_detect_failure_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_instance_label_last_check_timestamp_seconds" type="gauge">
          Unix timestamp of the last instance label verification.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** seconds since Unix epoch  
            **Path:** `Collector.InstanceLabelLastCheck`  
            **Example:** `pangolin_kube_controller_instance_label_last_check_timestamp_seconds 1767225600`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Leader Election">
        <ResponseField name="pangolin_kube_controller_leader_state" type="gauge">
          Current leader election state of the controller.
          <Expandable title="Details">
            **Labels:** —  
            **Unit:** 1  
            **Value:** `1` = leader, `0` = follower, `-1` = leader election disabled  
            **Path:** `Collector.LeaderState`  
            **Example:** `pangolin_kube_controller_leader_state 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Runtime">
        <ResponseField name="go_*" type="runtime metrics">
          Go runtime metrics registered by the Prometheus Go collector.
          <Expandable title="Details">
            **Labels:** varies  
            **Unit:** varies  
            **Path:** `collectors.NewGoCollector()`  
            **Example:** `go_goroutines 14`
          </Expandable>
        </ResponseField>

        <ResponseField name="process_*" type="process metrics">
          Process metrics registered by the Prometheus process collector.
          <Expandable title="Details">
            **Labels:** varies  
            **Unit:** varies  
            **Path:** `collectors.NewProcessCollector()`  
            **Example:** `process_resident_memory_bytes 52428800`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

---

## References

* <Link href="https://opentelemetry.io/docs/">OpenTelemetry Documentation</Link>
* <Link href="https://prometheus.io/docs/introduction/overview/">Prometheus Documentation</Link>

<Tip>
Have improvements or a missing metric? Open an issue or PR referencing this page.
</Tip>

