[client] Feature/client metrics (#5512)

* Add client metrics * Add client metrics system with OpenTelemetry and VictoriaMetrics support Implements a comprehensive client metrics system to track peer connection stages and performance. The system supports multiple backend implementations (OpenTelemetry, VictoriaMetrics, and no-op) and tracks detailed connection stage durations from creation through WireGuard handshake. Key changes: - Add metrics package with pluggable backend implementations - Implement OpenTelemetry metrics backend - Implement VictoriaMetrics metrics backend - Add no-op metrics implementation for disabled state - Track connection stages: creation, semaphore, signaling, connection ready, and WireGuard handshake - Move WireGuard watcher functionality to conn.go - Refactor engine to integrate metrics tracking - Add metrics export endpoint in debug server * Add signaling metrics tracking for initial and reconnection attempts * Reset connection stage timestamps during reconnections to exclude unnecessary metrics tracking * Delete otel lib from client * Update unit tests * Invoke callback on handshake success in WireGuard watcher * Add Netbird version tracking to client metrics Integrate Netbird version into VictoriaMetrics backend and metrics labels. Update `ClientMetrics` constructor and metric name formatting to include version information. * Add sync duration tracking to client metrics Introduce `RecordSyncDuration` for measuring sync message processing time. Update all metrics implementations (VictoriaMetrics, no-op) to support the new method. Refactor `ClientMetrics` to use `AgentInfo` for static agent data. * Remove no-op metrics implementation and simplify ClientMetrics constructor Eliminate unused `noopMetrics` and refactor `ClientMetrics` to always use the VictoriaMetrics implementation. Update associated logic to reflect these changes. * Add total duration tracking for connection attempts Calculate total duration for both initial connections and reconnections, accounting for different timestamp scenarios. Update `Export` method to include Prometheus HELP comments. * Add metrics push support to VictoriaMetrics integration * [client] anchor connection metrics to first signal received * Remove creation_to_semaphore connection stage metric The semaphore queuing stage (Created → SemaphoreAcquired) is no longer tracked. Connection metrics now start from SignalingReceived. Updated docs and Grafana dashboard accordingly. * [client] Add remote push config for metrics with version-based eligibility Introduce remoteconfig.Manager that fetches a remote JSON config to control metrics push interval and restrict pushing to a specific agent version range. When NB_METRICS_INTERVAL is set, remote config is bypassed entirely for local override. * [client] Add WASM-compatible NewClientMetrics implementation Replace NewClientMetrics in metrics.go with a WASM-specific stub in metrics_js.go, returning nil for compatibility with JS builds. Simplify method usage for WASM targets. * Add missing file * Update default case in DeploymentType.String to return "unknown" instead of "selfhosted" * [client] Rework metrics to use timestamped samples instead of histograms Replace cumulative Prometheus histograms with timestamped point-in-time samples that are pushed once and cleared. This fixes metrics for sparse events (connections/syncs that happen once at startup) where rate() and increase() produced incorrect or empty results. Changes: - Switch from VictoriaMetrics histogram library to raw Prometheus text format with explicit millisecond timestamps - Reset samples after successful push (no resending stale data) - Rename connection_to_handshake → connection_to_wg_handshake - Add netbird_peer_connection_count metric for ICE vs Relay tracking - Simplify dashboard: point-based scatter plots, donut pie chart - Add maxStalenessInterval=1m to VictoriaMetrics to prevent forward-fill - Fix deployment_type Unknown returning "selfhosted" instead of "unknown" - Fix inverted shouldPush condition in push.go * [client] Add InfluxDB metrics backend alongside VictoriaMetrics Add influxdb.go with timestamped line protocol export for sparse one-shot events. Restore victoria.go to use proper Prometheus histograms. Update Grafana dashboards, add InfluxDB datasource, and update docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [client] Fix metrics issues and update dev docker setup - Fix StopPush not clearing push state, preventing restart - Fix race condition reading currentConnPriority without lock in recordConnectionMetrics - Fix stale comment referencing old metrics server URL - Update docker-compose for InfluxDB: add scoped tokens, .env config, init scripts - Rename docker-compose.victoria.yml to docker-compose.yml * [client] Add anonymised peer tracking to pushed metrics Introduce peer_id and connection_pair_id tags to InfluxDB metrics. Public keys are hashed (truncated SHA-256) for anonymisation. The connection pair ID is deterministic regardless of which side computes it, enabling deduplication of reconnections in the ICE vs Relay dashboard. Also pin Grafana to v11.6.0 for file-based provisioning and fix datasource UID references. * Remove unused dependencies from go.mod and go.sum * Refactor InfluxDB ingest pipeline: extract validation logic - Move line validation logic to `validateLine` and `validateField` helper functions. - Improve error handling with structured validation and clearer separation of concerns. - Add stderr redirection for error messages in `create-tokens.sh`. * Set non-root user in Dockerfile for Ingest service * Fix Windows CI: command line too long * Remove Victoria metrics * Add hashed peer ID as Authorization header in metrics push * Revert influxdb in docker compose * Enable gzip compression and authorization validation for metrics push and ingest * Reducate code of complexity * Update debug documentation to include metrics.txt description * Increase `maxBodySize` limit to 50 MB and update gzip reader wrapping logic * Refactor deployment type detection to use URL parsing for improved accuracy * Update readme * Throttle remote config retries on fetch failure * Preserve first WG handshake timestamp, ignore rekeys * Skip adding empty metrics.txt to debug bundle in debug mode * Update default metrics server URL to https://ingest.netbird.io * Atomic metrics export-and-reset to prevent sample loss between Export and Reset calls * Fix doc * Refactor Push configuration to improve clarity and enforce minimum push interval * Remove `minPushInterval` and update push interval validation logic * Revert ExportAndReset, it is acceptable data loss * Fix metrics review issues: rename env var, remove stale infra, add tests - Rename NB_METRICS_ENABLED to NB_METRICS_PUSH_ENABLED to clarify that collection is always active (for debug bundles) and only push is opt-in - Change default config URL from staging to production (ingest.netbird.io) - Delete broken Prometheus dashboard (used non-existent metric names) - Delete unused VictoriaMetrics datasource config - Replace committed .env with .env.example containing placeholder values - Wire Grafana admin credentials through env vars in docker-compose - Make metricsStages a pointer to prevent reset-vs-write race on reconnect - Fix typed-nil interface in debug bundle path (GetClientMetrics) - Use deterministic field order in InfluxDB Export (sorted keys) - Replace Authorization header with X-Peer-ID for metrics push - Fix ingest server timeout to use time.Second instead of float - Fix gzip double-close, stale comments, trim log levels - Add tests for influxdb.go and MetricsStages * Add login duration metric, ingest tag validation, and duration bounds - Add netbird_login measurement recording login/auth duration to management server, with success/failure result tag - Validate InfluxDB tags against per-measurement allowlists in ingest server to prevent arbitrary tag injection - Cap all duration fields (*_seconds) at 300s instead of only total_seconds - Add ingest server tests for tag/field validation, bounds, and auth * Add arch tag to all metrics * Fix Grafana dashboard: add arch to drop columns, add login panels * Validate NB_METRICS_SERVER_URL is an absolute HTTP(S) URL * Address review comments: fix README wording, update stale comments * Clarify env var precedence does not bypass remote config eligibility * Remove accidentally committed pprof files --------- Co-authored-by: Viktor Liu <viktor@netbird.io>
2026-04-18 08:16:39 +00:00 · 2026-03-22 12:45:41 +01:00
parent 82762280ee
commit 91f0d5cefd
40 changed files with 3405 additions and 42 deletions
--- a/client/internal/peer/conn.go
+++ b/client/internal/peer/conn.go
@@ -15,6 +15,7 @@ import (

 	"github.com/netbirdio/netbird/client/iface/configurer"
 	"github.com/netbirdio/netbird/client/iface/wgproxy"
+	"github.com/netbirdio/netbird/client/internal/metrics"
 	"github.com/netbirdio/netbird/client/internal/peer/conntype"
 	"github.com/netbirdio/netbird/client/internal/peer/dispatcher"
 	"github.com/netbirdio/netbird/client/internal/peer/guard"
@@ -26,6 +27,17 @@ import (
 	relayClient "github.com/netbirdio/netbird/shared/relay/client"
 )

+// MetricsRecorder is an interface for recording peer connection metrics
+type MetricsRecorder interface {
+	RecordConnectionStages(
+		ctx context.Context,
+		remotePubKey string,
+		connectionType metrics.ConnectionType,
+		isReconnection bool,
+		timestamps metrics.ConnectionStageTimestamps,
+	)
+}
+
 type ServiceDependencies struct {
 	StatusRecorder     *Status
 	Signaler           *Signaler
@@ -33,6 +45,7 @@ type ServiceDependencies struct {
 	RelayManager       *relayClient.Manager
 	SrWatcher          *guard.SRWatcher
 	PeerConnDispatcher *dispatcher.ConnectionDispatcher
+	MetricsRecorder    MetricsRecorder
 }

 type WgConfig struct {
@@ -115,6 +128,10 @@ type Conn struct {
 	dumpState *stateDump

 	endpointUpdater *EndpointUpdater
+
+	// Connection stage timestamps for metrics
+	metricsRecorder MetricsRecorder
+	metricsStages   *MetricsStages
 }

 // NewConn creates a new not opened Conn to the remote peer.
@@ -140,6 +157,7 @@ func NewConn(config ConnConfig, services ServiceDependencies) (*Conn, error) {
 		dumpState:       dumpState,
 		endpointUpdater: NewEndpointUpdater(connLog, config.WgConfig, isController(config)),
 		wgWatcher:       NewWGWatcher(connLog, config.WgConfig.WgInterface, config.Key, dumpState),
+		metricsRecorder: services.MetricsRecorder,
 	}

 	return conn, nil
@@ -156,6 +174,9 @@ func (conn *Conn) Open(engineCtx context.Context) error {
 		return nil
 	}

+	// Allocate new metrics stages so old goroutines don't corrupt new state
+	conn.metricsStages = &MetricsStages{}
+
 	conn.ctx, conn.ctxCancel = context.WithCancel(engineCtx)

 	conn.workerRelay = NewWorkerRelay(conn.ctx, conn.Log, isController(conn.config), conn.config, conn, conn.relayManager)
@@ -167,7 +188,7 @@ func (conn *Conn) Open(engineCtx context.Context) error {
 	}
 	conn.workerICE = workerICE

-	conn.handshaker = NewHandshaker(conn.Log, conn.config, conn.signaler, conn.workerICE, conn.workerRelay)
+	conn.handshaker = NewHandshaker(conn.Log, conn.config, conn.signaler, conn.workerICE, conn.workerRelay, conn.metricsStages)

 	conn.handshaker.AddRelayListener(conn.workerRelay.OnNewOffer)
 	if !isForceRelayed() {
@@ -335,7 +356,7 @@ func (conn *Conn) onICEConnectionIsReady(priority conntype.ConnPriority, iceConn
 	if conn.currentConnPriority > priority {
 		conn.Log.Infof("current connection priority (%s) is higher than the new one (%s), do not upgrade connection", conn.currentConnPriority, priority)
 		conn.statusICE.SetConnected()
-		conn.updateIceState(iceConnInfo)
+		conn.updateIceState(iceConnInfo, time.Now())
 		return
 	}

@@ -375,7 +396,8 @@ func (conn *Conn) onICEConnectionIsReady(priority conntype.ConnPriority, iceConn
 	}

 	conn.Log.Infof("configure WireGuard endpoint to: %s", ep.String())
-	conn.enableWgWatcherIfNeeded()
+	updateTime := time.Now()
+	conn.enableWgWatcherIfNeeded(updateTime)

 	presharedKey := conn.presharedKey(iceConnInfo.RosenpassPubKey)
 	if err = conn.endpointUpdater.ConfigureWGEndpoint(ep, presharedKey); err != nil {
@@ -391,8 +413,8 @@ func (conn *Conn) onICEConnectionIsReady(priority conntype.ConnPriority, iceConn

 	conn.currentConnPriority = priority
 	conn.statusICE.SetConnected()
-	conn.updateIceState(iceConnInfo)
-	conn.doOnConnected(iceConnInfo.RosenpassPubKey, iceConnInfo.RosenpassAddr)
+	conn.updateIceState(iceConnInfo, updateTime)
+	conn.doOnConnected(iceConnInfo.RosenpassPubKey, iceConnInfo.RosenpassAddr, updateTime)
 }

 func (conn *Conn) onICEStateDisconnected(sessionChanged bool) {
@@ -444,6 +466,10 @@ func (conn *Conn) onICEStateDisconnected(sessionChanged bool) {

 	conn.disableWgWatcherIfNeeded()

+	if conn.currentConnPriority == conntype.None {
+		conn.metricsStages.Disconnected()
+	}
+
 	peerState := State{
 		PubKey:           conn.config.Key,
 		ConnStatus:       conn.evalStatus(),
@@ -484,7 +510,7 @@ func (conn *Conn) onRelayConnectionIsReady(rci RelayConnInfo) {
 		conn.Log.Debugf("do not switch to relay because current priority is: %s", conn.currentConnPriority.String())
 		conn.setRelayedProxy(wgProxy)
 		conn.statusRelay.SetConnected()
-		conn.updateRelayStatus(rci.relayedConn.RemoteAddr().String(), rci.rosenpassPubKey)
+		conn.updateRelayStatus(rci.relayedConn.RemoteAddr().String(), rci.rosenpassPubKey, time.Now())
 		return
 	}

@@ -493,7 +519,8 @@ func (conn *Conn) onRelayConnectionIsReady(rci RelayConnInfo) {
 	if controller {
 		wgProxy.Work()
 	}
-	conn.enableWgWatcherIfNeeded()
+	updateTime := time.Now()
+	conn.enableWgWatcherIfNeeded(updateTime)
 	if err := conn.endpointUpdater.ConfigureWGEndpoint(wgProxy.EndpointAddr(), conn.presharedKey(rci.rosenpassPubKey)); err != nil {
 		if err := wgProxy.CloseConn(); err != nil {
 			conn.Log.Warnf("Failed to close relay connection: %v", err)
@@ -504,13 +531,16 @@ func (conn *Conn) onRelayConnectionIsReady(rci RelayConnInfo) {
 	if !controller {
 		wgProxy.Work()
 	}
+
+	wgConfigWorkaround()
+
 	conn.rosenpassRemoteKey = rci.rosenpassPubKey
 	conn.currentConnPriority = conntype.Relay
 	conn.statusRelay.SetConnected()
 	conn.setRelayedProxy(wgProxy)
-	conn.updateRelayStatus(rci.relayedConn.RemoteAddr().String(), rci.rosenpassPubKey)
+	conn.updateRelayStatus(rci.relayedConn.RemoteAddr().String(), rci.rosenpassPubKey, updateTime)
 	conn.Log.Infof("start to communicate with peer via relay")
-	conn.doOnConnected(rci.rosenpassPubKey, rci.rosenpassAddr)
+	conn.doOnConnected(rci.rosenpassPubKey, rci.rosenpassAddr, updateTime)
 }

 func (conn *Conn) onRelayDisconnected() {
@@ -548,6 +578,10 @@ func (conn *Conn) handleRelayDisconnectedLocked() {

 	conn.disableWgWatcherIfNeeded()

+	if conn.currentConnPriority == conntype.None {
+		conn.metricsStages.Disconnected()
+	}
+
 	peerState := State{
 		PubKey:           conn.config.Key,
 		ConnStatus:       conn.evalStatus(),
@@ -588,10 +622,10 @@ func (conn *Conn) onWGDisconnected() {
 	}
 }

-func (conn *Conn) updateRelayStatus(relayServerAddr string, rosenpassPubKey []byte) {
+func (conn *Conn) updateRelayStatus(relayServerAddr string, rosenpassPubKey []byte, updateTime time.Time) {
 	peerState := State{
 		PubKey:             conn.config.Key,
-		ConnStatusUpdate:   time.Now(),
+		ConnStatusUpdate:   updateTime,
 		ConnStatus:         conn.evalStatus(),
 		Relayed:            conn.isRelayed(),
 		RelayServerAddress: relayServerAddr,
@@ -604,10 +638,10 @@ func (conn *Conn) updateRelayStatus(relayServerAddr string, rosenpassPubKey []by
 	}
 }

-func (conn *Conn) updateIceState(iceConnInfo ICEConnInfo) {
+func (conn *Conn) updateIceState(iceConnInfo ICEConnInfo, updateTime time.Time) {
 	peerState := State{
 		PubKey:                     conn.config.Key,
-		ConnStatusUpdate:           time.Now(),
+		ConnStatusUpdate:           updateTime,
 		ConnStatus:                 conn.evalStatus(),
 		Relayed:                    iceConnInfo.Relayed,
 		LocalIceCandidateType:      iceConnInfo.LocalIceCandidateType,
@@ -645,11 +679,13 @@ func (conn *Conn) setStatusToDisconnected() {
 	}
 }

-func (conn *Conn) doOnConnected(remoteRosenpassPubKey []byte, remoteRosenpassAddr string) {
+func (conn *Conn) doOnConnected(remoteRosenpassPubKey []byte, remoteRosenpassAddr string, updateTime time.Time) {
 	if runtime.GOOS == "ios" {
 		runtime.GC()
 	}

+	conn.metricsStages.RecordConnectionReady(updateTime)
+
 	if conn.onConnected != nil {
 		conn.onConnected(conn.config.Key, remoteRosenpassPubKey, conn.config.WgConfig.AllowedIps[0].Addr().String(), remoteRosenpassAddr)
 	}
@@ -701,14 +737,14 @@ func (conn *Conn) isConnectedOnAllWay() (connected bool) {
 	return true
 }

-func (conn *Conn) enableWgWatcherIfNeeded() {
+func (conn *Conn) enableWgWatcherIfNeeded(enabledTime time.Time) {
 	if !conn.wgWatcher.IsEnabled() {
 		wgWatcherCtx, wgWatcherCancel := context.WithCancel(conn.ctx)
 		conn.wgWatcherCancel = wgWatcherCancel
 		conn.wgWatcherWg.Add(1)
 		go func() {
 			defer conn.wgWatcherWg.Done()
-			conn.wgWatcher.EnableWgWatcher(wgWatcherCtx, conn.onWGDisconnected)
+			conn.wgWatcher.EnableWgWatcher(wgWatcherCtx, enabledTime, conn.onWGDisconnected, conn.onWGHandshakeSuccess)
 		}()
 	}
 }
@@ -783,6 +819,41 @@ func (conn *Conn) setRelayedProxy(proxy wgproxy.Proxy) {
 	conn.wgProxyRelay = proxy
 }

+// onWGHandshakeSuccess is called when the first WireGuard handshake is detected
+func (conn *Conn) onWGHandshakeSuccess(when time.Time) {
+	conn.metricsStages.RecordWGHandshakeSuccess(when)
+	conn.recordConnectionMetrics()
+}
+
+// recordConnectionMetrics records connection stage timestamps as metrics
+func (conn *Conn) recordConnectionMetrics() {
+	if conn.metricsRecorder == nil {
+		return
+	}
+
+	// Determine connection type based on current priority
+	conn.mu.Lock()
+	priority := conn.currentConnPriority
+	conn.mu.Unlock()
+
+	var connType metrics.ConnectionType
+	switch priority {
+	case conntype.Relay:
+		connType = metrics.ConnectionTypeRelay
+	default:
+		connType = metrics.ConnectionTypeICE
+	}
+
+	// Record metrics with timestamps - duration calculation happens in metrics package
+	conn.metricsRecorder.RecordConnectionStages(
+		context.Background(),
+		conn.config.Key,
+		connType,
+		conn.metricsStages.IsReconnection(),
+		conn.metricsStages.GetTimestamps(),
+	)
+}
+
 // AllowedIP returns the allowed IP of the remote peer
 func (conn *Conn) AllowedIP() netip.Addr {
 	return conn.config.WgConfig.AllowedIps[0].Addr()
--- a/client/internal/peer/handshaker.go
+++ b/client/internal/peer/handshaker.go
@@ -44,12 +44,13 @@ type OfferAnswer struct {
 }

 type Handshaker struct {
-	mu       sync.Mutex
-	log      *log.Entry
-	config   ConnConfig
-	signaler *Signaler
-	ice      *WorkerICE
-	relay    *WorkerRelay
+	mu            sync.Mutex
+	log           *log.Entry
+	config        ConnConfig
+	signaler      *Signaler
+	ice           *WorkerICE
+	relay         *WorkerRelay
+	metricsStages *MetricsStages
 	// relayListener is not blocking because the listener is using a goroutine to process the messages
 	// and it will only keep the latest message if multiple offers are received in a short time
 	// this is to avoid blocking the handshaker if the listener is doing some heavy processing
@@ -64,13 +65,14 @@ type Handshaker struct {
 	remoteAnswerCh chan OfferAnswer
 }

-func NewHandshaker(log *log.Entry, config ConnConfig, signaler *Signaler, ice *WorkerICE, relay *WorkerRelay) *Handshaker {
+func NewHandshaker(log *log.Entry, config ConnConfig, signaler *Signaler, ice *WorkerICE, relay *WorkerRelay, metricsStages *MetricsStages) *Handshaker {
 	return &Handshaker{
 		log:            log,
 		config:         config,
 		signaler:       signaler,
 		ice:            ice,
 		relay:          relay,
+		metricsStages:  metricsStages,
 		remoteOffersCh: make(chan OfferAnswer),
 		remoteAnswerCh: make(chan OfferAnswer),
 	}
@@ -89,6 +91,12 @@ func (h *Handshaker) Listen(ctx context.Context) {
 		select {
 		case remoteOfferAnswer := <-h.remoteOffersCh:
 			h.log.Infof("received offer, running version %s, remote WireGuard listen port %d, session id: %s", remoteOfferAnswer.Version, remoteOfferAnswer.WgListenPort, remoteOfferAnswer.SessionIDString())
+
+			// Record signaling received for reconnection attempts
+			if h.metricsStages != nil {
+				h.metricsStages.RecordSignalingReceived()
+			}
+
 			if h.relayListener != nil {
 				h.relayListener.Notify(&remoteOfferAnswer)
 			}
@@ -103,6 +111,12 @@ func (h *Handshaker) Listen(ctx context.Context) {
 			}
 		case remoteOfferAnswer := <-h.remoteAnswerCh:
 			h.log.Infof("received answer, running version %s, remote WireGuard listen port %d, session id: %s", remoteOfferAnswer.Version, remoteOfferAnswer.WgListenPort, remoteOfferAnswer.SessionIDString())
+
+			// Record signaling received for reconnection attempts
+			if h.metricsStages != nil {
+				h.metricsStages.RecordSignalingReceived()
+			}
+
 			if h.relayListener != nil {
 				h.relayListener.Notify(&remoteOfferAnswer)
 			}
--- a/client/internal/peer/metrics_saver.go
+++ b/client/internal/peer/metrics_saver.go
@@ -0,0 +1,73 @@
+package peer
+
+import (
+	"sync"
+	"time"
+
+	"github.com/netbirdio/netbird/client/internal/metrics"
+)
+
+type MetricsStages struct {
+	isReconnectionAttempt bool // Track if current attempt is a reconnection
+	stageTimestamps       metrics.ConnectionStageTimestamps
+	mu                    sync.Mutex
+}
+
+// RecordSignalingReceived records when the first signal is received from the remote peer.
+// Used as the base for all subsequent stage durations to avoid inflating metrics when
+// the remote peer was offline.
+func (s *MetricsStages) RecordSignalingReceived() {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if s.stageTimestamps.SignalingReceived.IsZero() {
+		s.stageTimestamps.SignalingReceived = time.Now()
+	}
+}
+
+func (s *MetricsStages) RecordConnectionReady(when time.Time) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.stageTimestamps.ConnectionReady.IsZero() {
+		s.stageTimestamps.ConnectionReady = when
+	}
+}
+
+func (s *MetricsStages) RecordWGHandshakeSuccess(handshakeTime time.Time) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	if !s.stageTimestamps.ConnectionReady.IsZero() && s.stageTimestamps.WgHandshakeSuccess.IsZero() {
+		// WireGuard only reports handshake times with second precision, but ConnectionReady
+		// is captured with microsecond precision. If handshake appears before ConnectionReady
+		// due to truncation (e.g., handshake at 6.042s truncated to 6.000s), normalize to
+		// ConnectionReady to avoid negative duration metrics.
+		if handshakeTime.Before(s.stageTimestamps.ConnectionReady) {
+			s.stageTimestamps.WgHandshakeSuccess = s.stageTimestamps.ConnectionReady
+		} else {
+			s.stageTimestamps.WgHandshakeSuccess = handshakeTime
+		}
+	}
+}
+
+// Disconnected sets the mode to reconnection. It is called only when both ICE and Relay have been disconnected at the same time.
+func (s *MetricsStages) Disconnected() {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+
+	// Reset all timestamps for reconnection
+	s.stageTimestamps = metrics.ConnectionStageTimestamps{}
+	s.isReconnectionAttempt = true
+}
+
+func (s *MetricsStages) IsReconnection() bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.isReconnectionAttempt
+}
+
+func (s *MetricsStages) GetTimestamps() metrics.ConnectionStageTimestamps {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.stageTimestamps
+}
--- a/client/internal/peer/metrics_saver_test.go
+++ b/client/internal/peer/metrics_saver_test.go
@@ -0,0 +1,125 @@
+package peer
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/netbirdio/netbird/client/internal/metrics"
+)
+
+func TestMetricsStages_RecordSignalingReceived(t *testing.T) {
+	s := &MetricsStages{}
+
+	s.RecordSignalingReceived()
+	ts := s.GetTimestamps()
+	require.False(t, ts.SignalingReceived.IsZero())
+
+	// Second call should not overwrite
+	first := ts.SignalingReceived
+	time.Sleep(time.Millisecond)
+	s.RecordSignalingReceived()
+	ts = s.GetTimestamps()
+	assert.Equal(t, first, ts.SignalingReceived, "should keep the first signaling timestamp")
+}
+
+func TestMetricsStages_RecordConnectionReady(t *testing.T) {
+	s := &MetricsStages{}
+
+	now := time.Now()
+	s.RecordConnectionReady(now)
+	ts := s.GetTimestamps()
+	assert.Equal(t, now, ts.ConnectionReady)
+
+	// Second call should not overwrite
+	later := now.Add(time.Second)
+	s.RecordConnectionReady(later)
+	ts = s.GetTimestamps()
+	assert.Equal(t, now, ts.ConnectionReady, "should keep the first connection ready timestamp")
+}
+
+func TestMetricsStages_RecordWGHandshakeSuccess(t *testing.T) {
+	s := &MetricsStages{}
+
+	connReady := time.Now()
+	s.RecordConnectionReady(connReady)
+
+	handshake := connReady.Add(500 * time.Millisecond)
+	s.RecordWGHandshakeSuccess(handshake)
+
+	ts := s.GetTimestamps()
+	assert.Equal(t, handshake, ts.WgHandshakeSuccess)
+}
+
+func TestMetricsStages_HandshakeBeforeConnectionReady_Normalizes(t *testing.T) {
+	s := &MetricsStages{}
+
+	connReady := time.Now()
+	s.RecordConnectionReady(connReady)
+
+	// WG handshake appears before ConnectionReady due to second-precision truncation
+	handshake := connReady.Add(-100 * time.Millisecond)
+	s.RecordWGHandshakeSuccess(handshake)
+
+	ts := s.GetTimestamps()
+	assert.Equal(t, connReady, ts.WgHandshakeSuccess, "should normalize to ConnectionReady when handshake appears earlier")
+}
+
+func TestMetricsStages_HandshakeIgnoredWithoutConnectionReady(t *testing.T) {
+	s := &MetricsStages{}
+
+	s.RecordWGHandshakeSuccess(time.Now())
+	ts := s.GetTimestamps()
+	assert.True(t, ts.WgHandshakeSuccess.IsZero(), "should not record handshake without connection ready")
+}
+
+func TestMetricsStages_HandshakeRecordedOnce(t *testing.T) {
+	s := &MetricsStages{}
+
+	connReady := time.Now()
+	s.RecordConnectionReady(connReady)
+
+	first := connReady.Add(time.Second)
+	s.RecordWGHandshakeSuccess(first)
+
+	// Second call (rekey) should be ignored
+	second := connReady.Add(2 * time.Second)
+	s.RecordWGHandshakeSuccess(second)
+
+	ts := s.GetTimestamps()
+	assert.Equal(t, first, ts.WgHandshakeSuccess, "should preserve first handshake, ignore rekeys")
+}
+
+func TestMetricsStages_Disconnected(t *testing.T) {
+	s := &MetricsStages{}
+
+	s.RecordSignalingReceived()
+	s.RecordConnectionReady(time.Now())
+	assert.False(t, s.IsReconnection())
+
+	s.Disconnected()
+
+	assert.True(t, s.IsReconnection())
+	ts := s.GetTimestamps()
+	assert.True(t, ts.SignalingReceived.IsZero(), "timestamps should be reset after disconnect")
+	assert.True(t, ts.ConnectionReady.IsZero(), "timestamps should be reset after disconnect")
+	assert.True(t, ts.WgHandshakeSuccess.IsZero(), "timestamps should be reset after disconnect")
+}
+
+func TestMetricsStages_GetTimestamps(t *testing.T) {
+	s := &MetricsStages{}
+
+	ts := s.GetTimestamps()
+	assert.Equal(t, metrics.ConnectionStageTimestamps{}, ts)
+
+	now := time.Now()
+	s.RecordSignalingReceived()
+	s.RecordConnectionReady(now)
+
+	ts = s.GetTimestamps()
+	assert.False(t, ts.SignalingReceived.IsZero())
+	assert.Equal(t, now, ts.ConnectionReady)
+	assert.True(t, ts.WgHandshakeSuccess.IsZero())
+}
--- a/client/internal/peer/wg_watcher.go
+++ b/client/internal/peer/wg_watcher.go
@@ -48,7 +48,7 @@ func NewWGWatcher(log *log.Entry, wgIfaceStater WGInterfaceStater, peerKey strin

 // EnableWgWatcher starts the WireGuard watcher. If it is already enabled, it will return immediately and do nothing.
 // The watcher runs until ctx is cancelled. Caller is responsible for context lifecycle management.
-func (w *WGWatcher) EnableWgWatcher(ctx context.Context, onDisconnectedFn func()) {
+func (w *WGWatcher) EnableWgWatcher(ctx context.Context, enabledTime time.Time, onDisconnectedFn func(), onHandshakeSuccessFn func(when time.Time)) {
 	w.muEnabled.Lock()
 	if w.enabled {
 		w.muEnabled.Unlock()
@@ -56,7 +56,6 @@ func (w *WGWatcher) EnableWgWatcher(ctx context.Context, onDisconnectedFn func()
 	}

 	w.log.Debugf("enable WireGuard watcher")
-	enabledTime := time.Now()
 	w.enabled = true
 	w.muEnabled.Unlock()

@@ -65,7 +64,7 @@ func (w *WGWatcher) EnableWgWatcher(ctx context.Context, onDisconnectedFn func()
 		w.log.Warnf("failed to read initial wg stats: %v", err)
 	}

-	w.periodicHandshakeCheck(ctx, onDisconnectedFn, enabledTime, initialHandshake)
+	w.periodicHandshakeCheck(ctx, onDisconnectedFn, onHandshakeSuccessFn, enabledTime, initialHandshake)

 	w.muEnabled.Lock()
 	w.enabled = false
@@ -89,7 +88,7 @@ func (w *WGWatcher) Reset() {
 }

 // wgStateCheck help to check the state of the WireGuard handshake and relay connection
-func (w *WGWatcher) periodicHandshakeCheck(ctx context.Context, onDisconnectedFn func(), enabledTime time.Time, initialHandshake time.Time) {
+func (w *WGWatcher) periodicHandshakeCheck(ctx context.Context, onDisconnectedFn func(), onHandshakeSuccessFn func(when time.Time), enabledTime time.Time, initialHandshake time.Time) {
 	w.log.Infof("WireGuard watcher started")

 	timer := time.NewTimer(wgHandshakeOvertime)
@@ -108,6 +107,9 @@ func (w *WGWatcher) periodicHandshakeCheck(ctx context.Context, onDisconnectedFn
 			if lastHandshake.IsZero() {
 				elapsed := calcElapsed(enabledTime, *handshake)
 				w.log.Infof("first wg handshake detected within: %.2fsec, (%s)", elapsed, handshake)
+				if onHandshakeSuccessFn != nil {
+					onHandshakeSuccessFn(*handshake)
+				}
 			}

 			lastHandshake = *handshake
--- a/client/internal/peer/wg_watcher_test.go
+++ b/client/internal/peer/wg_watcher_test.go
@@ -35,9 +35,11 @@ func TestWGWatcher_EnableWgWatcher(t *testing.T) {
 	defer cancel()

 	onDisconnected := make(chan struct{}, 1)
-	go watcher.EnableWgWatcher(ctx, func() {
+	go watcher.EnableWgWatcher(ctx, time.Now(), func() {
 		mlog.Infof("onDisconnectedFn")
 		onDisconnected <- struct{}{}
+	}, func(when time.Time) {
+		mlog.Infof("onHandshakeSuccess: %v", when)
 	})

 	// wait for initial reading
@@ -64,7 +66,7 @@ func TestWGWatcher_ReEnable(t *testing.T) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
-		watcher.EnableWgWatcher(ctx, func() {})
+		watcher.EnableWgWatcher(ctx, time.Now(), func() {}, func(when time.Time) {})
 	}()
 	cancel()

@@ -75,9 +77,9 @@ func TestWGWatcher_ReEnable(t *testing.T) {
 	defer cancel()

 	onDisconnected := make(chan struct{}, 1)
-	go watcher.EnableWgWatcher(ctx, func() {
+	go watcher.EnableWgWatcher(ctx, time.Now(), func() {
 		onDisconnected <- struct{}{}
-	})
+	}, func(when time.Time) {})

 	time.Sleep(2 * time.Second)
 	mocWgIface.disconnect()