Replace the fixed 1.5s pre-dial sleep with a wait driven by the OS
availability verdict: while online only a 200ms settle window is spent
(the offline flag lands a few milliseconds after the disconnect that
triggered the reconnect), and while offline the guard waits for the
network to return, bounded by the same 1.5s budget as before. Without
an injected netState the behavior is unchanged.
Measured on Android: relay recovery after a network switch drops from
1.7-1.9s to under 500ms after the cut.
WrapDialContext and WrapConn registered the dial and the connection
independently, so a sweep landing between the dial finishing and WrapConn
cancelled only the dial registration: the connection dialed on the old
network entered the fresh registry and survived the network change.
Replace the pair with a Dial handle. Sweep marks pending dials under the
sweeper mutex, and WrapConn decides under the same mutex: a swept dial's
connection is closed and ErrSwept returned, so the caller redials on the
new network; otherwise the connection transfers to the registry with no
window in between.
sweptConn embeds net.Conn, so it does not promote Protocol() from the
concrete relay connection. Asserting transportConn on the wrapper always
failed and left c.transport empty. Read the transport off the dialed
connection first, then wrap it.
On a network switch (e.g. cellular to WiFi) the management, signal and
relay sockets stay bound to the old network and look alive until the OS
tears them down — measured at 5 seconds of dead air on Android, while
the UI kept claiming Connected. The Android client papered over this
with a full engine restart, paying for it with a torn-down TUN device
and discarded peer state.
Introduce client/netsweep: connections register on dial and deregister
on close, and a sweep closes everything registered while aborting
in-flight dials through sweep-cancellable dial contexts. The aborted
dials matter: a relay dial started on the dying network would otherwise
hold the reconnect loop hostage for the QUIC handshake timeout. After a
sweep every failure surfaces as an ordinary read/write error and the
existing retry loops redial immediately on the new network.
The sweeper reaches the three long-lived connections through the same
options that carry the netstate gate: a gRPC dial option wraps the
management and signal transports (reconnects included), and the relay
client wraps its connection in one place for the picker, the guard and
foreign relays alike. Everything is nil-safe; platforms that inject no
sweeper are untouched.
Mobile clients expose the sweep as NotifyNetworkChange. Measured on
Android against the engine restart it replaces: recovery in 1.6s
instead of 3.2s, no Disconnected flash, and the TUN device, WireGuard
config and peer state survive.
On mobile the client kept dialing management, signal, relay and peer
connections while the device had no usable network at all (airplane
mode), burning battery for attempts that cannot succeed. Stopping the
engine is not an option: tearing it down destroys the TUN device, and
traffic can leak outside the tunnel until it is rebuilt.
Add client/netstate, a small gate the platform feeds from its own
connectivity callbacks. Every reconnection loop waits on it instead of
retrying blindly, and resets its backoff when the network returns so
recovery is immediate. The state is injected through functional options
and consumers hold a *State that may be nil, so every platform that does
not report availability behaves exactly as before.
The relay quick-reconnect rechecks availability after its 1.5s wait: the
disconnect that triggers it is usually the first symptom of the network
going away, so the flag typically arrives while it sleeps.
Report the suspension to the UI as well. peer.Listener grows
OnStateChanged with a typed ClientState, re-exported across the gomobile
boundary as integer constants, and the notifier maps Connecting to a new
NoNetwork state while the OS reports no network, so mobile clients can
show "no network available" instead of a misleading "connecting".
Finally, exit the client retry loop cleanly when its context is
cancelled. backoff.WithContext surfaces the bare context error, which
callers could not distinguish from a real failure — on Android that
turned an engine restart into an unrecoverable error.
## Describe your changes
The relay client's reconnect backoff was constructed without a
`RandomizationFactor`
([guard.go:156-165](https://github.com/netbirdio/netbird/blob/main/shared/relay/client/guard.go#L156-L165)),
so it kept the zero value: every client that lost the same relay server
retried on the identical 2/4/8/16/32/60, potentially in waves.
It was the only exponential backoff in the codebase without a
randomization factor.
Use `backoff.DefaultRandomizationFactor`, as random factor.
## Issue ticket number and link
No public issue. Found while reviewing the relay reconnect path for the
client-metrics review:
22k relay reconnection events in 24h, and the shared transport's own
retry schedule was
identical across all clients.
## Stack
<!-- branch-stack -->
### Checklist
- [x] Is it a bug fix
- [ ] Is a typo/documentation fix
- [ ] Is a feature enhancement
- [ ] It is a refactor
- [x] Created tests that fail without the change (if possible)
> By submitting this pull request, you confirm that you have read and
agree to the terms of the [Contributor License
Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md).
## Documentation
Select exactly one:
- [ ] I added/updated documentation for this change
- [x] Documentation is **not needed** for this change (explain why)
Internal retry-timing change with no user-visible surface: no CLI flag,
configuration option
or API field is added or altered, and the mean reconnect delay is
unchanged.
### Docs PR URL (required if "docs added" is checked)
Paste the PR link from https://github.com/netbirdio/docs here:
N/A
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved reconnect timing to distribute repeated connection attempts
more evenly and reduce synchronized retry spikes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* [relay] evict foreign client cache on disconnect
When a foreign relay's TCP connection drops, the manager's
onServerDisconnected handler only triggered reconnect logic for the
home server; the disconnected foreign entry stayed in the relayClients
cache. Subsequent OpenConn calls reused the closed client until the
60-second cleanup tick evicted it, breaking peer connectivity through
that relay for up to a minute.
Evict the foreign entry from the cache on disconnect so the next
OpenConn dials a fresh client.
Also:
- Make the reconnect backoff cap configurable via WithMaxBackoffInterval
ManagerOption; the previous hard-coded 60s constant forced
TestAutoReconnect to sleep ~61s. Test now polls Ready() and finishes
in ~2s.
- Add NB_HOME_RELAY_SERVERS env var that overrides the relay URL list
received from management, so a peer can be pinned to a specific home
relay (used by the netbird-conn-lab Edge 4 reproducer).
* [client] treat empty NB_HOME_RELAY_SERVERS as unset
Returning (urls=[], ok=true) when the env var contained only separators or
whitespace caused callers to wipe the mgmt-provided relay list, leaving the
peer with no relays. Treat a parsed-empty result the same as an unset env.
* [client] Use `net.JoinHostPort` for consistency in constructing host-port pairs
* [client] Fix handling of IPv6 addresses by trimming brackets in `net.JoinHostPort`
* Fix race condition and ensure correct message ordering in
connection establishment
Reorder operations in OpenConn to register the connection before
waiting for peer availability. This ensures:
- Connection is ready to receive messages before peer subscription
completes
- Transport messages and onconnected events maintain proper ordering
- No messages are lost during the connection establishment window
- Concurrent OpenConn calls cannot create duplicate connections
If peer availability check fails, the pre-registered connection is
properly cleaned up.
* Handle service shutdown during relay connection initialization
Ensure relay connections are properly cleaned up when the service is not running by verifying `serviceIsRunning` and removing stale entries from `c.conns` to prevent unintended behaviors.
* Refactor relay client Conn/connContainer ownership and decouple Conn from Client
Conn previously held a direct *Client pointer and called client methods
(writeTo, closeConn, LocalAddr) directly, creating a tight bidirectional
coupling. The message channel was also created externally in OpenConn and
shared between Conn and connContainer with unclear ownership.
Now connContainer fully owns the lifecycle of both the channel and the
Conn it wraps:
- connContainer creates the channel (sized by connChannelSize const)
and the Conn internally via newConnContainer
- connContainer feeds messages into the channel (writeMsg), closes and
drains it on shutdown (close)
- Conn reads from the channel (Read) but never closes it
Conn is decoupled from *Client by replacing the *Client field with
three function closures (writeFn, closeFn, localAddrFn) that are wired
by newConnContainer at construction time. Write, Close, and LocalAddr
delegate to these closures. This removes the direct dependency while
keeping the identity-check logic: writeTo and closeConn now compare
connContainer pointers instead of Conn pointers to verify the caller
is the current active connection for that peer.
* Fix race condition and ensure correct message ordering in
connection establishment
Reorder operations in OpenConn to register the connection before
waiting for peer availability. This ensures:
- Connection is ready to receive messages before peer subscription
completes
- Transport messages and onconnected events maintain proper ordering
- No messages are lost during the connection establishment window
- Concurrent OpenConn calls cannot create duplicate connections
If peer availability check fails, the pre-registered connection is
properly cleaned up.
* Handle service shutdown during relay connection initialization
Ensure relay connections are properly cleaned up when the service is not running by verifying `serviceIsRunning` and removing stale entries from `c.conns` to prevent unintended behaviors.
- Move `util/grpc` and `util/net` to `client` so `internal` packages can be accessed
- Add methods to return the next best interface after the NetBird interface.
- Use `IP_UNICAST_IF` sock opt to force the outgoing interface for the NetBird `net.Dialer` and `net.ListenerConfig` to avoid routing loops. The interface is picked by the new route lookup method.
- Some refactoring to avoid import cycles
- Old behavior is available through `NB_USE_LEGACY_ROUTING=true` env var
The health check endpoint listens on a dedicated HTTP server.
By default, it is available at 0.0.0.0:9000/health. This can be configured using the --health-listen-address flag.
The results are cached for 3 seconds to avoid excessive calls.
The health check performs the following:
Checks the number of active listeners.
Validates each listener via WebSocket and QUIC dials, including TLS certificate verification.
