A worker that saw a new remote session ID rebuilt its agent and also
picked a new local ID. On the answer path nothing carries that ID back,
so the next offer made the remote see a changed session, rebuild, and
answer with yet another ID. Two peers kept tearing down working ICE
connections on every offer and answer; nearly every answer in the
affected logs carried a new remote session ID.
Only a local restart changes the local ID now: a failed negotiation, as
before, and an explicit Close, which previously kept the old ID and left
the remote answering from a negotiation this side had abandoned.
Following a remote restart keeps the ID the remote already knows, so the
pair settles after one rebuild, also against peers that still pick a
new ID when following a restart.
* [relay] Signal relay disconnects through the conn context
AddCloseListener deduplicated listeners by comparing
reflect.ValueOf(callback).Pointer(). For a method value that pointer is
the address of the compiler-generated wrapper, not an identity bound to
the receiver, so every peer's w.onRelayClientDisconnected compared equal.
All peers on the home relay register under the same connectionURL key, so
only the first registration survived and the rest were silently dropped.
On a relay disconnect those peers were never notified: statusRelay stayed
connected and the reconnect guard never fired. The relayed net.Conn itself
was closed by closeAllConns, so nothing leaked, but the peer state machine
did not learn about it. Foreign relays had the same defect scoped to the
peers sharing that server.
Rather than fixing the deduplication, drop the peer-level listener registry
entirely. A relayed Conn now exposes Context(), cancelled when the
connection is torn down, with a cancellation cause naming the reason. This
is the same shape quic-go uses for its Conn and Stream types, and it
removes the whole class of problems around listener identity, lifetime and
deregistration: the signal belongs to the resource instead of a side table.
WorkerRelay watches that context in a goroutine whose lifetime matches the
connection. A watcher that wakes up for a superseded connection compares
the conn pointer against the current one and returns without touching the
state machine, so a fast relay reconnect cannot have a stale watcher tear
down the connection that replaced it.
Client.SetOnDisconnectListener stays: it is server-level and drives the
reconnect guard and foreign relay eviction, unrelated to peers.
handleRelayReady also checks the conn context, closing the race where the
relay dies between OpenConn and the readiness handoff and the peer would
otherwise build a WireGuard endpoint over a dead connection.
TestNotifierDoubleAdd covered the removed mechanism and is gone.
TestForeignAutoClose asserted nothing (both branches logged); it now waits
for the relay to leave the client map and fails if it does not.
* [relay] Fix build: return the concrete conn from Client.OpenConn
OpenConn now returns *Conn, but it still went through connContainer.netConn(),
which widens to net.Conn. The helper had one caller and only existed to produce
the interface value the signature no longer wants, so return container.conn
directly and drop it.
* [relay] Assert the local-close cancellation cause explicitly
The local-close test only rejected ErrServerDisconnected, so it would also
have passed for ErrPeerDisconnected or a bare context.Canceled. closeConn
cancels with net.ErrClosed, so assert that.
* [client] Ignore relay disconnects from superseded connections
The relayed conn watcher compared the conn pointer under relayLock, released
it, and only then tore the connection down. A new offer could install its
replacement in that window, so a watcher that validated the old pointer went
on to close the proxy of the connection that had already replaced it and
report the peer as disconnected while it was up.
Move the decision to where the teardown happens. Conn records which relayed
connection the current proxy was built from, and onRelayDisconnected takes the
connection the signal belongs to and drops it under conn.mu when it is no
longer the current one. Check and effect are now in the same critical section,
so the verdict cannot go stale before it is acted on.
This also covers the proxy read loops, whose disconnect listener took no
argument and had the same defect: it now names the connection it belongs to.
The WG timeout path keeps passing nil, since it deliberately tears down
whatever is current.
* [client] Bind the relayed conn reference to the proxy swap
relayedConnRef was set at the top of the readiness path, but wgProxyRelay only
changes at the end, in setRelayedProxy. The two failure returns in between —
newProxy and ConfigureWGEndpoint — left the reference pointing at a connection
that never became active while the old proxy was still installed. A disconnect
of that old, live relay would then be dismissed as belonging to a superseded
connection and never cleaned up.
Set the reference in setRelayedProxy, next to the proxy it belongs to. Both
success paths go through it and neither failure path does, so no failure branch
has to remember to roll anything back.
GetSessionExpiresAt took the exclusive lock for a plain field read, so
every caller queued behind writers and behind each other. The Android
SessionMonitor polls it from the main thread, and in the captured ANR
that is exactly where the main thread was blocked while hundreds of
peer-list callbacks held or waited on the same mutex.
d.mux is already an RWMutex and the other getters use RLock; this brings
the deadline read in line with them.
Read the relay instance URL and IP atomically to prevent reconnects from mixing values from different connections. Extend existing connection and offer/answer logs with relay URLs and IPs to help trace mismatched advertisements.
pqControllerReoffer returned true whenever we are the controller running the
KEM, even when ShouldSendBootstrapOffer was false — which is the steady state
for every PQ peer past its bootstrap (exchange in awaitingRekey) and for
non-capable peers. handleRemoteOffer then returned early, dropping the ICE
credentials and relay info carried in the peer's offer, so a responder-initiated
reconnect (the normal path under lazy connections) stalled: the controller only
recovered on its own guard timing.
Return true only when a re-offer was actually sent; otherwise fall through to the
normal offer handling (sendAnswer + notifyListeners) so the connection can come
up on the retained PSK. A stale PSK still self-heals via the WG watcher, which
triggers a fresh signal offer that re-bootstraps.
- strict-kem vs strict-rp said "Connected + Quantum resistance: true" but it is actually blocked
- perm-kem vs perm-rp "Connected + Quantum resistance: true" but it's a classic WG link, without PQ safety
Reduce Listen's cognitive complexity (SonarCloud S3776, 30 -> under 25) by
extracting the offer and answer cases into handleRemoteOffer/handleRemoteAnswer,
with shared onSignalReceived/notifyListeners helpers and a pqControllerReoffer
helper for the controller re-offer branch. No functional change.
The responder configures WireGuard with endpoint=nil first, then a delayed
update (scheduleDelayedUpdate) applies the real endpoint after fallbackDelay.
It captured the preshared key at schedule time and re-applied it. With the
post-quantum exchange the PSK can change within that window (a fresher PSK
derived and applied via SetPresharedKey), so re-applying the captured one
reverted WireGuard to a key the remote peer no longer used — a mismatch that
stalled the handshake until the WGWatcher timeout forced a retry (~30s).
Pass a nil PSK in the delayed update so it only sets the endpoint and leaves
the current PSK in place; the latest SetPresharedKey wins.
When the controller receives the responder's (KEM-less) offer it replies with
its own KEM offer instead of answering, so the only transaction that brings the
tunnel up is the one that also carries the PSK. Guard that reply with
ShouldSendBootstrapOffer so it fires only when no exchange is in flight: without
it, every responder offer triggered another offer (an offer-per-offer runaway).
The whole behaviour is isolated to the KEM path (config.PQ != nil); non-PQ
connections answer as before.
To ensure two peers agree on a key, we need asymmetry. one peer is
the controller ("initiator") the other is the "responder".
Otherwise imagine two offers in parallel driving two answers at the same time
A B
| <----B-OFFER----- |
| -----A-OFFER----> |
| |
| |
---------------------------------
|****** ICE + WG Handshake ****** |
---------------------------------
| |
| <----B-ANSWER---- |
| -----A-ANSWER---> |
PSK is derived on receive of offer, so A and B derive different PSKs.
When WG handshake takes place it picks misaligned PSKs.
So we impair the two nodes and only the offer of one of the two (the controller/initiator)
is allowed to progress and drive the answer (and carry the KEM material).
If a responder initiates an offer, we redo the offer towards it. This is oK
since the ICEworker don't treat offer/answers differently.
To ensure two peers agree on a key, we need asymmetry. one peer is
the controller ("initiator") the other is the "responder".
Otherwise imagine two offers in parallel driving two answers at the same time
A B
| <----B-OFFER----- |
| -----A-OFFER----> |
| |
| |
---------------------------------
|****** ICE + WG Handshake ****** |
---------------------------------
| |
| <----B-ANSWER---- |
| -----A-ANSWER---> |
PSK is derived on receive of offer, so A and B derive different PSKs.
When WG handshake takes place it picks misaligned PSKs.
So we impair the two nodes and only the offer of one of the two (the controller/initiator)
carries the KEM material.
This means that if the responder OFFER/ANSWER comes first, when the controller/initiator's one
completes (and the genuine PSK is shared between A and B, we need to force a new WG handshake with
the proper keys.
OnRekeyFailed now re-runs the KEM bootstrap over Signal (conn.RequestReoffer ->
handshaker.SendOffer) instead of only logging: a fresh signalling offer starts a new
exchange that overwrites the stalled PSK on both sides, resyncing after a persistent
data-path desync. Chosen over a responder-side awaitingAck revert (which fights the
confirm-less ack timing) and a full tunnel teardown (heavier). The tunnel stays up on
the previous PSK meanwhile since Signal is independent of the broken data path.
Strict mode (NB_PQ_MLKEM_STRICT, default off) closes the initial PQ-vulnerable
window (NET-1408): when enabled, conn.presharedKey programs a per-conn random
sentinel PSK until the ML-KEM exchange derives the real one, so no session can form
on a non-PQ key (the real PSK is pushed via SetPresharedKey once it converges).
Default stays opportunistic.
Also surface PQ status: the peer 'Quantum resistance' flag (RosenpassEnabled) is now
true when an ML-KEM PSK has been derived for the peer, not only for Rosenpass.
The idle-gate reads LastActivities, which only tracks per-peer data in userspace;
in kernel mode it is empty, so the gate treated every kernel peer as idle and
disabled data-path rotation entirely. Detect the bind via IsUserspaceBind and, in
kernel mode, report zero activity age (always 'active') so rotation runs on every
rekey. Lazy back-to-idle is already limited in kernel; the eBPF WG-activity
detection will later supply a real signal that excludes handshake/pqkem traffic.
Source OnDataPathRekeyed from the WGWatcher's per-handshake callback
(onWGCheckSuccess), which fires only on a fresh handshake, and OnDataPathDown
from the handshake-timeout path. A fresh handshake clocks the next chained
KEM exchange pushed over the data-path UDP transport.
Learn the peer's data-path endpoint from the signalling offer/answer: its WG
overlay IP combined with the advertised pq UDP port (SetRemotePort -> AddPeer).
Registering here is safe before the tunnel is up because sends only ever fire
once it is (clocked by OnDataPathRekeyed). RemovePeer is wired at peer teardown
(engine.removePeer), not on transient disconnect.
* [client] Drop agentConnecting whenever ICE session state clears
Closing a WorkerICE raced a blocked dial goroutine: Close released the
agent while connect() was still inside Dial, and the goroutine's own
cleanup skipped its flag reset because w.agent no longer matched. With
agentConnecting stuck on true, evalConnStatus read the peer as
connected, the reconnection guard stopped sending offers and
same-session offers were dropped, so the peer could not recover without
a restart. An aborted recreate in OnNewOffer reaches the same wedged
state without any race.
Route every teardown path through one abandonNegotiation helper so the
agent and flag fields always clear together; Close now also cleans up
residual state left by an aborted recreate.
* [client] Drive the ICE teardown race test through the real dial goroutine
The regression test simulated the stale goroutine by calling closeAgent
directly, so it pinned the symptom rather than the mechanism. Rework it
to start a real negotiation, tear it down mid-flight and let the actual
goroutine run its own cleanup: with no remote responder the dial can
only fail once Close cancels it, so the interleaving stays deterministic
without sleeps or injection points.
Assert the full idle state that abandonNegotiation owns (agent nil,
connecting false, remote session ID empty) instead of only InProgress,
and make the stale-cleanup ownership test verify that the newer session
survives field by field.
* [client] Assert live remote session ID after stale ICE cleanup
The stale-cleanup test compared a snapshot captured before closeAgent
ran, so clearing the field during cleanup would have gone unnoticed.
Read the field under the mutex after the cleanup instead.
* [client] Give the ICE race tests a no-op signal client
The candidate callback fires from a real gather and dereferences the
signaler, so a nil one crashes the test package intermittently when
gather wins the race against Close. Build the worker with a stub
signal.Client instead.
* [client] Read the ICE dial cancel func from an argument in connect
The error paths read w.agentDialerCancel without holding muxAgent while
OnNewOffer rewrites the field for a newer negotiation, a data race the
new teardown test trips under -race. Reading a stale value also let an
old goroutine cancel another session's dial. Capture the cancel func at
goroutine spawn, like the dial context already is.
* [client] Guard the ICE dial success path against stale negotiations
The stale-cleanup guard in closeAgent only protected teardown. Its
success-path counterpart was missing: an older negotiation could complete
agentDial after a newer one replaced w.agent, then clear the newer
session's agentConnecting, record lastSuccess and publish its dead
connection via onICEConnectionIsReady.
Verify ownership under muxAgent twice: right after the dial returns, so a
stale goroutine drops its connection before touching a closed agent, and
again at the state-commit point, atomic with the agentConnecting and
lastSuccess writes, so a replacement arriving in the meantime cannot get
its state clobbered. Both paths close the stale connection and return
without modifying worker state. A regression test holds session A's dial
open until session B is installed, then releases it; the stale connection
must be discarded and B's agent, connecting flag and remote session ID
must survive.
* [client] Fix ICE teardown test leak and document the stale delivery window
A code review of the stale-negotiation guard found a leftover resource
leak in TestWorkerICE_StaleCloseAgentKeepsCurrentSession: session B is
never closed, so its ICE sockets and blocked dial goroutine live as long
as the test process. Register t.Cleanup(w.Close).
The delivery race flagged after the success-path guard is pre-existing
and self-correcting - the newer negotiation overwrites the transient
endpoint - so document it in the existing todo instead of locking the
callback, which would invert lock order against Conn.Close. Adjust the
teardown test comment to match the now-synchronous Close flag clearing.
Losing the last network only flipped the availability state: the dead management, signal and relay sockets stayed silently connected until their own timeouts, so the client kept reporting Connected with no network at all.
Introduce client/netevents with a Manager that ties the availability state, the connection sweeper and the status recorder together, and move the netstate and netsweep packages under it (netsweep renamed to sweep). SetNetworkAvailable(false) now also sweeps the registered connections so their owners redial and the listener reaches the NoNetwork state.
The Android and iOS bindings own a Manager instance and inject it through the constructors; consumers hold the concrete *Manager whose nil zero value reports always-online and never sweeps, with interfaces kept only as parameter contracts. The relay guard settle wait moved into the Manager as WaitSettled, removing the netevents import from the relay package.
On network changes the client restarted the whole engine. That is heavy-handed and slow: it tears down working state to recover from a transition the engine could handle itself. This replaces the restart with proper network event handling.
Suspend the retry loops while no network is available. Instead of burning through backoff intervals against an unreachable network, the reconnection loops park until the OS reports a usable network again.
Reconnect immediately on a network switch. When the OS hands us a new network, connections bound to the old one are swept and re-dialed right away, rather than waiting for a timeout to notice they are dead.
## Describe your changes
`recordConnectionMetrics` mapped only `conntype.Relay` to `relay` and
let a `default` branch
record everything else as `ice`. That silently included `ICETurn` — an
ICE connection through
a TURN server, which
[`conn.isRelayed`](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L788-L795)
itself counts as relayed — and `None`, the transient state set when the
relay drops
([conn.go:632](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L632))
or the peer state is reset
([conn.go:757](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L757)).
Both were reported as direct peer-to-peer, so the `ice` share overstated
direct connections on
every platform.
The mapping now lists every priority explicitly and emits `ice_p2p`,
`ice_turn`, `relay` or
`unknown`. The new values deliberately do not reuse `ice` to avoid
ambuguity.
## Issue ticket number and link
No public issue. Found while reviewing the first production sample of
client metrics: 38% of iOS
connection events were tagged `ice` on a platform that forces relay by
default, which traced back
to the `default` branch at
[client/internal/peer/conn.go#L963-L968](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L963-L968).
## Stack
<!-- branch-stack -->
### Checklist
- [x] Is it a bug fix
- [ ] Is a typo/documentation fix
- [ ] Is a feature enhancement
- [ ] It is a refactor
- [x] Created tests that fail without the change (if possible)
> By submitting this pull request, you confirm that you have read and
agree to the terms of the [Contributor License
Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md).
## Documentation
Select exactly one:
- [] I added/updated documentation for this change
- [x] Documentation is **not needed** for this change (explain why)
Internal metrics documentation only, in
`client/internal/metrics/infra/README.md`: the four
`connection_type` values with their derivation, and a note that pre-fix
`ice` samples are not
comparable with `ice_p2p`. No public API, CLI or configuration change,
so no netbirdio/docs PR.
### Docs PR URL (required if "docs added" is checked)
Paste the PR link from https://github.com/netbirdio/docs here:
N/A
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
* **New Features**
* Connection metrics now distinguish direct peer-to-peer, TURN-assisted,
relay, and unknown connection types.
* Metrics include clearer connection and peer identification details.
* **Documentation**
* Updated connection timing metric values, traffic semantics, priority
behavior, and historical data guidance.
* **Bug Fixes**
* Unset or unrecognized connection priorities are no longer incorrectly
classified as peer-to-peer.
* Unknown-transport metrics are skipped to prevent misleading connection
data.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Describe your changes
The Conn struct is reused across lazy-connection deactivate/activate.
Close
cancels the WireGuard watcher (via wgWatcherCancel, and ctxCancel also
tears
down its context) but left conn.wgWatcher pointing at the stopped
instance.
enableWgWatcherIfNeeded skips while conn.wgWatcher is non-nil, so the
next Open
never started a fresh watcher: once a lazy connection had idled and
woken, the
peer ran with no watcher at all — no WireGuard handshake-timeout
detection and
none of the escalation that depends on it.
Clear conn.wgWatcher and conn.wgWatcherCancel in Close so the next Open
re-arms
a fresh watcher.
## Issue ticket number and link
<!--
Required for anything that changes behavior. Link the issue (or the
validated
discussion it came from) that the NetBird team already agreed on. See
https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#ticket-first-pr-second
-->
## Stack
<!-- branch-stack -->
### Checklist
- [X] Is it a bug fix
- [ ] Is a typo/documentation fix
- [ ] Is a feature enhancement
- [ ] It is a refactor
- [ ] Created tests that fail without the change (if possible)
- [ ] I ran and tested this change locally — I did not rely on CI to
find out whether it works
- [ ] This PR has a single purpose (not a fix + refactor + feature in
one)
- [ ] This change is a trivial fix, **OR** it links an issue the NetBird
team agreed on beforehand. Changes to the public API, gRPC protocols,
functionality behavior, CLI / service flags, or new features always need
that agreement first. See
[CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#ticket-first-pr-second).
> By submitting this pull request, you confirm that you have read and
agree to the terms of the [Contributor License
Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md).
## Documentation
Select exactly one:
- [ ] I added/updated documentation for this change
- [X] Documentation is **not needed** for this change (explain why)
### Docs PR URL (required if "docs added" is checked)
Paste the PR link from https://github.com/netbirdio/docs here:
https://github.com/netbirdio/docs/pull/__
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved connection cleanup by fully releasing WireGuard watcher
resources when a connection closes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Stick new watcher creation to actual existence of af the conn
and its removal to the removal of such same conn.
Avoid debouncing and cross lock dead locking
* Discriminate not updated from timeout handshakes
* [Recheck watcher ctx cancellation under conn.mu in onWGDisconnected
onWGDisconnected only checked conn.ctx (the engine-scoped context), never
the watcher's own context. disableWgWatcherIfNeeded cancels the wgWatcherCtx,
not conn.ctx, so a disabled watcher's timeout callback did not see the
cancellation.
handshakeCheck runs lock-free, so between the ctx check in periodicHandshakeCheck
and acquiring conn.mu a fast disconnect/reconnect can slip in: the stale watcher
then acquires the lock and tears down the *new*, healthy connection based on the
old timeout, forcing the guard into an unnecessary reconnect (flap).
Recheck watcherCtx.Err() under conn.mu so a superseded watcher exits without
touching the connection that replaced it.
* Remove verbose comments
* Fixup merge conflict leftovers
* Fixup context brought by onWGDisconnected
- **Wails v3 application** (`client/ui`) with a React + TypeScript + Tailwind frontend replacing the Fyne UI: main connection view, exit-node switcher, networks/peers browser with detail panels, profile management, settings (general, network, SSH, security, troubleshooting, appearance), debug-bundle creation, and a first-run welcome flow.
- **Internationalization**: go-i18n bundle with 9 locales (en, de, es, fr, hu, it, pt, ru, zh-CN) shared between the tray and the frontend.
- **New system tray** implementation with per-platform theme-aware icons, including a native XEmbed host for Linux (`xembed_tray_linux.c`) and a Linux theme watcher.
- **Session handling**: auth session watcher (`client/internal/auth/sessionwatch`), pending login flow, session-expiration dialog and tray notifications, and `netbird login` improvements.
- **Daemon API extensions** (`daemon.proto`): status stream subscription, event stream, networks/exit-node selection endpoints, and richer full status — with probe throttling on the daemon side to protect against UI-driven request storms.
- **UI preferences store** persisted per profile, autostart management via the daemon (single source of truth in HKCU on Windows).
- **Build system**: Taskfile-based builds per platform (macOS, Linux, Windows), Docker cross-compilation images, MSIX/NSIS/nfpm/AppImage packaging, and a new `frontend-ui` CI workflow.
Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
Co-authored-by: Eduard Gert <kontakt@eduardgert.de>
Co-authored-by: braginini <bangvalo@gmail.com>
Co-authored-by: Pascal Fischer <32096965+pascal-fischer@users.noreply.github.com>
Co-authored-by: riccardom <riccardomanfrin@gmail.com>
* [client] Fix race between WG watcher initial handshake read and endpoint config
The watcher's initial handshake read ran in a separate goroutine with no
ordering guarantee relative to the WireGuard endpoint configuration, so it
would sometimes race with the peer being added to the interface. Split
enabling into a synchronous PrepareInitialHandshake, called before the
endpoint is configured, and an EnableWgWatcher that only runs the
monitoring loop, making the baseline read deterministic and keeping it
correct for reconnects where the peer's WireGuard entry survives.
* [client] Skip WG watcher disconnect callback when context is cancelled
A superseded or cancelled watcher whose handshake-check timer fires before
it observes ctx.Done() would still invoke onDisconnectedFn, tearing down a
now-healthy connection. Re-check ctx before firing the disconnect and
handshake-success callbacks and stand down silently if it was cancelled.
* peer/status: move relay-state reads off the main mux
GetRelayStates held d.mux (RLock) while calling into the relay
Manager (RelayStates/RelayConnectError/ServerURLs). Those calls can be
slow or block on the relay manager's own locks while it is reconnecting,
which kept the central Status mutex held and stalled every peer state
writer (UpdatePeerState, ReplaceOfflinePeers, etc.) contending for it.
Guard relayMgr/relayStates with a dedicated muxRelays mutex and release
it before invoking the relay Manager, so the relay read path no longer
contends with the hot peer-state writers on d.mux.
* peer/status: clone relay states in nil-manager path
Return a cloned snapshot of d.relayStates when relayMgr is nil so callers
cannot mutate the shared cached state, matching the non-nil path.
riccardom
