mirror of
https://github.com/netbirdio/netbird.git
synced 2026-10-14 09:29:07 +02:00
70ddb2a39380605c8b0d928f15736489a14d86c0
213
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Riccardo Manfrin
|
70ddb2a393 |
[client] Bound the Signal stream registration wait and track its state based on stream health (#8175)
* [client] Move the Signal registration wait into its own helper Pure move, no behavior change. The wait for the registration header and the check on its value leave connect() for awaitRegistration(), so the next commit can bound that wait in one place. * [client] Split the Signal test server helper from its listener setup Pure move, no behavior change. serveTestSignalExchange serves any SignalExchangeServer on a local listener; startTestSignalServer keeps serving the real server through it. Lets a test wrap the real server. * [client] Bound the wait for the Signal registration header After an outage the Signal client could wait forever for the registration header of a new stream. Keepalive cannot end that wait when whatever answers the transport (a proxy in front of Signal, or Signal itself) accepts the stream and never answers it, and the receive watchdog only starts once registration succeeds. Signal then stayed down until the daemon was restarted, while Management and Relay recovered. Cancel the stream when the header does not arrive within 10 seconds, so connect() fails and the existing retry loop opens a new stream. A timer is used instead of a context deadline because the stream context must outlive registration. * [client] Report Signal unhealthy while no stream is registered IsHealthy judged Signal by the gRPC connection state alone and marked it connected whenever the connection was Ready or Idle. With the connection up but no stream registered, the client cannot receive offers or answers, yet netbird status showed Signal: Connected, hiding a stuck registration for as long as it lasted. Require a registered stream before reporting healthy. When there is none, return false without touching the recorder: the disconnect that dropped the stream already reported its reason. * [client] Name the server that reconnected in the SR watcher log Signal and Relay shared one reconnect callback, which logged "reconnected to Signal or Relay server" for either. A Relay reconnect while the Signal stream was down read as Signal having recovered, which misled diagnosis of a stuck Signal registration. Register one callback per server so the log names the one that actually reconnected. Which listeners get notified, and when, is unchanged. * [client] Trim comments in the Signal registration and health code Comment-only change, no behavior change. Drop comments that restate the code and keep only the reason a timer is used instead of a context deadline. * [client] Check Signal stream registration before the connection state IsHealthy returned true for a Connecting or Shutdown channel before the registration check ran, so a client with no registered stream could still be reported healthy while the channel was reconnecting. Check the stream first; the connection-state cases now only apply to a client that has a registered stream. * [client] Log a plain connect in the SR watcher The Signal and Relay callbacks also fire on the first connection, not only after a disconnect, so "reconnected" was wrong at startup. Log "connected to <server> server" for both cases. * [client] Say what the SR watcher does when it logs The line was read as a per-server connect log, which Signal and Relay already emit themselves. What it marks is the peer guards being told to reset their reconnection backoff, which happens only once both are ready. Log that, naming the server whose event triggered it. |
||
|
Riccardo Manfrin
|
3e85e40be2 |
[client] Cache the WireGuard interface check shared by ICE agents (#8001)
* [client] Take a WireGuard detector through the interface filter The interface filter answers whether an interface is a WireGuard device by opening a wgctrl client and asking for it, and it does that for every interface it is given. Nothing about that call is tied to the caller, so it can be answered by a shared object instead of being repeated, but the filter has no way to receive one. InterfaceFilter and the constructors that build one now take a detector, and the ICE config carries it so that every agent can be handed the same one. Nobody supplies a detector yet: a nil one probes on every call, which is what the filter did before, so this changes no behaviour. * [client] Share one WireGuard detector across every ICE agent Creating an ICE agent builds two interface filters, one for the agent and one for the transport net it sits on, and each is asked about every host interface. For an interface the disallow list does not settle, answering means opening a wgctrl client, which builds a kernel and a userspace client and resolves the netlink family, and then a round trip that usually just reports the device does not exist. An agent is created per peer connection attempt, so on a large network that runs constantly: on a routing peer with ~16000 peers it measured 2.40s of a 66.59s CPU profile, 3.6%, split evenly between opening the client and the round trip. The engine now owns a detector and passes it to every agent through the ICE config, so the answer for an interface is reused instead of being asked again for each agent. It is kept for a second, short enough that a WireGuard interface appearing is picked up before ICE settles on candidates over it. The callers that build one filter and keep it, the relay and the UDP mux, keep passing nil and so keep probing, which costs them nothing at their rate. * [client] Recheck the WireGuard cache inside the singleflight group A caller that saw an expired entry could enter the singleflight group after another caller had already refreshed the entry and left it, and probe the interface a second time. Read the cache again inside the group before probing. This also makes the concurrent probe test independent of scheduling: a late caller finds the fresh entry instead of starting a new probe. * [client] Drop expired WireGuard detector entries The detector lives as long as the engine and kept an entry for every interface name it was ever asked about. On hosts that churn interfaces, such as container veths, the map only grew. Remove expired entries when a new answer is stored; the map holds a few dozen names at most, so the sweep is cheap and runs at most once per interface per TTL. * [client] Skip the disallow-list filter test on iOS InterfaceFilter does not apply the disallow list on iOS, so the subtest reaches the probe there and its no-probe assertion cannot hold. |
||
|
Pascal Fischer
|
2623feeb5b | [management] remove ingress ports (#8062) | ||
|
Zoltan Papp
|
d56e6fc5f4 |
[client, android, ios] Coalesce peer list change notifications to the mobile listener (#7546)
* [client] Coalesce peer list change notifications to the mobile listener Every peer state change spawned a goroutine to call the platform listener. During a reconnect storm this pinned hundreds of OS threads in JNI and let the UI call back into the engine from each of them. Deliver peer list changes from a single goroutine per listener and collapse pending changes into the latest count. * [client] Cover a pending wake-up when the peer list deliverer is replaced The replacement test waited for the old callback to finish before swapping listeners, so it never exercised the stop check that runs after a wake-up. Block the old callback, queue a peer list change and swap while it is blocked, then assert the old listener never sees the new count. * [client] Signal peer list deliverer exit and wait for it in the test The replacement test sampled the old listener after a fixed sleep, so a late stale delivery could slip past it. Close a done channel when the deliverer goroutine returns and let the test wait on it instead. * [client] Drop the test-only peer list deliverer exit channel The done channel was only read by the replacement test. Production code cannot wait on it, since joining the deliverer would block on a mobile callback. The tests now drive the deliverer loop directly and check that setListener and removeListener close its stop channel. |
||
|
Riccardo Manfrin
|
f175e402c7 |
[client] stop offering to every peer when the relay transport drops (#7092)
* [client] stop offering to every peer when the relay transport drops The relay transport is shared: one connection per relay server carries the streams of every peer using it. When it drops, each of those peers gets a Disconnected verdict from evalConnStatus even when ICE is still carrying its traffic, because peerUsesRelay comes from HasRelayAddress(), which only reports that management offered relay servers, not that we are connected to one. The guard answers Disconnected with the aggressive retry, so every peer starts sending offers over signal for a transport that no offer can restore: the relay client's own guard is what reconnects it. Feed relayManager.Ready() into the status inputs and return PartiallyConnected when ICE is up and the missing side is the shared transport. That is the existing "one path works, the other does not" branch, which retries three times and then hourly instead of walking the exponential ladder forever. Peers are not left waiting for the hourly tick: when the transport comes back, Manager.onServerConnected notifies srWatcher, the guard resets the ticker to 800ms and iceState.reset() clears the hourly mode. The verdict is unchanged when the transport is up but this peer is unreachable over relay - it may have moved to another server, and only an offer carries its new relay address - and in force-relay mode, where relay is the only transport. * Renaming according to actual meanings * Don't consider an in progress ICE as "partially connected" when the relay is not.. * Aligns tests * Address wrong comments |
||
|
Zoltan Papp
|
6b3cfbabd2 |
[client] Resolve the Android network route peer by HA unique ID instead of scanning the full status (#7705)
* [client] Build the Android network list from a single peer snapshot Networks() called GetFullStatus() once per network to find the peer that serves it, copying every peer state and taking eight recorder locks each time. With 100+ peers and the UI calling Networks() from every peer list change, this queued hundreds of callers on the status recorder lock. Take one snapshot per call and index it by route. * [client] Track the active route peer by HA unique ID in the status recorder The Android network list resolved the route owner by scanning peer state route keys. Those keys are the handler string: a prefix for static routes and the domain pattern for dynamic ones, so the prefix-based lookup never matched dynamic routes, and two networks sharing a prefix resolved to the same owner. The route watcher now records the chosen route peer under the route's HA unique ID in the status recorder, and the Android binding looks the owner up by that ID. The key is unique per network and independent of the handler string format, so both anomalies are gone. The prefix-based routeOwners helper is removed. * [client] Record the active route peer before notifying listeners AddPeerStateRoute and RemovePeerStateRoute fire the peer list change callback and wake the status subscribers. The active route peer mapping was written after those calls, so a Networks() call landing in between found no mapping for the network and fell back to the first connected peer, or kept showing the previous peer on removal. Nothing re-notified after the mapping write, so the wrong peer stayed until the next peer list change. Write and delete the mapping before the notifying calls so a listener reacting to the notification always reads the current owner. |
||
|
Riccardo Manfrin
|
9f8ddc7131 |
[client] Discover interfaces lazily in stdnet instead of at construction (#7346)
* [client] Discover interfaces lazily in stdnet instead of at construction
stdnet.NewNet and NewNetWithDiscover ended with
return n, n.UpdateInterfaces()
handing back a non-nil *Net together with the discovery error. Three of the
five call sites (Engine.newWgIface, ice.NewAgent, SingleSocketUDPMux) logged
the error and kept using the instance, which is only safe as long as the
instance still works after a failed discovery.
That stopped being true when Interfaces() gained a lazily refreshed cache:
updateInterfaces sets lastUpdate only on success, so after a failed
construction the 30s cache guard never holds and Interfaces() returns an
error rather than the empty list it used to return. Feeding such an instance
to pion is worse than passing nothing at all - ice.NewAgent falls back to its
own stdnet when Net is nil, and the interface blacklist is applied separately
through AgentConfig.InterfaceFilter, so the fallback loses nothing. Instead,
a transient discovery failure (the Android bridge at boot, or an interface
disappearing between net.Interfaces() and Interface.Addrs()) turned into a
hard "error getting local interfaces" from ice.NewAgent, and aborted the STUN
and TURN probes, which never even need the interface list.
Since the accessors already refresh a stale cache on demand, the eager
discovery in the constructors is redundant: drop it, make both constructors
infallible, and let the discovery error surface at the call that actually
needs the interfaces. UpdateInterfaces had no callers left and is not part of
transport.Net, so it is removed along with it.
InterfaceByIndex and InterfaceByName read the cached slice directly and never
refreshed it, so they would have kept reporting ErrInterfaceNotFound forever
on an instance whose first discovery failed. They now go through the same
refresh path as Interfaces().
* [client] Warm the stdnet interface cache at construction
Moving discovery to first use regressed the privileged suites on the three
platforms that always build an ICE bind: Darwin, FreeBSD and Windows time out
in TestWGIface_UpdateAddr, TestRecreation, TestEngine_SSH and
TestEngine_MultiplePeers, while Linux stays green because a host with the
WireGuard kernel module takes the kernel-device branch and never drives the
mux that asks for interfaces.
interfaceFilter probes with wgctrl every interface the disallow list does not
already exclude. Discovering at construction ran that probe before the caller
had an overlay interface of its own; discovering at first use runs it after,
so on a userspace WireGuard platform the probe reaches the UAPI socket of the
same process. The tests reach it because they construct with a nil disallow
list, where the client passes DefaultInterfaceBlacklist and its own interface
is excluded by prefix.
Restore the original timing with an explicit warm-up. The constructors stay
infallible and the error is still reported by the accessor that needs the
interfaces, so the contract this branch is about is unchanged.
* Revert "[client] Warm the stdnet interface cache at construction"
This reverts commit
|
||
|
Maycon Santos
|
e72be6698f |
[client] Keep the advertised ICE session ID when following a remote restart (#7814)
A worker that saw a new remote session ID rebuilt its agent and also picked a new local ID. On the answer path nothing carries that ID back, so the next offer made the remote see a changed session, rebuild, and answer with yet another ID. Two peers kept tearing down working ICE connections on every offer and answer; nearly every answer in the affected logs carried a new remote session ID. Only a local restart changes the local ID now: a failed negotiation, as before, and an explicit Close, which previously kept the old ID and left the remote answering from a negotiation this side had abandoned. Following a remote restart keeps the ID the remote already knows, so the pair settles after one rebuild, also against peers that still pick a new ID when following a restart. |
||
|
Zoltan Papp
|
bc0671fd21 |
[client] Fix peers not being notified when the relay connection drops (#7490)
* [relay] Signal relay disconnects through the conn context AddCloseListener deduplicated listeners by comparing reflect.ValueOf(callback).Pointer(). For a method value that pointer is the address of the compiler-generated wrapper, not an identity bound to the receiver, so every peer's w.onRelayClientDisconnected compared equal. All peers on the home relay register under the same connectionURL key, so only the first registration survived and the rest were silently dropped. On a relay disconnect those peers were never notified: statusRelay stayed connected and the reconnect guard never fired. The relayed net.Conn itself was closed by closeAllConns, so nothing leaked, but the peer state machine did not learn about it. Foreign relays had the same defect scoped to the peers sharing that server. Rather than fixing the deduplication, drop the peer-level listener registry entirely. A relayed Conn now exposes Context(), cancelled when the connection is torn down, with a cancellation cause naming the reason. This is the same shape quic-go uses for its Conn and Stream types, and it removes the whole class of problems around listener identity, lifetime and deregistration: the signal belongs to the resource instead of a side table. WorkerRelay watches that context in a goroutine whose lifetime matches the connection. A watcher that wakes up for a superseded connection compares the conn pointer against the current one and returns without touching the state machine, so a fast relay reconnect cannot have a stale watcher tear down the connection that replaced it. Client.SetOnDisconnectListener stays: it is server-level and drives the reconnect guard and foreign relay eviction, unrelated to peers. handleRelayReady also checks the conn context, closing the race where the relay dies between OpenConn and the readiness handoff and the peer would otherwise build a WireGuard endpoint over a dead connection. TestNotifierDoubleAdd covered the removed mechanism and is gone. TestForeignAutoClose asserted nothing (both branches logged); it now waits for the relay to leave the client map and fails if it does not. * [relay] Fix build: return the concrete conn from Client.OpenConn OpenConn now returns *Conn, but it still went through connContainer.netConn(), which widens to net.Conn. The helper had one caller and only existed to produce the interface value the signature no longer wants, so return container.conn directly and drop it. * [relay] Assert the local-close cancellation cause explicitly The local-close test only rejected ErrServerDisconnected, so it would also have passed for ErrPeerDisconnected or a bare context.Canceled. closeConn cancels with net.ErrClosed, so assert that. * [client] Ignore relay disconnects from superseded connections The relayed conn watcher compared the conn pointer under relayLock, released it, and only then tore the connection down. A new offer could install its replacement in that window, so a watcher that validated the old pointer went on to close the proxy of the connection that had already replaced it and report the peer as disconnected while it was up. Move the decision to where the teardown happens. Conn records which relayed connection the current proxy was built from, and onRelayDisconnected takes the connection the signal belongs to and drops it under conn.mu when it is no longer the current one. Check and effect are now in the same critical section, so the verdict cannot go stale before it is acted on. This also covers the proxy read loops, whose disconnect listener took no argument and had the same defect: it now names the connection it belongs to. The WG timeout path keeps passing nil, since it deliberately tears down whatever is current. * [client] Bind the relayed conn reference to the proxy swap relayedConnRef was set at the top of the readiness path, but wgProxyRelay only changes at the end, in setRelayedProxy. The two failure returns in between — newProxy and ConfigureWGEndpoint — left the reference pointing at a connection that never became active while the old proxy was still installed. A disconnect of that old, live relay would then be dismissed as belonging to a superseded connection and never cleaned up. Set the reference in setRelayedProxy, next to the proxy it belongs to. Both success paths go through it and neither failure path does, so no failure branch has to remember to roll anything back. |
||
|
Zoltan Papp
|
e70ec07320 |
Read the session deadline under the status read lock (#7550)
GetSessionExpiresAt took the exclusive lock for a plain field read, so every caller queued behind writers and behind each other. The Android SessionMonitor polls it from the main thread, and in the captured ANR that is exactly where the main thread was blocked while hundreds of peer-list callbacks held or waited on the same mutex. d.mux is already an RWMutex and the other getters use RLock; this brings the deadline read in line with them. |
||
|
Zoltan Papp
|
794956a7a3 |
[client] Fix relay instance address race (#7498)
Read the relay instance URL and IP atomically to prevent reconnects from mixing values from different connections. Extend existing connection and offer/answer logs with relay URLs and IPs to help trace mismatched advertisements. |
||
|
Laotree
|
24959e1ed9 |
[client] Drop agentConnecting whenever ICE session state clears (#7327)
* [client] Drop agentConnecting whenever ICE session state clears Closing a WorkerICE raced a blocked dial goroutine: Close released the agent while connect() was still inside Dial, and the goroutine's own cleanup skipped its flag reset because w.agent no longer matched. With agentConnecting stuck on true, evalConnStatus read the peer as connected, the reconnection guard stopped sending offers and same-session offers were dropped, so the peer could not recover without a restart. An aborted recreate in OnNewOffer reaches the same wedged state without any race. Route every teardown path through one abandonNegotiation helper so the agent and flag fields always clear together; Close now also cleans up residual state left by an aborted recreate. * [client] Drive the ICE teardown race test through the real dial goroutine The regression test simulated the stale goroutine by calling closeAgent directly, so it pinned the symptom rather than the mechanism. Rework it to start a real negotiation, tear it down mid-flight and let the actual goroutine run its own cleanup: with no remote responder the dial can only fail once Close cancels it, so the interleaving stays deterministic without sleeps or injection points. Assert the full idle state that abandonNegotiation owns (agent nil, connecting false, remote session ID empty) instead of only InProgress, and make the stale-cleanup ownership test verify that the newer session survives field by field. * [client] Assert live remote session ID after stale ICE cleanup The stale-cleanup test compared a snapshot captured before closeAgent ran, so clearing the field during cleanup would have gone unnoticed. Read the field under the mutex after the cleanup instead. * [client] Give the ICE race tests a no-op signal client The candidate callback fires from a real gather and dereferences the signaler, so a nil one crashes the test package intermittently when gather wins the race against Close. Build the worker with a stub signal.Client instead. * [client] Read the ICE dial cancel func from an argument in connect The error paths read w.agentDialerCancel without holding muxAgent while OnNewOffer rewrites the field for a newer negotiation, a data race the new teardown test trips under -race. Reading a stale value also let an old goroutine cancel another session's dial. Capture the cancel func at goroutine spawn, like the dial context already is. * [client] Guard the ICE dial success path against stale negotiations The stale-cleanup guard in closeAgent only protected teardown. Its success-path counterpart was missing: an older negotiation could complete agentDial after a newer one replaced w.agent, then clear the newer session's agentConnecting, record lastSuccess and publish its dead connection via onICEConnectionIsReady. Verify ownership under muxAgent twice: right after the dial returns, so a stale goroutine drops its connection before touching a closed agent, and again at the state-commit point, atomic with the agentConnecting and lastSuccess writes, so a replacement arriving in the meantime cannot get its state clobbered. Both paths close the stale connection and return without modifying worker state. A regression test holds session A's dial open until session B is installed, then releases it; the stale connection must be discarded and B's agent, connecting flag and remote session ID must survive. * [client] Fix ICE teardown test leak and document the stale delivery window A code review of the stale-negotiation guard found a leftover resource leak in TestWorkerICE_StaleCloseAgentKeepsCurrentSession: session B is never closed, so its ICE sockets and blocked dial goroutine live as long as the test process. Register t.Cleanup(w.Close). The delivery race flagged after the success-path guard is pre-existing and self-correcting - the newer negotiation overwrites the transient endpoint - so document it in the existing todo instead of locking the callback, which would invert lock order against Conn.Close. Adjust the teardown test comment to match the now-synchronous Close flag clearing. |
||
|
Viktor Liu
|
63c26be72f | [client] Add local Prometheus metrics endpoint (#6689) | ||
|
Zoltan Papp
|
15fff4c164 |
[client] Sweep connections on network loss via a shared netevents manager (#7254)
Losing the last network only flipped the availability state: the dead management, signal and relay sockets stayed silently connected until their own timeouts, so the client kept reporting Connected with no network at all. Introduce client/netevents with a Manager that ties the availability state, the connection sweeper and the status recorder together, and move the netstate and netsweep packages under it (netsweep renamed to sweep). SetNetworkAvailable(false) now also sweeps the registered connections so their owners redial and the listener reaches the NoNetwork state. The Android and iOS bindings own a Manager instance and inject it through the constructors; consumers hold the concrete *Manager whose nil zero value reports always-online and never sweeps, with interfaces kept only as parameter contracts. The relay guard settle wait moved into the Manager as WaitSettled, removing the netevents import from the relay package. |
||
|
Viktor Liu
|
7f03a2e86f | [client] Hold a peer offer or answer that arrives before the handshaker starts listening (#7255) | ||
|
Viktor Liu
|
335adfe9c3 | [client] Move the PCP implementation to the go-nat fork (#7282) | ||
|
Viktor Liu
|
917ad880e3 | [client] Rename TURN-specific wg proxy naming to relayed connections (#7231) | ||
|
Zoltan Papp
|
070a0a7bf1 |
[client, android] Handle network changes without restarting the engine (#7144)
On network changes the client restarted the whole engine. That is heavy-handed and slow: it tears down working state to recover from a transition the engine could handle itself. This replaces the restart with proper network event handling. Suspend the retry loops while no network is available. Instead of burning through backoff intervals against an unreachable network, the reconnection loops park until the OS reports a usable network again. Reconnect immediately on a network switch. When the OS hands us a new network, connections bound to the old one are swept and re-dialed right away, rather than waiting for a timeout to notice they are dead. |
||
|
Riccardo Manfrin
|
eb619fc7e3 |
[client] disambiguate the connection_type metric tag (#7043)
## Describe your changes `recordConnectionMetrics` mapped only `conntype.Relay` to `relay` and let a `default` branch record everything else as `ice`. That silently included `ICETurn` — an ICE connection through a TURN server, which [`conn.isRelayed`](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L788-L795) itself counts as relayed — and `None`, the transient state set when the relay drops ([conn.go:632](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L632)) or the peer state is reset ([conn.go:757](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L757)). Both were reported as direct peer-to-peer, so the `ice` share overstated direct connections on every platform. The mapping now lists every priority explicitly and emits `ice_p2p`, `ice_turn`, `relay` or `unknown`. The new values deliberately do not reuse `ice` to avoid ambuguity. ## Issue ticket number and link No public issue. Found while reviewing the first production sample of client metrics: 38% of iOS connection events were tagged `ice` on a platform that forces relay by default, which traced back to the `default` branch at [client/internal/peer/conn.go#L963-L968](https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L963-L968). ## Stack <!-- branch-stack --> ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) Internal metrics documentation only, in `client/internal/metrics/infra/README.md`: the four `connection_type` values with their derivation, and a note that pre-fix `ice` samples are not comparable with `ice_p2p`. No public API, CLI or configuration change, so no netbirdio/docs PR. ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: N/A <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit * **New Features** * Connection metrics now distinguish direct peer-to-peer, TURN-assisted, relay, and unknown connection types. * Metrics include clearer connection and peer identification details. * **Documentation** * Updated connection timing metric values, traffic semantics, priority behavior, and historical data guidance. * **Bug Fixes** * Unset or unrecognized connection priorities are no longer incorrectly classified as peer-to-peer. * Unknown-transport metrics are skipped to prevent misleading connection data. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Riccardo Manfrin
|
f63fd21e0c |
[client] peer: re-arm the WireGuard watcher after a lazy wake (#7091)
## Describe your changes The Conn struct is reused across lazy-connection deactivate/activate. Close cancels the WireGuard watcher (via wgWatcherCancel, and ctxCancel also tears down its context) but left conn.wgWatcher pointing at the stopped instance. enableWgWatcherIfNeeded skips while conn.wgWatcher is non-nil, so the next Open never started a fresh watcher: once a lazy connection had idled and woken, the peer ran with no watcher at all — no WireGuard handshake-timeout detection and none of the escalation that depends on it. Clear conn.wgWatcher and conn.wgWatcherCancel in Close so the next Open re-arms a fresh watcher. ## Issue ticket number and link <!-- Required for anything that changes behavior. Link the issue (or the validated discussion it came from) that the NetBird team already agreed on. See https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#ticket-first-pr-second --> ## Stack <!-- branch-stack --> ### Checklist - [X] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [ ] Created tests that fail without the change (if possible) - [ ] I ran and tested this change locally — I did not rely on CI to find out whether it works - [ ] This PR has a single purpose (not a fix + refactor + feature in one) - [ ] This change is a trivial fix, **OR** it links an issue the NetBird team agreed on beforehand. Changes to the public API, gRPC protocols, functionality behavior, CLI / service flags, or new features always need that agreement first. See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#ticket-first-pr-second). > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [X] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: https://github.com/netbirdio/docs/pull/__ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved connection cleanup by fully releasing WireGuard watcher resources when a connection closes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Zoltan Papp
|
a12a3e4603 |
[client] Keep session deadline visible across reconnects (#6847)
## Describe your changes The tray session row flapped on every transient engine restart and went blank at the exact moment the session expired: - Watcher.Close zeroed the server-scoped recorder deadline on every engine restart (sleep/wake, network change, stream errors), hiding the tray row until the next successful login re-seeded it. Close now only stops the timers; the client run loop clears the recorder when it exits for real. - Status.GetSessionExpiresAt masked past deadlines as zero, so the status snapshot dropped the field right when the UI should have shown "session expired". The getter now returns the raw value and the tray renders past deadlines as "Session expired — sign in again", routing the click to the login flow. - The watcher rejected recently-expired deadlines (30s skew) delivered by login/sync, clearing the recorder instead of surfacing the expiry. Deadlines up to 30 days past are now recorded as expired; only older values are rejected as garbage. ## Issue ticket number and link ## Stack <!-- branch-stack --> ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [ ] Created tests that fail without the change (if possible) - [ ] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: https://github.com/netbirdio/docs/pull/__ <!-- codesmith:footer --> --- <a href="https://app.blacksmith.sh/netbirdio/codesmith/netbird/pr/6847"><picture><source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v2.svg"><source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-light-v2.svg"><img alt="View with Codesmith" src="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v2.svg"></picture></a> <a href="https://backend.blacksmith.sh/track/enable-autofix?expires=1787220767&installation_model_id=427504&pr_number=6847&repository=netbirdio%2Fnetbird&return_to=https%3A%2F%2Fgithub.com%2Fnetbirdio%2Fnetbird%2Fpull%2F6847&signature=77977ad47c5eda51e1d76e59bc736203b4592462bf94076811a20ea0f7f23aee"><picture><source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-light.svg"><img alt="Autofix with Codesmith" src="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"></picture></a> <sup>Need help on this PR? Tag <code>/codesmith</code> with what you need. Autofix is disabled.</sup> <!-- codesmith:autofix:disabled --> <!-- /codesmith:footer --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tray “session expires” and countdown now accurately reflect expired sessions and remaining time near boundary moments (including improved rounding). * **Bug Fixes** * Recently expired deadlines are retained for correct status reporting, while very old past deadlines are rejected and cleared. * Session expiry “expires at” is preserved when the session watcher closes during reconnect scenarios. * Clicking an already-expired session in the tray now routes the user to the login screen. * **Tests** * Updated and expanded coverage for recent/ancient past handling and watcher close behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Riccardo Manfrin
|
39193396f5 |
[client] Fix WGWatcher silently failing to restart on fast disconnect/reconnect (#6664)
* Stick new watcher creation to actual existence of af the conn and its removal to the removal of such same conn. Avoid debouncing and cross lock dead locking * Discriminate not updated from timeout handshakes * [Recheck watcher ctx cancellation under conn.mu in onWGDisconnected onWGDisconnected only checked conn.ctx (the engine-scoped context), never the watcher's own context. disableWgWatcherIfNeeded cancels the wgWatcherCtx, not conn.ctx, so a disabled watcher's timeout callback did not see the cancellation. handshakeCheck runs lock-free, so between the ctx check in periodicHandshakeCheck and acquiring conn.mu a fast disconnect/reconnect can slip in: the stale watcher then acquires the lock and tears down the *new*, healthy connection based on the old timeout, forcing the guard into an unnecessary reconnect (flap). Recheck watcherCtx.Err() under conn.mu so a superseded watcher exits without touching the connection that replaced it. * Remove verbose comments * Fixup merge conflict leftovers * Fixup context brought by onWGDisconnected |
||
|
|
30d15ecc3d |
[client,management] sync 0.74.4 changes (#6727)
* [management] fix: prevent reverse proxy domain from being pushed as DNS search domain by @blaugrau90 in https://github.com/netbirdio/netbird/pull/6498 * [client] Recover from rosenpass key desync by @lixmal in https://github.com/netbirdio/netbird/pull/6714 * [client] Bump golang.org/x/crypto to v0.54.0 by @lixmal in https://github.com/netbirdio/netbird/pull/6709 * [client] fix MDM managementURL conflict on default-port URL echo by @riccardomanfrin in https://github.com/netbirdio/netbird/pull/6672 * [client] Update gopsutil to v4 by @mlsmaycon in https://github.com/netbirdio/netbird/pull/6688 * [client] Fix hanging status command during relay dial by @theodorsm in https://github.com/netbirdio/netbird/pull/6694 --------- Co-authored-by: Theodor Midtlien <theodor@midtlien.com> Co-authored-by: blaugrau90 <61945343+blaugrau90@users.noreply.github.com> Co-authored-by: Viktor Liu <17948409+lixmal@users.noreply.github.com> |
||
|
|
91acb8147c |
[management,client] 0.75.0 release with new desktop UI (#6473)
- **Wails v3 application** (`client/ui`) with a React + TypeScript + Tailwind frontend replacing the Fyne UI: main connection view, exit-node switcher, networks/peers browser with detail panels, profile management, settings (general, network, SSH, security, troubleshooting, appearance), debug-bundle creation, and a first-run welcome flow. - **Internationalization**: go-i18n bundle with 9 locales (en, de, es, fr, hu, it, pt, ru, zh-CN) shared between the tray and the frontend. - **New system tray** implementation with per-platform theme-aware icons, including a native XEmbed host for Linux (`xembed_tray_linux.c`) and a Linux theme watcher. - **Session handling**: auth session watcher (`client/internal/auth/sessionwatch`), pending login flow, session-expiration dialog and tray notifications, and `netbird login` improvements. - **Daemon API extensions** (`daemon.proto`): status stream subscription, event stream, networks/exit-node selection endpoints, and richer full status — with probe throttling on the daemon side to protect against UI-driven request storms. - **UI preferences store** persisted per profile, autostart management via the daemon (single source of truth in HKCU on Windows). - **Build system**: Taskfile-based builds per platform (macOS, Linux, Windows), Docker cross-compilation images, MSIX/NSIS/nfpm/AppImage packaging, and a new `frontend-ui` CI workflow. Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com> Co-authored-by: Eduard Gert <kontakt@eduardgert.de> Co-authored-by: braginini <bangvalo@gmail.com> Co-authored-by: Pascal Fischer <32096965+pascal-fischer@users.noreply.github.com> Co-authored-by: riccardom <riccardomanfrin@gmail.com> |
||
|
Theodor Midtlien
|
3aa6c02b93 | [client] Fix backoff.Ticker goroutine leak in reconnect guard | ||
|
Zoltan Papp
|
06839a4731 |
[client] Fix race between WG watcher initial handshake read and endpoint creation (#6626)
* [client] Fix race between WG watcher initial handshake read and endpoint config The watcher's initial handshake read ran in a separate goroutine with no ordering guarantee relative to the WireGuard endpoint configuration, so it would sometimes race with the peer being added to the interface. Split enabling into a synchronous PrepareInitialHandshake, called before the endpoint is configured, and an EnableWgWatcher that only runs the monitoring loop, making the baseline read deterministic and keeping it correct for reconnects where the peer's WireGuard entry survives. * [client] Skip WG watcher disconnect callback when context is cancelled A superseded or cancelled watcher whose handshake-check timer fires before it observes ctx.Done() would still invoke onDisconnectedFn, tearing down a now-healthy connection. Re-check ctx before firing the disconnect and handshake-success callbacks and stand down silently if it was cancelled. |
||
|
Viktor Liu
|
4ef65294e9 | [client] Reinject captured first packet on lazy connection activation (#6572) | ||
|
Riccardo Manfrin
|
2bb5421631 | These logs are needed for troubleshooting (debug) (#6565) | ||
|
Zoltan Papp
|
1b29995ece |
[client] Fix blocked status lock via relay manager path (#6547)
* peer/status: move relay-state reads off the main mux GetRelayStates held d.mux (RLock) while calling into the relay Manager (RelayStates/RelayConnectError/ServerURLs). Those calls can be slow or block on the relay manager's own locks while it is reconnecting, which kept the central Status mutex held and stalled every peer state writer (UpdatePeerState, ReplaceOfflinePeers, etc.) contending for it. Guard relayMgr/relayStates with a dedicated muxRelays mutex and release it before invoking the relay Manager, so the relay read path no longer contends with the hot peer-state writers on d.mux. * peer/status: clone relay states in nil-manager path Return a cloned snapshot of d.relayStates when relayMgr is nil so callers cannot mutate the shared cached state, matching the non-nil path. |
||
|
Viktor Liu
|
6fbc90b4d3 | [client, relay] Expose relay transport and connection errors in status and metrics (#6342) | ||
|
Viktor Liu
|
b57f714350 | [client] Drop signaling-side ICE candidate filter, drop overlay STUN at mux read-side instead (#6142) | ||
|
Maycon Santos
|
8ff3b06cf1 |
[client] Index peer tunnel IPs for faster PeerStateByIP lookup (#6412)
* [client] Index peer tunnel IPs for O(1) PeerStateByIP lookup Replace the linear scan over all peers with an ipToKey map maintained by AddPeer/RemovePeer, covering both IPv4 and IPv6 tunnel addresses. Offline peers are intentionally no longer resolvable by IP: only active peers can carry traffic, so IdentityForIP and the DNS disconnected-peer filter now treat them as unknown, same as foreign IPs. Skip the DNS answer filter for single-record responses; dropping the only answer was always restored by the empty-answer escape hatch, so the fast path is behavior-neutral. * Ensure `ipToKey` entries are only removed if they match the peer being deleted, preventing accidental removal of unrelated mappings. |
||
|
Theodor Midtlien
|
5993ec6e43 |
[client] Allow wireguard port to be zero in UI and show port in status command (#6158)
* Allow wireguard port to be set to 0 in UI * Add wireguard port to cmd status * Correct protoc version |
||
|
Maycon Santos
|
fa1e241aea |
[management, client, proxy] Follow-up fixes for private reverse-proxy services (#6268)
* fix(proxy): gate tunnel-peer fast-path on inbound listener marker
forwardWithTunnelPeer previously accepted any RFC1918 / ULA / CGNAT
source IP, so a public client whose address happened to fall in those
ranges could bypass the configured operator auth scheme by colliding
with a known tunnel IP. The fast-path is now gated on
TunnelLookupFromContext(r.Context()) being present — that context value
is attached only by the per-account inbound (overlay) listener, so the
host-facing listener never enters this branch.
Tests updated to reflect the new requirement: requests that don't
carry the inbound marker now fall through to the regular auth flow.
* fix(proxy): harden inbound listener resource + startup-ctx handling
Three correctness fixes on the per-account inbound path, with tests:
- Close the logrus ErrorLog PipeWriter on tearDown. WriterLevel hands
back an *io.PipeWriter backed by a pipe + scanner goroutine that the
caller owns; the two writers per account (https + plain) were never
closed, leaking the pipe and goroutine on every teardown.
- Run the post-Start hooks on context.Background(). runClientStartup
is launched in a goroutine from AddPeer and was inheriting the
caller's request-scoped ctx, so a cancelled request could abort the
inbound bring-up or fail the management status notification. The
tail is split into notifyClientReady so the contract is testable.
Tests cover the PipeWriter close behaviour and assert the readyHandler
+ NotifyStatus calls receive a non-cancelled background context.
* feat(proxy): short-circuit peer-own-target loops with 421
When a peer that hosts the target of a private service dials its own
service URL the request was being looped through the proxy and back
over WireGuard to the same peer — twice the WG round-trip for no
benefit, with no signal to the caller that something was wrong.
Add isSelfTargetLoop to ReverseProxy.ServeHTTP: when the request
arrived on the per-account overlay listener (IsOverlayOrigin) and the
source tunnel IP matches the target host, refuse the request with 421
Misdirected Request and a body pointing the operator at the backend
directly.
The gate is scoped to overlay origin so requests on the public
listener that happen to share a source IP with the target host are
forwarded normally.
* fix(management): private-service validation + tunnel-IP lookup semantics
- Require an explicit port for L4 cluster targets. validateL4Target
exempted TargetTypeCluster from the port check, but buildPathMappings
serializes every L4 target via net.JoinHostPort(host, port) — port=0
shipped a ":0" upstream. Cluster targets use the same Host/Port
fields, so the same requirement applies.
- GetPeerByIP returns NotFound on a tunnel-IP miss instead of mapping
every error to Internal. The proxy's ValidateTunnelPeer probes IPs
that legitimately aren't in the roster; the miss is expected and now
distinguishable from a real store failure.
- Thread ctx into getClusterCapability's gorm query so a cancelled
request doesn't keep the store busy.
Tests updated for the L4-cluster port requirement and the GetPeerByIP
NotFound path.
* fix(client): include offlinePeers in PeerStateByIP lookup
ReplaceOfflinePeers moves peers into d.offlinePeers but PeerStateByIP
only scanned d.peers. Callers (the local DNS filter via
localPeerConnectivity, embed.Client.IdentityForIP used by the
proxy's tunnel-peer validator) were treating known-but-offline peers
as unknown, which:
- causes the DNS filter to keep returning records pointing at peers
that have no live tunnel, AND
- makes the proxy's local-roster check deny a request from such a
peer rather than letting the cached management RPC carry the
authorisation decision.
Search both slices in PeerStateByIP. Adds a unit test for the IPv4
and IPv6 offline-match paths.
* fix(rest): reject empty Delete path params in reverse-proxy clients
ReverseProxyClustersAPI.Delete and ReverseProxyTokensAPI.Delete passed
the path parameter into url.PathEscape without an empty check.
PathEscape("") returns "" which collapses the request onto the
collection endpoint ("/api/reverse-proxies/clusters/" /
"/api/reverse-proxies/proxy-tokens/"), so a caller bug delete with no
id reached a routable URL with surprising semantics (typically 405).
Short-circuit with a typed error before the request is built. Tests
mount a handler on the collection path that fails the test if hit, so
the regression is impossible to reintroduce silently.
* chore(api,ci,docs,test): private-service schema, proto-check, fixups
Non-functional cleanups and contract/CI hardening around the
private-service work:
API schema (openapi.yml):
- Require a non-empty access_groups and mode=http when private=true,
on both Service and ServiceRequest, mirroring
validatePrivateRequirements. mode stays optional-but-constrained
(empty defaults to http server-side), matching runtime.
CI (proto-version-check.yml):
- Cover renamed .pb.go files (read base via previous_filename).
- Match protoc-gen-go-grpc version headers (optional "- " prefix and
-gen-go-grpc suffix) so grpc-generated files are in scope.
Docs / comments:
- Reword Config field docs to say defaults are applied at Server.Start
(initDefaults), not New.
- Rename the obsolete --private-inbound flag to --private across
comments and the proto doc.
Pre-existing test fixups surfaced by review:
- Repair the integration-tagged validate_session_test.go (SignToken
signature growth + new Manager interface methods).
- Fix the CI-skip boolean precedence so Windows isn't skipped
unconditionally.
- Guard the router.HTTPListener type assertion with comma-ok.
* fix(proxy): background ctx for already-started AddPeer notification
The earlier ctx fix covered the async runClientStartup path but missed
the synchronous branch: when a service is added to an already-started
client, AddPeer called NotifyStatus with the caller's request-scoped
ctx. A cancelled request/stream could drop the connected notification
to management. Use context.Background() here too, matching
notifyClientReady.
Extends TestNetBird_AddPeer_ExistingStartedClient_NotifiesStatus to
pass a pre-cancelled caller ctx and assert the notification still ran
on a non-cancelled context.
* use the cmd context for roundtripper
|
||
|
Riccardo Manfrin
|
7ea5e37dd4 |
[client] Improve rosenpass support (#6136)
* Updates rosenpass version go-rosenpass v0.4.0 → v0.5.42 bump — detailed findings Change summary cunicu.li/go-rosenpass v0.4.0 → v0.5.42 (target) cilium/ebpf v0.15.0 → v0.19.0 (transitive) gopacket/gopacket v1.1.1 → v1.4.0 (transitive) wireguard 2023-07 → 2023-12 (transitive) wireguard/wgctrl 2023-04 → 2024-12 (transitive) Wire interop v0.4.0 (in v0.70.5) <-> v0.5.42 OK v0.5.42 <-> v0.5.42 OK Quantum resistance: true both ends --- **Replay error eliminated.** Before (on v0.4.0): `ERROR Failed to handle message: failed to load biscuit (ICR1): detected replay` Recurring every ~50ms for minutes at a time. Gone entirely after both ends upgraded to v0.5.42. Upstream fix in biscuit/replay handling between v0.4.x and v0.5.x series. * Fixup [::]:port socket trying to send to v4 * Adds more tests on netbird<->rosenpass interactions * Anticipates rp handler creation before generateConfig * [client] Moves deterministic key gen into rosenpass * go mod tidy * Adds reminder to reason about rosenpass surface area * Apply code rabbit suggestions |
||
|
Maycon Santos
|
7aebdd69dd |
[management, client, proxy] add expose NetBird-only services over tunnel peers (#6226)
Adds a new "private" service mode for the reverse proxy: services reachable exclusively over the embedded WireGuard tunnel, gated by per-peer group membership instead of operator auth schemes. Wire contract - ProxyMapping.private (field 13): the proxy MUST call ValidateTunnelPeer and fail closed; operator schemes are bypassed. - ProxyCapabilities.private (4) + supports_private_service (5): capability gate. Management never streams private mappings to proxies that don't claim the capability; the broadcast path applies the same filter via filterMappingsForProxy. - ValidateTunnelPeer RPC: resolves an inbound tunnel IP to a peer, checks the peer's groups against service.AccessGroups, and mints a session JWT on success. checkPeerGroupAccess fails closed when a private service has empty AccessGroups. - ValidateSession/ValidateTunnelPeer responses now carry peer_group_ids + peer_group_names so the proxy can authorise policy-aware middlewares without an extra management round-trip. - ProxyInboundListener + SendStatusUpdate.inbound_listener: per-account inbound listener state surfaced to dashboards. - PathTargetOptions.direct_upstream (11): bypass the embedded NetBird client and dial the target via the proxy host's network stack for upstreams reachable without WireGuard. Data model - Service.Private (bool) + Service.AccessGroups ([]string, JSON- serialised). Validate() rejects bearer auth on private services. Copy() deep-copies AccessGroups. pgx getServices loads the columns. - DomainConfig.Private threaded into the proxy auth middleware. Request handler routes private services through forwardWithTunnelPeer and returns 403 on validation failure. - Account-level SynthesizePrivateServiceZones (synthetic DNS) and injectPrivateServicePolicies (synthetic ACL) gate on len(svc.AccessGroups) > 0. Proxy - /netbird proxy --private (embedded mode) flag; Config.Private in proxy/lifecycle.go. - Per-account inbound listener (proxy/inbound.go) binding HTTP/HTTPS on the embedded NetBird client's WireGuard tunnel netstack. - proxy/internal/auth/tunnel_cache: ValidateTunnelPeer response cache with single-flight de-duplication and per-account eviction. - Local peerstore short-circuit: when the inbound IP isn't in the account roster, deny fast without an RPC. - proxy/server.go reports SupportsPrivateService=true and redacts the full ProxyMapping JSON from info logs (auth_token + header-auth hashed values now only at debug level). Identity forwarding - ValidateSessionJWT returns user_id, email, method, groups, group_names. sessionkey.Claims carries Email + Groups + GroupNames so the proxy can stamp identity onto upstream requests without an extra management round-trip on every cookie-bearing request. - CapturedData carries userEmail / userGroups / userGroupNames; the proxy stamps X-NetBird-User and X-NetBird-Groups on r.Out from the authenticated identity (strips client-supplied values first to prevent spoofing). - AccessLog.UserGroups: access-log enrichment captures the user's group memberships at write time so the dashboard can render group context without reverse-resolving stale memberships. OpenAPI/dashboard surface - ReverseProxyService gains private + access_groups; ReverseProxyCluster gains private + supports_private. ReverseProxyTarget target_type enum gains "cluster". ServiceTargetOptions gains direct_upstream. ProxyAccessLog gains user_groups. |
||
|
Viktor Liu
|
205ebcfda2 | [management, client] Add IPv6 overlay support (#5631) | ||
|
Zoltan Papp
|
a21f6ecb0a |
[client] release Status.mux before invoking notifier callbacks (#6039)
The Status recorder used to fire notifier callbacks while holding d.mux: - notifyPeerListChanged / notifyPeerStateChangeListeners ran from inside the locked section of every Update*/AddPeerStateRoute/etc. - notifyAddressChanged ran from UpdateLocalPeerState and CleanLocalPeerState while d.mux was held. - onConnectionChanged was registered with a defer above defer d.mux.Unlock, so it executed before the mutex was released in the Mark*Connected/ Disconnected helpers. - notifyPeerStateChangeListeners did a blocking channel send under d.mux, so a slow subscriber stalled every other d.mux holder. A listener that re-enters the recorder (e.g. calls GetFullStatus from within a callback) deadlocks against d.mux, and any callback that takes longer than expected stalls every other state query for its duration. Capture the values needed for notification under the lock, release d.mux, then call the notifier. Build per-peer router-state snapshots inside the lock and dispatch them via dispatchRouterPeers afterwards. The router-peer channel send stays blocking, but now happens outside d.mux so a slow consumer cannot stall any other d.mux holder, and no peer state transitions are silently dropped. The notifier itself is unchanged: its internal state was already protected by its own locks, and the field d.notifier is set once in NewRecorder and never reassigned, so reading it without d.mux is safe. Also fix a pre-existing race in Test_notifier_RemoveListener / Test_notifier_SetListener: setListener spawns a goroutine that writes listener.peers, but the tests read listener.peers without waiting for it. |
||
|
Viktor Liu
|
50b58a6828 | [client, relay] Advertise relay server IP via signal for foreign-relay fallback dial (#6004) | ||
|
Zoltan Papp
|
8fc4265995 |
[relay] evict foreign client cache on disconnect (#6015)
* [relay] evict foreign client cache on disconnect When a foreign relay's TCP connection drops, the manager's onServerDisconnected handler only triggered reconnect logic for the home server; the disconnected foreign entry stayed in the relayClients cache. Subsequent OpenConn calls reused the closed client until the 60-second cleanup tick evicted it, breaking peer connectivity through that relay for up to a minute. Evict the foreign entry from the cache on disconnect so the next OpenConn dials a fresh client. Also: - Make the reconnect backoff cap configurable via WithMaxBackoffInterval ManagerOption; the previous hard-coded 60s constant forced TestAutoReconnect to sleep ~61s. Test now polls Ready() and finishes in ~2s. - Add NB_HOME_RELAY_SERVERS env var that overrides the relay URL list received from management, so a peer can be pinned to a specific home relay (used by the netbird-conn-lab Edge 4 reproducer). * [client] treat empty NB_HOME_RELAY_SERVERS as unset Returning (urls=[], ok=true) when the env var contained only separators or whitespace caused callers to wipe the mgmt-provided relay list, leaving the peer with no relays. Treat a parsed-empty result the same as an unset env. |
||
|
Zoltan Papp
|
5a89e6621b |
[client] Supress ICE signaling (#5820)
* [client] Suppress ICE signaling and periodic offers in force-relay mode When NB_FORCE_RELAY is enabled, skip WorkerICE creation entirely, suppress ICE credentials in offer/answer messages, disable the periodic ICE candidate monitor, and fix isConnectedOnAllWay to only check relay status so the guard stops sending unnecessary offers. * [client] Dynamically suppress ICE based on remote peer's offer credentials Track whether the remote peer includes ICE credentials in its offers/answers. When remote stops sending ICE credentials, skip ICE listener dispatch, suppress ICE credentials in responses, and exclude ICE from the guard connectivity check. When remote resumes sending ICE credentials, re-enable all ICE behavior. * [client] Fix nil SessionID panic and force ICE teardown on relay-only transition Fix nil pointer dereference in signalOfferAnswer when SessionID is nil (relay-only offers). Close stale ICE agent immediately when remote peer stops sending ICE credentials to avoid traffic black-hole during the ICE disconnect timeout. * [client] Add relay-only fallback check when ICE is unavailable Ensure the relay connection is supported with the peer when ICE is disabled to prevent connectivity issues. * [client] Add tri-state connection status to guard for smarter ICE retry (#5828) * [client] Add tri-state connection status to guard for smarter ICE retry Refactor isConnectedOnAllWay to return a ConnStatus enum (Connected, Disconnected, PartiallyConnected) instead of a boolean. When relay is up but ICE is not (PartiallyConnected), limit ICE offers to 3 retries with exponential backoff then fall back to hourly attempts, reducing unnecessary signaling traffic. Fully disconnected peers continue to retry aggressively. External events (relay/ICE disconnect, signal/relay reconnect) reset retry state to give ICE a fresh chance. * [client] Clarify guard ICE retry state and trace log trigger Split iceRetryState.attempt into shouldRetry (pure predicate) and enterHourlyMode (explicit state transition) so the caller in reconnectLoopWithRetry reads top-to-bottom. Restore the original trace-log behavior in isConnectedOnAllWay so it only logs on full disconnection, not on the new PartiallyConnected state. * [client] Extract pure evalConnStatus and add unit tests Split isConnectedOnAllWay into a thin method that snapshots state and a pure evalConnStatus helper that takes a connStatusInputs struct, so the tri-state decision logic can be exercised without constructing full Worker or Handshaker objects. Add table-driven tests covering force-relay, ICE-unavailable and fully-available code paths, plus unit tests for iceRetryState budget/hourly transitions and reset. * [client] Improve grammar in logs and refactor ICE credential checks |
||
|
Viktor Liu
|
d33cd4c95b | [client] Add NAT-PMP/UPnP support (#5202) | ||
|
Zoltan PappandViktor Liu
|
91f0d5cefd |
[client] Feature/client metrics (#5512)
* Add client metrics * Add client metrics system with OpenTelemetry and VictoriaMetrics support Implements a comprehensive client metrics system to track peer connection stages and performance. The system supports multiple backend implementations (OpenTelemetry, VictoriaMetrics, and no-op) and tracks detailed connection stage durations from creation through WireGuard handshake. Key changes: - Add metrics package with pluggable backend implementations - Implement OpenTelemetry metrics backend - Implement VictoriaMetrics metrics backend - Add no-op metrics implementation for disabled state - Track connection stages: creation, semaphore, signaling, connection ready, and WireGuard handshake - Move WireGuard watcher functionality to conn.go - Refactor engine to integrate metrics tracking - Add metrics export endpoint in debug server * Add signaling metrics tracking for initial and reconnection attempts * Reset connection stage timestamps during reconnections to exclude unnecessary metrics tracking * Delete otel lib from client * Update unit tests * Invoke callback on handshake success in WireGuard watcher * Add Netbird version tracking to client metrics Integrate Netbird version into VictoriaMetrics backend and metrics labels. Update `ClientMetrics` constructor and metric name formatting to include version information. * Add sync duration tracking to client metrics Introduce `RecordSyncDuration` for measuring sync message processing time. Update all metrics implementations (VictoriaMetrics, no-op) to support the new method. Refactor `ClientMetrics` to use `AgentInfo` for static agent data. * Remove no-op metrics implementation and simplify ClientMetrics constructor Eliminate unused `noopMetrics` and refactor `ClientMetrics` to always use the VictoriaMetrics implementation. Update associated logic to reflect these changes. * Add total duration tracking for connection attempts Calculate total duration for both initial connections and reconnections, accounting for different timestamp scenarios. Update `Export` method to include Prometheus HELP comments. * Add metrics push support to VictoriaMetrics integration * [client] anchor connection metrics to first signal received * Remove creation_to_semaphore connection stage metric The semaphore queuing stage (Created → SemaphoreAcquired) is no longer tracked. Connection metrics now start from SignalingReceived. Updated docs and Grafana dashboard accordingly. * [client] Add remote push config for metrics with version-based eligibility Introduce remoteconfig.Manager that fetches a remote JSON config to control metrics push interval and restrict pushing to a specific agent version range. When NB_METRICS_INTERVAL is set, remote config is bypassed entirely for local override. * [client] Add WASM-compatible NewClientMetrics implementation Replace NewClientMetrics in metrics.go with a WASM-specific stub in metrics_js.go, returning nil for compatibility with JS builds. Simplify method usage for WASM targets. * Add missing file * Update default case in DeploymentType.String to return "unknown" instead of "selfhosted" * [client] Rework metrics to use timestamped samples instead of histograms Replace cumulative Prometheus histograms with timestamped point-in-time samples that are pushed once and cleared. This fixes metrics for sparse events (connections/syncs that happen once at startup) where rate() and increase() produced incorrect or empty results. Changes: - Switch from VictoriaMetrics histogram library to raw Prometheus text format with explicit millisecond timestamps - Reset samples after successful push (no resending stale data) - Rename connection_to_handshake → connection_to_wg_handshake - Add netbird_peer_connection_count metric for ICE vs Relay tracking - Simplify dashboard: point-based scatter plots, donut pie chart - Add maxStalenessInterval=1m to VictoriaMetrics to prevent forward-fill - Fix deployment_type Unknown returning "selfhosted" instead of "unknown" - Fix inverted shouldPush condition in push.go * [client] Add InfluxDB metrics backend alongside VictoriaMetrics Add influxdb.go with timestamped line protocol export for sparse one-shot events. Restore victoria.go to use proper Prometheus histograms. Update Grafana dashboards, add InfluxDB datasource, and update docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [client] Fix metrics issues and update dev docker setup - Fix StopPush not clearing push state, preventing restart - Fix race condition reading currentConnPriority without lock in recordConnectionMetrics - Fix stale comment referencing old metrics server URL - Update docker-compose for InfluxDB: add scoped tokens, .env config, init scripts - Rename docker-compose.victoria.yml to docker-compose.yml * [client] Add anonymised peer tracking to pushed metrics Introduce peer_id and connection_pair_id tags to InfluxDB metrics. Public keys are hashed (truncated SHA-256) for anonymisation. The connection pair ID is deterministic regardless of which side computes it, enabling deduplication of reconnections in the ICE vs Relay dashboard. Also pin Grafana to v11.6.0 for file-based provisioning and fix datasource UID references. * Remove unused dependencies from go.mod and go.sum * Refactor InfluxDB ingest pipeline: extract validation logic - Move line validation logic to `validateLine` and `validateField` helper functions. - Improve error handling with structured validation and clearer separation of concerns. - Add stderr redirection for error messages in `create-tokens.sh`. * Set non-root user in Dockerfile for Ingest service * Fix Windows CI: command line too long * Remove Victoria metrics * Add hashed peer ID as Authorization header in metrics push * Revert influxdb in docker compose * Enable gzip compression and authorization validation for metrics push and ingest * Reducate code of complexity * Update debug documentation to include metrics.txt description * Increase `maxBodySize` limit to 50 MB and update gzip reader wrapping logic * Refactor deployment type detection to use URL parsing for improved accuracy * Update readme * Throttle remote config retries on fetch failure * Preserve first WG handshake timestamp, ignore rekeys * Skip adding empty metrics.txt to debug bundle in debug mode * Update default metrics server URL to https://ingest.netbird.io * Atomic metrics export-and-reset to prevent sample loss between Export and Reset calls * Fix doc * Refactor Push configuration to improve clarity and enforce minimum push interval * Remove `minPushInterval` and update push interval validation logic * Revert ExportAndReset, it is acceptable data loss * Fix metrics review issues: rename env var, remove stale infra, add tests - Rename NB_METRICS_ENABLED to NB_METRICS_PUSH_ENABLED to clarify that collection is always active (for debug bundles) and only push is opt-in - Change default config URL from staging to production (ingest.netbird.io) - Delete broken Prometheus dashboard (used non-existent metric names) - Delete unused VictoriaMetrics datasource config - Replace committed .env with .env.example containing placeholder values - Wire Grafana admin credentials through env vars in docker-compose - Make metricsStages a pointer to prevent reset-vs-write race on reconnect - Fix typed-nil interface in debug bundle path (GetClientMetrics) - Use deterministic field order in InfluxDB Export (sorted keys) - Replace Authorization header with X-Peer-ID for metrics push - Fix ingest server timeout to use time.Second instead of float - Fix gzip double-close, stale comments, trim log levels - Add tests for influxdb.go and MetricsStages * Add login duration metric, ingest tag validation, and duration bounds - Add netbird_login measurement recording login/auth duration to management server, with success/failure result tag - Validate InfluxDB tags against per-measurement allowlists in ingest server to prevent arbitrary tag injection - Cap all duration fields (*_seconds) at 300s instead of only total_seconds - Add ingest server tests for tag/field validation, bounds, and auth * Add arch tag to all metrics * Fix Grafana dashboard: add arch to drop columns, add login panels * Validate NB_METRICS_SERVER_URL is an absolute HTTP(S) URL * Address review comments: fix README wording, update stale comments * Clarify env var precedence does not bypass remote config eligibility * Remove accidentally committed pprof files --------- Co-authored-by: Viktor Liu <viktor@netbird.io> |
||
|
Zoltan Papp
|
4a54f0d670 |
[Client] Remove connection semaphore (#5419)
* [Client] Remove connection semaphore Remove the semaphore and the initial random sleep time (300ms) from the connectivity logic to speed up the initial connection time. Note: Implement limiter logic that can prioritize router peers and keep the fast connection option for the first few peers. * Remove unused function |
||
|
Zoltan Papp
|
2dbdb5c1a7 |
[client] Refactor WG endpoint setup with role-based proxy activation (#5277)
* Refactor WG endpoint setup with role-based proxy activation For relay connections, the controller (initiator) now activates the wgProxy before configuring the WG endpoint, while the non-controller (responder) configures the endpoint first with a delayed update, then activates the proxy after. This prevents the responder from sending traffic through the proxy before WireGuard is ready to receive it, avoiding handshake congestion when both sides try to initiate simultaneously. For ICE connections, pass hasRelayBackup as the setEndpointNow flag so the responder sets the endpoint immediately when a relay fallback exists (avoiding the delayed update path since relay is already available as backup). On ICE disconnect with relay fallback, remove the duplicate wgProxyRelay.Work() calls — the relay proxy is already active from initial setup, so re-activating it is unnecessary. In EndpointUpdater, split ConfigureWGEndpoint into explicit configureAsInitiator and configureAsResponder paths, and add the setEndpointNow parameter to let the caller control whether the responder applies the endpoint immediately or defers it. Add unused SwitchWGEndpoint and RemoveEndpointAddress methods. Remove the wgConfigWorkaround sleep from the relay setup path. * Fix redundant wgProxyRelay.Work() call during relay fallback setup * Simplify WireGuard endpoint configuration by removing unused parameters and redundant logic |
||
|
Zoltan Papp
|
baed6e46ec |
Reset WireGuard endpoint on ICE session change during relay fallback (#5283)
When an ICE connection disconnects and falls back to relay, reset the WireGuard endpoint and handshake watcher if the remote peer's ICE session has changed. This ensures the controller re-establishes a fresh WireGuard handshake rather than waiting on a stale endpoint from the previous session. |
||
|
Zoltan Papp
|
d2f9653cea |
Fix nil pointer panic in ICE agent during sleep/wake cycles (#5261)
Add defensive nil checks in ThreadSafeAgent.Close() to prevent panic when agent field is nil. This can occur during Windows suspend/resume when network interfaces are disrupted or the pion/ice library returns nil without error. Also capture agent pointer in local variable before goroutine execution to prevent race conditions. Fixes service crashes on laptop wake-up. |
||
|
Zoltan Papp
|
5333e55a81 |
Fix WG watcher missing initial handshake (#5213)
Start the WireGuard watcher before configuring the WG endpoint to ensure it captures the initial handshake timestamp. Previously, the watcher was started after endpoint configuration, causing it to miss the handshake that occurred during setup. |
||
|
Zoltan Papp
|
ee3a67d2d8 |
[client] Fix/health result in bundle (#5164)
* Add support for optional status refresh callback during debug bundle generation * Always update wg status * Remove duplicated wg status call |
||
|
Viktor Liu
|
ee54827f94 | [client] Add IPv6 support to usersace bind (#5147) |
