Start tripped SonarCloud's cognitive-complexity gate (S3776) after the ML-KEM
block landed. Pull the Rosenpass and ML-KEM manager startup into two helpers
that read as one line each from Start. Pure refactor, behavior unchanged.
A worker that saw a new remote session ID rebuilt its agent and also
picked a new local ID. On the answer path nothing carries that ID back,
so the next offer made the remote see a changed session, rebuild, and
answer with yet another ID. Two peers kept tearing down working ICE
connections on every offer and answer; nearly every answer in the
affected logs carried a new remote session ID.
Only a local restart changes the local ID now: a failed negotiation, as
before, and an explicit Close, which previously kept the old ID and left
the remote answering from a negotiation this side had abandoned.
Following a remote restart keeps the ID the remote already knows, so the
pair settles after one rebuild, also against peers that still pick a
new ID when following a restart.
Connections to the daemon were left on gRPC's own defaults, which cap a received
message at 4 MB. A detailed status carries an entry per peer, so on a large
deployment the response outgrows that cap and the command fails outright:
netbird status -d
Error: status failed: grpc: received message larger than max (4287609 vs. 4194304)
The limit is raised where the daemon dial options are built, so every caller
inherits it: the CLI, the desktop UI, the JSON gateway, and the SSH client and
proxy. It is overridable through NB_DAEMON_GRPC_MAX_MSG_SIZE for a deployment
that outgrows the new default too, mirroring what the management client already
does with NB_MANAGEMENT_GRPC_MAX_MSG_SIZE, and reusing its 16 MB default.
Only the receive direction needs raising. Requests to the daemon are small, and
gRPC does not cap the send side by default, so the daemon could already send a
response the caller then refused to read.
* [client] Export the only-owner-writable path check from elevate
Pure refactor, no behavior change: the existing checkOnlyOwnerWritable gets a
thin exported wrapper so callers outside the elevation path can reuse it. No
call site changes here.
* [client] Validate the saved service parameters before applying them
The install reads <stateDir>/service.json and applies it to the service it then
registers: its arguments, its config path and its environment. The restricted
ACL that saveServiceParams puts on the state directory is applied when the file
is written, which is not necessarily before the file is first read, so the
install now checks the file rather than assuming it.
A file whose ownership or permissions are not the ones saveServiceParams
produces is treated as absent, and the install proceeds with its defaults. The
check covers the directories above the file as well, so what is checked is what
is read.
* [client] Restrict which environment variables the service is registered with
--service-env, and the service.json it persists to, accepted any name. A small
set of them decides how a process resolves the executables and libraries it
loads, and the daemon needs none of those: it now refuses them when they are
passed explicitly, and drops them with a warning when they come back from a
service.json written by an older version, so an upgrade does not fail over a
variable nobody needs.
* [client] Resolve netsh by absolute path
The lookup consulted PATH first and fell back to System32, in both the copy the
userspace firewall uses and the one that tears the interface down. It now asks
Windows for the system directory, so the resolution no longer depends on the
environment the service happens to be started with.
* [client] Move the System32 lookup into a package both callers share
Pure refactor, no behavior change: client/iface and client/firewall/uspfilter
carried a copy each of the same function, and neither imports the other, so the
body moves to client/internal/wincmd — alongside winregistry, which is where
the client's other Windows-only helper already lives. Both call sites now read
wincmd.System32("netsh").
* [client] Cover the System32 lookup with a test
Asserts what the previous commits changed: the lookup is absolute, and neither
PATH nor %SystemRoot% moves it.
* [client] Refuse the loader environment families by prefix
Review follow-up on the previous commit:
- LD_* and DYLD_* are now refused whole rather than name by name. Their members
differ per platform and libc and grow with new OS releases, so a list of them
is out of date as soon as it is written — DYLD_FALLBACK_LIBRARY_PATH and
DYLD_FALLBACK_FRAMEWORK_PATH were already missing from it.
- The names are folded to upper case only on Windows, where a variable is the
same one however it is spelled. Elsewhere the environment is case-sensitive,
so Path and PATH are two variables and only the exact spelling is the one that
is read; the fold refused the wrong one.
- TEMP and TMP stay in the denylist, but the rationale and the message now say
what they actually decide: where the service writes, not what it loads.
* [client,management] Skip route firewall rule computation when no firewall
A peer that runs with the firewall disabled has no ACL manager and no
firewall to program, so nothing ever reads RoutesFirewallRules: the only
consumers are acl.Manager, which is reached solely when e.acl is set, and
the legacy-management probe in updateNetworkMap, which is guarded by a
non-nil firewall.
Building those rules is the most expensive part of a sync on a peer that
routes many network resources. On a 15k-peer deployment a debug bundle
showed getPeerNetworkResourceFirewallRules accounting for 62% of the
allocations of Calculate, and Calculate for effectively all of the
allocations of handleSync, which was taking 3.2s on average and holding
the engine lock for the duration.
Let the caller ask Calculate to leave the rules out. The client passes
its existing DisableFirewall setting; the management server keeps the
default and still produces them.
RoutesFirewallRulesIsEmpty is set from the resulting empty list, so a
receiver that would otherwise infer legacy management from an empty rule
set does not misread the skip.
* [client,management] Cover the skip flag through the envelope
Review feedback on #7624.
The components test compared only the length of the peer firewall rules, so
a change to their content would have passed while the message claimed they
came out unchanged. Compare the slices.
The skip path was also only exercised by setting the field directly on the
components, which bypasses the envelope conversion where
RoutesFirewallRulesIsEmpty is derived. That bit is what keeps the client from
reading skipped rules as a legacy management server, so it gets a test that
goes through EnvelopeToNetworkMap with the flag set.
* [management] Give the router a peer ACL so the rule comparison bites
Review feedback on #7624.
peer-router-1 appears in no peer ACL in the shared fixture, so its
FirewallRules came out empty and the equality assertion compared two empty
slices — it would have passed even if the peer rules were dropped entirely.
Add a policy covering the router and require the baseline to be non-empty
before comparing.
* [relay] Signal relay disconnects through the conn context
AddCloseListener deduplicated listeners by comparing
reflect.ValueOf(callback).Pointer(). For a method value that pointer is
the address of the compiler-generated wrapper, not an identity bound to
the receiver, so every peer's w.onRelayClientDisconnected compared equal.
All peers on the home relay register under the same connectionURL key, so
only the first registration survived and the rest were silently dropped.
On a relay disconnect those peers were never notified: statusRelay stayed
connected and the reconnect guard never fired. The relayed net.Conn itself
was closed by closeAllConns, so nothing leaked, but the peer state machine
did not learn about it. Foreign relays had the same defect scoped to the
peers sharing that server.
Rather than fixing the deduplication, drop the peer-level listener registry
entirely. A relayed Conn now exposes Context(), cancelled when the
connection is torn down, with a cancellation cause naming the reason. This
is the same shape quic-go uses for its Conn and Stream types, and it
removes the whole class of problems around listener identity, lifetime and
deregistration: the signal belongs to the resource instead of a side table.
WorkerRelay watches that context in a goroutine whose lifetime matches the
connection. A watcher that wakes up for a superseded connection compares
the conn pointer against the current one and returns without touching the
state machine, so a fast relay reconnect cannot have a stale watcher tear
down the connection that replaced it.
Client.SetOnDisconnectListener stays: it is server-level and drives the
reconnect guard and foreign relay eviction, unrelated to peers.
handleRelayReady also checks the conn context, closing the race where the
relay dies between OpenConn and the readiness handoff and the peer would
otherwise build a WireGuard endpoint over a dead connection.
TestNotifierDoubleAdd covered the removed mechanism and is gone.
TestForeignAutoClose asserted nothing (both branches logged); it now waits
for the relay to leave the client map and fails if it does not.
* [relay] Fix build: return the concrete conn from Client.OpenConn
OpenConn now returns *Conn, but it still went through connContainer.netConn(),
which widens to net.Conn. The helper had one caller and only existed to produce
the interface value the signature no longer wants, so return container.conn
directly and drop it.
* [relay] Assert the local-close cancellation cause explicitly
The local-close test only rejected ErrServerDisconnected, so it would also
have passed for ErrPeerDisconnected or a bare context.Canceled. closeConn
cancels with net.ErrClosed, so assert that.
* [client] Ignore relay disconnects from superseded connections
The relayed conn watcher compared the conn pointer under relayLock, released
it, and only then tore the connection down. A new offer could install its
replacement in that window, so a watcher that validated the old pointer went
on to close the proxy of the connection that had already replaced it and
report the peer as disconnected while it was up.
Move the decision to where the teardown happens. Conn records which relayed
connection the current proxy was built from, and onRelayDisconnected takes the
connection the signal belongs to and drops it under conn.mu when it is no
longer the current one. Check and effect are now in the same critical section,
so the verdict cannot go stale before it is acted on.
This also covers the proxy read loops, whose disconnect listener took no
argument and had the same defect: it now names the connection it belongs to.
The WG timeout path keeps passing nil, since it deliberately tears down
whatever is current.
* [client] Bind the relayed conn reference to the proxy swap
relayedConnRef was set at the top of the readiness path, but wgProxyRelay only
changes at the end, in setRelayedProxy. The two failure returns in between —
newProxy and ConfigureWGEndpoint — left the reference pointing at a connection
that never became active while the old proxy was still installed. A disconnect
of that old, live relay would then be dismissed as belonging to a superseded
connection and never cleaned up.
Set the reference in setRelayedProxy, next to the proxy it belongs to. Both
success paths go through it and neither failure path does, so no failure branch
has to remember to roll anything back.
GetSessionExpiresAt took the exclusive lock for a plain field read, so
every caller queued behind writers and behind each other. The Android
SessionMonitor polls it from the main thread, and in the captured ANR
that is exactly where the main thread was blocked while hundreds of
peer-list callbacks held or waited on the same mutex.
d.mux is already an RWMutex and the other getters use RLock; this brings
the deadline read in line with them.
Read the relay instance URL and IP atomically to prevent reconnects from mixing values from different connections. Extend existing connection and offer/answer logs with relay URLs and IPs to help trace mismatched advertisements.
The per-exchange lifecycle logs were all at LevelTrace (below the default field
level) and did not distinguish a signalling re-bootstrap from a data-path rekey,
which made diagnosing exchange activity in the field guesswork (e.g. telling an
ICE-retry-driven re-bootstrap storm apart from normal KEM rotation).
Promote the two pivotal events — an offer going out and a PSK converging — to
LevelDebug and tag every lifecycle log with four dimensions:
- msg: offer vs answer (implicit in the message)
- role: initiator vs responder for this peer
- via: signal vs data-path (threaded into processOffer/processAnswer)
- kind: bootstrap (a new connection / reconnect) vs rotation (a rekey)
kind is derived from the exchange's AckID (zero = bootstrap) on the responder
side and from viaSignal on the initiator side. No behavioural change.
pqControllerReoffer returned true whenever we are the controller running the
KEM, even when ShouldSendBootstrapOffer was false — which is the steady state
for every PQ peer past its bootstrap (exchange in awaitingRekey) and for
non-capable peers. handleRemoteOffer then returned early, dropping the ICE
credentials and relay info carried in the peer's offer, so a responder-initiated
reconnect (the normal path under lazy connections) stalled: the controller only
recovered on its own guard timing.
Return true only when a re-offer was actually sent; otherwise fall through to the
normal offer handling (sendAnswer + notifyListeners) so the connection can come
up on the retained PSK. A stale PSK still self-heals via the WG watcher, which
triggers a fresh signal offer that re-bootstraps.
Two convergence bugs surfaced by the security review:
- Role guard (finding B): processOffer accepted an offer even when we are the
KEM initiator for the peer, and processAnswer accepted an answer when we are
the responder. The KEM is unidirectional (initiator offers, responder
answers), so a role-violating message is anomalous — a desync, a duplicate,
or an injected/spoofed data-path packet. Processing it derived and committed
a fresh PSK, overwriting a live one and silently dropping any in-flight
exchange (whose retry loop then exited without raising a failure or
re-bootstrapping). Reject offers when we are the initiator and answers when
we are not; this drops only anomalous traffic and leaves the normal flow
untouched.
- Re-bootstrap on signal re-negotiation (finding A): SignalOffer was idempotent
in stateAwaitingRekey too, replaying the frozen bootstrap offer. After the
responder restarted and lost its state it derived a different PSK from fresh
material, which our awaitingRekey side then rejected — a permanent desync with
no recovery (in strict mode the peer stays blocked). Make the idempotency
apply only while a bootstrap is still in flight (awaitingAnswer); once a PSK
is derived, a fresh signal offer starts a new exchange so both sides converge.
The controller-double-offer case the idempotency guarded is already covered by
ShouldSendBootstrapOffer. Reusing the cached offer also reused the same
ephemeral keys across exchanges, reducing forward secrecy.
Both paths have a failing-without-the-fix regression test.
- strict-kem vs strict-rp said "Connected + Quantum resistance: true" but it is actually blocked
- perm-kem vs perm-rp "Connected + Quantum resistance: true" but it's a classic WG link, without PQ safety
Reduce Listen's cognitive complexity (SonarCloud S3776, 30 -> under 25) by
extracting the offer and answer cases into handleRemoteOffer/handleRemoteAnswer,
with shared onSignalReceived/notifyListeners helpers and a pqControllerReoffer
helper for the controller re-offer branch. No functional change.
The responder configures WireGuard with endpoint=nil first, then a delayed
update (scheduleDelayedUpdate) applies the real endpoint after fallbackDelay.
It captured the preshared key at schedule time and re-applied it. With the
post-quantum exchange the PSK can change within that window (a fresher PSK
derived and applied via SetPresharedKey), so re-applying the captured one
reverted WireGuard to a key the remote peer no longer used — a mismatch that
stalled the handshake until the WGWatcher timeout forced a retry (~30s).
Pass a nil PSK in the delayed update so it only sets the endpoint and leaves
the current PSK in place; the latest SetPresharedKey wins.
When the controller receives the responder's (KEM-less) offer it replies with
its own KEM offer instead of answering, so the only transaction that brings the
tunnel up is the one that also carries the PSK. Guard that reply with
ShouldSendBootstrapOffer so it fires only when no exchange is in flight: without
it, every responder offer triggered another offer (an offer-per-offer runaway).
The whole behaviour is isolated to the KEM path (config.PQ != nil); non-PQ
connections answer as before.
The controller sends its KEM offer both on its own guard event and in reply
to the responder's offer. SignalOffer was idempotent only while awaiting the
answer; once the answer arrived (awaitingRekey) a repeat call started a fresh
exchange with a different PSK, desyncing the two peers (one on the old PSK,
one on the new) so WireGuard derived misaligned transport keys and dropped all
data. Treat awaitingRekey as in-flight too and return the same offer.
To ensure two peers agree on a key, we need asymmetry. one peer is
the controller ("initiator") the other is the "responder".
Otherwise imagine two offers in parallel driving two answers at the same time
A B
| <----B-OFFER----- |
| -----A-OFFER----> |
| |
| |
---------------------------------
|****** ICE + WG Handshake ****** |
---------------------------------
| |
| <----B-ANSWER---- |
| -----A-ANSWER---> |
PSK is derived on receive of offer, so A and B derive different PSKs.
When WG handshake takes place it picks misaligned PSKs.
So we impair the two nodes and only the offer of one of the two (the controller/initiator)
is allowed to progress and drive the answer (and carry the KEM material).
If a responder initiates an offer, we redo the offer towards it. This is oK
since the ICEworker don't treat offer/answers differently.
To ensure two peers agree on a key, we need asymmetry. one peer is
the controller ("initiator") the other is the "responder".
Otherwise imagine two offers in parallel driving two answers at the same time
A B
| <----B-OFFER----- |
| -----A-OFFER----> |
| |
| |
---------------------------------
|****** ICE + WG Handshake ****** |
---------------------------------
| |
| <----B-ANSWER---- |
| -----A-ANSWER---> |
PSK is derived on receive of offer, so A and B derive different PSKs.
When WG handshake takes place it picks misaligned PSKs.
So we impair the two nodes and only the offer of one of the two (the controller/initiator)
carries the KEM material.
This means that if the responder OFFER/ANSWER comes first, when the controller/initiator's one
completes (and the genuine PSK is shared between A and B, we need to force a new WG handshake with
the proper keys.
- RecoversViaResignalAfterDataPathBreak: a data-path rotation that can no longer
converge raises OnRekeyFailed, and re-bootstrapping over signalling resyncs both
peers on a fresh PSK even while the data path stays broken.
- ConcurrentRekeysNoRace: hammers the single-lock state machine with concurrent
rotation clocks from many goroutines (run with -race) and asserts no split-brain
via a final deterministic bootstrap.
OnRekeyFailed now re-runs the KEM bootstrap over Signal (conn.RequestReoffer ->
handshaker.SendOffer) instead of only logging: a fresh signalling offer starts a new
exchange that overwrites the stalled PSK on both sides, resyncing after a persistent
data-path desync. Chosen over a responder-side awaitingAck revert (which fights the
confirm-less ack timing) and a full tunnel teardown (heavier). The tunnel stays up on
the previous PSK meanwhile since Signal is independent of the broken data path.
Strict mode (NB_PQ_MLKEM_STRICT, default off) closes the initial PQ-vulnerable
window (NET-1408): when enabled, conn.presharedKey programs a per-conn random
sentinel PSK until the ML-KEM exchange derives the real one, so no session can form
on a non-PQ key (the real PSK is pushed via SetPresharedKey once it converges).
Default stays opportunistic.
Also surface PQ status: the peer 'Quantum resistance' flag (RosenpassEnabled) is now
true when an ML-KEM PSK has been derived for the peer, not only for Rosenpass.
The idle-gate reads LastActivities, which only tracks per-peer data in userspace;
in kernel mode it is empty, so the gate treated every kernel peer as idle and
disabled data-path rotation entirely. Detect the bind via IsUserspaceBind and, in
kernel mode, report zero activity age (always 'active') so rotation runs on every
rekey. Lazy back-to-idle is already limited in kernel; the eBPF WG-activity
detection will later supply a real signal that excludes handshake/pqkem traffic.
Replace the raw SHA-256 concat combiner with HKDF-SHA256 (crypto/hkdf, Go 1.24):
IKM = ML-KEM_ss || X25519_ss (draft-ietf-tls-ecdhe-mlkem order), salt = the
domain-separation label, info = full transcript (offer || answer) || canonicalised
peer identities. Keeps the transcript + identity binding while using a proper KDF.
- Add a trace slog level (NB_PQ_MLKEM_LOG_LEVEL=trace) and move the verbose
per-exchange lifecycle logs (offer/answer/PSK/ack/rotation) to it, so debug
stays quiet and troubleshooting is opt-in.
- Stop logging the raw preshared key; drop the temporary pqkem-dbg OnRemoteOffer/
OnRemoteAnswer probes.
- Demote the per-handshake conn log to trace.
Source OnDataPathRekeyed from the WGWatcher's per-handshake callback
(onWGCheckSuccess), which fires only on a fresh handshake, and OnDataPathDown
from the handshake-timeout path. A fresh handshake clocks the next chained
KEM exchange pushed over the data-path UDP transport.
Learn the peer's data-path endpoint from the signalling offer/answer: its WG
overlay IP combined with the advertised pq UDP port (SetRemotePort -> AddPeer).
Registering here is safe before the tunnel is up because sends only ever fire
once it is (clocked by OnDataPathRekeyed). RemovePeer is wired at peer teardown
(engine.removePeer), not on transient disconnect.
riccardom