Riccardo Manfrin
a419e770d9
* [client] Track the WireGuard device on the engine as a lock-free handle Add an atomic handle on the wg device next to wgInterface, stored once the interface is up and cleared when it is closed. Nothing reads it yet, so this is a pure addition with no behavior change; it exists so the next commit can reach the device without taking syncMsgMux. * [client] Retune the WireGuard buffer pool without the engine lock SetPerformance took syncMsgMux before reaching the device. That lock is held by handleSync while it adds and removes peers, and peer removal is exactly what blocks when a device's buffer pool is exhausted: Peer.Stop waits on a keepalive timer callback that is itself parked in WaitPool.Get. Raising the cap is the way out of that state, so the call must not queue behind the lock the stall is holding. Read the device through the atomic handle instead. Device.SetPreallocatedBuffersPerPool takes the pool's own lock and broadcasts, so the waiters wake up. * [proxy] Extract the buffer-cap apply loop out of the perf handler Pure move: the loop over the registered clients becomes applyBufferCap, with the same sequential behavior and the same return values. Split out so the next commit can change how it iterates without the diff also carrying the move. * [proxy] Bound the perf endpoint so one wedged client cannot hold it The apply loop was sequential and unbounded. embed.Client.SetPerformance goes through the client lock, which Start holds for the whole of a startup, so a single account that is busy or wedged delayed the new buffer cap for every other account on the node -- on the endpoint whose whole purpose is to un-wedge a node. Apply to all clients concurrently and give the whole call a 5s budget. Accounts that do not answer in time are reported in "failed" instead of blocking the response. * [client] Drop the device handle before closing the interface close() cleared the atomic handle only after wgInterface.Close() returned, so a concurrent SetPerformance could still load it, retune a device that is being torn down, and report the change as applied for an engine that has stopped. Clear it first, so the window closes before the teardown begins. Reported by cubic on PR #7452. * [proxy] Put the per-client retune behind a field Pure refactor: applyBufferCap calls h.setPerformance instead of the client method directly, and NewHandler wires it to setClientPerformance. Same call, same behavior; the seam is what lets the next two commits be tested without a live embedded client. * [proxy] Do not report a finished retune as timed out When the deadline fires, select chooses at random among the ready cases, so a result already sitting in the buffered channel could be skipped and its account reported as timed out even though the cap had been applied. Drain what is buffered before declaring the rest pending. Reported by cubic on PR #7452. * [proxy] Keep one retune per account in flight The 5s budget bounds how long the endpoint waits, not the work: SetPerformance goes through the embedded client's lock, and on a wedged account Stop holds that lock forever, so every retry left one more goroutine parked there. Route each account through a single worker. A request that finds one already running takes its result if it has landed, and otherwise reports the account under "in_flight" instead of starting a second attempt. One stuck account now costs one goroutine, no matter how often the endpoint is called. Reported by CodeRabbit and cubic on PR #7452. * [proxy] Make the retune budget a var Pure refactor: perfApplyTimeout becomes a var so a test can shorten it instead of waiting five seconds. Same value, same behavior in production. * [proxy] Extract the buffered-result drain Pure refactor: the loop that empties the results channel when the deadline fires becomes collectBuffered. Same behavior; split out so it can be tested on its own, which the inline version could not be without racing the deadline. * [proxy] Cover the retune single-flight and the deadline drain TestApplyBufferCapSingleFlightPerAccount fails without the worker registry: five calls against a client stuck in its own lock start five blocked workers instead of one. TestCollectBufferedCountsResultsReadyAtTheDeadline pins the drain helper's contract - buffered results counted, errors recorded, only unanswered accounts left pending. It drives collectBuffered directly: through applyBufferCap the two select cases race by construction, so an end-to-end version of it would pass on the unfixed code about half the time. * [proxy] Keep the worker alongside each pending account Pure refactor: the pending set becomes a map to the account's worker instead of an empty struct. Same membership and same behavior; the next commit needs the worker to resolve an account whose result has not reached the channel yet. * [proxy] Publish a retune result before releasing its slot The worker sent its result last, after taking perfMu to remove itself from the registry. That lock is taken once per account by every caller walking the fleet, so a worker that finished on time could queue behind an apply over thousands of accounts and land after the deadline. Send first, deregister after. Reported by cubic on PR #7452. * [proxy] Read the worker, not the clock, for a finished retune Publishing earlier only narrows the window: a client that answers just before the deadline can still be reported as timed out. At the deadline the workers themselves are authoritative - a closed done channel means the retune finished and w.err carries its outcome, ordered by the close. Consult them instead of declaring every pending account timed out, and keep the timeout label for the ones actually still running. Reported by cubic on PR #7452. * [proxy] Cover the finished-worker resolution at the deadline Fails on the previous behavior with "applied = 0, want 1": every pending account was labelled a timeout, including the one whose retune had already completed.
Netbird Reverse Proxy
The NetBird Reverse Proxy is a separate service that can act as a public entrypoint to certain resources within a NetBird network. At a high level, the way that it operates is:
- Configured routes are communicated from the Management server to the proxy.
- For each route the proxy creates a NetBird connection to the NetBird Peer that hosts the resource.
- When traffic hits the proxy at the address and path configured for the proxied resource, the NetBird Proxy brings up a relevant authentication method for that resource.
- On successful authentication the proxy will forward traffic onwards to the NetBird Peer.
Proxy Authentication methods supported are:
- No authentication
- Oauth2/OIDC
- Emailed Magic Link
- Simple PIN
- HTTP Basic Auth Username and Password
Management Connection and Authentication
The Proxy communicates with the Management server over a gRPC connection. Proxies act as clients to the Management server, the following RPCs are used:
- Server-side streaming for proxied service updates.
- Client-side streaming for proxy logs.
To authenticate with the Management server, the proxy server uses Machine-to-Machine OAuth2. If you are using the embedded IdP //TODO: explain how to get credentials. Otherwise, create a new machine-to-machine profile in your IdP for proxy servers and set the relevant settings in the proxy's environment or flags (see below).
User Authentication
When a request hits the Proxy, it looks up the permitted authentication methods for the Host domain. If no authentication methods are registered for the Host domain, then no authentication will be applied (for fully public resources). If any authentication methods are registered for the Host domain, then the Proxy will first serve an authentication page allowing the user to select an authentication method (from the permitted methods) and enter the required information for that authentication method. If the user is successfully authenticated, their request will be forwarded through to the Proxy to be proxied to the relevant Peer. Successful authentication does not guarantee a successful forwarding of the request as there may be failures behind the Proxy, such as with Peer connectivity or the underlying resource.
TLS
Due to the authentication provided, the Proxy uses HTTPS for its endpoint, even if the underlying service is HTTP.
Certificate generation can either be via ACME (by default, using Let's Encrypt, but alternative ACME providers can be used) or through certificate files.
When not using ACME, the proxy server attempts to load a certificate and key from the files tls.crt and tls.key in a specified certificate directory.
When using ACME, the proxy server will store generated certificates in the specified certificate directory.
Auth UI
The authentication UI is a Vite + React application located in the web/ directory. It is embedded into the Go binary at build time.
To build the UI:
cd web
npm install
npm run build
For UI development with hot reload (served at http://localhost:3031):
npm run dev
The built assets in web/dist/ are embedded via //go:embed and served by the web.ServeHTTP handler.
Configuration
NetBird Proxy deployment configuration is via flags or environment variables, with flags taking precedence over the environment. The following deployment configuration is available:
| Flag | Env | Purpose | Default |
|---|---|---|---|
-debug |
NB_PROXY_DEBUG_LOGS |
Enable debug logging | false |
-mgmt |
NB_PROXY_MANAGEMENT_ADDRESS |
The address of the management server for the proxy to get configuration from. | "https://api.netbird.io:443" |
-addr |
NB_PROXY_ADDRESS |
The address that the reverse proxy will listen on. | ":443 |
-url |
NB_PROXY_URL |
The URL that the proxy will be reached at (where endpoints will be CNAMEd to). If unset, this will fall back to the proxy address. | "proxy.netbird.io" |
-cert-dir |
NB_PROXY_CERTIFICATE_DIRECTORY |
The location that certificates are stored in. | "./certs" |
-acme-certs |
NB_PROXY_ACME_CERTIFICATES |
Whether to use ACME to generate certificates. | false |
-acme-addr |
NB_PROXY_ACME_ADDRESS |
The HTTP address the proxy will listen on to respond to HTTP-01 ACME challenges | ":80" |
-acme-dir |
NB_PROXY_ACME_DIRECTORY |
The directory URL of the ACME server to be used | "https://acme-v02.api.letsencrypt.org/directory" |
-oidc-id |
NB_PROXY_OIDC_CLIENT_ID |
The OAuth2 Client ID for OIDC User Authentication | "netbird-proxy" |
-oidc-secret |
NB_PROXY_OIDC_CLIENT_SECRET |
The OAuth2 Client Secret for OIDC User Authentication | "" |
-oidc-endpoint |
NB_PROXY_OIDC_ENDPOINT |
The OAuth2 provider endpoint for OIDC User Authentication | "https://api.netbird.io/oauth2" |
-oidc-scopes |
NB_PROXY_OIDC_SCOPES |
The OAuth2 scopes for OIDC User Authentication, comma separated | "openid,profile,email" |