Compare commits

...

26 Commits

Author SHA1 Message Date
Zoltán Papp
43659ba4ab Merge branch 'main' into android-airplane 2026-08-17 14:19:38 +02:00
Viktor Liu
6210399e65 [client] Declare multi-buffer support for the loopback XDP program (#7230) 2026-08-17 13:10:12 +02:00
Viktor Liu
939b686d05 [client] Delete NRPT rules by enumerating the registry instead of a rule count (#7195) 2026-08-17 12:52:17 +02:00
Zoltan Papp
70f192344b [client] Update golang.org/x/mobile to v0.0.0-20260816165457-f98cc9b3c733 (#7229) 2026-08-17 12:10:19 +02:00
Misha Bragin
4e5b632490 [infrastructure] Don't override the dashboard image on enterprise migration (#7206) 2026-08-16 16:40:20 +02:00
Misha Bragin
93e97f4bf1 [doc] Agent network docs update (#7020)
* [docs] Update agent-network docs for management-owned pricing

  The docs still described the retired proxy-side pricing: pricing.Loader,
  pricing_path, MiddlewareDataDir, embedded defaults_pricing.yaml, and the
  symlink-safe Unix loader. Rewrite them for the current design — management
  synthesizes the whole table and ships it in cost_meter's ConfigJSON, so the
  proxy carries no price list and has nothing to reload.
2026-08-15 19:31:49 +02:00
Zoltan Papp
16544dbc58 [client] Pass stored email as login hint from UI and keep it on logout (#7199)
* [client] Pass stored email as login hint from UI and keep it on logout

Follow the CLI pattern: the Wails UI now reads the account email from the
user-owned profile state file and passes it as the OIDC login_hint on login
and session extend, since the daemon-side fallback runs as root and cannot
see the user's state file. Logout no longer deletes the stored email, so a
later login preselects the account at the IdP; profile removal remains the
operation that deletes it.

* [client] Log ignored profile lookup errors in extend-session hint fallback
2026-08-15 11:21:57 +02:00
Zoltan Papp
f458c1f265 [client] Skip IPv6 route tests when the default nexthop is unusable (#7212)
* [client] Skip IPv6 route tests when the default nexthop is unusable

ensureIPv6DefaultRoute treated a successful netlink RouteAdd as proof that
a usable IPv6 nexthop exists. Installing ::/0 via loopback can succeed while
the kernel still rejects that nexthop for a concrete prefix, which surfaced
on ubuntu22/20260810.260 runners as:

    add route to table: netlink add route: invalid argument

Probe the resolved nexthop by installing and removing a discard-prefix route
through the same code path the tests use, and skip when it fails. EEXIST
means the nexthop already carries a route, so it counts as usable.

* [client] Probe the IPv6 nexthop through raw netlink

addRoute swallows EAFNOSUPPORT and EOPNOTSUPP via isOpErr, so a nil return
did not prove the probe route was installed. Call netlink directly so an
unsupported operation skips the test instead of passing as usable.
2026-08-15 10:13:06 +02:00
Viktor Liu
ec6f1b8c27 [client] Rank Windows route candidates by combined route and interface metric (#7210) 2026-08-15 09:06:22 +02:00
Zoltán Papp
cfd3a231e2 [client] Reconnect management and signal promptly after a network change
Three pieces, each verified on Android across WiFi/cellular switches:

- A sweep-aware backoff wrapper: the first retry after a disconnect
  that follows a recent network-change mark comes after 200ms instead
  of the randomized [0..1.6s] interval. Any other failure keeps the
  unchanged spread, so the clients of a restarted server still scatter
  their reconnects.
- The retry sleep wakes on OS network availability transitions
  (nbgrpc.Retry): a disconnect that precedes the offline flag by a few
  milliseconds no longer sleeps blindly through the recovery - the
  loop parks on the netstate gate and resumes the moment the network
  returns.
- The connection state is re-checked after WaitForStateChange: a dial
  settling in Ready proceeds immediately instead of burning another
  backoff round on an already-usable channel.

Measured after a network switch: management and signal recover in
270-470ms deterministically, down from a 312-1593ms lottery.
2026-08-13 15:49:44 +02:00
Zoltán Papp
31e558a64d [client] Sweep only stale connections after a network change
Stamp every registered connection and in-flight dial with a network
generation, bumped by MarkNetworkChange, which replaces the immediate
full sweep with one delayed by a configurable 500ms. The sweep then
cuts only registrations older than the last change: subsystems that
redialed on their own hold fresh-generation connections and survive,
so the callers no longer need cancellation logic around the sweep.
A connection inherits its dial's generation, because the socket was
bound to the network that was default when the dial started.

This fixes the sweep being cancelled by the engine's management-level
reconnect while the relay was still down, and lets the mobile
notifiers shrink to plain forwarders.
2026-08-13 14:54:19 +02:00
Zoltán Papp
f986b33b92 [relay] Gate the quick reconnect on network availability
Replace the fixed 1.5s pre-dial sleep with a wait driven by the OS
availability verdict: while online only a 200ms settle window is spent
(the offline flag lands a few milliseconds after the disconnect that
triggered the reconnect), and while offline the guard waits for the
network to return, bounded by the same 1.5s budget as before. Without
an injected netState the behavior is unchanged.

Measured on Android: relay recovery after a network switch drops from
1.7-1.9s to under 500ms after the cut.
2026-08-13 14:54:04 +02:00
Zoltán Papp
fc52cdcedd [client] Fix lint issues in management and signal grpc clients
Rename ClientOption to Option in the management and signal client
packages so callers do not repeat the package name, and silence nilerr
on the offline-wait path where a cancelled context means shutdown
rather than a retryable failure.
2026-08-12 12:24:13 +02:00
Zoltán Papp
b4cc8a16d9 Merge remote-tracking branch 'origin/main' into android-airplane
# Conflicts:
#	client/android/client.go
#	client/ios/NetBirdSDK/client.go
2026-08-12 11:35:59 +02:00
Zoltán Papp
3c973c98e3 [client] Serialize notifier state publication
The effective state was computed under serverStateLock but published after
releasing it, so two transitions could reorder between compute and notify
and leave the listener on a state the notifier had already superseded,
e.g. NoNetwork surviving after the network came back.

Hold a publish lock across compute and notify on every publishing path.
2026-08-12 10:46:16 +02:00
Zoltán Papp
1fcb6d4bed [client] Do not wrap a nil connection listener in the mobile bindings
The listener adapter introduced with the network state work turned a nil
listener into a non-nil peer.Listener holding a nil delegate, so the
notifier's nil check passed it through and setListener panicked on its
immediate OnAddressChanged callback. EngineRunner already forwards null,
so the path is reachable.

Drop the listener instead when it is nil, on Android and iOS alike.
2026-08-12 10:23:23 +02:00
Zoltán Papp
7612b4d299 [client] Stop the guard test poller from outliving the test
pollUntil span a goroutine that looped forever when the condition never
held, which is exactly the path the test takes when it fails. Pass the
test context in and give up when it is done.
2026-08-12 10:14:59 +02:00
Zoltán Papp
ecce1aa2a1 [client] Give the netsweep registry a named id type
The connection and dial registries keyed on a bare uint64, which says
nothing about what the number identifies. Introduce sweepID so the maps,
the counter and the id fields state their intent. No behavior change.
2026-08-12 10:14:51 +02:00
Zoltán Papp
53f2518af1 [client] Resume peer reconnection when the network comes back
Ticks taken while the OS reported no network were skipped, but they still
advanced the exponential backoff, so a peer could be tens of seconds from its
next attempt by the time connectivity returned. Recovery then depended on a
signal or relay event, which never arrives when both stayed up across the
outage, e.g. a short airplane mode toggle over Wi-Fi. A peer parked in ICE
hourly mode stayed there for the same reason.

React to the offline-to-online transition directly: re-arm the reconnect
ticker and reset the ICE retry state, so the peer retries at once. Add
netstate.Changed for callers that own a select loop and cannot block in Wait.
2026-08-12 09:43:50 +02:00
Zoltán Papp
71a4276426 [client] Group the netsweep dial API and the Android network hooks
Move the Dial type and its Ctx/Release methods above the Sweeper in
netsweep.go, and relocate SetNetworkAvailable / NotifyNetworkChange below
GetTunSettings in the Android binding. Pure code moves, no behavior change.
2026-08-12 09:39:18 +02:00
Zoltán Papp
71bfc73cd1 [client] Hand off dialed connections to the sweeper atomically
WrapDialContext and WrapConn registered the dial and the connection
independently, so a sweep landing between the dial finishing and WrapConn
cancelled only the dial registration: the connection dialed on the old
network entered the fresh registry and survived the network change.

Replace the pair with a Dial handle. Sweep marks pending dials under the
sweeper mutex, and WrapConn decides under the same mutex: a swept dial's
connection is closed and ErrSwept returned, so the caller redials on the
new network; otherwise the connection transfers to the registry with no
window in between.
2026-08-11 18:09:37 +02:00
Zoltán Papp
77e7d82d5a [client] Keep the peer endpoint open in the sweep test
connPair closed the accepted connection right after the handshake, so the
reads in TestSweepClosesRegisteredConns failed on the peer's own close
rather than on the sweep. The test passed even with Sweep's close loop
removed. Hold the peer until cleanup so the read errors come from Sweep.
2026-08-11 17:59:48 +02:00
Zoltán Papp
e3e5a25605 [client] Read the relay transport before the sweeper wraps the connection
sweptConn embeds net.Conn, so it does not promote Protocol() from the
concrete relay connection. Asserting transportConn on the wrapper always
failed and left c.transport empty. Read the transport off the dialed
connection first, then wrap it.
2026-08-11 17:55:19 +02:00
Zoltan Papp
072fa8143b [client] Sweep network-bound connections when the OS switches networks
On a network switch (e.g. cellular to WiFi) the management, signal and
relay sockets stay bound to the old network and look alive until the OS
tears them down — measured at 5 seconds of dead air on Android, while
the UI kept claiming Connected. The Android client papered over this
with a full engine restart, paying for it with a torn-down TUN device
and discarded peer state.

Introduce client/netsweep: connections register on dial and deregister
on close, and a sweep closes everything registered while aborting
in-flight dials through sweep-cancellable dial contexts. The aborted
dials matter: a relay dial started on the dying network would otherwise
hold the reconnect loop hostage for the QUIC handshake timeout. After a
sweep every failure surfaces as an ordinary read/write error and the
existing retry loops redial immediately on the new network.

The sweeper reaches the three long-lived connections through the same
options that carry the netstate gate: a gRPC dial option wraps the
management and signal transports (reconnects included), and the relay
client wraps its connection in one place for the picker, the guard and
foreign relays alike. Everything is nil-safe; platforms that inject no
sweeper are untouched.

Mobile clients expose the sweep as NotifyNetworkChange. Measured on
Android against the engine restart it replaces: recovery in 1.6s
instead of 3.2s, no Disconnected flash, and the TUN device, WireGuard
config and peer state survive.
2026-08-11 00:58:50 +02:00
Zoltan Papp
8fb3e707af [client] Keep the iOS ConnectionListener source-compatible
Adding OnStateChanged to the gomobile interface forces every Swift
implementation to grow the method before the app builds again. Drop it
from the iOS binding for now — the adapter satisfies the internal
listener with a no-op and the legacy per-state callbacks keep firing —
so the app upgrades on its own schedule. The state constants stay
exported for that follow-up.
2026-08-11 00:58:50 +02:00
Zoltan Papp
902263ac96 [client] Suspend reconnection loops while the OS reports no network
On mobile the client kept dialing management, signal, relay and peer
connections while the device had no usable network at all (airplane
mode), burning battery for attempts that cannot succeed. Stopping the
engine is not an option: tearing it down destroys the TUN device, and
traffic can leak outside the tunnel until it is rebuilt.

Add client/netstate, a small gate the platform feeds from its own
connectivity callbacks. Every reconnection loop waits on it instead of
retrying blindly, and resets its backoff when the network returns so
recovery is immediate. The state is injected through functional options
and consumers hold a *State that may be nil, so every platform that does
not report availability behaves exactly as before.

The relay quick-reconnect rechecks availability after its 1.5s wait: the
disconnect that triggers it is usually the first symptom of the network
going away, so the flag typically arrives while it sleeps.

Report the suspension to the UI as well. peer.Listener grows
OnStateChanged with a typed ClientState, re-exported across the gomobile
boundary as integer constants, and the notifier maps Connecting to a new
NoNetwork state while the OS reports no network, so mobile clients can
show "no network available" instead of a misleading "connecting".

Finally, exit the client retry loop cleanly when its context is
cancelled. backoff.WithContext surfaces the bare context error, which
callers could not distinguish from a real failure — on Android that
turned an engine restart into an unrecoverable error.
2026-08-11 00:58:50 +02:00
58 changed files with 2600 additions and 383 deletions

View File

@@ -1,6 +1,6 @@
# NetBird Agent Guidelines
**NetBird** is an open-source connectivity platform: a WireGuard®-based overlay
**NetBird** is an open source connectivity platform: a WireGuard®-based overlay
network with a control plane. The **agent** (`client/`) runs on user machines as
a privileged daemon and manages the WireGuard interface, routing, firewall, and
DNS. **Management** (`management/`) is the control plane and REST/gRPC API,

View File

@@ -479,7 +479,7 @@ go test -race ./client/internal/dns/...
## Checklist before submitting a PR
As a critical network service and open-source project, we must enforce a few
As a critical network service and open source project, we must enforce a few
things before submitting a pull request. The
[pull request template](/.github/pull_request_template.md) mirrors this list —
fill it in rather than deleting it.

View File

@@ -130,7 +130,7 @@ In November 2022, NetBird joined the [StartUpSecure program](https://www.forschu
![CISPA_Logo_BLACK_EN_RZ_RGB (1)](https://user-images.githubusercontent.com/700848/203091324-c6d311a0-22b5-4b05-a288-91cbc6cdcc46.png)
### Acknowledgements
We build on open-source technologies like [WireGuard®](https://www.wireguard.com/), [Pion ICE](https://github.com/pion/ice), and [Rosenpass](https://rosenpass.eu). We greatly appreciate the work these projects are doing, and we'd love it if you could support them too (e.g., by starring or contributing).
We build on open source technologies like [WireGuard®](https://www.wireguard.com/), [Pion ICE](https://github.com/pion/ice), and [Rosenpass](https://rosenpass.eu). We greatly appreciate the work these projects are doing, and we'd love it if you could support them too (e.g., by starring or contributing).
### Legal
This repository is licensed under the BSD-3-Clause license, which applies to all parts of the repository except for the directories management/, signal/ and relay/.

View File

@@ -14,7 +14,7 @@ Report security issues one of these two ways:
on this repository. This is the preferred route: it keeps the discussion, the draft advisory, and the credit in one place.
- **Email** — `security@netbird.io`.
If the finding affects NetBird Cloud or our hosted infrastructure rather than the open-source code, email us rather than
If the finding affects NetBird Cloud or our hosted infrastructure rather than the open source code, email us rather than
filing a repository report.
### What to include

View File

@@ -26,6 +26,8 @@ import (
"github.com/netbirdio/netbird/client/internal/routemanager"
"github.com/netbirdio/netbird/client/internal/stdnet"
"github.com/netbirdio/netbird/client/net"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
"github.com/netbirdio/netbird/client/system"
"github.com/netbirdio/netbird/formatter"
"github.com/netbirdio/netbird/route"
@@ -40,11 +42,6 @@ const (
AnonymizeLevelStrict = nbAnonymize.LevelStrictString
)
// ConnectionListener export internal Listener for mobile
type ConnectionListener interface {
peer.Listener
}
// TunAdapter export internal TunAdapter for mobile
type TunAdapter interface {
device.TunAdapter
@@ -85,6 +82,13 @@ type Client struct {
deviceName string
uiVersion string
networkChangeListener listener.NetworkChangeListener
// netState outlives engine restarts: it mirrors the OS connectivity, not
// the engine lifecycle. Run and RunWithoutLogin inject it into each new
// ConnectClient, which distributes it to every reconnection loop.
netState *netstate.State
// sweeper also outlives engine restarts; NotifyNetworkChange sweeps it.
sweeper *netsweep.Sweeper
stateMu sync.RWMutex
connectClient *internal.ConnectClient
@@ -156,6 +160,8 @@ func NewClient(androidSDKVersion int, deviceName string, uiVersion string, tunAd
recorder: peer.NewRecorder(""),
ctxCancelLock: &sync.Mutex{},
networkChangeListener: networkChangeListener,
netState: netstate.New(),
sweeper: netsweep.New(),
}
}
@@ -196,7 +202,8 @@ func (c *Client) Run(platformFiles PlatformFiles, urlOpener URLOpener, isAndroid
}
// todo do not throw error in case of cancelled context
ctx = internal.CtxInitState(ctx)
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder)
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder,
internal.WithNetworkState(c.netState), internal.WithSweeper(c.sweeper))
c.setState(cfg, cacheDir, cfgFile, connectClient)
// This path runs the interactive SSO flow, so reaching here means the peer
// is authenticated again — release the latch Status() reports from. Clear
@@ -237,7 +244,8 @@ func (c *Client) RunWithoutLogin(platformFiles PlatformFiles, dns *DNSList, dnsR
// todo do not throw error in case of cancelled context
ctx = internal.CtxInitState(ctx)
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder)
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder,
internal.WithNetworkState(c.netState), internal.WithSweeper(c.sweeper))
c.setState(cfg, cacheDir, cfgFile, connectClient)
return connectClient.RunOnAndroid(c.tunAdapter, c.iFaceDiscover, c.networkChangeListener, slices.Clone(dns.items), dnsReadyListener, stateFile, cacheDir)
}
@@ -285,6 +293,24 @@ func (c *Client) GetTunSettings() (*TunSettings, error) {
}, nil
}
// SetNetworkAvailable feeds OS-reported network availability into the client.
// While unavailable, the internal reconnect loops suspend their attempts and
// the connection listener reports NoNetwork instead of Connecting; when
// availability returns, the loops resume immediately with a fresh backoff.
func (c *Client) SetNetworkAvailable(available bool) {
c.netState.Set(available)
c.recorder.SetNetworkAvailable(available)
}
// NotifyNetworkChange marks the management, signal and relay connections
// stale after the OS switched networks and schedules a sweep that cuts
// whatever has not redialed on the new network by then. The engine and the
// TUN device stay untouched.
func (c *Client) NotifyNetworkChange() {
c.sweeper.MarkNetworkChange()
log.Infof("network change: connections marked stale")
}
// DebugBundle generates a debug bundle, uploads it, and returns the upload key.
// It works both with and without a running engine. anonymizeLevel is "default"
// or "strict"; strict also anonymizes internal IP ranges, peer names, and
@@ -525,7 +551,11 @@ func (c *Client) OnUpdatedHostDNS(list *DNSList) error {
// SetConnectionListener set the network connection listener
func (c *Client) SetConnectionListener(listener ConnectionListener) {
c.recorder.SetConnectionListener(listener)
if listener == nil {
c.recorder.RemoveConnectionListener()
return
}
c.recorder.SetConnectionListener(connectionListenerAdapter{listener})
}
// RemoveConnectionListener remove connection listener

View File

@@ -0,0 +1,41 @@
//go:build android
package android
import (
"github.com/netbirdio/netbird/client/internal/peer"
)
// Client state values delivered via ConnectionListener.OnStateChanged,
// re-exported as basic constants so gomobile emits them into the generated
// Java bindings. They mirror peer.ClientState*: append-only, never reorder.
const (
ClientStateDisconnected = int(peer.ClientStateDisconnected)
ClientStateConnected = int(peer.ClientStateConnected)
ClientStateConnecting = int(peer.ClientStateConnecting)
ClientStateDisconnecting = int(peer.ClientStateDisconnecting)
ClientStateNoNetwork = int(peer.ClientStateNoNetwork)
)
// ConnectionListener export internal Listener for mobile. It mirrors
// peer.Listener with OnStateChanged taking a plain int (one of the
// ClientState* constants), because gomobile cannot bind named types.
type ConnectionListener interface {
OnStateChanged(state int)
OnConnected()
OnDisconnected()
OnConnecting()
OnDisconnecting()
OnAddressChanged(string, string)
OnPeersListChanged(int)
}
// connectionListenerAdapter adapts the gomobile-facing ConnectionListener to
// peer.Listener, converting the typed state to the int the binding carries.
type connectionListenerAdapter struct {
ConnectionListener
}
func (a connectionListenerAdapter) OnStateChanged(state peer.ClientState) {
a.ConnectionListener.OnStateChanged(int(state))
}

View File

@@ -16,28 +16,47 @@ import (
"google.golang.org/grpc"
nbnet "github.com/netbirdio/netbird/client/net"
"github.com/netbirdio/netbird/client/netsweep"
)
func WithCustomDialer(_ bool, _ string) grpc.DialOption {
return grpc.WithContextDialer(dialContext)
}
// WithSweeper dials like WithCustomDialer but registers connections and
// dials with the sweeper. Append it after WithCustomDialer: gRPC applies
// dial options in order, so the later context dialer wins.
func WithSweeper(sweeper *netsweep.Sweeper) grpc.DialOption {
return grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
if runtime.GOOS == "linux" {
currentUser, err := user.Current()
if err != nil {
return nil, status.Errorf(codes.FailedPrecondition, "failed to get current user: %v", err)
}
dial := sweeper.StartDial(ctx)
defer dial.Release()
// the custom dialer requires root permissions which are not required for use cases run as non-root
if currentUser.Uid != "0" {
log.Debug("Not running as root, using standard dialer")
dialer := &net.Dialer{}
return dialer.DialContext(ctx, "tcp", addr)
}
}
conn, err := nbnet.NewDialer().DialContext(ctx, "tcp", addr)
conn, err := dialContext(dial.Ctx(), addr)
if err != nil {
return nil, fmt.Errorf("nbnet.NewDialer().DialContext: %w", err)
return nil, err
}
return conn, nil
return dial.WrapConn(conn)
})
}
func dialContext(ctx context.Context, addr string) (net.Conn, error) {
if runtime.GOOS == "linux" {
currentUser, err := user.Current()
if err != nil {
return nil, status.Errorf(codes.FailedPrecondition, "failed to get current user: %v", err)
}
// the custom dialer requires root permissions which are not required for use cases run as non-root
if currentUser.Uid != "0" {
log.Debug("Not running as root, using standard dialer")
dialer := &net.Dialer{}
return dialer.DialContext(ctx, "tcp", addr)
}
}
conn, err := nbnet.NewDialer().DialContext(ctx, "tcp", addr)
if err != nil {
return nil, fmt.Errorf("nbnet.NewDialer().DialContext: %w", err)
}
return conn, nil
}

View File

@@ -3,6 +3,7 @@ package grpc
import (
"google.golang.org/grpc"
"github.com/netbirdio/netbird/client/netsweep"
"github.com/netbirdio/netbird/util/wsproxy/client"
)
@@ -11,3 +12,8 @@ import (
func WithCustomDialer(tlsEnabled bool, component string) grpc.DialOption {
return client.WithWebSocketDialer(tlsEnabled, component)
}
// WithSweeper is a no-op on WASM/JS: there is no network change signal.
func WithSweeper(_ *netsweep.Sweeper) grpc.DialOption {
return grpc.EmptyDialOption{}
}

49
client/grpc/retry.go Normal file
View File

@@ -0,0 +1,49 @@
package grpc
import (
"context"
"errors"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/netbirdio/netbird/client/netstate"
)
// Retry mirrors backoff.Retry, but the sleep between attempts also wakes on
// OS network availability transitions: an operation cut down by a network
// change retries the moment the network settles instead of sleeping through
// the recovery. A nil netState never fires, leaving plain backoff.Retry
// behavior.
func Retry(ctx context.Context, operation backoff.Operation, bo backoff.BackOff, netState *netstate.State) error {
bo.Reset()
for {
err := operation()
if err == nil {
return nil
}
var permanent *backoff.PermanentError
if errors.As(err, &permanent) {
return permanent.Err
}
next := bo.NextBackOff()
if next == backoff.Stop {
if cerr := ctx.Err(); cerr != nil {
return cerr
}
return err
}
timer := time.NewTimer(next)
select {
case <-timer.C:
case <-netState.Changed():
timer.Stop()
case <-ctx.Done():
timer.Stop()
return ctx.Err()
}
}
}

91
client/grpc/retry_test.go Normal file
View File

@@ -0,0 +1,91 @@
package grpc
import (
"context"
"errors"
"testing"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/netbirdio/netbird/client/netstate"
)
func TestRetryWakesOnNetworkChange(t *testing.T) {
ns := netstate.New()
attempts := 0
operation := func() error {
attempts++
if attempts == 1 {
return errors.New("cut by network change")
}
return nil
}
go func() {
time.Sleep(20 * time.Millisecond)
ns.Set(false)
}()
start := time.Now()
err := Retry(context.Background(), operation, backoff.NewConstantBackOff(time.Minute), ns)
require.NoError(t, err)
assert.Equal(t, 2, attempts)
assert.Less(t, time.Since(start), time.Second, "the transition must cut the minute-long sleep short")
}
func TestRetryPermanentError(t *testing.T) {
sentinel := errors.New("permission denied")
operation := func() error {
return backoff.Permanent(sentinel)
}
err := Retry(context.Background(), operation, backoff.NewConstantBackOff(time.Millisecond), nil)
assert.ErrorIs(t, err, sentinel)
}
func TestRetryNilNetState(t *testing.T) {
attempts := 0
operation := func() error {
attempts++
if attempts < 3 {
return errors.New("transient")
}
return nil
}
err := Retry(context.Background(), operation, backoff.NewConstantBackOff(time.Millisecond), nil)
require.NoError(t, err)
assert.Equal(t, 3, attempts)
}
func TestRetryStops(t *testing.T) {
failure := errors.New("still failing")
operation := func() error {
return failure
}
err := Retry(context.Background(), operation, &backoff.StopBackOff{}, nil)
assert.ErrorIs(t, err, failure)
}
func TestRetryCtxCancelDuringSleep(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
operation := func() error {
return errors.New("failing")
}
go func() {
time.Sleep(20 * time.Millisecond)
cancel()
}()
start := time.Now()
err := Retry(ctx, operation, backoff.NewConstantBackOff(time.Minute), netstate.New())
assert.ErrorIs(t, err, context.Canceled)
assert.Less(t, time.Since(start), time.Second)
}

View File

@@ -38,6 +38,8 @@ import (
"github.com/netbirdio/netbird/client/internal/updater"
"github.com/netbirdio/netbird/client/internal/updater/installer"
nbnet "github.com/netbirdio/netbird/client/net"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
cProto "github.com/netbirdio/netbird/client/proto"
"github.com/netbirdio/netbird/client/ssh"
sshconfig "github.com/netbirdio/netbird/client/ssh/config"
@@ -70,18 +72,42 @@ type ConnectClient struct {
updateManager *updater.Manager
persistSyncResponse bool
// netState gates every reconnection loop on OS-reported network
// availability. Nil (the default) disables gating; mobile platforms
// inject it via WithNetworkState.
netState *netstate.State
// sweeper cuts the management, signal and relay connections on network
// change; nil disables it.
sweeper *netsweep.Sweeper
}
// ConnectClientOption configures optional ConnectClient behavior.
type ConnectClientOption func(*ConnectClient)
// WithNetworkState injects the OS network availability state that gates every
// reconnection loop; without it gating is disabled.
func WithNetworkState(netState *netstate.State) ConnectClientOption {
return func(c *ConnectClient) { c.netState = netState }
}
// WithSweeper injects the network change sweeper.
func WithSweeper(sweeper *netsweep.Sweeper) ConnectClientOption {
return func(c *ConnectClient) { c.sweeper = sweeper }
}
func NewConnectClient(
ctx context.Context,
config *profilemanager.Config,
statusRecorder *peer.Status,
opts ...ConnectClientOption,
) *ConnectClient {
// Derive the run context here so Stop owns the cancel that unblocks the run
// loop. runCancel is set once at construction, so Stop can call it without
// racing the run loop's startup. Callers therefore need not cancel before Stop.
runCtx, runCancel := context.WithCancel(ctx)
return &ConnectClient{
c := &ConnectClient{
ctx: runCtx,
runCancel: runCancel,
runExited: make(chan struct{}),
@@ -89,6 +115,10 @@ func NewConnectClient(
statusRecorder: statusRecorder,
engineMutex: sync.Mutex{},
}
for _, opt := range opts {
opt(c)
}
return c
}
func (c *ConnectClient) SetUpdateManager(um *updater.Manager) {
@@ -274,6 +304,13 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
return nil
}
// suspend connection attempts while the OS reports no usable network
if waited, err := c.netState.Wait(c.ctx); err != nil {
return nil
} else if waited {
backOff.Reset()
}
state.Set(StatusConnecting)
engineCtx, cancel := context.WithCancel(c.ctx)
@@ -285,7 +322,8 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
}()
log.Debugf("connecting to the Management service %s", c.config.ManagementURL.Host)
mgmClient, err := mgm.NewClient(engineCtx, c.config.ManagementURL.Host, myPrivateKey, mgmTlsEnabled)
mgmClient, err := mgm.NewClient(engineCtx, c.config.ManagementURL.Host, myPrivateKey, mgmTlsEnabled,
mgm.WithNetworkState(c.netState), mgm.WithSweeper(c.sweeper))
if err != nil {
// On daemon shutdown / Down() the parent context is cancelled
// and the dial fails with "context canceled". Wrapping that
@@ -360,7 +398,7 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
}()
// with the global Netbird config in hand connect (just a connection, no stream yet) Signal
signalClient, err := connectToSignal(engineCtx, loginResp.GetNetbirdConfig(), myPrivateKey)
signalClient, err := connectToSignal(engineCtx, loginResp.GetNetbirdConfig(), myPrivateKey, c.netState, c.sweeper)
if err != nil {
log.Error(err)
return wrapErr(err)
@@ -396,7 +434,8 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
engineConfig.StateDir = filepath.Dir(path)
}
relayManager := relayClient.NewManager(engineCtx, relayURLs, myPrivateKey.PublicKey().String(), engineConfig.MTU)
relayManager := relayClient.NewManager(engineCtx, relayURLs, myPrivateKey.PublicKey().String(), engineConfig.MTU,
relayClient.WithNetworkState(c.netState), relayClient.WithSweeper(c.sweeper))
c.statusRecorder.SetRelayMgr(relayManager)
if len(relayURLs) > 0 {
if token != nil {
@@ -424,6 +463,7 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
UpdateManager: c.updateManager,
ClientMetrics: c.clientMetrics,
MetricsCtx: c.ctx,
NetState: c.netState,
}, mobileDependency)
engine.SetSyncResponsePersistence(c.persistSyncResponse)
c.engine = engine
@@ -480,6 +520,16 @@ func (c *ConnectClient) run(mobileDependency MobileDependency, runningChan chan
// status stream stuck at Connecting.
err = backoff.Retry(operation, backoff.WithContext(backOff, c.ctx))
if err != nil {
// Once the client context is cancelled backoff.WithContext surfaces the
// bare context error, and any attempt torn down mid-flight reports the
// same. That cancellation is the caller asking us to stop (Stop, Down or
// an engine restart), so exit cleanly instead of handing back a failure
// the caller would have to distinguish from a real one.
if c.ctx.Err() != nil && errors.Is(err, context.Canceled) {
log.Info("exiting client retry loop, context cancelled")
return nil
}
log.Debugf("exiting client retry loop due to unrecoverable error: %s", err)
if s, ok := gstatus.FromError(err); ok && (s.Code() == codes.PermissionDenied) {
state.Set(StatusNeedsLogin)
@@ -673,7 +723,7 @@ func selectMTU(localMTU uint16, peerMTU int32) uint16 {
}
// connectToSignal creates Signal Service client and established a connection
func connectToSignal(ctx context.Context, wtConfig *mgmProto.NetbirdConfig, ourPrivateKey wgtypes.Key) (*signal.GrpcClient, error) {
func connectToSignal(ctx context.Context, wtConfig *mgmProto.NetbirdConfig, ourPrivateKey wgtypes.Key, netState *netstate.State, sweeper *netsweep.Sweeper) (*signal.GrpcClient, error) {
var sigTLSEnabled bool
if wtConfig.Signal.Protocol == mgmProto.HostConfig_HTTPS {
sigTLSEnabled = true
@@ -681,7 +731,8 @@ func connectToSignal(ctx context.Context, wtConfig *mgmProto.NetbirdConfig, ourP
sigTLSEnabled = false
}
signalClient, err := signal.NewClient(ctx, wtConfig.Signal.Uri, ourPrivateKey, sigTLSEnabled)
signalClient, err := signal.NewClient(ctx, wtConfig.Signal.Uri, ourPrivateKey, sigTLSEnabled,
signal.WithNetworkState(netState), signal.WithSweeper(sweeper))
if err != nil {
log.Errorf("error while connecting to the Signal Exchange Service %s: %s", wtConfig.Signal.Uri, err)
return nil, gstatus.Errorf(codes.FailedPrecondition, "failed connecting to Signal Service : %s", err)

View File

@@ -35,6 +35,8 @@ var (
// exported so a diagnostic reader reports the same locations that are written.
const (
// NRPTKeyPrefix starts the name of every NRPT rule key this client creates.
// Older versions used different layouts under the same prefix: a single
// unsuffixed key, then one key per domain, now one key per batch of domains.
NRPTKeyPrefix = "NetBird-Match"
// DNSPolicyConfigRoot holds the NRPT rules of the local policy store.
@@ -89,7 +91,6 @@ type registryConfigurator struct {
guid string
routingAll bool
gpo bool
nrptEntryCount int
origNameservers []netip.Addr
}
@@ -322,14 +323,9 @@ func (r *registryConfigurator) applyDNSConfig(config HostDNSConfig, stateManager
}
if len(matchDomains) != 0 {
count, err := r.addDNSMatchPolicy(matchDomains, config.ServerIP)
// Update count even on error to ensure cleanup covers partially created rules
r.nrptEntryCount = count
if err != nil {
if err := r.addDNSMatchPolicy(matchDomains, config.ServerIP); err != nil {
return fmt.Errorf("add dns match policy: %w", err)
}
} else {
r.nrptEntryCount = 0
}
r.updateState(stateManager)
@@ -345,9 +341,8 @@ func (r *registryConfigurator) applyDNSConfig(config HostDNSConfig, stateManager
func (r *registryConfigurator) updateState(stateManager *statemanager.Manager) {
if err := stateManager.UpdateState(&ShutdownState{
Guid: r.guid,
GPO: r.gpo,
NRPTEntryCount: r.nrptEntryCount,
Guid: r.guid,
GPO: r.gpo,
}); err != nil {
log.Errorf("failed to update shutdown state: %s", err)
}
@@ -362,7 +357,7 @@ func (r *registryConfigurator) addDNSSetupForAll(ip netip.Addr) error {
return nil
}
func (r *registryConfigurator) addDNSMatchPolicy(domains []string, ip netip.Addr) (int, error) {
func (r *registryConfigurator) addDNSMatchPolicy(domains []string, ip netip.Addr) error {
// if the gpo key is present, we need to put our DNS settings there, otherwise our config might be ignored
// see https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-gpnrpt/8cc31cb9-20cb-4140-9e85-3e08703b4745
@@ -379,19 +374,17 @@ func (r *registryConfigurator) addDNSMatchPolicy(domains []string, ip netip.Addr
gpoPath := fmt.Sprintf("%s-%d", gpoDnsPolicyConfigMatchPath, ruleIndex)
if err := r.configureDNSPolicy(localPath, batchDomains, ip); err != nil {
return ruleIndex, fmt.Errorf("configure DNS Local policy for rule %d: %w", ruleIndex, err)
return fmt.Errorf("configure DNS Local policy for rule %d: %w", ruleIndex, err)
}
// Increment immediately so the caller's cleanup path knows about this rule
ruleIndex++
if r.gpo {
if err := r.configureDNSPolicy(gpoPath, batchDomains, ip); err != nil {
return ruleIndex, fmt.Errorf("configure gpo DNS policy for rule %d: %w", ruleIndex-1, err)
return fmt.Errorf("configure gpo DNS policy for rule %d: %w", ruleIndex, err)
}
}
log.Debugf("added NRPT rule %d with %d domains", ruleIndex-1, len(batchDomains))
log.Debugf("added NRPT rule %d with %d domains", ruleIndex, len(batchDomains))
ruleIndex++
}
if r.gpo {
@@ -401,7 +394,7 @@ func (r *registryConfigurator) addDNSMatchPolicy(domains []string, ip netip.Addr
}
log.Infof("added %d NRPT rules for %d domains", ruleIndex, len(domains))
return ruleIndex, nil
return nil
}
func (r *registryConfigurator) configureDNSPolicy(policyPath string, domains []string, ip netip.Addr) error {
@@ -534,28 +527,28 @@ func (r *registryConfigurator) restoreHostDNS() error {
return nil
}
// removeDNSMatchPolicies deletes every NRPT rule this client may have created,
// from the local and the GPO policy store. The rules are found by enumerating
// the registry, the only authoritative record of what was written. Cleanup must
// not depend on a rule count: the in-memory one is scoped to a single
// registryConfigurator and the persisted one is deleted on every clean
// disconnect, and a rule left behind keeps resolving names over an interface
// that is gone, until reboot discards the volatile key.
func (r *registryConfigurator) removeDNSMatchPolicies() error {
var merr *multierror.Error
// Try to remove the base entries (for backward compatibility)
if err := removeRegistryKeyFromDNSPolicyConfig(dnsPolicyConfigMatchPath); err != nil {
merr = multierror.Append(merr, fmt.Errorf("remove local base entry: %w", err))
}
if err := removeRegistryKeyFromDNSPolicyConfig(gpoDnsPolicyConfigMatchPath); err != nil {
merr = multierror.Append(merr, fmt.Errorf("remove GPO base entry: %w", err))
}
for i := 0; i < r.nrptEntryCount; i++ {
localPath := fmt.Sprintf("%s-%d", dnsPolicyConfigMatchPath, i)
gpoPath := fmt.Sprintf("%s-%d", gpoDnsPolicyConfigMatchPath, i)
if err := removeRegistryKeyFromDNSPolicyConfig(localPath); err != nil {
merr = multierror.Append(merr, fmt.Errorf("remove local entry %d: %w", i, err))
for _, root := range []string{DNSPolicyConfigRoot, GPODNSPolicyConfigRoot} {
names, err := listNRPTRuleKeys(root)
if err != nil {
merr = multierror.Append(merr, fmt.Errorf("list rule keys under %s: %w", root, err))
continue
}
if err := removeRegistryKeyFromDNSPolicyConfig(gpoPath); err != nil {
merr = multierror.Append(merr, fmt.Errorf("remove GPO entry %d: %w", i, err))
for _, name := range names {
path := root + `\` + name
if err := removeRegistryKeyFromDNSPolicyConfig(path); err != nil {
merr = multierror.Append(merr, fmt.Errorf("remove entry %s: %w", path, err))
}
}
}
@@ -570,6 +563,39 @@ func (r *registryConfigurator) restoreUncleanShutdownDNS() error {
return r.restoreHostDNS()
}
// listNRPTRuleKeys returns the names of our NRPT rule keys under a policy store
// root. An absent root holds nothing to clean up, which is the normal state of
// the GPO store on a machine without DNS Client policy.
func listNRPTRuleKeys(root string) ([]string, error) {
k, err := registry.OpenKey(registry.LOCAL_MACHINE, root, registry.ENUMERATE_SUB_KEYS)
switch {
case errors.Is(err, registry.ErrNotExist), errors.Is(err, syscall.ERROR_PATH_NOT_FOUND):
// the GPO store is absent on a machine without DNS client policy
log.Debugf("HKEY_LOCAL_MACHINE\\%s does not exist", root)
return nil, nil
case err != nil:
// any other failure has to reach the caller: reporting no rules would
// report a successful cleanup while leaving the rules in place
return nil, fmt.Errorf("open HKEY_LOCAL_MACHINE\\%s: %w", root, err)
}
defer closer(k)
names, err := k.ReadSubKeyNames(-1)
if err != nil {
return nil, fmt.Errorf("read subkey names: %w", err)
}
var ruleKeys []string
for _, name := range names {
// registry key names are case insensitive
if strings.HasPrefix(strings.ToLower(name), strings.ToLower(NRPTKeyPrefix)) {
ruleKeys = append(ruleKeys, name)
}
}
return ruleKeys, nil
}
func removeRegistryKeyFromDNSPolicyConfig(regKeyPath string) error {
k, err := registry.OpenKey(registry.LOCAL_MACHINE, regKeyPath, registry.QUERY_VALUE)
if err != nil {

View File

@@ -25,7 +25,7 @@ func TestNRPTEntriesCleanupOnConfigChange(t *testing.T) {
// Create a test interface registry key so updateSearchDomains doesn't fail
testGUID := "{12345678-1234-1234-1234-123456789ABC}"
interfacePath := `SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\` + testGUID
interfacePath := InterfaceConfigPath + `\` + testGUID
testKey, _, err := registry.CreateKey(registry.LOCAL_MACHINE, interfacePath, registry.SET_VALUE)
require.NoError(t, err, "Should create test interface registry key")
testKey.Close()
@@ -56,7 +56,7 @@ func TestNRPTEntriesCleanupOnConfigChange(t *testing.T) {
require.NoError(t, err)
// Verify 3 NRPT rules exist
assert.Equal(t, 3, cfg.nrptEntryCount, "Should create 3 NRPT rules for 125 domains")
assert.Equal(t, 3, countNRPTRuleKeys(t), "Should create 3 NRPT rules for 125 domains")
for i := 0; i < 3; i++ {
exists, err := registryKeyExists(fmt.Sprintf("%s-%d", dnsPolicyConfigMatchPath, i))
require.NoError(t, err)
@@ -81,7 +81,7 @@ func TestNRPTEntriesCleanupOnConfigChange(t *testing.T) {
require.NoError(t, err)
// Verify first 2 NRPT rules exist
assert.Equal(t, 2, cfg.nrptEntryCount, "Should create 2 NRPT rules for 75 domains")
assert.Equal(t, 2, countNRPTRuleKeys(t), "Should create 2 NRPT rules for 75 domains")
for i := 0; i < 2; i++ {
exists, err := registryKeyExists(fmt.Sprintf("%s-%d", dnsPolicyConfigMatchPath, i))
require.NoError(t, err)
@@ -106,9 +106,65 @@ func registryKeyExists(path string) (bool, error) {
return true, nil
}
// TestNRPTCleanupWithoutRuleCount verifies that rules written by a previous run
// are removed by a configurator that has no record of how many there are: an
// unclean exit loses the in-memory count and a clean disconnect deletes the
// persisted one, so cleanup cannot depend on either.
func TestNRPTCleanupWithoutRuleCount(t *testing.T) {
if testing.Short() {
t.Skip("skipping registry integration test in short mode")
}
defer cleanupRegistryKeys(t)
cleanupRegistryKeys(t)
testIP := netip.MustParseAddr("100.64.0.1")
// 75 domains produce two indexed rules, as the current layout does
domains := make([]string, 75)
for i := range domains {
domains[i] = fmt.Sprintf(".domain%d.com", i+1)
}
previousRun := &registryConfigurator{}
require.NoError(t, previousRun.addDNSMatchPolicy(domains, testIP))
// the unsuffixed key an older version would have written
require.NoError(t, previousRun.configureDNSPolicy(dnsPolicyConfigMatchPath, []string{".legacy.example.com"}, testIP))
// a policy owned by someone else, which cleanup must not touch
foreignPath := DNSPolicyConfigRoot + `\DnsPolicyConfigTestForeign`
foreignKey, _, err := registry.CreateKey(registry.LOCAL_MACHINE, foreignPath, registry.SET_VALUE)
require.NoError(t, err, "Should create foreign policy key")
foreignKey.Close()
defer func() {
_ = registry.DeleteKey(registry.LOCAL_MACHINE, foreignPath)
}()
require.Equal(t, 3, countNRPTRuleKeys(t), "Should have two indexed rules and the legacy one")
// a configurator that never applied a DNS config, as one built after a
// restart or from a shutdown state without a count is
freshRun := &registryConfigurator{}
require.NoError(t, freshRun.removeDNSMatchPolicies())
assert.Equal(t, 0, countNRPTRuleKeys(t), "Should remove every rule left by the previous run")
exists, err := registryKeyExists(foreignPath)
require.NoError(t, err)
assert.True(t, exists, "Should not remove a policy that is not ours")
}
func countNRPTRuleKeys(t *testing.T) int {
t.Helper()
names, err := listNRPTRuleKeys(DNSPolicyConfigRoot)
require.NoError(t, err, "Should list NRPT rule keys")
return len(names)
}
func cleanupRegistryKeys(*testing.T) {
// Clean up more entries to account for batching tests with many domains
cfg := &registryConfigurator{nrptEntryCount: 20}
cfg := &registryConfigurator{}
_ = cfg.removeDNSMatchPolicies()
}
@@ -125,7 +181,7 @@ func TestNRPTDomainBatching(t *testing.T) {
// Create a test interface registry key so updateSearchDomains doesn't fail
testGUID := "{12345678-1234-1234-1234-123456789ABC}"
interfacePath := `SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\` + testGUID
interfacePath := InterfaceConfigPath + `\` + testGUID
testKey, _, err := registry.CreateKey(registry.LOCAL_MACHINE, interfacePath, registry.SET_VALUE)
require.NoError(t, err, "Should create test interface registry key")
testKey.Close()
@@ -193,7 +249,7 @@ func TestNRPTDomainBatching(t *testing.T) {
require.NoError(t, err)
// Verify that exactly expectedRuleCount rules were created
assert.Equal(t, tc.expectedRuleCount, cfg.nrptEntryCount,
assert.Equal(t, tc.expectedRuleCount, countNRPTRuleKeys(t),
"Should create %d NRPT rules for %d domains", tc.expectedRuleCount, tc.domainCount)
// Verify all expected rules exist

View File

@@ -5,9 +5,8 @@ import (
)
type ShutdownState struct {
Guid string
GPO bool
NRPTEntryCount int
Guid string
GPO bool
}
func (s *ShutdownState) Name() string {
@@ -16,9 +15,8 @@ func (s *ShutdownState) Name() string {
func (s *ShutdownState) Cleanup() error {
manager := &registryConfigurator{
guid: s.Guid,
gpo: s.GPO,
nrptEntryCount: s.NRPTEntryCount,
guid: s.Guid,
gpo: s.GPO,
}
if err := manager.restoreUncleanShutdownDNS(); err != nil {

View File

@@ -2,17 +2,21 @@ package ebpf
import (
_ "embed"
"fmt"
"net"
"sync"
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/rlimit"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
"github.com/netbirdio/netbird/client/internal/ebpf/manager"
)
const (
xdpProgName = "nb_xdp_prog"
mapKeyFeatures uint32 = 0
featureFlagWGProxy = 0b00000001
@@ -68,21 +72,50 @@ func (tf *GeneralManager) loadXdp() error {
return err
}
// load pre-compiled programs into the kernel.
err = loadBpfObjects(&tf.bpfObjs, nil)
// lo has no native XDP, so the program runs in generic mode. Unless it
// declares multi-buffer support the kernel must linearize every non-linear
// skb before running it. Loopback packets are up to 64 KB, so that is a
// contiguous GFP_ATOMIC allocation per packet, and when it fails the packet
// is dropped before the program runs, stalling local TCP connections.
// Multi-buffer XDP in generic mode requires kernel 6.3, so fall back to a
// plain attach when the kernel rejects it.
err = tf.attachXdp(iFace.Index, true)
if err == nil {
return nil
}
log.Debugf("failed to attach multi-buffer xdp program, retrying without it: %s", err)
return tf.attachXdp(iFace.Index, false)
}
func (tf *GeneralManager) attachXdp(iFaceIndex int, multiBuffer bool) error {
spec, err := loadBpf()
if err != nil {
return err
return fmt.Errorf("load bpf spec: %w", err)
}
if multiBuffer {
prog, ok := spec.Programs[xdpProgName]
if !ok {
return fmt.Errorf("program %s not found in bpf spec", xdpProgName)
}
prog.Flags |= unix.BPF_F_XDP_HAS_FRAGS
}
if err := spec.LoadAndAssign(&tf.bpfObjs, nil); err != nil {
return fmt.Errorf("load bpf objects: %w", err)
}
tf.link, err = link.AttachXDP(link.XDPOptions{
Program: tf.bpfObjs.NbXdpProg,
Interface: iFace.Index,
Interface: iFaceIndex,
})
if err != nil {
_ = tf.bpfObjs.Close()
if closeErr := tf.bpfObjs.Close(); closeErr != nil {
log.Debugf("failed to close bpf objects after xdp attach error: %s", closeErr)
}
tf.link = nil
return err
return fmt.Errorf("attach xdp: %w", err)
}
return nil
}

View File

@@ -59,6 +59,7 @@ import (
"github.com/netbirdio/netbird/client/internal/syncstore"
"github.com/netbirdio/netbird/client/internal/updater"
"github.com/netbirdio/netbird/client/jobexec"
"github.com/netbirdio/netbird/client/netstate"
cProto "github.com/netbirdio/netbird/client/proto"
"github.com/netbirdio/netbird/client/system"
nbdns "github.com/netbirdio/netbird/dns"
@@ -181,6 +182,9 @@ type EngineServices struct {
UpdateManager *updater.Manager
ClientMetrics *metrics.ClientMetrics
MetricsCtx context.Context
// NetState gates the reconnection loops on OS-reported network
// availability; nil disables gating.
NetState *netstate.State
}
// Engine is a mechanism responsible for reacting on Signal and Management stream events and managing connections to the remote peers.
@@ -204,6 +208,10 @@ type Engine struct {
config *EngineConfig
mobileDep MobileDependency
// netState gates the peer reconnection guards on OS-reported network
// availability; nil disables gating.
netState *netstate.State
// STUNs is a list of STUN servers used by ICE
STUNs []*stun.URI
// TURNs is a list of STUN servers used by ICE
@@ -337,6 +345,7 @@ func NewEngine(
syncMsgMux: &sync.Mutex{},
config: config,
mobileDep: mobileDep,
netState: services.NetState,
STUNs: []*stun.URI{},
TURNs: []*stun.URI{},
networkSerial: 0,
@@ -1893,7 +1902,8 @@ func (e *Engine) createPeerConn(pubKey string, allowedIPs []netip.Prefix, agentV
Addr: e.getRosenpassAddr(),
PermissiveMode: e.config.RosenpassPermissive,
},
ICEConfig: e.createICEConfig(),
ICEConfig: e.createICEConfig(),
NetworkState: e.netState,
}
serviceDependencies := peer.ServiceDependencies{

View File

@@ -26,6 +26,7 @@ import (
"github.com/netbirdio/netbird/client/internal/portforward"
"github.com/netbirdio/netbird/client/internal/rosenpass"
"github.com/netbirdio/netbird/client/internal/stdnet"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/route"
relayClient "github.com/netbirdio/netbird/shared/relay/client"
)
@@ -93,6 +94,10 @@ type ConnConfig struct {
// ICEConfig ICE protocol configuration
ICEConfig icemaker.Config
// NetworkState gates the reconnection guard on OS-reported network
// availability; nil disables gating.
NetworkState *netstate.State
}
type Conn struct {
@@ -254,7 +259,7 @@ func (conn *Conn) open(engineCtx context.Context, firstPacket []byte) error {
conn.handshaker.AddICEListener(conn.workerICE.OnNewOffer)
}
conn.guard = guard.NewGuard(conn.Log, conn.isConnectedOnAllWay, conn.config.Timeout, conn.srWatcher)
conn.guard = guard.NewGuard(conn.Log, conn.isConnectedOnAllWay, conn.config.Timeout, conn.srWatcher, conn.config.NetworkState)
conn.wg.Add(1)
go func() {

View File

@@ -6,6 +6,8 @@ import (
"github.com/cenkalti/backoff/v4"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netstate"
)
// ConnStatus represents the connection state as seen by the guard.
@@ -31,20 +33,26 @@ type connStatusFunc func() ConnStatus
// - Relayed connection disconnected
// - ICE candidate changes
type Guard struct {
log *log.Entry
isConnectedOnAllWay connStatusFunc
timeout time.Duration
srWatcher *SRWatcher
log *log.Entry
isConnectedOnAllWay connStatusFunc
timeout time.Duration
srWatcher *SRWatcher
// netState gates reconnect attempts on OS-reported network availability;
// nil disables gating.
netState *netstate.State
relayedConnDisconnected chan struct{}
iCEConnDisconnected chan struct{}
}
func NewGuard(log *log.Entry, isConnectedFn connStatusFunc, timeout time.Duration, srWatcher *SRWatcher) *Guard {
// NewGuard creates a reconnection guard for a peer connection. A nil netState
// disables network availability gating.
func NewGuard(log *log.Entry, isConnectedFn connStatusFunc, timeout time.Duration, srWatcher *SRWatcher, netState *netstate.State) *Guard {
return &Guard{
log: log,
isConnectedOnAllWay: isConnectedFn,
timeout: timeout,
srWatcher: srWatcher,
netState: netState,
relayedConnDisconnected: make(chan struct{}, 1),
iCEConnDisconnected: make(chan struct{}, 1),
}
@@ -96,9 +104,16 @@ func (g *Guard) reconnectLoopWithRetry(ctx context.Context, callback func()) {
iceState := &iceRetryState{log: g.log}
defer iceState.reset()
netChanged := g.netState.Changed()
for {
select {
case <-tickerChannel:
// skip attempts while the OS reports no usable network; the
// netChanged case below resumes the loop once it returns
if !g.netState.IsOnline() {
continue
}
switch g.isConnectedOnAllWay() {
case ConnStatusConnected:
// all good, nothing to do
@@ -135,6 +150,23 @@ func (g *Guard) reconnectLoopWithRetry(ctx context.Context, callback func()) {
tickerChannel = ticker.C
iceState.reset()
case <-netChanged:
// Re-arm for the next transition before acting on this one.
netChanged = g.netState.Changed()
if !g.netState.IsOnline() {
continue
}
// Ticks skipped while offline drove the backoff towards its
// maximum without ever attempting, and left the ICE budget
// frozen — possibly in hourly mode. Recover on our own so the
// peer does not depend on a signal or relay event that never
// comes when both stayed up across the outage.
g.log.Debugf("network is back, reset reconnection ticker")
ticker.Stop()
ticker = g.newReconnectTicker(ctx)
tickerChannel = ticker.C
iceState.reset()
case <-ctx.Done():
g.log.Debugf("context is done, stop reconnect loop")
return

View File

@@ -15,7 +15,7 @@ import (
func newTestGuard(status connStatusFunc) *Guard {
srw := NewSRWatcher(nil, nil, nil, ice.Config{})
return NewGuard(log.WithField("test", "guard"), status, 50*time.Millisecond, srw)
return NewGuard(log.WithField("test", "guard"), status, 50*time.Millisecond, srw, nil)
}
// countBackoffTickerGoroutines returns how many goroutines are currently sitting

View File

@@ -0,0 +1,107 @@
package guard
import (
"context"
"sync/atomic"
"testing"
"time"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/internal/peer/ice"
"github.com/netbirdio/netbird/client/netstate"
)
// newTestGuardWithNetState builds a guard with a realistic MaxInterval: the
// backoff must be able to grow well past the outage, as it does in production
// where the timeout is seconds to minutes.
func newTestGuardWithNetState(status connStatusFunc, netState *netstate.State) *Guard {
srw := NewSRWatcher(nil, nil, nil, ice.Config{})
return NewGuard(log.WithField("test", "guard"), status, 30*time.Second, srw, netState)
}
// TestGuard_RecoversAfterOfflineToOnline covers a peer that stays disconnected
// across a network outage while neither signal nor relay reports an event —
// both stayed up, as on a short airplane mode toggle over Wi-Fi.
//
// Every tick taken while offline is skipped, but it still advances the
// exponential backoff, so by the time the network returns the next tick can be
// tens of seconds away. Without an explicit reaction to the transition the
// peer waits out that interval for a recovery that could start immediately.
func TestGuard_RecoversAfterOfflineToOnline(t *testing.T) {
netState := netstate.New()
var attempts atomic.Int32
g := newTestGuardWithNetState(func() ConnStatus { return ConnStatusDisconnected }, netState)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start from the reconnect ticker (800ms initial interval), the state a
// peer is in after it loses its connection.
go g.Start(ctx, func() { attempts.Add(1) })
g.SetRelayedConnDisconnected()
// Let the backoff climb: 0.8s, 1.6s, 3.2s, 6.4s ... every tick is skipped
// while offline, but each one doubles the wait for the next.
netState.Set(false)
time.Sleep(8 * time.Second)
offlineAttempts := attempts.Load()
if offlineAttempts != 0 {
t.Fatalf("callback ran %d times while offline, want 0", offlineAttempts)
}
netState.Set(true)
// The next organic tick is now several seconds out, so anything within
// this window can only come from reacting to the transition itself.
pollCtx, stopPolling := context.WithTimeout(ctx, 2*time.Second)
defer stopPolling()
select {
case <-pollCtx.Done():
t.Fatal("peer was not retried within 2s of the network coming back, " +
"with neither a signal nor a relay event to fall back on")
case <-pollUntil(pollCtx, func() bool { return attempts.Load() > 0 }):
}
}
// TestGuard_OfflineTransitionDoesNotRetry checks the other direction: going
// offline must not itself trigger an attempt.
func TestGuard_OfflineTransitionDoesNotRetry(t *testing.T) {
netState := netstate.New()
var attempts atomic.Int32
g := newTestGuardWithNetState(func() ConnStatus { return ConnStatusDisconnected }, netState)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go g.Start(ctx, func() { attempts.Add(1) })
netState.Set(false)
time.Sleep(5 * time.Second)
if got := attempts.Load(); got != 0 {
t.Fatalf("callback ran %d times after going offline, want 0", got)
}
}
// pollUntil closes the returned channel once cond holds. It gives up when ctx
// is done, so the polling goroutine never outlives the test that started it.
func pollUntil(ctx context.Context, cond func() bool) <-chan struct{} {
done := make(chan struct{})
go func() {
for {
if cond() {
close(done)
return
}
select {
case <-ctx.Done():
return
case <-time.After(10 * time.Millisecond):
}
}
}()
return done
}

View File

@@ -1,11 +1,40 @@
package peer
// ClientState identifies the client connection state delivered via
// Listener.OnStateChanged.
type ClientState int
// Client states. The numeric values cross the gomobile boundary (the mobile
// bindings re-export them as integer constants), so they are a wire format:
// append new states at the end, never reorder or insert.
const (
ClientStateDisconnected ClientState = iota
ClientStateConnected
ClientStateConnecting
ClientStateDisconnecting
// ClientStateNoNetwork is an overlay state: it is never stored as the
// last notification, only derived from ClientStateConnecting while the
// OS reports no usable network (see notifier.effectiveState).
ClientStateNoNetwork
)
// Listener is a callback type about the NetBird network connection state
type Listener interface {
// OnStateChanged reports every client state transition. New states are
// delivered only through this callback; the per-state callbacks below
// are kept for compatibility and will be removed once all consumers
// have migrated.
OnStateChanged(state ClientState)
// Deprecated: consume OnStateChanged instead.
OnConnected()
// Deprecated: consume OnStateChanged instead.
OnDisconnected()
// Deprecated: consume OnStateChanged instead.
OnConnecting()
// Deprecated: consume OnStateChanged instead.
OnDisconnecting()
OnAddressChanged(string, string)
OnPeersListChanged(int)
}

View File

@@ -4,31 +4,64 @@ import (
"sync"
)
const (
stateDisconnected = iota
stateConnected
stateConnecting
stateDisconnecting
)
type notifier struct {
// publishLock orders state publication: it is held across computing the
// effective state and handing it to the listener, so a transition cannot
// overtake a newer one and leave the listener on a stale state.
publishLock sync.Mutex
serverStateLock sync.Mutex
listenersLock sync.Mutex
listener Listener
currentClientState bool
lastNotification int
lastNotification ClientState
lastNumberOfPeers int
lastFqdnAddress string
lastIPAddress string
networkAvailable bool
}
func newNotifier() *notifier {
return &notifier{}
return &notifier{
networkAvailable: true,
}
}
// effectiveState maps the computed state to what listeners should see:
// while the OS reports no usable network, "Connecting" would be a lie —
// connection attempts are suspended — so it is reported as NoNetwork.
// Caller must hold serverStateLock.
func (n *notifier) effectiveState(state ClientState) ClientState {
if !n.networkAvailable && state == ClientStateConnecting {
return ClientStateNoNetwork
}
return state
}
// setNetworkAvailable records the OS network availability and re-notifies
// the listener when the flag flips the effective state (Connecting <->
// NoNetwork).
func (n *notifier) setNetworkAvailable(available bool) {
n.publishLock.Lock()
defer n.publishLock.Unlock()
n.serverStateLock.Lock()
if n.networkAvailable == available {
n.serverStateLock.Unlock()
return
}
previous := n.effectiveState(n.lastNotification)
n.networkAvailable = available
current := n.effectiveState(n.lastNotification)
n.serverStateLock.Unlock()
if previous != current {
n.notify(current)
}
}
func (n *notifier) setListener(listener Listener) {
n.serverStateLock.Lock()
lastNotification := n.lastNotification
lastNotification := n.effectiveState(n.lastNotification)
numOfPeers := n.lastNumberOfPeers
fqdnAddress := n.lastFqdnAddress
address := n.lastIPAddress
@@ -52,6 +85,9 @@ func (n *notifier) removeListener() {
}
func (n *notifier) updateServerStates(mgmState bool, signalState bool) {
n.publishLock.Lock()
defer n.publishLock.Unlock()
n.serverStateLock.Lock()
calculatedState := n.calculateState(mgmState, signalState)
@@ -61,43 +97,54 @@ func (n *notifier) updateServerStates(mgmState bool, signalState bool) {
}
n.lastNotification = calculatedState
effective := n.effectiveState(calculatedState)
n.serverStateLock.Unlock()
n.notify(calculatedState)
n.notify(effective)
}
func (n *notifier) clientStart() {
n.publishLock.Lock()
defer n.publishLock.Unlock()
n.serverStateLock.Lock()
n.currentClientState = true
n.lastNotification = stateConnecting
n.lastNotification = ClientStateConnecting
effective := n.effectiveState(ClientStateConnecting)
n.serverStateLock.Unlock()
n.notify(stateConnecting)
n.notify(effective)
}
func (n *notifier) clientStop() {
n.publishLock.Lock()
defer n.publishLock.Unlock()
n.serverStateLock.Lock()
n.currentClientState = false
n.lastNotification = stateDisconnected
n.lastNotification = ClientStateDisconnected
n.serverStateLock.Unlock()
n.notify(stateDisconnected)
n.notify(ClientStateDisconnected)
}
func (n *notifier) clientTearDown() {
n.publishLock.Lock()
defer n.publishLock.Unlock()
n.serverStateLock.Lock()
n.currentClientState = false
n.lastNotification = stateDisconnecting
n.lastNotification = ClientStateDisconnecting
n.serverStateLock.Unlock()
n.notify(stateDisconnecting)
n.notify(ClientStateDisconnecting)
}
func (n *notifier) isServerStateChanged(newState int) bool {
func (n *notifier) isServerStateChanged(newState ClientState) bool {
return n.lastNotification != newState
}
func (n *notifier) notify(state int) {
func (n *notifier) notify(state ClientState) {
n.listenersLock.Lock()
listener := n.listener
n.listenersLock.Unlock()
@@ -109,20 +156,20 @@ func (n *notifier) notify(state int) {
notifyListener(listener, state)
}
func (n *notifier) calculateState(managementConn, signalConn bool) int {
func (n *notifier) calculateState(managementConn, signalConn bool) ClientState {
if managementConn && signalConn {
return stateConnected
return ClientStateConnected
}
if !managementConn && !signalConn && !n.currentClientState {
return stateDisconnected
return ClientStateDisconnected
}
if n.lastNotification == stateDisconnecting {
return stateDisconnecting
if n.lastNotification == ClientStateDisconnecting {
return ClientStateDisconnecting
}
return stateConnecting
return ClientStateConnecting
}
func (n *notifier) peerListChanged(numOfPeers int) {
@@ -159,15 +206,19 @@ func (n *notifier) localAddressChanged(fqdn, address string) {
listener.OnAddressChanged(fqdn, address)
}
func notifyListener(l Listener, state int) {
func notifyListener(l Listener, state ClientState) {
// legacy per-state callbacks; NoNetwork is delivered only via
// OnStateChanged below
switch state {
case stateDisconnected:
case ClientStateDisconnected:
l.OnDisconnected()
case stateConnected:
case ClientStateConnected:
l.OnConnected()
case stateConnecting:
case ClientStateConnecting:
l.OnConnecting()
case stateDisconnecting:
case ClientStateDisconnecting:
l.OnDisconnecting()
}
l.OnStateChanged(state)
}

View File

@@ -0,0 +1,108 @@
package peer
import (
"sync"
"testing"
"time"
)
type recordingListener struct {
mu sync.Mutex
states []ClientState
onState func(ClientState)
}
func (l *recordingListener) OnStateChanged(state ClientState) {
l.mu.Lock()
l.states = append(l.states, state)
hook := l.onState
l.mu.Unlock()
if hook != nil {
hook(state)
}
}
func (l *recordingListener) last() (ClientState, bool) {
l.mu.Lock()
defer l.mu.Unlock()
if len(l.states) == 0 {
return 0, false
}
return l.states[len(l.states)-1], true
}
func (l *recordingListener) snapshot() []ClientState {
l.mu.Lock()
defer l.mu.Unlock()
return append([]ClientState(nil), l.states...)
}
func (l *recordingListener) OnConnected() {}
func (l *recordingListener) OnDisconnected() {}
func (l *recordingListener) OnConnecting() {}
func (l *recordingListener) OnDisconnecting() {}
func (l *recordingListener) OnAddressChanged(string, string) {}
func (l *recordingListener) OnPeersListChanged(int) {}
// TestNotifier_ConcurrentAvailabilityFlipOrdersPublication holds the first
// transition inside the listener callback and flips availability again from
// another goroutine while it is parked. The second flip must not publish
// ahead of the one in flight, otherwise the listener ends up on a state the
// notifier already superseded.
func TestNotifier_ConcurrentAvailabilityFlipOrdersPublication(t *testing.T) {
n := newNotifier()
n.currentClientState = true
n.lastNotification = ClientStateConnecting
entered := make(chan struct{})
release := make(chan struct{})
l := &recordingListener{}
l.onState = func(state ClientState) {
if state != ClientStateNoNetwork {
return
}
l.mu.Lock()
l.onState = nil
l.mu.Unlock()
close(entered)
<-release
}
n.listener = l
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
n.setNetworkAvailable(false)
}()
<-entered
flipped := make(chan struct{})
go func() {
defer close(flipped)
n.setNetworkAvailable(true)
}()
select {
case <-flipped:
t.Fatal("the online transition published while the offline one was " +
"still in flight; publication is not serialized")
case <-time.After(200 * time.Millisecond):
}
close(release)
<-flipped
wg.Wait()
got, ok := l.last()
if !ok {
t.Fatal("listener never observed a state")
}
if got != ClientStateConnecting {
t.Fatalf("listener holds %v after the network came back, want Connecting; sequence: %v",
got, l.snapshot())
}
}

View File

@@ -6,29 +6,32 @@ import (
)
type mocListener struct {
lastState int
lastState ClientState
wg sync.WaitGroup
peersWg sync.WaitGroup
peers int
}
func (l *mocListener) OnConnected() {
l.lastState = stateConnected
l.lastState = ClientStateConnected
l.wg.Done()
}
func (l *mocListener) OnDisconnected() {
l.lastState = stateDisconnected
l.lastState = ClientStateDisconnected
l.wg.Done()
}
func (l *mocListener) OnConnecting() {
l.lastState = stateConnecting
l.lastState = ClientStateConnecting
l.wg.Done()
}
func (l *mocListener) OnDisconnecting() {
l.lastState = stateDisconnecting
l.lastState = ClientStateDisconnecting
l.wg.Done()
}
func (l *mocListener) OnStateChanged(state ClientState) {
}
func (l *mocListener) OnAddressChanged(host, addr string) {
}
@@ -57,15 +60,15 @@ func Test_notifier_serverState(t *testing.T) {
type scenario struct {
name string
expected int
expected ClientState
mgmState bool
signalState bool
}
scenarios := []scenario{
{"connected", stateConnected, true, true},
{"mgm down", stateConnecting, false, true},
{"signal down", stateConnecting, true, false},
{"disconnected", stateDisconnected, false, false},
{"connected", ClientStateConnected, true, true},
{"mgm down", ClientStateConnecting, false, true},
{"signal down", ClientStateConnecting, true, false},
{"disconnected", ClientStateDisconnected, false, false},
}
for _, tt := range scenarios {
@@ -85,7 +88,7 @@ func Test_notifier_SetListener(t *testing.T) {
listener.setPeersWaiter()
n := newNotifier()
n.lastNotification = stateConnecting
n.lastNotification = ClientStateConnecting
n.setListener(listener)
listener.wait()
listener.waitPeers()
@@ -99,7 +102,7 @@ func Test_notifier_RemoveListener(t *testing.T) {
listener.setWaiter()
listener.setPeersWaiter()
n := newNotifier()
n.lastNotification = stateConnecting
n.lastNotification = ClientStateConnecting
n.setListener(listener)
// setListener replays cached state on a goroutine; wait for both the state
// and peers callbacks to finish so we don't race on listener.peers.

View File

@@ -1211,6 +1211,12 @@ func (d *Status) ClientTeardown() {
d.notifyStateChange()
}
// SetNetworkAvailable records the OS-reported network availability; while
// unavailable, listeners see NoNetwork instead of Connecting.
func (d *Status) SetNetworkAvailable(available bool) {
d.notifier.setNetworkAvailable(available)
}
// SetConnectionListener set a listener to the notifier
func (d *Status) SetConnectionListener(listener Listener) {
d.notifier.setListener(listener)

View File

@@ -87,9 +87,10 @@ func (pm *ProfileManager) SetActiveProfileState(state *ProfileState) error {
// RemoveProfileState deletes the per-profile state file (which holds the
// account email used for the SSO login hint and the UI display). Called after
// a successful logout so a logged-out profile no longer shows a stale account
// email. The state file only stores the email, so deleting it is equivalent to
// clearing it; the next SSO login recreates it. A missing file is not an error.
// profile removal; logout keeps the file so the next login can pass the email
// as the login_hint. The state file only stores the email, so deleting it is
// equivalent to clearing it; the next SSO login recreates it. A missing file
// is not an error.
func (pm *ProfileManager) RemoveProfileState(profileName string) error {
configDir, err := getConfigDir()
if err != nil {

View File

@@ -0,0 +1,82 @@
//go:build windows
package systemops
import (
"math"
"testing"
"github.com/stretchr/testify/assert"
)
func TestSortRouteCandidates(t *testing.T) {
tests := []struct {
name string
candidates []candidateRoute
wantOrder []uint32
}{
{
name: "longest prefix wins over metrics",
candidates: []candidateRoute{
{interfaceIndex: 1, prefixLength: 0, routeMetric: 0, interfaceMetric: 5},
{interfaceIndex: 2, prefixLength: 24, routeMetric: 100, interfaceMetric: 50},
},
wantOrder: []uint32{2, 1},
},
{
// Windows ranks equal-length prefixes by route metric + interface metric,
// so a higher route metric on a low metric interface can still win.
name: "combined metric beats route metric alone",
candidates: []candidateRoute{
{interfaceIndex: 8, prefixLength: 0, routeMetric: 0, interfaceMetric: 100},
{interfaceIndex: 5, prefixLength: 0, routeMetric: 10, interfaceMetric: 5},
},
wantOrder: []uint32{5, 8},
},
{
name: "lower combined metric wins",
candidates: []candidateRoute{
{interfaceIndex: 5, prefixLength: 0, routeMetric: 300, interfaceMetric: 5},
{interfaceIndex: 8, prefixLength: 0, routeMetric: 0, interfaceMetric: 100},
},
wantOrder: []uint32{8, 5},
},
{
name: "equal combined metric falls back to route metric",
candidates: []candidateRoute{
{interfaceIndex: 1, prefixLength: 0, routeMetric: 20, interfaceMetric: 10},
{interfaceIndex: 2, prefixLength: 0, routeMetric: 5, interfaceMetric: 25},
},
wantOrder: []uint32{2, 1},
},
{
// The metrics are uint32 on the Windows side, so the sum must not wrap.
name: "combined metric beyond the uint32 range",
candidates: []candidateRoute{
{interfaceIndex: 1, prefixLength: 0, routeMetric: math.MaxUint32, interfaceMetric: 5},
{interfaceIndex: 2, prefixLength: 0, routeMetric: math.MaxUint32 - 10, interfaceMetric: 5},
},
wantOrder: []uint32{2, 1},
},
{
name: "unknown interface metric ranks on route metric only",
candidates: []candidateRoute{
{interfaceIndex: 1, prefixLength: 0, routeMetric: 30, interfaceMetric: -1},
{interfaceIndex: 2, prefixLength: 0, routeMetric: 5, interfaceMetric: 10},
},
wantOrder: []uint32{2, 1},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
sortRouteCandidates(tt.candidates)
got := make([]uint32, 0, len(tt.candidates))
for _, c := range tt.candidates {
got = append(got, c.interfaceIndex)
}
assert.Equal(t, tt.wantOrder, got)
})
}
}

View File

@@ -882,26 +882,40 @@ func getInterfaceMetric(interfaceIndex uint32, family int16) int {
return int(ipInterfaceRow.Metric)
}
// sortRouteCandidates sorts route candidates by priority: prefix length -> route metric -> interface metric
// sortRouteCandidates sorts route candidates by priority: prefix length -> combined metric -> route metric.
// Windows prefers the longest matching prefix and, among prefixes of the same length, the lowest metric, see
// https://learn.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/microsoft-windows-tcpip-interfaces-interface-routes-route-metric
func sortRouteCandidates(candidates []candidateRoute) {
sort.Slice(candidates, func(i, j int) bool {
if candidates[i].prefixLength != candidates[j].prefixLength {
return candidates[i].prefixLength > candidates[j].prefixLength
}
if candidates[i].routeMetric != candidates[j].routeMetric {
return candidates[i].routeMetric < candidates[j].routeMetric
mi, mj := combinedMetric(candidates[i]), combinedMetric(candidates[j])
if mi != mj {
return mi < mj
}
return candidates[i].interfaceMetric < candidates[j].interfaceMetric
return candidates[i].routeMetric < candidates[j].routeMetric
})
}
// combinedMetric returns the effective metric Windows uses to rank routes with an equal prefix length:
// the sum of the route metric and the metric of the interface the route is on, see
// https://learn.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-interface-metric
// An unknown interface metric contributes nothing.
func combinedMetric(candidate candidateRoute) uint64 {
if candidate.interfaceMetric < 0 {
return uint64(candidate.routeMetric)
}
return uint64(candidate.routeMetric) + uint64(candidate.interfaceMetric)
}
// GetBestInterface finds the best interface for reaching a destination,
// excluding the VPN interface to avoid routing loops.
//
// Route selection priority:
// 1. Longest prefix match (most specific route)
// 2. Lowest route metric
// 3. Lowest interface metric
// 2. Lowest combined metric (route metric + interface metric)
// 3. Lowest route metric.
func GetBestInterface(dest netip.Addr, vpnIntf string) (*net.Interface, error) {
var skipInterfaceIndex int
if vpnIntf != "" {
@@ -925,7 +939,6 @@ func GetBestInterface(dest netip.Addr, vpnIntf string) (*net.Interface, error) {
return nil, fmt.Errorf("no route to %s", dest)
}
// Sort routes: prefix length -> route metric -> interface metric
sortRouteCandidates(candidates)
for _, candidate := range candidates {

View File

@@ -5,6 +5,7 @@ package systemops
import (
"errors"
"net"
"net/netip"
"syscall"
"testing"
@@ -29,6 +30,7 @@ func ensureIPv6DefaultRoute(t *testing.T) {
}
if err := netlink.RouteAdd(route); err != nil {
if errors.Is(err, syscall.EEXIST) {
requireUsableIPv6Nexthop(t)
return
}
t.Skipf("install IPv6 fallback default route: %v", err)
@@ -38,4 +40,36 @@ func ensureIPv6DefaultRoute(t *testing.T) {
t.Logf("delete IPv6 fallback default route: %v", err)
}
})
requireUsableIPv6Nexthop(t)
}
// requireUsableIPv6Nexthop skips the test unless the resolved IPv6 default
// nexthop can actually carry a route. Installing the default route succeeding
// does not imply the kernel accepts it as a nexthop for a concrete prefix.
func requireUsableIPv6Nexthop(t *testing.T) {
t.Helper()
nexthop, err := GetNextHop(netip.IPv6Unspecified())
if err != nil {
t.Skipf("resolve IPv6 default nexthop: %v", err)
}
probe := &netlink.Route{
Scope: netlink.SCOPE_UNIVERSE,
Table: syscall.RT_TABLE_MAIN,
Family: netlink.FAMILY_V6,
Dst: &net.IPNet{IP: net.ParseIP("100::64"), Mask: net.CIDRMask(128, 128)},
}
require.NoError(t, addNextHop(nexthop, probe), "build IPv6 probe route")
switch err := netlink.RouteAdd(probe); {
case err == nil:
if err := netlink.RouteDel(probe); err != nil && !errors.Is(err, syscall.ESRCH) {
t.Logf("delete IPv6 probe route: %v", err)
}
case errors.Is(err, syscall.EEXIST):
default:
t.Skipf("IPv6 nexthop %s unusable for route installation: %v", nexthop, err)
}
}

View File

@@ -22,6 +22,8 @@ import (
"github.com/netbirdio/netbird/client/internal/listener"
"github.com/netbirdio/netbird/client/internal/peer"
"github.com/netbirdio/netbird/client/internal/profilemanager"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
"github.com/netbirdio/netbird/client/system"
"github.com/netbirdio/netbird/formatter"
"github.com/netbirdio/netbird/route"
@@ -36,11 +38,6 @@ const (
AnonymizeLevelStrict = nbAnonymize.LevelStrictString
)
// ConnectionListener export internal Listener for mobile
type ConnectionListener interface {
peer.Listener
}
// RouteListener export internal RouteListener for mobile
type NetworkChangeListener interface {
listener.NetworkChangeListener
@@ -87,6 +84,12 @@ type Client struct {
onHostDnsFn func([]string)
dnsManager dns.IosDnsManager
loginComplete bool
// netState outlives engine restarts: it mirrors the OS connectivity, not
// the engine lifecycle. Run injects it into each new ConnectClient, which
// distributes it to every reconnection loop.
netState *netstate.State
// sweeper also outlives engine restarts; NotifyNetworkChange sweeps it.
sweeper *netsweep.Sweeper
// preloadedConfig holds config loaded from JSON (used on tvOS where file writes are blocked)
preloadedConfig *profilemanager.Config
@@ -109,6 +112,8 @@ func NewClient(cfgFile, stateFile, cacheDir, logFilePath, deviceName string, osV
ctxCancelLock: &sync.Mutex{},
networkChangeListener: networkChangeListener,
dnsManager: dnsManager,
netState: netstate.New(),
sweeper: netsweep.New(),
}
}
@@ -184,7 +189,8 @@ func (c *Client) Run(fd int32, interfaceName string, envList *EnvList) error {
c.onHostDnsFn = func([]string) {}
cfg.WgIface = interfaceName
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder)
connectClient := internal.NewConnectClient(ctx, cfg, c.recorder,
internal.WithNetworkState(c.netState), internal.WithSweeper(c.sweeper))
c.setState(cfg, connectClient)
// Persist the latest sync response so DebugBundle can include the network
// map. On iOS this is backed by disk to keep it out of the constrained
@@ -193,6 +199,25 @@ func (c *Client) Run(fd int32, interfaceName string, envList *EnvList) error {
return connectClient.RunOniOS(fd, c.networkChangeListener, c.dnsManager, c.stateFile, c.cacheDir, c.logFilePath)
}
// SetNetworkAvailable feeds OS-reported network availability into the client
// (e.g. from NWPathMonitor). While unavailable, the internal reconnect loops
// suspend their attempts and the connection listener reports NoNetwork
// instead of Connecting; when availability returns, the loops resume
// immediately with a fresh backoff.
func (c *Client) SetNetworkAvailable(available bool) {
c.netState.Set(available)
c.recorder.SetNetworkAvailable(available)
}
// NotifyNetworkChange marks the management, signal and relay connections
// stale after the OS switched networks and schedules a sweep that cuts
// whatever has not redialed on the new network by then. The engine and the
// TUN device stay untouched.
func (c *Client) NotifyNetworkChange() {
c.sweeper.MarkNetworkChange()
log.Infof("network change: connections marked stale")
}
// Stop the internal client and free the resources
func (c *Client) Stop() {
c.ctxCancelLock.Lock()
@@ -331,7 +356,11 @@ func (c *Client) GetStatusDetails() *StatusDetails {
// SetConnectionListener set the network connection listener
func (c *Client) SetConnectionListener(listener ConnectionListener) {
c.recorder.SetConnectionListener(listener)
if listener == nil {
c.recorder.RemoveConnectionListener()
return
}
c.recorder.SetConnectionListener(connectionListenerAdapter{listener})
}
// RemoveConnectionListener remove connection listener

View File

@@ -0,0 +1,43 @@
//go:build ios
package NetBirdSDK
import (
"github.com/netbirdio/netbird/client/internal/peer"
)
// Client state values, re-exported as basic constants so gomobile emits them
// into the generated bindings. They mirror peer.ClientState*: append-only,
// never reorder.
const (
ClientStateDisconnected = int(peer.ClientStateDisconnected)
ClientStateConnected = int(peer.ClientStateConnected)
ClientStateConnecting = int(peer.ClientStateConnecting)
ClientStateDisconnecting = int(peer.ClientStateDisconnecting)
ClientStateNoNetwork = int(peer.ClientStateNoNetwork)
)
// ConnectionListener export internal Listener for mobile.
//
// It intentionally lacks OnStateChanged for now: adding a method to a gomobile
// interface breaks every Swift implementation, so the iOS app keeps building
// against the legacy per-state callbacks. A follow-up will extend it together
// with the app.
type ConnectionListener interface {
OnConnected()
OnDisconnected()
OnConnecting()
OnDisconnecting()
OnAddressChanged(string, string)
OnPeersListChanged(int)
}
// connectionListenerAdapter adapts the gomobile-facing ConnectionListener to
// peer.Listener.
type connectionListenerAdapter struct {
ConnectionListener
}
// OnStateChanged is dropped on iOS until the app adopts the state callback;
// the legacy per-state callbacks continue to fire.
func (a connectionListenerAdapter) OnStateChanged(peer.ClientState) {}

107
client/netstate/netstate.go Normal file
View File

@@ -0,0 +1,107 @@
// Package netstate tracks OS-reported network availability for the client.
//
// A State instance is owned by the platform integration (e.g. the Android or
// iOS bindings, fed from ConnectivityManager callbacks or NWPathMonitor) and
// is injected into the connection retry loops (management, signal, relay,
// peer guards and the top-level connect loop), which consult it to avoid
// burning CPU and battery on reconnect attempts while the device has no
// network at all (e.g. airplane mode), and to reset their backoff as soon as
// the network returns.
//
// Consumers hold a *State that may be nil — every non-mobile platform leaves
// it unset. The read methods are safe on a nil receiver: they report online
// and never block, so consumers behave as if this package did not exist.
package netstate
import (
"context"
"sync"
log "github.com/sirupsen/logrus"
)
// State holds the OS-reported network availability. The zero value is not
// usable; create instances with New.
type State struct {
mu sync.Mutex
online bool
changed chan struct{}
}
// New creates a State that starts online.
func New() *State {
return &State{
online: true,
changed: make(chan struct{}),
}
}
// Set records whether the OS reports any usable network. Transitions wake up
// all Wait callers immediately.
func (s *State) Set(online bool) {
s.mu.Lock()
defer s.mu.Unlock()
if s.online == online {
return
}
s.online = online
close(s.changed)
s.changed = make(chan struct{})
log.Infof("OS network availability changed: online=%t", online)
}
// IsOnline reports whether the OS reports at least one usable network. On a
// nil receiver — no State injected — it reports online.
func (s *State) IsOnline() bool {
if s == nil {
return true
}
s.mu.Lock()
defer s.mu.Unlock()
return s.online
}
// Changed returns a channel closed on the next availability transition, for
// callers that already own a select loop and cannot block in Wait. Re-read it
// after every fire: each transition installs a fresh channel. On a nil
// receiver — no State injected — it returns nil, which blocks forever in a
// select, so the caller simply never observes a transition.
func (s *State) Changed() <-chan struct{} {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
return s.changed
}
// Wait blocks while the network is offline. It reports whether it had to
// wait, so callers can reset their backoff after an outage. It returns early
// with the context error when ctx is done. On a nil receiver — no State
// injected — it returns immediately.
func (s *State) Wait(ctx context.Context) (bool, error) {
if s == nil {
return false, nil
}
waited := false
for {
s.mu.Lock()
if s.online {
s.mu.Unlock()
return waited, nil
}
ch := s.changed
s.mu.Unlock()
if !waited {
waited = true
log.Debugf("network is offline, pausing connection attempts")
}
select {
case <-ctx.Done():
return waited, ctx.Err()
case <-ch:
}
}
}

View File

@@ -0,0 +1,170 @@
package netstate
import (
"context"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewStateIsOnline(t *testing.T) {
assert.True(t, New().IsOnline(), "a fresh State should start online")
}
func TestSetTogglesOnlineState(t *testing.T) {
s := New()
s.Set(false)
assert.False(t, s.IsOnline(), "state should be offline after Set(false)")
s.Set(true)
assert.True(t, s.IsOnline(), "state should be online after Set(true)")
}
func TestWaitReturnsImmediatelyWhenOnline(t *testing.T) {
s := New()
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
waited, err := s.Wait(ctx)
require.NoError(t, err)
assert.False(t, waited, "Wait should not block when the network is online")
}
func TestWaitBlocksUntilOnline(t *testing.T) {
s := New()
s.Set(false)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
result := make(chan bool, 1)
go func() {
waited, err := s.Wait(ctx)
if err != nil {
result <- false
return
}
result <- waited
}()
// Verify Wait is actually blocking while offline
select {
case <-result:
t.Fatal("Wait should block while the network is offline")
case <-time.After(100 * time.Millisecond):
}
s.Set(true)
select {
case waited := <-result:
assert.True(t, waited, "Wait should report that it had to wait for the network")
case <-time.After(2 * time.Second):
t.Fatal("Wait should return promptly after the network becomes available")
}
}
func TestWaitReturnsOnContextCancel(t *testing.T) {
s := New()
s.Set(false)
ctx, cancel := context.WithCancel(context.Background())
result := make(chan error, 1)
go func() {
_, err := s.Wait(ctx)
result <- err
}()
cancel()
select {
case err := <-result:
assert.ErrorIs(t, err, context.Canceled)
case <-time.After(2 * time.Second):
t.Fatal("Wait should return promptly after context cancellation")
}
}
func TestWaitWakesAllWaiters(t *testing.T) {
s := New()
s.Set(false)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
const waiters = 10
var wg sync.WaitGroup
results := make(chan bool, waiters)
for i := 0; i < waiters; i++ {
wg.Add(1)
go func() {
defer wg.Done()
waited, err := s.Wait(ctx)
if err != nil {
results <- false
return
}
results <- waited
}()
}
time.Sleep(100 * time.Millisecond)
s.Set(true)
wg.Wait()
close(results)
count := 0
for waited := range results {
assert.True(t, waited, "every waiter should report that it waited")
count++
}
assert.Equal(t, waiters, count, "all waiters should have returned")
}
func TestNilStateReadsAreNoops(t *testing.T) {
var s *State
assert.True(t, s.IsOnline(), "nil State should report online")
waited, err := s.Wait(context.Background())
require.NoError(t, err)
assert.False(t, waited, "nil State's Wait should not block")
}
func TestConcurrentSetAndWait(t *testing.T) {
s := New()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
var wg sync.WaitGroup
for i := 0; i < 4; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := 0; j < 100; j++ {
s.Set(j%2 == 0)
s.IsOnline()
}
}()
}
for i := 0; i < 4; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := 0; j < 100; j++ {
if _, err := s.Wait(ctx); err != nil {
return
}
}
}()
}
wg.Wait()
}

267
client/netsweep/netsweep.go Normal file
View File

@@ -0,0 +1,267 @@
// Package netsweep cuts network-bound activity when the OS switches networks:
// a sweep closes the registered connections and aborts the in-flight dials, so
// their owners redial immediately instead of waiting for the old sockets to
// time out.
//
// A nil *Sweeper disables everything: all methods are nil-safe no-ops.
package netsweep
import (
"context"
"errors"
"net"
"sync"
"time"
"github.com/cenkalti/backoff/v4"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netstate"
)
// DefaultSweepDelay absorbs network flapping while the OS settles on a
// default network before the stale registrations are cut.
const DefaultSweepDelay = 500 * time.Millisecond
const recentMarkWindow = 3 * time.Second
// Config customizes a Sweeper. The zero value applies the defaults.
type Config struct {
// SweepDelay overrides DefaultSweepDelay when positive.
SweepDelay time.Duration
}
// ErrSwept reports that a dial finished after a network change swept its
// registration. The connection is already closed; the caller must treat it
// as a failed dial and redial on the new network.
var ErrSwept = errors.New("netsweep: connection swept by network change")
// sweepID identifies one registration in a sweeper. Connections and dials
// draw from the same counter, so an id is unique across both registries.
type sweepID uint64
type connEntry struct {
conn net.Conn
gen uint64
}
// Dial tracks one dial from start to connection registration. It hands the
// dialed connection to the sweeper atomically, so a sweep can never fall
// between the dial finishing and the connection being registered.
type Dial struct {
sweeper *Sweeper
ctx context.Context
cancel context.CancelFunc
id sweepID
done bool // set by a sweep, WrapConn or Release; guarded by sweeper.mu
gen uint64
}
// Ctx returns the dial's context. A sweep cancels it, so a dial started on the
// old network aborts instead of waiting out its handshake timeout.
func (d *Dial) Ctx() context.Context {
return d.ctx
}
// Release ends the dial's registration and cancels its context. It is
// idempotent and safe after WrapConn, so callers can defer it.
func (d *Dial) Release() {
s := d.sweeper
if s == nil {
return
}
s.mu.Lock()
d.done = true
delete(s.dials, d.id)
s.mu.Unlock()
d.cancel()
}
// sweptConn deregisters itself from the sweeper when closed.
type sweptConn struct {
net.Conn
sweeper *Sweeper
id sweepID
}
func (c *sweptConn) Close() error {
c.sweeper.deregister(c.id)
return c.Conn.Close()
}
// Sweeper registers live connections and in-flight dials so the
// network-change sweep can cut everything registered before the change.
type Sweeper struct {
mu sync.Mutex
conns map[sweepID]connEntry
dials map[sweepID]*Dial
nextID sweepID
gen uint64
timer *time.Timer
sweepDelay time.Duration
lastMark time.Time
}
// New creates an empty sweeper with the default configuration.
func New() *Sweeper {
return NewWithConfig(Config{})
}
// NewWithConfig creates an empty sweeper customized by cfg.
func NewWithConfig(cfg Config) *Sweeper {
delay := cfg.SweepDelay
if delay <= 0 {
delay = DefaultSweepDelay
}
return &Sweeper{
conns: make(map[sweepID]connEntry),
dials: make(map[sweepID]*Dial),
sweepDelay: delay,
}
}
// StartDial registers an in-flight dial. Dial with Ctx, hand the result to
// WrapConn, and Release the dial when the attempt is over, typically deferred.
func (s *Sweeper) StartDial(ctx context.Context) *Dial {
if s == nil {
return &Dial{ctx: ctx}
}
ctx, cancel := context.WithCancel(ctx)
d := &Dial{sweeper: s, ctx: ctx, cancel: cancel}
s.mu.Lock()
d.id = s.nextID
s.nextID++
d.gen = s.gen
s.dials[d.id] = d
s.mu.Unlock()
return d
}
// WrapConn hands conn over to the sweeper. If a sweep ran since StartDial,
// the connection belongs to the old network: it is closed and ErrSwept is
// returned. Otherwise conn is registered against the next sweep and returned
// wrapped, deregistering itself on Close. Call it once, before Release.
func (d *Dial) WrapConn(conn net.Conn) (net.Conn, error) {
s := d.sweeper
if s == nil {
return conn, nil
}
s.mu.Lock()
if d.done {
s.mu.Unlock()
if err := conn.Close(); err != nil {
log.Debugf("swept dial close error: %v", err)
}
return nil, ErrSwept
}
d.done = true
delete(s.dials, d.id)
id := s.nextID
s.nextID++
// The conn inherits the dial's generation: the socket was bound to the
// network that was default when the dial started, not when it finished.
s.conns[id] = connEntry{conn: conn, gen: d.gen}
s.mu.Unlock()
return &sweptConn{Conn: conn, sweeper: s, id: id}, nil
}
// MarkNetworkChange records that the OS switched networks: everything
// registered so far becomes stale, and a sweep is (re)scheduled after the
// configured delay to cut whatever is still stale by then. Owners that
// redialed in the meantime hold fresh-generation registrations and survive,
// so no cancellation is needed around the sweep.
func (s *Sweeper) MarkNetworkChange() {
if s == nil {
return
}
s.mu.Lock()
s.gen++
cutoff := s.gen
s.lastMark = time.Now()
if s.timer != nil {
s.timer.Stop()
}
s.timer = time.AfterFunc(s.sweepDelay, func() {
n := s.sweep(cutoff)
log.Infof("network change sweep: closed %d stale connections", n)
})
s.mu.Unlock()
}
// QuickRetryBackoff wraps bo so that after each Reset the first retry comes
// quickly when the disconnect followed a recent network change and the
// network is online. Any other failure keeps bo's spread, so the clients of
// a restarted server still scatter their reconnects. A nil sweeper returns
// bo unchanged.
func (s *Sweeper) QuickRetryBackoff(ctx context.Context, bo backoff.BackOff, netState *netstate.State) backoff.BackOff {
if s == nil {
return bo
}
return backoff.WithContext(newQuickRetryBackoff(bo, s, netState), ctx)
}
func (s *Sweeper) markedRecently() bool {
if s == nil {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
return !s.lastMark.IsZero() && time.Since(s.lastMark) < recentMarkWindow
}
// sweep closes the registered connections and aborts the in-flight dials
// older than cutoff, and returns how many connections it closed. A dial
// whose connection was not yet handed to WrapConn is marked, so the late
// WrapConn closes it instead of registering it.
func (s *Sweeper) sweep(cutoff uint64) int {
if s == nil {
return 0
}
s.mu.Lock()
var conns []net.Conn
for id, e := range s.conns {
if e.gen < cutoff {
delete(s.conns, id)
conns = append(conns, e.conn)
}
}
var dials []*Dial
for id, d := range s.dials {
if d.gen < cutoff {
d.done = true
delete(s.dials, id)
dials = append(dials, d)
}
}
s.mu.Unlock()
if len(dials) > 0 {
log.Debugf("aborting %d in-flight dials", len(dials))
for _, d := range dials {
d.cancel()
}
}
for _, conn := range conns {
log.Debugf("sweeping connection %s -> %s", conn.LocalAddr(), conn.RemoteAddr())
if err := conn.Close(); err != nil {
log.Debugf("swept connection close error: %v", err)
}
}
return len(conns)
}
func (s *Sweeper) deregister(id sweepID) {
s.mu.Lock()
delete(s.conns, id)
s.mu.Unlock()
}

View File

@@ -0,0 +1,241 @@
package netsweep
import (
"context"
"math"
"net"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestSweepClosesRegisteredConns(t *testing.T) {
sweeper := New()
c1 := wrap(t, sweeper, connPair(t))
c2 := wrap(t, sweeper, connPair(t))
assert.Equal(t, 2, sweeper.sweepAll(), "both live connections should be closed")
// The wrappers must report closed now.
buf := make([]byte, 1)
_, err := c1.Read(buf)
assert.Error(t, err, "first connection should be unusable after the sweep")
_, err = c2.Read(buf)
assert.Error(t, err, "second connection should be unusable after the sweep")
assert.Equal(t, 0, sweeper.sweepAll(), "second sweep should find nothing")
}
func TestCloseDeregisters(t *testing.T) {
sweeper := New()
conn := wrap(t, sweeper, connPair(t))
require.NoError(t, conn.Close())
assert.Equal(t, 0, sweeper.sweepAll(), "closed connection must leave the registry")
}
func TestCloseIsIdempotent(t *testing.T) {
sweeper := New()
conn := wrap(t, sweeper, connPair(t))
require.NoError(t, conn.Close())
assert.Error(t, conn.Close(), "double close surfaces the underlying error but must not panic")
}
func TestSweepOnlyAffectsOlderConns(t *testing.T) {
sweeper := New()
_ = wrap(t, sweeper, connPair(t))
assert.Equal(t, 1, sweeper.sweepAll())
// A connection dialed after the sweep must survive until the next one.
_ = wrap(t, sweeper, connPair(t))
assert.Equal(t, 1, sweeper.sweepAll(), "post-sweep connection belongs to the next sweep")
}
func TestSweepAbortsInFlightDials(t *testing.T) {
sweeper := New()
dial := sweeper.StartDial(context.Background())
defer dial.Release()
sweeper.sweepAll()
assert.ErrorIs(t, dial.Ctx().Err(), context.Canceled, "sweep must cancel the in-flight dial context")
}
func TestReleasedDialIsNotAborted(t *testing.T) {
sweeper := New()
// Simulate a dial that finished before the sweep.
released := sweeper.StartDial(context.Background())
released.Release()
// A dial still in flight during the sweep.
pending := sweeper.StartDial(context.Background())
defer pending.Release()
sweeper.sweepAll()
assert.ErrorIs(t, pending.Ctx().Err(), context.Canceled, "pending dial must be aborted")
}
func TestSweepBetweenDialAndHandoffClosesConn(t *testing.T) {
sweeper := New()
dial := sweeper.StartDial(context.Background())
defer dial.Release()
// The dial succeeds on the old network, then the sweep lands before the
// connection is handed over.
conn := connPair(t)
assert.Equal(t, 0, sweeper.sweepAll(), "the connection is not registered yet")
wrapped, err := dial.WrapConn(conn)
require.ErrorIs(t, err, ErrSwept)
require.Nil(t, wrapped)
buf := make([]byte, 1)
_, err = conn.Read(buf)
assert.Error(t, err, "the old-network connection must be closed, not leaked")
assert.Equal(t, 0, sweeper.sweepAll(), "nothing may leak into the next sweep")
}
func TestMarkNetworkChangeSparesFreshConns(t *testing.T) {
sweeper := NewWithConfig(Config{SweepDelay: 10 * time.Millisecond})
stale := wrap(t, sweeper, connPair(t))
sweeper.MarkNetworkChange()
_ = wrap(t, sweeper, connPair(t))
_ = stale.SetReadDeadline(time.Now().Add(time.Second))
buf := make([]byte, 1)
_, err := stale.Read(buf)
require.ErrorIs(t, err, net.ErrClosed, "stale connection must be closed by the delayed sweep")
assert.Equal(t, 1, sweeper.sweepAll(), "the fresh connection must survive the stale sweep")
}
func TestMarkNetworkChangeAbortsStaleDials(t *testing.T) {
sweeper := NewWithConfig(Config{SweepDelay: 10 * time.Millisecond})
stale := sweeper.StartDial(context.Background())
defer stale.Release()
sweeper.MarkNetworkChange()
fresh := sweeper.StartDial(context.Background())
defer fresh.Release()
assert.Eventually(t, func() bool {
return stale.Ctx().Err() != nil
}, time.Second, 5*time.Millisecond, "stale dial must be aborted by the delayed sweep")
assert.NoError(t, fresh.Ctx().Err(), "post-mark dial must not be aborted")
}
func TestConnInheritsDialGeneration(t *testing.T) {
sweeper := NewWithConfig(Config{SweepDelay: 20 * time.Millisecond})
// The dial starts before the network change but completes after it: the
// socket is bound to the old network, so the sweep must still cut it.
dial := sweeper.StartDial(context.Background())
defer dial.Release()
sweeper.MarkNetworkChange()
wrapped, err := dial.WrapConn(connPair(t))
require.NoError(t, err)
_ = wrapped.SetReadDeadline(time.Now().Add(time.Second))
buf := make([]byte, 1)
_, err = wrapped.Read(buf)
require.ErrorIs(t, err, net.ErrClosed, "old-generation connection must be swept")
}
func TestRepeatedMarksCoalesce(t *testing.T) {
sweeper := NewWithConfig(Config{SweepDelay: 20 * time.Millisecond})
first := wrap(t, sweeper, connPair(t))
sweeper.MarkNetworkChange()
second := wrap(t, sweeper, connPair(t))
sweeper.MarkNetworkChange()
_ = wrap(t, sweeper, connPair(t))
buf := make([]byte, 1)
for _, conn := range []net.Conn{first, second} {
_ = conn.SetReadDeadline(time.Now().Add(time.Second))
_, err := conn.Read(buf)
require.ErrorIs(t, err, net.ErrClosed, "every pre-mark connection must be swept by the rescheduled sweep")
}
assert.Equal(t, 1, sweeper.sweepAll(), "only the newest-generation connection may remain")
}
func TestNilSweeperIsNoop(t *testing.T) {
var sweeper *Sweeper
conn := connPair(t)
dial := sweeper.StartDial(context.Background())
defer dial.Release()
wrapped, err := dial.WrapConn(conn)
require.NoError(t, err)
assert.Equal(t, conn, wrapped, "nil sweeper must return the conn unchanged")
assert.NoError(t, dial.Ctx().Err(), "nil sweeper must not cancel the dial context")
assert.Equal(t, 0, sweeper.sweepAll(), "nil sweeper closes nothing")
}
// wrap registers conn with the sweeper through a completed dial.
func wrap(t *testing.T, sweeper *Sweeper, conn net.Conn) net.Conn {
t.Helper()
dial := sweeper.StartDial(context.Background())
defer dial.Release()
wrapped, err := dial.WrapConn(conn)
require.NoError(t, err)
return wrapped
}
// connPair dials a loopback TCP connection and keeps the accepted peer open
// until the test ends: a peer that closed early would make the connection
// unreadable on its own, so a read error after the sweep would prove nothing.
func connPair(t *testing.T) net.Conn {
t.Helper()
l, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
t.Cleanup(func() {
if err := l.Close(); err != nil {
t.Logf("listener close error: %v", err)
}
})
accepted := make(chan net.Conn, 1)
go func() {
conn, err := l.Accept()
if err != nil {
close(accepted)
return
}
accepted <- conn
}()
conn, err := net.Dial("tcp", l.Addr().String())
require.NoError(t, err)
peer, ok := <-accepted
require.True(t, ok, "listener must accept the dialed connection")
t.Cleanup(func() {
if err := peer.Close(); err != nil {
t.Logf("peer close error: %v", err)
}
})
return conn
}
// sweepAll cuts every registration regardless of generation.
func (s *Sweeper) sweepAll() int {
return s.sweep(math.MaxUint64)
}

View File

@@ -0,0 +1,39 @@
package netsweep
import (
"time"
"github.com/cenkalti/backoff/v4"
"github.com/netbirdio/netbird/client/netstate"
)
const quickRetryDelay = 200 * time.Millisecond
type quickRetryBackoff struct {
backoff.BackOff
sweeper *Sweeper
netState *netstate.State
used bool
}
func newQuickRetryBackoff(bo backoff.BackOff, sweeper *Sweeper, netState *netstate.State) *quickRetryBackoff {
return &quickRetryBackoff{
BackOff: bo,
sweeper: sweeper,
netState: netState,
}
}
func (b *quickRetryBackoff) NextBackOff() time.Duration {
if !b.used && b.sweeper.markedRecently() && b.netState.IsOnline() {
b.used = true
return quickRetryDelay
}
return b.BackOff.NextBackOff()
}
func (b *quickRetryBackoff) Reset() {
b.used = false
b.BackOff.Reset()
}

View File

@@ -0,0 +1,58 @@
package netsweep
import (
"context"
"testing"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/stretchr/testify/assert"
)
func TestQuickRetryAfterRecentMark(t *testing.T) {
sweeper := New()
sweeper.MarkNetworkChange()
slow := backoff.NewConstantBackOff(5 * time.Second)
bo := sweeper.QuickRetryBackoff(context.Background(), slow, nil)
assert.Equal(t, quickRetryDelay, bo.NextBackOff(), "first retry after a mark must be quick")
assert.Equal(t, 5*time.Second, bo.NextBackOff(), "second retry must fall back to the wrapped backoff")
bo.Reset()
assert.Equal(t, quickRetryDelay, bo.NextBackOff(), "reset must re-arm the quick retry")
}
func TestQuickRetryWithoutMarkKeepsSpread(t *testing.T) {
sweeper := New()
slow := backoff.NewConstantBackOff(5 * time.Second)
bo := sweeper.QuickRetryBackoff(context.Background(), slow, nil)
assert.Equal(t, 5*time.Second, bo.NextBackOff(), "without a mark the wrapped backoff decides")
sweeper.mu.Lock()
sweeper.lastMark = time.Now().Add(-recentMarkWindow)
sweeper.mu.Unlock()
assert.Equal(t, 5*time.Second, bo.NextBackOff(), "a stale mark must not trigger the quick retry")
}
func TestQuickRetryNilSweeperPassthrough(t *testing.T) {
var sweeper *Sweeper
slow := backoff.NewConstantBackOff(5 * time.Second)
bo := sweeper.QuickRetryBackoff(context.Background(), slow, nil)
assert.Equal(t, backoff.BackOff(slow), bo, "nil sweeper must return the backoff unchanged")
}
func TestQuickRetryHonorsContext(t *testing.T) {
sweeper := New()
sweeper.MarkNetworkChange()
ctx, cancel := context.WithCancel(context.Background())
cancel()
bo := sweeper.QuickRetryBackoff(ctx, backoff.NewConstantBackOff(time.Millisecond), nil)
assert.Equal(t, backoff.Stop, bo.NextBackOff(), "cancelled context must stop the retry loop")
}

View File

@@ -6,9 +6,11 @@ import (
"context"
"time"
log "github.com/sirupsen/logrus"
"google.golang.org/grpc/codes"
gstatus "google.golang.org/grpc/status"
"github.com/netbirdio/netbird/client/internal/profilemanager"
"github.com/netbirdio/netbird/client/proto"
)
@@ -60,9 +62,19 @@ func (s *Session) RequestExtend(ctx context.Context, p ExtendStartParams) (Exten
// a request from the UI implies a graphical session, which the daemon cannot detect itself
req := &proto.RequestExtendAuthSessionRequest{HasGraphicalSession: true}
if p.Hint != "" {
h := p.Hint
req.Hint = &h
hint := p.Hint
if hint == "" {
pm := profilemanager.NewProfileManager()
if active, perr := pm.GetActiveProfile(); perr != nil {
log.Debugf("failed to get active profile for login hint: %v", perr)
} else if state, serr := pm.GetProfileState(active.ID); serr != nil {
log.Debugf("failed to get profile state for login hint: %v", serr)
} else {
hint = state.Email
}
}
if hint != "" {
req.Hint = &hint
}
resp, err := cli.RequestExtendAuthSession(ctx, req)

View File

@@ -123,8 +123,16 @@ func (s *Connection) Login(ctx context.Context, p LoginParams) (LoginResult, err
if p.PreSharedKey != "" {
req.OptionalPreSharedKey = ptrStr(p.PreSharedKey)
}
if p.Hint != "" {
req.Hint = ptrStr(p.Hint)
hint := p.Hint
if hint == "" && profileID != "" {
if state, serr := profilemanager.NewProfileManager().GetProfileState(profilemanager.ID(profileID)); serr == nil {
hint = state.Email
} else {
log.Debugf("failed to get profile state for login hint: %v", serr)
}
}
if hint != "" {
req.Hint = ptrStr(hint)
}
resp, err := cli.Login(ctx, req)
@@ -228,16 +236,6 @@ func (s *Connection) Logout(ctx context.Context, p LogoutParams) error {
return s.classifyDaemonError(err)
}
// The daemon runs as root and can't reach the user-owned per-profile state
// file holding the account email (see Profiles.List), so clear the stale
// email here; the next SSO login recreates it.
if p.ProfileName != "" {
if err := profilemanager.NewProfileManager().RemoveProfileState(p.ProfileName); err != nil {
// Non-fatal: the logout itself succeeded.
log.Warnf("failed to remove profile state for %s: %v", p.ProfileName, err)
}
}
return nil
}
@@ -261,7 +259,7 @@ func (s *Connection) waitSSOLogin(ctx context.Context, p WaitSSOParams) (string,
// Persist the account email the same way the CLI does after its own
// WaitSSOLogin: the daemon returns it but cannot store it, since it runs as
// root and the per-profile state file is user-owned (see Logout below).
// root and the per-profile state file is user-owned (see Profiles.List).
// Without this the profile has no email, so Profiles.List shows no account
// and later logins and session extends go out without a login_hint —
// leaving the IdP to guess which account was meant.

View File

@@ -162,8 +162,9 @@ func (s *Profiles) Remove(ctx context.Context, p ProfileRef) error {
}
// The daemon deletes what it owns but runs as root, so it leaves the
// user-owned state file holding the account email behind (same split as
// Connection.Logout). Legacy profiles are keyed by name rather than by a
// user-owned state file holding the account email behind. Logout keeps the
// email on purpose so later logins can pass it as the login_hint; profile
// removal is what deletes it. Legacy profiles are keyed by name rather than by a
// generated ID, so a recreated profile of the same name would inherit the
// deleted one's email and offer it as the login_hint.
//

View File

@@ -115,7 +115,7 @@ sequenceDiagram
Resp->>Resp: parse usage tokens, completion
Note over Resp: capture_completion gates raw<br/>completion capture
Resp->>Cost: tokens
Cost->>Cost: lookup pricing.yaml + compute cost
Cost->>Cost: lookup rates from config-delivered<br/>pricing table + compute cost
Cost->>Rec: tokens + cost
Rec->>MgmtGrpc: RecordLLMUsage(provider, model, prompt_t, completion_t, cost, groups, user)
Rec-->>Log: emit access-log entry<br/>(if EnableLogCollection)

View File

@@ -15,6 +15,10 @@ Inside the package: `manager.go` is the CRUD + permissions-gated facade; `synthe
| ---- | ---- |
| `agentnetwork/manager.go` | Manager interface + CRUD + permission gates + bootstrap-settings + reconcile trigger |
| `agentnetwork/synthesizer.go` | Settings/policy → wire-format synthesis; sole writer of the proxy middleware chain |
| `agentnetwork/synthesizer_pricing.go` | `buildCostMeterConfigJSON` — default table + per-provider prices → `cost_meter` config |
| `agentnetwork/pricing/defaults.go` | Default pricing table derived from the catalog + supplementals; `DefaultTable`, `LookupDefault`, wire `Entry` |
| `agentnetwork/pricing/override.go` | `LoadFile`/`StartReloader` for `AgentNetwork.PricingDefaultsFile` (mtime poll, merge over compiled-in base) |
| `agentnetwork/pricing/{exampleyaml,gen}.go` | Generates `defaults_llm_pricing.example.yaml` from the compiled-in table (golden-tested) |
| `agentnetwork/policyselect.go` | Per-request policy attribution + account-budget ceiling (min-wins) |
| `agentnetwork/reconcile.go` | Per-account synth diff vs in-memory cache → Create/Update/Delete |
| `agentnetwork/catalog/catalog.go` | Static provider catalogue (auth headers, identity-injection shapes) |
@@ -48,6 +52,8 @@ flowchart TD
I --> J[indexProviderGroups: providerID -> sorted source groups]
J --> K[buildRouterConfigJSON drops orphan providers]
J --> L[buildIdentityInjectConfigJSON per catalog entry]
J --> K2[buildCostMeterConfigJSON: default table + per-provider prices]
K2 --> P
H --> M[mergeGuardrails: union allowlist, OR redact]
M --> N[applyAccountCollectionControls account toggle = SOLE capture control]
N --> O[marshalGuardrailConfig]
@@ -60,6 +66,84 @@ flowchart TD
R --> T[accountManager.UpdateAccountPeers — fans synth ACLs into network map]
```
### LLM pricing (management is the sole authority)
**The proxy carries no price list.** Management synthesizes the entire pricing
table and ships it inside `cost_meter`'s `ConfigJSON`, so a price change reaches
the proxies as an ordinary mapping push — the chain rebuild installs a fresh
table and there is nothing to reload on the proxy side.
```mermaid
flowchart TD
A[catalog.All — PricingSurfaces x Models] --> B[buildDefaultTable + supplementalDefaults]
B --> C{AgentNetwork.PricingDefaultsFile}
C -- absent --> D[compiled-in table serves]
C -- loaded --> E[LoadFile: merge file entries WHOLE over compiled base]
E --> F[mergedTable atomic.Pointer]
D --> G[DefaultTable]
F --> G
G --> H[buildCostMeterConfigJSON — pricing.defaults]
I[types.Provider.Models operator prices] --> J[normalizePricingModelID<br/>bedrock ARN/region/version, vertex @version]
J --> K[materializeEntry: default entry as base,<br/>operator input/output verbatim,<br/>cache pointers only when non-nil]
K --> L[pricing.providers keyed by provider record ID]
H --> M[cost_meter ConfigJSON]
L --> M
G --> N[GET /catalog — applyDefaultPricing prefills dashboard rows]
O[StartReloader: mtime poll every ReloadInterval 1m] --> E
```
**Two tiers, resolved per request on the proxy** (`synthesizer_pricing.go:22-35`):
- `pricing.defaults` — surface (`openai`/`anthropic`/`bedrock`) → normalized model
id → rates. The **full** default table ships to every account: it is small
(~10 KB) and it is what keeps gateway-style providers (which enumerate no
models, so they claim every model) priced.
- `pricing.providers` — provider **record** id → normalized model id → rates,
matched against the `llm.resolved_provider_id` the router stamps. Entries are
**fully materialized here**, at synth time: `materializeEntry` starts from the
default entry for that model so cache rates the operator didn't state are
inherited, overlays operator `input`/`output` verbatim (**including an explicit
0**, which prices a self-hosted or internal endpoint as free rather than
silently reverting to list price), and overlays cache-rate **pointers only when
non-nil** — `nil` means "inherit the default", an explicit `0` means "no
discount, bill this bucket at the input rate". The proxy therefore does two map
lookups and no merging.
Same orphan rule as the router: a provider no enabled policy authorises is
unreachable, so its prices aren't shipped. Model ids are normalized with the
**same** functions the request parser uses (`NormalizeBedrockModel` /
`NormalizeVertexModel`), which is what makes the per-record lookup key compare
equal to the `llm.model` the proxy meters. Post-normalization duplicates resolve
first-occurrence-wins, matching the routing dedup order.
**`AgentNetwork.PricingDefaultsFile`** (`config.go:190-207`) lets an operator
replace default rates without a rebuild. Schema is `surface → model → rates`
(`input_per_1k`, `output_per_1k`, and optional `cached_input_per_1k` /
`cache_read_per_1k` / `cache_creation_per_1k`). Semantics:
- A **relative** path resolves against `<Datadir>`, so a bare filename lands
alongside the store. Empty config probes `<Datadir>/defaults_llm_pricing.yaml`.
- An **explicitly configured** path is *required to load*: a typo or malformed
file fails startup, because the operator believes those rates are live. The
conventional probe is optional — an absent file just serves compiled-in
defaults, and the path stays watched in case it appears later.
- File entries **replace** the compiled-in entry for the same (surface, model)
**whole** — they are not field-merged, so an entry must repeat the cache rates
it wants to keep. Everything the file doesn't mention keeps built-in rates.
- Unknown YAML fields are rejected (`KnownFields(true)`) and every rate must be
finite and non-negative — the same constraints the HTTP API enforces on
operator per-provider prices.
- Reload is an mtime poll (`ReloadInterval`, 1 min) and is **lenient at runtime**:
a parse error keeps the previous table, a deleted file reverts to compiled-in
defaults. A mid-edit save can never take pricing down.
The live table feeds **both** consumers, which is what keeps them consistent: the
synthesizer (what proxies actually bill with) and `GET /api/agent-network/catalog`
via `applyDefaultPricing` (what the dashboard's model-row prices prefill with).
`defaults_llm_pricing.example.yaml` is generated from the compiled-in table
(`go generate ./management/internals/modules/agentnetwork/pricing`) and
golden-tested, so operators start from a file matching the built-in rates exactly.
### Budget rule resolution (min-wins, group+user bound)
```mermaid
@@ -124,7 +208,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
| on_request | 3 | `llm_identity_inject` | `{"providers":[{provider_id, header_pair?, json_metadata?, extra_headers?}]}` | **true** |
| on_request | 4 | `llm_guardrail` | `{"provider_allowlists"?: {providerID: []model}, "prompt_capture":{enabled,redact_pii}}` | |
| on_response | 5 | `llm_limit_record` | `{}` (runs LAST at runtime) | |
| on_response | 6 | `cost_meter` | `{}` | |
| on_response | 6 | `cost_meter` | `{"pricing":{"defaults":{surface:{model:rates}},"providers"?:{providerRecordID:{model:rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}` | |
| on_response | 7 | `llm_response_parser` | `{"capture_completion": <bool>, "redact_pii"?: true}` | |
- **Synthesized service shape** (`synthesizer.go:739`): `Mode=HTTP`, `Private=true`, `Domain=<subdomain>.<cluster>`, `AccessGroups=unionSourceGroups(enabledPolicies)`, one `TargetTypeCluster` target with `Host=noop.invalid:443` (router rewrites per request), `Options.{DirectUpstream,AgentNetwork}=true`, `DisableAccessLog=!settings.EnableLogCollection`, `CaptureMax{Req,Resp}Bytes=1<<20`, `CaptureContentTypes=["application/json","text/event-stream"]`.
@@ -139,6 +223,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Orphan providers (no enabled policy authorises them) NEVER reach the router** (`synthesizer.go:351-357`); skipped from `identity_inject` for symmetry.
- **Provider creation refuses empty `api_key`** (`manager.go:175`); **deletion refuses while any policy still references it** (`manager.go:265-273`).
- **Session keypair stability across provider edits** (`manager.go:226-228`) — server-managed, copied through every `UpdateProvider`, never API-surfaced.
- **Management is the sole pricing authority.** The proxy has no embedded price list, so an account whose `cost_meter` config carries no `pricing` block bills **nothing** (`cost.skipped=unknown_model`, $0) rather than falling back to stale built-ins. The top-level `pricing` wrapper is also the feature-detection signal in both directions: an old proxy ignores it as an unknown field, and a new proxy reads its absence as "old management".
- **Per-provider prices are materialized at synth time, not merged on the proxy** (`synthesizer_pricing.go:114-131`). A per-record entry is always complete, so the proxy's lookup is per-record-then-defaults with no field-level fallback between tiers.
- **An explicit operator price of `0` prices the model as free** — it must not be treated as "unset" and reverted to list price (`synthesizer_pricing.go:49-54`). Only *cache*-rate fields distinguish unset from zero, via `*float64`.
- **Pricing model ids are normalized with the same functions the request parser uses** (`normalizePricingModelID`). If the two ever diverge, per-record prices silently stop matching and every request falls through to surface defaults.
- **The default table's coverage is structural, not curated.** It is derived from the catalog via each provider's `PricingSurfaces`; `TestDefaultTable_CoversEveryCatalogModel` fails on an unpriced catalog model and `TestDefaultTable_NoConflictingContributions` fails if two providers contribute the same (surface, model) at different rates.
- **A pricing-defaults file failure is fatal only at startup, and only for an explicitly configured path.** Runtime reload failures keep the previous table; a deleted file reverts to compiled-in defaults (`pricing/override.go:62-81, 113-148`).
## Things to scrutinize
@@ -176,10 +266,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Capture-pointer semantics (restated):** non-agent-network callers see no field → legacy nil-default emit, identical to pre-PR. Agent-network targets always carry an explicit `capture_*` value.
- **`TestSynthesizeServices_HappyPath` was updated:** request-parser config moved from `{}` to `{"capture_prompt":false}` (`synthesizer_test.go:174`). External snapshot tests against synth output need updating.
- **`MergedGuardrails` retains zeroed `TokenLimits`/`Budget`/`Retention`** even though `Policy.Limits` carries the real values now; `llm_limit_check` is the authoritative enforcement. Comment at `synthesizer.go:940-948` calls this out.
- **`cost_meter`'s `pricing` block is version-skew-safe in both directions.** A proxy predating config-delivered pricing ignores the field as unknown JSON (it previously priced from its own embedded table, so it keeps billing — at its own rates, which is the skew to be aware of during a rolling upgrade). A current proxy paired with old management sees no `pricing` block, logs one warning at chain-build time, and records `cost.skipped=unknown_model` — token counting and cap enforcement are unaffected, only the USD annotation goes to $0.
### Performance
- **`SynthesizeServices` runs on every controller tick / mutation reconcile.** Cost: 4 store reads + optional per-provider keypair backfill. Sort + index + merge are O(N log N) / O(P × G); dominant cost is JSON marshalling. No nested loops escape these dimensions.
- **The full default pricing table is marshalled into every account's `cost_meter` config on every synth** (~10 KB serialized). This is a deliberate trade: it keeps gateway-style providers priced for every catalog model, and it is the largest single contributor to the synth JSON. `DefaultTable()` itself is a pointer load (or a `sync.Once`-built map) — the cost is the marshal, not the build.
- **`reconcile.diffMappings` is O(N + M)** with N=M=1 per account today — effectively constant.
- **`SynthesizeServicesForCluster`** (`synthesizer.go:71`) walks every account on a cluster; per-account failures are **swallowed** (`synthesizer.go:91-93`) so a single misconfigured account doesn't drop the cluster. Runs per proxy reconnect.
@@ -188,6 +280,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Activity codes:** `AgentNetwork{Provider,Policy,Guardrail,BudgetRule}{Created,Updated,Deleted}`; `AgentNetworkSettingsUpdated` with `log_collection/prompt_collection/redact_pii` payload (`manager.go:567-571`). **No activity code for `SelectPolicyForRequest` denies** — surfaced via proxy access log only (likely intentional given volume).
- **Deny codes** namespaced: `llm_policy.{token,budget}_cap_exceeded`, `llm_account.{token,budget}_cap_exceeded` (`policyselect.go:18-26`).
- **Reconcile failures are logged at warn and swallowed** (`reconcile.go:42-44`). Persistent synth failures (e.g. unknown catalog id) silently keep the proxy out of sync — consider a manager-level synth-health surface if this becomes a support burden.
- **Pricing-file lifecycle logs at info** (load, reload, revert-to-built-ins) and **at warn** for a runtime reload failure; the mtime check itself is `Debugf`. There is no metric on reload failures, so an operator who breaks the file mid-flight keeps billing at the previous table with only a log line to show it (`pricing/override.go:113-148`).
## Test coverage
@@ -198,6 +291,9 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
| `synthesizer_guardrail_realstore_test.go` | `PromptCaptureAccountIsSoleControl`; `PromptCaptureFlowsWhenAccountOptsIn`; `AccountRedactWithoutGuardrailRedact`; `NoGuardrail_CaptureOff`. |
| `synthesizer_log_collection_realstore_test.go` | `LogCollection{Off_SuppressesAccessLog,On_PermitsAccessLog}` — verifies `DisableAccessLog` propagation through `ToProtoMapping`. |
| `synthesizer_parser_redact_realstore_test.go` | **Capture-pointer regression suite:** `ParserConfigsCarryRedactPii`; `ParserConfigsSuppressCaptureWhenLogCollectionOnly` (log=on/prompt=off ⇒ both capture flags false); `ParserConfigsOmitRedactPiiWhenOff`. |
| `synthesizer_pricing_test.go` | `BuildCostMeterConfig_{BedrockModelNormalization,CacheRateNilVsZero,OrphanAndGatewayProviders}` — the per-record tier's three load-bearing rules: keys normalized like the parser's, `nil` cache pointer inherits vs explicit `0` bills at input rate, and orphan / gateway (empty `Models`) providers ship no per-record entry. |
| `pricing/defaults_test.go` | `DefaultTable_{CoversEveryCatalogModel,NoConflictingContributions,AllRatesFiniteNonNegative,PinnedRates}`; `LookupDefault_SurfaceOrder`. Catalog-derived coverage + rate sanity are structural, not curated. |
| `pricing/override_test.go` | `LoadFile_{MergesOverCompiledDefaults,MissingPath,RejectsInvalid}`; `Reload_LifeCycle` (mtime detect, parse error keeps previous, delete reverts to built-ins); `ExampleYAML_InSyncWithBuiltins` golden. |
| `policyselect_test.go` | Mock-store: `NoApplicablePolicies`; `AllowWithLowestGroupAttribution`; `LargerPoolWinsAcrossUsageLevels`; `StaysOnLargerPoolAfterPartialDrain`; `FallsThroughToSmallerPoolWhenLargerExhausted`; `TiebreakBy{LargerGroupPool,CreatedAt}`; `DeniesWhenAllExhausted`; `UncappedPolicyAlwaysWinsAgainstCapped`; `DisabledPolicyIgnored`; `StoreErrorPropagates`; `RejectsEmptyAccount`; `SharesGroupCounterAcrossPolicies`; `AntiFallThroughOnLowestGroup`; `BudgetOnlyExhaustionDenies`; `BudgetTighterThanTokenWins`. |
| `policyselect_realstore_test.go` | Real-sqlite regression guard: `NoApplicablePolicies`; `AllowAndLowestGroupAttribution`; `LargerPoolWins_FallsThroughWhenExhausted`; `BudgetCapDenies`; `GroupCounterSharedAcrossPolicies`; `DisabledPolicyIgnored`. |
| `policyselect_account_realstore_test.go` | Account budget rules: `AccountCeilingBindsEvenWithUncappedPolicy` (min-wins); `AccountGroupCeiling`; `AccountTargetUsersBindsOnlyThatUser`; `AccountRuleRecordsToOwnWindow`. |

View File

@@ -5,7 +5,7 @@ LLM request. The two highest-blast-radius areas are the **capture-pointer
semantics** and the **limit_check ⇒ limit_record** record-once invariant.
Sibling module: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — the SDK
adapters + pricing catalog this chain delegates to.
adapters + pricing table and cost formula this chain delegates to.
---
@@ -34,7 +34,7 @@ rewrites.
| `llm_identity_inject` | OnRequest | `llm.{resolved_provider_id,authorising_groups}`, `Input.{UserEmail,UserID,UserGroups,UserGroupNames}` | none | header strip/inject + optional body rewrite |
| `llm_guardrail` | OnRequest | `llm.{model,request_prompt_raw}` | `llm_policy.{decision,reason}`, `llm.request_prompt` | none (model allowlist deny) |
| `llm_response_parser` | OnResponse | `llm.provider`, `Input.{RespHeaders,RespBody,Status}` | `llm.{input,output,total,cached_input,cache_creation}_tokens`, `llm.response_completion` | none |
| `cost_meter` | OnResponse | `llm.{provider,model}`, token buckets | `cost.usd_total` or `cost.skipped` | pricing lookup |
| `cost_meter` | OnResponse | `llm.{provider,model,resolved_provider_id}`, token buckets | `cost.usd_{input,cached_input,cache_creation,output,total,cache}` or `cost.skipped` | none (in-memory pricing lookup) |
| `llm_limit_record` | OnResponse | `llm.{attribution_group_id,attribution_window_seconds,input_tokens,output_tokens}`, `cost.usd_total` | none | gRPC `RecordLLMUsage` |
[all_test.go:2640](../../../proxy/internal/middleware/builtin/all_test.go)
@@ -44,7 +44,7 @@ locks the ID set; adding or removing one is a conscious extension.
| File | LOC | Notes |
|---|---:|---|
| `builtin.go` | 86 | Registry + `FactoryContext` (ctx, data dir, meter, logger, mgmt client) |
| `builtin.go` | 90 | Registry + `FactoryContext` (ctx, meter, logger, mgmt client) |
| `all_test.go` | 41 | Locks the 8-ID registry surface |
| `agentnetwork_chain_integration_test.go` | 319 | Live sqlite + real gRPC bufconn; gate→recorder wire path |
| `llm_request_parser/*` | 162 / 66 / 356 | Provider detection, body parse, prompt extraction with capture-pointer gating |
@@ -53,7 +53,7 @@ locks the ID set; adding or removing one is a conscious extension.
| `llm_identity_inject/*` | 440 / 108 / 666 | HeaderPair (LiteLLM) + JSONMetadata (Portkey) + ExtraHeaders |
| `llm_guardrail/*` | 176 / 82 / 75 / 219 / 217 | Model allowlist + optional prompt capture with PII redaction |
| `llm_response_parser/*` | 258 / 222 / 43 / 433 / 169 / 111 | Buffered + SSE accumulation; AWS event-stream accumulator (`streaming_bedrock.go`) for Bedrock; capture-pointer gates completion emit |
| `cost_meter/*` | 181 / 84 / 439 | Token → USD via `proxy/internal/llm/pricing` |
| `cost_meter/*` | 236 / 98 / 586 | Token → USD via `proxy/internal/llm/pricing`; both pricing tiers arrive in the middleware config |
| `llm_limit_record/*` | 144 / 35 / 191 | Post-flight `RecordLLMUsage` (5s, debug-on-error) |
## Per-middleware
@@ -168,12 +168,46 @@ token schema.
### cost_meter
Reads `llm.provider` + `llm.model` + token buckets, looks up per-1k rate via
`pricing.Loader`, emits `cost.usd_total` or a closed-set `cost.skipped`
reason (`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
`unknown_model`). Loader's hot-reload goroutine is bound to proxy-lifetime
context via `startReloader`. **Key invariant:** provider-shape switch lives
in `pricing.Table.Cost` (sibling doc) — `cost_meter` stays provider-agnostic.
Reads `llm.provider` + `llm.model` + token buckets, looks up the per-1k rates,
and emits the full `cost.usd_*` breakdown (four per-bucket values plus the
`_total` and `_cache` aggregates) or a closed-set `cost.skipped` reason
(`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
`unknown_model`).
**Management owns pricing.** The proxy carries no embedded price list: the whole
table arrives in this middleware's `ConfigJSON` as
`{pricing: {defaults, providers}}`, synthesized by management from the catalog
plus the operator's stored per-provider prices
([factory.go:1334](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
Both tiers are validated by `pricing.NewTable` / `pricing.NewEntries` at
construction, so a non-finite or negative rate fails the chain build. A price
change is an ordinary mapping push — the chain rebuild yields a fresh instance
over a fresh immutable table, so there is no data dir, no pricing file, no
reload goroutine, and nothing to invalidate.
**Two-tier lookup**
([middleware.go:165183](../../../proxy/internal/middleware/builtin/cost_meter/middleware.go)):
1. **Per-provider-record** — the operator's stored price for the route that
actually served the request, keyed by the `llm.resolved_provider_id` that
`llm_router` stamped on the allow path, then by normalized model id. Entries
arrive fully materialized (management folds default cache rates in at synth
time), so there is no merging here. Absent metadata — no router in the chain
— skips this tier.
2. **Surface defaults** — the catalog-derived table keyed by `llm.provider`
(`openai`/`anthropic`/`bedrock`). This is also what prices gateway-style
providers, which enumerate no models and therefore get no per-record entry.
**Backward compatibility:** a config with no `pricing` block means management
predates config-delivered pricing. The factory logs one warning at build time
and the instance records `cost.skipped=unknown_model` ($0) for every request
rather than falling back to a stale built-in price list
([factory.go:5560](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
**Key invariant:** the provider-shape switch lives in `pricing.EntryCosts`
(sibling doc) and is selected by the **surface**, not by which tier the entry
came from — `cost_meter` stays provider-agnostic, and a per-record override on
an Anthropic route still bills its cache buckets additively.
### llm_limit_record
@@ -246,12 +280,14 @@ no mocks. Tests: `TestChain_AllowPath_StampsAttributionAndRecordsCounter`
| `llm_identity_inject` | `{providers: [{provider_id, header_pair?|json_metadata?, extra_headers?}]}` |
| `llm_guardrail` | `{provider_allowlists: {providerID: []string}, prompt_capture: {enabled, redact_pii}}` — allowlist keyed by resolved provider id; a provider absent from the map is unrestricted (fail-closed backstop; authoritative per-policy/group check is management's `CheckLLMPolicyLimits`) |
| `llm_response_parser` | `{redact_pii?, capture_completion?: *bool}` |
| `cost_meter` | `{pricing_path?}` (basename inside data-dir; defaults `pricing.yaml`) |
| `cost_meter` | `{pricing: {defaults: {surface: {model: rates}}, providers: {providerRecordID: {model: rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}`. A missing `pricing` key means "management predates config-delivered pricing": every request records `cost.skipped=unknown_model` |
| `llm_limit_record` | `{}` — same pattern as `llm_limit_check` |
All factories accept empty / null / `{}` / whitespace as zero-value config;
only structurally invalid JSON is rejected so misconfig surfaces at chain
build time.
build time. `cost_meter` adds a semantic check on top of that: a `pricing`
block carrying a negative or non-finite rate fails the build too, rather than
mispricing live traffic.
## Invariants
@@ -320,10 +356,11 @@ non-object `metadata` field
— header path still attributes, but body-level tag-budget enforcement
doesn't run for that request.
**Concurrency.** `cost_meter` shares a `pricing.Loader` via
`atomic.Pointer[Table]`; readers always see a consistent table. Every
middleware is a stateless value receiver. Integration test uses real bufconn
gRPC — race detector is the meaningful bar.
**Concurrency.** `cost_meter`'s two pricing tables are built once from the
middleware config and never mutated, so the lookup path needs no lock or atomic
swap — a price change replaces the whole instance. Every middleware is
otherwise a stateless value receiver. Integration test uses real bufconn gRPC —
race detector is the meaningful bar.
**Perf.** Hot path is `lookupKV` linear scan over <10 KVs; `cost_meter.Cost`
is O(1); SSE accumulation is single-pass. No map allocation per call.
@@ -349,13 +386,13 @@ counter accuracy.
| `llm_guardrail/redact_test.go` | 15 | Email, SSN, phone (E.164 + NA), bearer, IPv4; fixture-driven |
| `llm_response_parser/middleware_test.go` | 18 | Buffered OAI+Anthro, capture-pointer, redact, truncation |
| `llm_response_parser/streaming_test.go` | 7 | OAI usage frame, Anthro message_delta, truncated body best-effort |
| `cost_meter/middleware_test.go` | 17 | Each skip reason, provider-shape, pricing loader integration |
| `cost_meter/middleware_test.go` | 22 | Each skip reason, provider-shape formulas, config-delivered defaults, per-record-beats-defaults + miss-falls-back, per-record uses surface formula, nil-pricing skips everything, invalid-rate rejection |
| `llm_limit_record/middleware_test.go` | 7 | Skip-on-no-signal, skip-on-missing-attribution, RPC failure swallowed |
## Cross-references
- Sibling: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — SDK adapters
+ SSE framer + pricing loader.
+ SSE framer + pricing table and cost formula.
- Path-routed providers (Vertex AI + Bedrock), `keyfile::` credential, GCP
token minting, `/bedrock` prefix:
[50-path-routed-providers.md](./50-path-routed-providers.md).

View File

@@ -9,7 +9,7 @@ pricing table's per-provider cost formula is the highest-leverage place a
small bug would silently mis-bill operators.
Sibling module: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
— the 8 middlewares that consume this package's parsers + pricing loader.
— the 8 middlewares that consume this package's parsers + pricing table.
---
@@ -24,8 +24,9 @@ proxy-framework dependencies:
- `openai.go` / `anthropic.go` / `bedrock.go` — per-provider `Parser` impls.
- `sse.go` — SSE scanner (`Scanner`, `Event`, `NewScanner`).
- `errors.go` — sentinels callers branch on with `errors.Is`.
- `pricing/`embedded-default + hot-reload override table with
symlink-safe Unix loader (build-tagged stub elsewhere).
- `pricing/`immutable pricing table + the per-surface cost formula. The
rates themselves come from management inside `cost_meter`'s middleware
config; this package holds no price list and reads no files.
- `fixtures/` — captured request/response/stream bodies the tests replay.
The package carries zero proxy-framework dependencies so the same parsers can
@@ -47,12 +48,9 @@ be reused later by a WASM adapter
| `sse_test.go` | 175 | 12 tests; fixture replay + multiline + size limits |
| `parser_test.go` | 53 | `Parsers()`, `DetectParser`, provider enum values |
| `errors.go` | 31 | 6 sentinels: `Err{Unknown,Unsupported}Provider/Model`, `Err{NotLLM,Malformed}Response`, `ErrStreamingUnsupported`, `ErrMalformedRequest` |
| `pricing/pricing.go` | 421 | `Loader`, `Table`, `Entry`; embedded defaults + atomic swap + mtime reload |
| `pricing/pricing_unix.go` | 69 | `O_NOFOLLOW` + fstat-from-FD + 1 MiB cap |
| `pricing/pricing_other.go` | 21 | Stub returning "not supported on this platform" |
| `pricing/pricing_test.go` | 432 | 21 tests — symlink rejection, reload race, path traversal, oversize |
| `pricing/defaults_pricing.yaml` | 85 | go:embed source of truth |
| `fixtures/*` | 2159 | OAI chat/responses/stream + Anthro messages/stream + pricing starter |
| `pricing/pricing.go` | 234 | `Table`, `Entry`, `EntryJSON`, `Costs`; `NewTable`/`NewEntries` validation + `EntryCosts` formula. No I/O, no reload, no embedded rates |
| `pricing/pricing_test.go` | 177 | 10 tests — provider-shape formulas, cached clamp, rate fallback, nil-safety, rate validation |
| `fixtures/*` | 2159 | OAI chat/responses/stream + Anthro messages/stream |
## Request body → parser dispatch
@@ -188,9 +186,11 @@ response leg, covering both Bedrock body shapes:
`totalTokens`). `firstNonZero` folds the two naming conventions into one
`Usage`; when Converse omits `totalTokens` the parser sums the buckets.
`ProviderName()` returns `"bedrock"` — its own `defaults_pricing.yaml` block,
keyed by the **normalised** model id (region prefix + version suffix stripped by
the request parser). `ParseResponse` returns `ErrStreamingUnsupported` for an
`ProviderName()` returns `"bedrock"` — its own pricing surface in the table
management ships, keyed by the **normalised** model id (region prefix + version
suffix stripped by the request parser; management normalises its keys the same
way at synth time so the two compare equal). `ParseResponse` returns
`ErrStreamingUnsupported` for an
AWS binary event-stream content-type (`application/vnd.amazon.eventstream`,
`isAWSEventStream`) so the caller routes to the streaming accumulator instead.
@@ -205,11 +205,34 @@ response body. Streaming accumulators live in the middleware package
([llm_response_parser/streaming.go](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go))
but use `llm.NewScanner` so the framing contract stays here.
### Pricing catalog
### Pricing table
`Table.Cost`
([pricing.go:129174](../../../proxy/internal/llm/pricing/pricing.go))
is the cost formula — most security-relevant math in this module:
**Management is the sole pricing authority.** The proxy carries no embedded
price list and reads no pricing file: the whole table arrives inside
`cost_meter`'s `ConfigJSON` on the ordinary mapping push, and a price change
is just another push — the chain rebuild constructs a fresh `Table`, so there
is nothing to reload
([pricing.go:17](../../../proxy/internal/llm/pricing/pricing.go)). The
management side of the contract (catalog defaults, the operator's stored
per-provider prices, and `AgentNetwork.PricingDefaultsFile`) is covered in the
management-side module guide; `cost_meter`'s wire shape is in
[31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md).
`EntryJSON`
([pricing.go:3645](../../../proxy/internal/llm/pricing/pricing.go)) is the
management→proxy contract — five USD-per-1k rates under `input_per_1k`,
`output_per_1k`, `cached_input_per_1k`, `cache_read_per_1k`,
`cache_creation_per_1k`. Management's `pricing.Entry` marshals the identical
names, and `EntryJSON`/`Entry` are field-identical so `NewEntries` converts by
direct struct conversion rather than field-by-field copying (a new rate can't
be silently dropped in transit).
`EntryCosts`
([pricing.go:183234](../../../proxy/internal/llm/pricing/pricing.go))
is the cost formula — most security-relevant math in this module. The
**surface** (the `llm.provider` value the request parser stamped) selects the
formula, never the tier the entry came from: a per-provider-record override on
an Anthropic route still bills its cache buckets additively.
| Provider | Formula |
|---|---|
@@ -218,7 +241,7 @@ is the cost formula — most security-relevant math in this module:
| default | `inTokens × InputPer1K + outTokens × OutputPer1K` |
`bedrock` shares the Anthropic additive-cache formula
([pricing.go:172-174](../../../proxy/internal/llm/pricing/pricing.go)):
([pricing.go:214229](../../../proxy/internal/llm/pricing/pricing.go)):
Anthropic-on-Bedrock reports the same additive cache buckets, while non-Anthropic
Bedrock models (Nova, Llama) simply report zero in those buckets so cost reduces
to `input + output`.
@@ -226,15 +249,12 @@ to `input + output`.
Each per-bucket rate falls back to `InputPer1K` when zero — operators opt in
to discounts by setting the field.
`Loader`
([pricing.go:212268](../../../proxy/internal/llm/pricing/pricing.go))
overlays an optional `pricing.yaml` from data-dir on top of the go:embed
defaults. Atomic pointer swap means readers never observe a partial update.
The mtime-poll reloader (30s default cadence) keeps the previous table on
parse failure so cost annotation never goes blank during a botched edit.
`defaults_pricing.yaml` is the source of truth for built-in pricing.
Operator overrides only carry the entries they want to change.
`Costs`
([pricing.go:143163](../../../proxy/internal/llm/pricing/pricing.go)) is the
per-request split. The four per-bucket fields are the base; `TotalUSD` and
`CacheUSD` are **derived** in `newCosts` so the aggregates can never drift from
the breakdown. `InputUSD` is always the non-cached input bucket on both
provider shapes, so input and cached-input never double-count.
## Public contracts
@@ -264,29 +284,38 @@ Order matters: `DetectFromURL` ties resolve by registration order.
`ProviderBedrock = 3`. Numeric values are persisted in nothing today but treat
them as wire-stable — new providers must take fresh numbers.
**`Pricing` lookup**
([pricing.go:129](../../../proxy/internal/llm/pricing/pricing.go)):
**`Pricing` construction + lookup**
([pricing.go:60130](../../../proxy/internal/llm/pricing/pricing.go)):
```go
func NewEntries(raw map[string]map[string]EntryJSON) (map[string]map[string]Entry, error)
func NewTable(raw map[string]map[string]EntryJSON) (*Table, error)
func (t *Table) Lookup(provider, model string) (Entry, bool)
func (t *Table) Cost(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (float64, bool)
func (t *Table) Costs(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (Costs, bool)
func EntryCosts(entry Entry, surface string, inTokens, outTokens, cachedInput, cacheCreation int64) Costs
```
Nil-safe: `t.Cost` on a nil receiver returns `(0, false)`
([pricing.go:130132](../../../proxy/internal/llm/pricing/pricing.go)).
`ok=false` means provider or model is absent from the loaded table; the caller
emits `cost.skipped=unknown_model`.
`NewTable` is the surface-keyed defaults table; `NewEntries` returns the raw
two-level map `cost_meter` uses for the per-provider-record tier (it looks up an
`Entry` directly and calls `EntryCosts`, so it needs no `Table` wrapper). Both
reject any non-finite or negative rate, so a corrupt config fails the chain
build rather than mispricing silently. Nil input yields an empty,
never-matching table.
Nil-safe: `t.Cost`/`t.Lookup` on a nil receiver returns `ok=false`
([pricing.go:9699](../../../proxy/internal/llm/pricing/pricing.go)).
`ok=false` means the surface or model is absent from the table management sent;
the caller emits `cost.skipped=unknown_model`.
## Invariants
1. **Cross-platform pricing build.** `pricing_unix.go` carries the only
functional `loadPricing` (uses `syscall.O_NOFOLLOW` and `f.Stat()` on an
open descriptor — both Unix-only). `pricing_other.go` is a build-tag
fallback that returns `"not supported on this platform"`
([pricing_other.go:1416](../../../proxy/internal/llm/pricing/pricing_other.go)).
The proxy is Linux-only in production today; a Windows port needs an
equivalent path-as-handle implementation. Reviewers building on Windows
should expect this surface to return an error at startup if an override
file is configured.
1. **The pricing package is pure and platform-independent.** No file I/O, no
`//go:embed`, no goroutines, no build tags — the rates arrive as config, so
there is nothing platform-specific left to port. Anything reintroducing a
read-from-disk path here re-splits pricing authority between management and
the proxy, which is exactly what this design removed.
2. **SSE scanner handles partial chunks.** A buffered prefix that doesn't end
in `\n\n` still yields its accumulated event before `io.EOF`
@@ -298,38 +327,45 @@ emits `cost.skipped=unknown_model`.
usage rather than aborting
([streaming.go:6873, 144150](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go)).
3. **`defaults_pricing.yaml` is the source of truth.** Compiled into the
binary via `//go:embed`
([pricing.go:2930](../../../proxy/internal/llm/pricing/pricing.go)).
`DefaultTable()` parses once and panics on parse failure
([pricing.go:4249](../../../proxy/internal/llm/pricing/pricing.go))
— by design: a broken embedded YAML must not ship to production.
3. **Management is the only source of rates.** `Table` has no constructor that
invents prices: the only way in is `NewTable`/`NewEntries` over the wire map
management sent. A missing or empty `pricing` block therefore means *no
prices at all* (`cost_meter` records `cost.skipped=unknown_model`, $0) —
never a stale built-in fallback that would silently bill list price.
4. **Loader path validation.** `resolveMiddlewareDataPath`
([pricing.go:370394](../../../proxy/internal/llm/pricing/pricing.go))
rejects absolute paths, traversal segments, and basenames that fail
`basenameRegex = ^[a-zA-Z0-9._-]+$`. The resolved path must remain
inside `baseDir` even after `filepath.Clean`. Tests:
`TestNewLoader_PathValidation`, `TestNewLoader_PathValidation_Extended`,
`TestNewLoader_SymlinkOutsideBaseDirRejected`, `TestNewLoader_SymlinkRejected`.
4. **Tables are immutable once built.** `Table.entries` is written only in
`NewEntries` and never mutated afterwards, and `cost_meter`'s `perRecord`
map is likewise build-time-only
([pricing.go:4752](../../../proxy/internal/llm/pricing/pricing.go)). This
is what makes the no-reload design safe: a price change arrives as a mapping
push that builds a new middleware instance over a new table, so concurrent
readers can't observe a half-updated price list and no atomic swap or lock
is needed on the hot path.
5. **Unix loader symlink safety.** `O_NOFOLLOW` on open, `f.Stat()` on the
open descriptor (never re-stat by path), `info.Mode().IsRegular()` check,
`io.LimitReader(f, maxPricingBytes+1)` with a final size assertion
([pricing_unix.go:2557](../../../proxy/internal/llm/pricing/pricing_unix.go)).
A mid-read symlink swap is detected because the fstat is on the original
fd. Test: `TestNewLoader_RejectsOversizedFile_FixesM4`.
5. **Rate validation happens at chain-build time, not per request.**
`NewEntries` rejects negative, NaN, and ±Inf rates field by field
([pricing.go:6083](../../../proxy/internal/llm/pricing/pricing.go)), naming
the offending surface/model/field in the error. Management enforces the same
constraints at its API boundary and in its YAML parser, so this is
defense-in-depth — but it means a corrupt push fails loudly at build instead
of producing negative costs on live traffic. Test:
`TestNewTable_ValidatesRates`.
6. **`yaml.NewDecoder(...).KnownFields(true)`**
([pricing.go:397398](../../../proxy/internal/llm/pricing/pricing.go))
rejects YAML files that carry fields not in the schema. A typo in an
operator override file fails loud instead of silently zeroing rates.
6. **New rates must be added to `Entry`, `EntryJSON`, *and* management's
`pricing.Entry` together.** `NewEntries` converts by direct struct
conversion `Entry(e)`
([pricing.go:7678](../../../proxy/internal/llm/pricing/pricing.go)), which
only compiles while the two structs stay field-identical — so the proxy half
is compiler-enforced. The management half is not: a rate added there but not
here unmarshals into nothing and prices that bucket at `InputPer1K`.
## Things to scrutinise
**Correctness.** Verify OpenAI cached-prompt clamp at
[pricing.go:147149](../../../proxy/internal/llm/pricing/pricing.go)
short-circuits before subtraction. `Anthropic.TotalTokens` sums all four
**Correctness.** Verify the OpenAI cached-prompt clamp at
[pricing.go:203206](../../../proxy/internal/llm/pricing/pricing.go)
short-circuits before subtraction. Negative token counts are clamped to zero up
front ([pricing.go:186197](../../../proxy/internal/llm/pricing/pricing.go)) so
no formula can yield a negative cost. `Anthropic.TotalTokens` sums all four
buckets (in + out + cache_read + cache_creation) — downstream dashboards
need to know this differs from `input + output`.
`OpenAIParser.ExtractPrompt` falls through `messages → input → prompt`; a
@@ -338,22 +374,27 @@ noting).
**Security.** `Scanner.maxLine = 1 MiB`; a 2 MiB single-line `data:` event
errors from `Scanner.Next` and both accumulators stop with partial usage.
Pricing file 1 MiB cap is orders of magnitude larger than realistic. Confirm
new schema additions are mirrored in both `pricingFile` and `Entry`;
`KnownFields(true)` will reject silently-typo'd operator overrides
otherwise.
Pricing is no longer file-backed, so the loader's path-traversal / symlink /
oversize surface is gone entirely — the config channel (an authenticated
mapping push from management) is now the only way rates enter the proxy, and
`NewEntries` is the validation boundary on it. A new rate added to management's
`pricing.Entry` but not to `EntryJSON` here is the remaining silent-mispricing
path (see invariant 6).
**Concurrency.** `Loader.table` is `atomic.Pointer[Table]`; readers never
block or see a torn table. `Loader.Reload` is one goroutine, cancelled via
context (`TestLoader_ReloadBackgroundLoopCancellation`). `DefaultTable()`
uses `sync.Once`. Per-call `Scanner` instances mean no shared state across
concurrent response-parser calls.
**Concurrency.** Nothing in this package is shared mutable state: tables are
built once and never written again, so `cost_meter`'s hot path is lock-free by
construction rather than by atomic swap. Per-call `Scanner` instances mean no
shared state across concurrent response-parser calls.
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1).
`Scanner.Next` is one `ReadString('\n')` per line. Pricing reload poll 30s.
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1); the
per-provider-record tier adds at most one more lookup. `Scanner.Next` is one
`ReadString('\n')` per line. No background goroutines and no per-request
allocation of pricing state.
**Observability.** Reload failures count via `metric.Int64Counter` keyed
`plugin`; warning log rate-limited at 5 min so a broken file doesn't flood.
**Observability.** A config carrying no `pricing` block logs one warning at
chain-build time (`cost_meter` factory) and then records
`cost.skipped=unknown_model` per request, so an old-management deployment is
visible in both logs and the access log rather than quietly reporting $0.
Parser errors return sentinels — middleware uses `errors.Is` to map to the
right `cost.skipped` reason.
@@ -365,7 +406,7 @@ right `cost.skipped` reason.
| `openai_test.go` | 11 | Chat Completions + Responses API + legacy `prompt`; cached-tokens subset for both naming conventions; fixture replays |
| `anthropic_test.go` | 7 | Messages + legacy `/v1/complete`; streaming REJECTED on `ParseResponse` (must use scanner); fixture replays |
| `sse_test.go` | 12 | Fixture replay both providers; multiline `data:`; CRLF; comment skip; trailing-event-without-blank-line; oversize rejection |
| `pricing/pricing_test.go` | 21 | Provider-shape switch; cached-rate fallback; cached-clamp; symlink rejection (target outside basedir + symlink to file); path validation matrix; oversize rejection; reload-keeps-previous-on-parse-error; mtime change detection; goroutine cancellation |
| `pricing/pricing_test.go` | 10 | Provider-shape switch (surface selects the formula); cached-rate + cache-read/creation fallback to `InputPer1K`; cached-clamp; negative-token clamp; nil-receiver safety; rate validation (negative / NaN / Inf rejected); nil + empty table |
**Fixtures** ([proxy/internal/llm/fixtures/](../../../proxy/internal/llm/fixtures/)):
`openai_chat_completion.json` (chat.completions with usage),
@@ -373,14 +414,15 @@ right `cost.skipped` reason.
`openai_stream.txt` (3 deltas + usage + `[DONE]`),
`anthropic_messages.json` (Messages API non-streaming),
`anthropic_stream.txt` (full 7-event sequence: message_start →
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop),
`pricing.yaml` (realistic-pricing starter for operator overrides).
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop).
No pricing fixture: the table is config-delivered, so pricing tests construct
it in-process from a wire-shape map.
## Cross-references
- Sibling: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
— the chain that calls `llm.Parsers()`, `llm.ParserByName`,
`llm.NewScanner`, `pricing.NewLoader`.
`llm.NewScanner`, `pricing.NewTable` / `pricing.NewEntries`.
- Path-routed providers (Vertex AI + Bedrock), credential syntax, and the
Bedrock AWS event-stream accumulator:
[50-path-routed-providers.md](./50-path-routed-providers.md).

View File

@@ -1,7 +1,7 @@
# proxy/runtime — translate + serve + log
> **Risk level:** High — every config push from management is translated here, and the chain runs on every HTTP request to a synth target.
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareDataDir`, `MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path.
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path. Middleware config is entirely wire-delivered — no proxy-side data dir is involved, including for LLM pricing, which management ships inside `cost_meter`'s config.
## Module boundary
@@ -114,8 +114,7 @@ At **request time** the access-log middleware stamps `CapturedData`; the auth ch
## Public contracts touched
- `proxy.Server.MiddlewareDataDir` (string) — base dir for file-backed middleware config (server.go:238-241).
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:248-250).
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:249-253). There is no `MiddlewareDataDir`: no built-in middleware reads config from disk, so `builtin.FactoryContext` carries only the proxy-lifetime context, meter, logger, and management client.
- `proxy/internal/proxy.WithMiddlewareManager(*middleware.Manager) Option` — new option on `NewReverseProxy`; nil keeps the fast path (reverseproxy.go:48-56).
- `proxy/internal/proxy.PathTarget` adds `Middlewares`, `CaptureConfig`, `AgentNetwork`, `DisableAccessLog` (servicemapping.go:27-51), all zero-default.
- `proxy/internal/proxy.CapturedData` adds `agentNetwork`, `suppressAccessLog`, `userGroupNames` behind `sync.RWMutex`; slices deep-copied (context.go:47-66, 183-258).

View File

@@ -87,9 +87,9 @@ strips the `@version` suffix from the model, and maps the publisher to a parser
surface via `vertexPublisherVendor`:
- `anthropic``llm.provider="anthropic"` → metered through the Anthropic
parser, priced under the **`anthropic`** block in `defaults_pricing.yaml`
(the parser emits the standard Anthropic provider label, so Vertex Claude
reuses first-party Anthropic prices).
parser, priced under the **`anthropic`** surface of the pricing table
management ships (the parser emits the standard Anthropic provider label, so
Vertex Claude reuses first-party Anthropic prices).
- `openai``llm.provider="openai"` (reserved; not in the catalog lineup
today).
- anything else (notably `google` / Gemini) → empty vendor → **no parser**.
@@ -104,8 +104,9 @@ is omitted from the catalog.
> Caveat: cross-region inference profiles in `eu` / `apac` carry a ~10% price
> premium that the base per-token rates do **not** model — cost annotations for
> those regions read low. Operators who need exact regional billing override
> the affected entries in `pricing.yaml`.
> those regions read low. Operators who need exact regional billing set the
> affected models' prices on the provider record, or replace the default entries
> via management's `AgentNetwork.PricingDefaultsFile`.
## AWS Bedrock (`bedrock_api`)
@@ -211,15 +212,19 @@ so a model-listing call can't be rewritten onto an upstream that would 404 it.
## Catalog ↔ pricing cross-check
Catalog prices and context windows are cross-checked against LiteLLM's
`model_prices_and_context_window.json`. The proxy's embedded
`defaults_pricing.yaml` covers **every metered first-party model** the catalog
enumerates — guarded by
`TestDefaultTable_FirstPartyModelCoverage`
([pricing/defaults_coverage_test.go](../../../proxy/internal/llm/pricing/defaults_coverage_test.go)),
which fails if a catalog model has no embedded price. Bedrock entries are keyed
by the **normalised** id the request parser emits (region prefix + version
suffix stripped). Vertex Claude carries no Bedrock-style prefix, so it prices
straight off the `anthropic` block.
`model_prices_and_context_window.json`. The **catalog is the source of default
prices**: management's `pricing.DefaultTable` folds every catalog provider's
models into the surfaces that provider declares (`PricingSurfaces`), so coverage
is structural rather than maintained in a parallel file
([pricing/defaults.go](../../../management/internals/modules/agentnetwork/pricing/defaults.go)).
`TestDefaultTable_CoversEveryCatalogModel` fails if a catalog model ends up
unpriced, and `TestDefaultTable_NoConflictingContributions` fails if two
providers contribute the same (surface, model) at different rates. Bedrock
entries are keyed by the **normalised** id the request parser emits (region
prefix + version suffix stripped) — management applies the same normalisation to
per-provider prices at synth time, so the two keys compare equal. Vertex Claude
carries no Bedrock-style prefix, so it prices straight off the `anthropic`
surface.
## Things to scrutinise
@@ -232,16 +237,17 @@ operator-misconfigured Vertex provider and unmetered Gemini traffic; verify
publishers).
**Correctness.** `normalizeBedrockModel` is the join between the wire id and the
pricing key — a model that normalises to something not in `defaults_pricing.yaml`
meters at `cost.skipped=unknown_model` rather than failing the request. The
pricing key — a model that normalises to something absent from the shipped
pricing table meters at `cost.skipped=unknown_model` rather than failing the
request. The
`/bedrock` prefix strip must run on both the parser side (so the model is
extracted) and the router side (so the upstream path is native); a regression in
either silently breaks the other.
**Metering caveats.** eu/apac cross-region Bedrock + Vertex profiles carry a
~10% premium not modelled by base pricing — flagged in both the catalog comment
and `defaults_pricing.yaml`. Operators needing exact regional billing override
the relevant entries.
~10% premium not modelled by base pricing — flagged in the catalog comment.
Operators needing exact regional billing set per-provider prices on the model
rows (or replace the default entries via `AgentNetwork.PricingDefaultsFile`).
## Cross-references

View File

@@ -6,7 +6,7 @@
"name": "NetBird GmbH",
"email": "hello@netbird.io",
"phone": "",
"description": "NetBird GmbH is a Berlin-based software company specializing in the development of open-source network security solutions. Network security is utterly complex and expensive, accessible only to companies with multi-million dollar IT budgets. In contrast, there are millions of companies left behind. Our mission is to create an advanced network and cybersecurity platform that is both easy-to-use and affordable for teams of all sizes and budgets. By leveraging the open-source strategy and technological advancements, NetBird aims to set the industry standard for connecting and securing IT infrastructure.",
"description": "NetBird GmbH is a Berlin-based software company specializing in the development of open source network security solutions. Network security is utterly complex and expensive, accessible only to companies with multi-million dollar IT budgets. In contrast, there are millions of companies left behind. Our mission is to create an advanced network and cybersecurity platform that is both easy-to-use and affordable for teams of all sizes and budgets. By leveraging the open source strategy and technological advancements, NetBird aims to set the industry standard for connecting and securing IT infrastructure.",
"webpageUrl": {
"url": "https://github.com/netbirdio"
}
@@ -15,7 +15,7 @@
{
"guid": "netbird",
"name": "NetBird",
"description": "NetBird is a configuration-free peer-to-peer private network and a centralized access control system combined in a single open-source platform. It makes it easy to create secure WireGuard-based private networks for your organization or home.",
"description": "NetBird is a configuration-free peer-to-peer private network and a centralized access control system combined in a single open source platform. It makes it easy to create secure WireGuard-based private networks for your organization or home.",
"webpageUrl": {
"url": "https://github.com/netbirdio/netbird"
},
@@ -59,7 +59,7 @@
"guid": "support-yearly",
"status": "active",
"name": "Support Open Source Development and Maintenance - Yearly",
"description": "This will help us partially cover the yearly cost of maintaining the open-source NetBird project.",
"description": "This will help us partially cover the yearly cost of maintaining the open source NetBird project.",
"amount": 100000,
"currency": "USD",
"frequency": "yearly",
@@ -72,7 +72,7 @@
"guid": "support-one-time-year",
"status": "active",
"name": "Support Open Source Development and Maintenance - One Year",
"description": "This will help us partially cover the yearly cost of maintaining the open-source NetBird project.",
"description": "This will help us partially cover the yearly cost of maintaining the open source NetBird project.",
"amount": 100000,
"currency": "USD",
"frequency": "one-time",
@@ -85,7 +85,7 @@
"guid": "support-one-time-monthly",
"status": "active",
"name": "Support Open Source Development and Maintenance - Monthly",
"description": "This will help us partially cover the monthly cost of maintaining the open-source NetBird project.",
"description": "This will help us partially cover the monthly cost of maintaining the open source NetBird project.",
"amount": 10000,
"currency": "USD",
"frequency": "monthly",
@@ -98,7 +98,7 @@
"guid": "support-monthly",
"status": "active",
"name": "Support Open Source Development and Maintenance - One Month",
"description": "This will help us partially cover the monthly cost of maintaining the open-source NetBird project.",
"description": "This will help us partially cover the monthly cost of maintaining the open source NetBird project.",
"amount": 10000,
"currency": "USD",
"frequency": "monthly",

12
go.mod
View File

@@ -19,7 +19,7 @@ require (
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/vishvananda/netlink v1.3.1
golang.org/x/crypto v0.54.0
golang.org/x/crypto v0.55.0
golang.org/x/sys v0.47.0
golang.zx2c4.com/wireguard v0.0.0-20231211153847-12269c276173
golang.zx2c4.com/wireguard/wgctrl v0.0.0-20241231184526-a9ab2273dd10
@@ -127,9 +127,9 @@ require (
go.uber.org/zap v1.27.0
goauthentik.io/api/v3 v3.2023051.3
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f
golang.org/x/mobile v0.0.0-20251113184115-a159579294ab
golang.org/x/mod v0.37.0
golang.org/x/net v0.56.0
golang.org/x/mobile v0.0.0-20260816165457-f98cc9b3c733
golang.org/x/mod v0.39.0
golang.org/x/net v0.58.0
golang.org/x/oauth2 v0.36.0
golang.org/x/sync v0.22.0
golang.org/x/term v0.45.0
@@ -313,8 +313,8 @@ require (
go.opentelemetry.io/otel/trace v1.43.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/tools v0.47.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/tools v0.49.0 // indirect
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect

24
go.sum
View File

@@ -728,13 +728,13 @@ golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1m
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20251113184115-a159579294ab h1:Iqyc+2zr7aGyLuEadIm0KRJP0Wwt+fhlXLa51Fxf1+Q=
golang.org/x/mobile v0.0.0-20251113184115-a159579294ab/go.mod h1:Eq3Nh/5pFSWug2ohiudJ1iyU59SO78QFuh4qTTN++I0=
golang.org/x/mobile v0.0.0-20260816165457-f98cc9b3c733 h1:XKMObIaAElmkdO+4SQh1iCfzwciZHJi1OblnX9BED9k=
golang.org/x/mobile v0.0.0-20260816165457-f98cc9b3c733/go.mod h1:jMwjxoDSx9jqhNaZqPnr6nnKzb7cs+Dy1Czk7wdX+R8=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
@@ -744,8 +744,8 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/mod v0.39.0 h1:UF5zwQdCRRUpHfyPwr7d4UrGiVeldIsogtzWVnczL74=
golang.org/x/mod v0.39.0/go.mod h1:bvIbwjQ0HUFFf5AKukeeYQG4ZBUG9yxQbR9aEweIwYY=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
@@ -764,8 +764,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
@@ -843,8 +843,8 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -858,8 +858,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

View File

@@ -221,9 +221,6 @@ render_override() {
# Remove this file (and config.yaml.enterprise if present) to revert.
services:
${DASHBOARD_SERVICE}:
image: \${NETBIRD_DASHBOARD_IMAGE:-ghcr.io/netbirdio/dashboard-cloud:latest}
${COMBINED_SERVICE}:
image: \${NETBIRD_SERVER_IMAGE:-ghcr.io/netbirdio/netbird-server-cloud:latest}
environment:

View File

@@ -21,6 +21,8 @@ import (
"google.golang.org/grpc/connectivity"
nbgrpc "github.com/netbirdio/netbird/client/grpc"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
"github.com/netbirdio/netbird/client/system"
"github.com/netbirdio/netbird/encryption"
"github.com/netbirdio/netbird/shared/management/domain"
@@ -62,6 +64,13 @@ type GrpcClient struct {
connStateCallbackLock sync.RWMutex
serverURL string
// netState gates the stream retry loop on OS-reported network
// availability; nil (the default) disables gating.
netState *netstate.State
// sweeper cuts the transport connections on network change; nil disables it.
sweeper *netsweep.Sweeper
// syncStreamErr holds the last Sync stream error, or nil while the stream
// is established and healthy. GetServerKey succeeds even when the peer
// cannot sync (e.g. the server returns "settings not found"), so the
@@ -111,16 +120,43 @@ func MaxRecvMsgSize() int {
return size
}
// Option configures optional GrpcClient behavior.
type Option func(*GrpcClient)
// WithNetworkState injects the OS network availability state that gates the
// stream retry loop; without it gating is disabled.
func WithNetworkState(netState *netstate.State) Option {
return func(c *GrpcClient) { c.netState = netState }
}
// WithSweeper injects the network change sweeper.
func WithSweeper(sweeper *netsweep.Sweeper) Option {
return func(c *GrpcClient) { c.sweeper = sweeper }
}
// NewClient creates a new client to Management service
func NewClient(ctx context.Context, addr string, ourPrivateKey wgtypes.Key, tlsEnabled bool) (*GrpcClient, error) {
var conn *grpc.ClientConn
func NewClient(ctx context.Context, addr string, ourPrivateKey wgtypes.Key, tlsEnabled bool, opts ...Option) (*GrpcClient, error) {
// Options apply before dialing: the sweeper must wrap the first connection too.
c := &GrpcClient{
key: ourPrivateKey,
ctx: ctx,
connStateCallbackLock: sync.RWMutex{},
serverURL: addr,
}
for _, opt := range opts {
opt(c)
}
var extraOpts []grpc.DialOption
if maxSize := MaxRecvMsgSize(); maxSize > 0 {
extraOpts = append(extraOpts, grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxSize)))
log.Infof("management gRPC max receive message size set to %d bytes", maxSize)
}
if c.sweeper != nil {
extraOpts = append(extraOpts, nbgrpc.WithSweeper(c.sweeper))
}
var conn *grpc.ClientConn
operation := func() error {
var err error
conn, err = nbgrpc.CreateConnection(ctx, addr, tlsEnabled, wsproxy.ManagementComponent, extraOpts...)
@@ -136,16 +172,9 @@ func NewClient(ctx context.Context, addr string, ourPrivateKey wgtypes.Key, tlsE
return nil, err
}
realClient := proto.NewManagementServiceClient(conn)
return &GrpcClient{
key: ourPrivateKey,
realClient: realClient,
ctx: ctx,
conn: conn,
connStateCallbackLock: sync.RWMutex{},
serverURL: addr,
}, nil
c.conn = conn
c.realClient = proto.NewManagementServiceClient(conn)
return c, nil
}
// GetServerURL returns the management server URL
@@ -206,16 +235,33 @@ func (c *GrpcClient) withMgmtStream(
ctx context.Context,
handler func(ctx context.Context, serverPubKey wgtypes.Key, backOff backoff.BackOff) error,
) error {
backOff := defaultBackoff(ctx)
backOff := c.sweeper.QuickRetryBackoff(ctx, defaultBackoff(ctx), c.netState)
operation := func() error {
log.Debugf("management connection state %v", c.conn.GetState())
connState := c.conn.GetState()
// suspend reconnect attempts while the OS reports no usable network.
// Wait only errors on a cancelled context, which means shutdown, so
// stop the loop without reporting a failure.
if waited, err := c.netState.Wait(ctx); err != nil {
log.Debugf("management connection context has been canceled while offline, this usually indicates shutdown")
return nil //nolint:nilerr // a cancelled context means shutdown, not a retryable failure
} else if waited {
backOff.Reset()
}
connState := c.conn.GetState()
log.Debugf("management connection state %v", connState)
if connState == connectivity.Shutdown {
return backoff.Permanent(fmt.Errorf("connection to management has been shut down"))
} else if !(connState == connectivity.Ready || connState == connectivity.Idle) {
}
if !(connState == connectivity.Ready || connState == connectivity.Idle) {
// A dial may already be in flight (e.g. the other stream triggered
// it after a network change); wait for it to settle and proceed if
// the channel became usable, instead of burning a backoff round on
// a successful dial. A failed dial errors out as before.
c.conn.WaitForStateChange(ctx, connState)
return fmt.Errorf("connection to management is not ready and in %s state", connState)
connState = c.conn.GetState()
if !(connState == connectivity.Ready || connState == connectivity.Idle) {
return fmt.Errorf("connection to management is not ready and in %s state", connState)
}
}
serverPubKey, err := c.getServerPublicKey()
@@ -227,7 +273,7 @@ func (c *GrpcClient) withMgmtStream(
return handler(ctx, *serverPubKey, backOff)
}
err := backoff.Retry(operation, backOff)
err := nbgrpc.Retry(ctx, operation, backOff, c.netState)
if err != nil {
log.Warnf("exiting the Management service connection retry loop due to the unrecoverable error: %s", err)
}

View File

@@ -4608,7 +4608,7 @@ components:
FleetDMMatchAttributes:
type: object
description: Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
description: Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
additionalProperties: false
properties:
disk_encryption_enabled:

View File

@@ -2876,7 +2876,7 @@ type EDRFleetDMRequest struct {
// LastSyncedInterval The devices last sync requirement interval in hours. Minimum value is 24 hours
LastSyncedInterval int `json:"last_synced_interval"`
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
MatchAttributes FleetDMMatchAttributes `json:"match_attributes"`
}
@@ -2909,7 +2909,7 @@ type EDRFleetDMResponse struct {
// LastSyncedInterval The devices last sync requirement interval in hours.
LastSyncedInterval int `json:"last_synced_interval"`
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
MatchAttributes FleetDMMatchAttributes `json:"match_attributes"`
// UpdatedAt Timestamp of when the integration was last updated.
@@ -3129,7 +3129,7 @@ type Event struct {
// EventActivityCode The string code of the activity that occurred during the event
type EventActivityCode string
// FleetDMMatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
// FleetDMMatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
type FleetDMMatchAttributes struct {
// DiskEncryptionEnabled Whether disk encryption (FileVault/BitLocker) must be enabled on the host
DiskEncryptionEnabled *bool `json:"disk_encryption_enabled,omitempty"`

View File

@@ -14,6 +14,7 @@ import (
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netsweep"
auth "github.com/netbirdio/netbird/shared/relay/auth/hmac"
"github.com/netbirdio/netbird/shared/relay/client/dialer"
netErr "github.com/netbirdio/netbird/shared/relay/client/dialer/net"
@@ -184,6 +185,10 @@ type Client struct {
// datagram-sized transport is avoided on subsequent connects. Shared via
// the manager.
transportFallback *transportFallback
// sweeper cuts the relay connection on network change; the read loop
// reports the disconnect and the guard reconnects. Shared via the manager.
sweeper *netsweep.Sweeper
// datagramFallbackTriggered guards a single fallback per connection so a
// burst of oversized datagrams triggers one reconnect, not many.
datagramFallbackTriggered atomic.Bool
@@ -393,6 +398,12 @@ func (c *Client) Close() error {
}
func (c *Client) connect(ctx context.Context) (*RelayAddr, error) {
// A sweep cancels this context, so a dial started on the old network
// aborts instead of waiting out its handshake timeout.
dial := c.sweeper.StartDial(ctx)
defer dial.Release()
ctx = dial.Ctx()
mode := transportModeFromEnv()
dialers := c.getDialers(mode)
@@ -417,12 +428,19 @@ func (c *Client) connect(ctx context.Context) (*RelayAddr, error) {
return nil, fmt.Errorf("dial via FQDN: %w", err)
}
}
c.relayConn = conn
c.datagramFallbackTriggered.Store(false)
// Read the transport off the concrete connection: the sweeper's wrapper
// embeds net.Conn only, so it does not promote Protocol().
if tc, ok := conn.(transportConn); ok {
c.transport = tc.Protocol()
}
conn, err := dial.WrapConn(conn)
if err != nil {
return nil, fmt.Errorf("register connection: %w", err)
}
c.relayConn = conn
c.datagramFallbackTriggered.Store(false)
instanceURL, err := c.handShake(ctx)
if err != nil {
cErr := conn.Close()

View File

@@ -7,9 +7,22 @@ import (
"github.com/cenkalti/backoff/v4"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netstate"
)
const defaultMaxBackoffInterval = 60 * time.Second
const (
defaultMaxBackoffInterval = 60 * time.Second
// quickReconnectBudget bounds how long a quick reconnect waits for the
// network before handing the retry over to the ticker.
quickReconnectBudget = 1500 * time.Millisecond
// verdictSettleWindow is how long an online verdict must hold before it
// is trusted: the disconnect often precedes the OS offline flag by a few
// milliseconds.
verdictSettleWindow = 200 * time.Millisecond
)
// Guard manage the reconnection tries to the Relay server in case of disconnection event.
type Guard struct {
@@ -22,14 +35,19 @@ type Guard struct {
// attempts.
maxBackoffInterval time.Duration
// netState gates reconnect attempts on OS-reported network availability;
// nil disables gating.
netState *netstate.State
// lastErr is the error from the most recent failed reconnect attempt,
// surfaced as the home relay status while disconnected.
lastErr atomic.Pointer[error]
}
// NewGuard creates a new guard for the relay client. A non-positive
// maxBackoffInterval falls back to defaultMaxBackoffInterval.
func NewGuard(sp *ServerPicker, maxBackoffInterval time.Duration) *Guard {
// maxBackoffInterval falls back to defaultMaxBackoffInterval. A nil netState
// disables network availability gating.
func NewGuard(sp *ServerPicker, maxBackoffInterval time.Duration, netState *netstate.State) *Guard {
if maxBackoffInterval <= 0 {
maxBackoffInterval = defaultMaxBackoffInterval
}
@@ -38,6 +56,7 @@ func NewGuard(sp *ServerPicker, maxBackoffInterval time.Duration) *Guard {
OnReconnected: make(chan struct{}, 1),
serverPicker: sp,
maxBackoffInterval: maxBackoffInterval,
netState: netState,
}
return g
}
@@ -70,11 +89,21 @@ func (g *Guard) StartReconnectTrys(ctx context.Context, relayClient *Client) {
// start a ticker to pick a new server
ticker := g.exponentTicker(ctx)
defer ticker.Stop()
defer func() {
ticker.Stop()
}()
for {
select {
case <-ticker.C:
// suspend reconnect attempts while the OS reports no usable network
if waited, err := g.netState.Wait(ctx); err != nil {
return
} else if waited {
ticker.Stop()
ticker = g.exponentTicker(ctx)
continue
}
if err := g.retry(ctx); err != nil {
log.Errorf("failed to pick new Relay server: %s", err)
g.setLastError(err)
@@ -100,7 +129,12 @@ func (g *Guard) tryToQuickReconnect(parentCtx context.Context, rc *Client) bool
return false
}
if cancelled := waiteBeforeRetry(parentCtx); !cancelled {
if ok := g.waitForNetwork(parentCtx); !ok {
return false
}
// Still offline after the budget: leave the retry to the ticker.
if !g.netState.IsOnline() {
return false
}
@@ -166,14 +200,40 @@ func (g *Guard) exponentTicker(ctx context.Context) *backoff.Ticker {
return backoff.NewTicker(bo)
}
func waiteBeforeRetry(ctx context.Context) bool {
timer := time.NewTimer(1500 * time.Millisecond)
defer timer.Stop()
// waitForNetwork waits out the settle window while online, or waits for the
// network to return while offline, within the budget. Returns false when ctx
// is cancelled. Without an injected netState it degrades to a fixed
// budget-long sleep, the pre-netstate behavior.
func (g *Guard) waitForNetwork(ctx context.Context) bool {
budget := time.NewTimer(quickReconnectBudget)
defer budget.Stop()
select {
case <-timer.C:
return true
case <-ctx.Done():
return false
settleWindow := verdictSettleWindow
if g.netState == nil {
settleWindow = quickReconnectBudget
}
settle := time.NewTimer(settleWindow)
defer settle.Stop()
for {
// Channel first, flag second: a flip in between still fires the channel.
changedCh := g.netState.Changed()
if g.netState.IsOnline() {
select {
case <-settle.C:
return true
case <-changedCh:
continue
case <-ctx.Done():
return false
}
}
select {
case <-budget.C:
return true
case <-changedCh:
case <-ctx.Done():
return false
}
}
}

View File

@@ -12,6 +12,8 @@ import (
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
relayAuth "github.com/netbirdio/netbird/shared/relay/auth/hmac"
)
@@ -65,6 +67,17 @@ func WithMaxBackoffInterval(d time.Duration) ManagerOption {
return func(m *Manager) { m.maxBackoffInterval = d }
}
// WithNetworkState injects the OS network availability state that gates the
// reconnect guard; without it reconnect attempts are not gated.
func WithNetworkState(netState *netstate.State) ManagerOption {
return func(m *Manager) { m.netState = netState }
}
// WithSweeper injects the network change sweeper.
func WithSweeper(sweeper *netsweep.Sweeper) ManagerOption {
return func(m *Manager) { m.sweeper = sweeper }
}
// Manager is a manager for the relay client instances. It establishes one persistent connection to the given relay URL
// and automatically reconnect to them in case disconnection.
// The manager also manage temporary relay connection. If a client wants to communicate with a client on a
@@ -92,6 +105,8 @@ type Manager struct {
mtu uint16
maxBackoffInterval time.Duration
netState *netstate.State
sweeper *netsweep.Sweeper
cleanupInterval time.Duration
keepUnusedServerTime time.Duration
@@ -128,8 +143,9 @@ func NewManager(ctx context.Context, serverURLs []string, peerID string, mtu uin
for _, opt := range opts {
opt(m)
}
m.serverPicker.Sweeper = m.sweeper
m.serverPicker.ServerURLs.Store(serverURLs)
m.reconnectGuard = NewGuard(m.serverPicker, m.maxBackoffInterval)
m.reconnectGuard = NewGuard(m.serverPicker, m.maxBackoffInterval, m.netState)
return m
}
@@ -354,6 +370,7 @@ func (m *Manager) openConnVia(ctx context.Context, serverAddress, peerKey string
relayClient := NewClientWithServerIP(serverAddress, serverIP, m.tokenStore, m.peerID, m.mtu)
relayClient.SetTransportFallback(m.transportFallback)
relayClient.sweeper = m.sweeper
err := relayClient.Connect(m.ctx)
if err != nil {
rt.Lock()

View File

@@ -9,6 +9,7 @@ import (
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/client/netsweep"
auth "github.com/netbirdio/netbird/shared/relay/auth/hmac"
)
@@ -30,6 +31,7 @@ type ServerPicker struct {
MTU uint16
ConnectionTimeout time.Duration
TransportFallback *transportFallback
Sweeper *netsweep.Sweeper
}
func (sp *ServerPicker) PickServer(parentCtx context.Context) (*Client, error) {
@@ -73,6 +75,7 @@ func (sp *ServerPicker) startConnection(ctx context.Context, resultChan chan con
log.Infof("try to connecting to relay server: %s", url)
relayClient := NewClient(url, sp.TokenStore, sp.PeerID, sp.MTU)
relayClient.SetTransportFallback(sp.TransportFallback)
relayClient.sweeper = sp.Sweeper
err := relayClient.Connect(ctx)
resultChan <- connResult{
RelayClient: relayClient,

View File

@@ -19,6 +19,8 @@ import (
"google.golang.org/grpc/status"
nbgrpc "github.com/netbirdio/netbird/client/grpc"
"github.com/netbirdio/netbird/client/netstate"
"github.com/netbirdio/netbird/client/netsweep"
"github.com/netbirdio/netbird/encryption"
"github.com/netbirdio/netbird/shared/management/client"
"github.com/netbirdio/netbird/shared/signal/proto"
@@ -65,6 +67,13 @@ type GrpcClient struct {
connStateCallback ConnStateNotifier
connStateCallbackLock sync.RWMutex
// netState gates the Receive retry loop on OS-reported network
// availability; nil (the default) disables gating.
netState *netstate.State
// sweeper cuts the transport connections on network change; nil disables it.
sweeper *netsweep.Sweeper
onReconnectedListenerFn func()
decryptionWorker *Worker
@@ -88,13 +97,43 @@ type GrpcClient struct {
watchdogWg sync.WaitGroup
}
// NewClient creates a new Signal client
func NewClient(ctx context.Context, addr string, key wgtypes.Key, tlsEnabled bool) (*GrpcClient, error) {
var conn *grpc.ClientConn
// Option configures optional GrpcClient behavior.
type Option func(*GrpcClient)
// WithNetworkState injects the OS network availability state that gates the
// Receive retry loop; without it gating is disabled.
func WithNetworkState(netState *netstate.State) Option {
return func(c *GrpcClient) { c.netState = netState }
}
// WithSweeper injects the network change sweeper.
func WithSweeper(sweeper *netsweep.Sweeper) Option {
return func(c *GrpcClient) { c.sweeper = sweeper }
}
// NewClient creates a new Signal client
func NewClient(ctx context.Context, addr string, key wgtypes.Key, tlsEnabled bool, opts ...Option) (*GrpcClient, error) {
// Options apply before dialing: the sweeper must wrap the first connection too.
c := &GrpcClient{
ctx: ctx,
key: key,
mux: sync.Mutex{},
status: StreamDisconnected,
connStateCallbackLock: sync.RWMutex{},
}
for _, opt := range opts {
opt(c)
}
var extraOpts []grpc.DialOption
if c.sweeper != nil {
extraOpts = append(extraOpts, nbgrpc.WithSweeper(c.sweeper))
}
var conn *grpc.ClientConn
operation := func() error {
var err error
conn, err = nbgrpc.CreateConnection(ctx, addr, tlsEnabled, wsproxy.SignalComponent)
conn, err = nbgrpc.CreateConnection(ctx, addr, tlsEnabled, wsproxy.SignalComponent, extraOpts...)
if err != nil {
return fmt.Errorf("create connection: %w", err)
}
@@ -109,15 +148,9 @@ func NewClient(ctx context.Context, addr string, key wgtypes.Key, tlsEnabled boo
log.Debugf("connected to Signal Service: %v", conn.Target())
return &GrpcClient{
realClient: proto.NewSignalExchangeClient(conn),
ctx: ctx,
signalConn: conn,
key: key,
mux: sync.Mutex{},
status: StreamDisconnected,
connStateCallbackLock: sync.RWMutex{},
}, nil
c.signalConn = conn
c.realClient = proto.NewSignalExchangeClient(conn)
return c, nil
}
func (c *GrpcClient) StreamConnected() bool {
@@ -165,19 +198,36 @@ func defaultBackoff(ctx context.Context) backoff.BackOff {
// The connection retry logic will try to reconnect for 30 min and if wasn't successful will propagate the error to the function caller.
func (c *GrpcClient) Receive(ctx context.Context, msgHandler func(msg *proto.Message) error) error {
var backOff = defaultBackoff(ctx)
backOff := c.sweeper.QuickRetryBackoff(ctx, defaultBackoff(ctx), c.netState)
operation := func() error {
// suspend reconnect attempts while the OS reports no usable network.
// Wait only errors on a cancelled context, which means shutdown, so
// stop the loop without reporting a failure.
if waited, err := c.netState.Wait(ctx); err != nil {
log.Debugf("signal connection context has been canceled while offline, this usually indicates shutdown")
return nil
} else if waited {
backOff.Reset()
}
c.notifyStreamDisconnected()
log.Debugf("signal connection state %v", c.signalConn.GetState())
connState := c.signalConn.GetState()
log.Debugf("signal connection state %v", connState)
if connState == connectivity.Shutdown {
return backoff.Permanent(fmt.Errorf("connection to signal has been shut down"))
} else if !(connState == connectivity.Ready || connState == connectivity.Idle) {
}
if !(connState == connectivity.Ready || connState == connectivity.Idle) {
// A dial may already be in flight (e.g. triggered by another RPC
// after a network change); wait for it to settle and proceed if
// the channel became usable, instead of burning a backoff round on
// a successful dial. A failed dial errors out as before.
c.signalConn.WaitForStateChange(ctx, connState)
return fmt.Errorf("connection to signal is not ready and in %s state", connState)
connState = c.signalConn.GetState()
if !(connState == connectivity.Ready || connState == connectivity.Idle) {
return fmt.Errorf("connection to signal is not ready and in %s state", connState)
}
}
// connect to Signal stream identifying ourselves with a public WireGuard key
@@ -231,7 +281,7 @@ func (c *GrpcClient) Receive(ctx context.Context, msgHandler func(msg *proto.Mes
return nil
}
err := backoff.Retry(operation, backOff)
err := nbgrpc.Retry(ctx, operation, backOff, c.netState)
if err != nil {
log.Errorf("exiting the Signal service connection retry loop due to the unrecoverable error: %v", err)
return err