[management] Scope the change to the private-capability check

The PR grew past its purpose. What it needs to do is refuse to bootstrap an
agent network endpoint onto a cluster that cannot serve it, which is the
private capability check on the picked cluster. Everything that accreted
around it — canonicalising proxy addresses at connect, refusing another
account's cluster or a host another account pinned, withdrawing a claim
lost to a concurrent one, folding casing on migrated settings rows — is
security work in its own right and moves to follow-up PRs, where each can
be reviewed against its own threat rather than as a rider on this one.

This restores main's version of every file outside that purpose and reduces
the validation to: a cluster the account can see must have a live embedded
proxy, and a cluster management holds no row for stays pinnable
(address-first). The e2e test and the fixture seeds are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sa3DsBDP3VciAi4PPG17L6
This commit is contained in:
mlsmaycon
2026-09-12 16:08:07 +00:00
co-authored by Claude Fable 5.1
parent 4ed71f8987
commit 5502ea08ac
17 changed files with 41 additions and 1323 deletions
@@ -1030,24 +1030,13 @@ func (m *managerImpl) CreateSettings(ctx context.Context, userID string, setting
// bootstrapSelfAddressed claims the given hostname as the account's endpoint,
// served only by a proxy declaring exactly that address (Domain ==
// ProxyAddress). The domain unique index arbitrates between pins; ownership
// against another account's proxy is asked the same way as for a labeled pin,
// because the hostname lands in proxy_address, which is what a proxy
// registration is refused on when another account holds it there.
// ProxyAddress). The domain unique index is the arbiter of availability.
func (m *managerImpl) bootstrapSelfAddressed(ctx context.Context, settings *types.Settings, endpoint string) error {
hostname, err := types.NormalizeHostname(endpoint)
if err != nil {
return status.Errorf(status.InvalidArgument, "invalid endpoint: %s", err)
}
foreign, err := m.store.HasForeignAccountProxyAtHost(ctx, hostname, settings.AccountID)
if err != nil {
return fmt.Errorf("check proxy cluster ownership: %w", err)
}
if foreign {
return errForeignCluster(hostname)
}
settings.Domain = hostname
settings.ProxyAddress = hostname
if err := m.store.CreateAgentNetworkSettings(ctx, settings); err != nil {
@@ -1062,7 +1051,7 @@ func (m *managerImpl) bootstrapSelfAddressed(ctx context.Context, settings *type
}
return fmt.Errorf("create agent network settings: %w", err)
}
return m.confirmGatewayClusterOwnership(ctx, settings)
return nil
}
// validateGatewayCluster rejects a labeled bootstrap pinned to a cluster that
@@ -1078,10 +1067,10 @@ func (m *managerImpl) bootstrapSelfAddressed(ctx context.Context, settings *type
// the `private` capability, the same flag the dashboard renders as
// supports_private when it gates NetBird-only services.
//
// Without this check the bootstrap happily pins to any hostname the caller
// names, including a cluster the account cannot use or one with no embedded
// proxy — and the endpoint it allocates is immutable, so the account is left
// with a dead gateway that only a DeleteSettings/re-bootstrap can undo.
// Without this check the bootstrap happily pins to any cluster the caller
// names, including one with no embedded proxy — and the endpoint it allocates
// is immutable, so the account is left with a dead gateway that only a
// DeleteSettings/re-bootstrap can undo.
//
// Whether management knows the cluster is decided on the proxy rows
// themselves, never on how fresh their heartbeats are: a cluster's rows
@@ -1094,23 +1083,6 @@ func (m *managerImpl) bootstrapSelfAddressed(ctx context.Context, settings *type
// all: pinning ahead of a proxy's first connection is a legitimate order — the
// dedicated path claims an address the same way, before any proxy declares it.
func (m *managerImpl) validateGatewayCluster(ctx context.Context, accountID, clusterAddr string) error {
// Ownership is decided first, before anything the account's own view can
// answer. A host another account's proxy declares is refused even when
// this account has a row for it too: two accounts claiming one hostname is
// the ambiguity the connect-time conflict check exists to prevent, and the
// endpoint pinned here cannot be moved afterwards, so the ambiguous case
// has to fail closed. Asking the account's view first would skip this
// whenever the account had any row of its own, which is exactly when a
// collision is worth catching. Shared proxies are not foreign — they are
// what most accounts pin to.
foreign, err := m.store.HasForeignAccountProxyAtHost(ctx, clusterAddr, accountID)
if err != nil {
return fmt.Errorf("check proxy cluster ownership: %w", err)
}
if foreign {
return errForeignCluster(clusterAddr)
}
declared, err := m.accountClusterSpellings(ctx, accountID, clusterAddr)
if err != nil {
return err
@@ -1146,16 +1118,13 @@ func (m *managerImpl) validateGatewayCluster(ctx context.Context, accountID, clu
// host as clusterAddr. Empty means management holds no proxy row for that host
// in this account's view.
//
// Addresses are canonicalised where they are written (canonicalProxyAddress on
// the proxy-connect path), so a stored spelling normally is the normalised
// form. Identity is still compared on the normalised form rather than
// byte-equal, which costs nothing here — this is an in-memory pass over the
// account's clusters, not a query — and covers a row written before that
// landed. What comes back is the stored spelling either way, because the
// capability lookup matches cluster_address exactly and would silently find
// nothing under a spelling the store never held. The cluster listing is not
// gated on heartbeats, so this answer does not change while a cluster's
// proxies are merely offline.
// A proxy declares its cluster address as the operator spelled it, so identity
// is compared on the normalised form rather than byte-equal — an in-memory pass
// over the account's clusters, not a query. What comes back is the stored
// spelling, because the capability lookup matches cluster_address exactly and
// would silently find nothing under a spelling the store never held. The
// cluster listing is not gated on heartbeats, so this answer does not change
// while a cluster's proxies are merely offline.
func (m *managerImpl) accountClusterSpellings(ctx context.Context, accountID, clusterAddr string) ([]string, error) {
clusters, err := m.store.GetProxyClusters(ctx, accountID)
if err != nil {
@@ -1237,40 +1206,10 @@ func (m *managerImpl) bootstrapLabeled(ctx context.Context, settings *types.Sett
}
return fmt.Errorf("create agent network settings: %w", err)
}
return m.confirmGatewayClusterOwnership(ctx, settings)
}
return fmt.Errorf("allocate agent network endpoint for account %s: %d attempts exhausted", settings.AccountID, maxDomainAllocationAttempts)
}
// confirmGatewayClusterOwnership re-reads ownership once the settings row is
// committed and withdraws the row if another account's proxy now declares the
// host; see proxy.ErrClusterAddressUnavailable for why the re-read is what
// closes the race with a concurrent proxy registration. Only ownership is
// re-read: the capability check is about what the cluster can do, not who
// holds it, and does not race a claim.
func (m *managerImpl) confirmGatewayClusterOwnership(ctx context.Context, settings *types.Settings) error {
foreign, err := m.store.HasForeignAccountProxyAtHost(ctx, settings.ProxyAddress, settings.AccountID)
if err == nil && !foreign {
return nil
}
if delErr := m.store.DeleteAgentNetworkSettings(ctx, settings.AccountID); delErr != nil {
log.WithContext(ctx).Errorf("failed to withdraw agent network settings for account %s after losing the claim on %s: %v",
settings.AccountID, settings.ProxyAddress, delErr)
}
if err != nil {
return fmt.Errorf("confirm proxy cluster ownership: %w", err)
}
log.WithContext(ctx).Warnf("proxy cluster %s was claimed by another account while account %s bootstrapped onto it, withdrawing the pin",
settings.ProxyAddress, settings.AccountID)
return errForeignCluster(settings.ProxyAddress)
}
// errForeignCluster is the refusal for a cluster another account's proxy
// declares, worded the same whether it is caught before or after the insert.
func errForeignCluster(clusterAddr string) error {
return status.Errorf(status.InvalidArgument, "proxy cluster %s is not available to this account", clusterAddr)
return fmt.Errorf("allocate agent network endpoint for account %s: %d attempts exhausted", settings.AccountID, maxDomainAllocationAttempts)
}
// isUniqueConstraintError reports whether err is a database unique-constraint
@@ -2,7 +2,6 @@ package agentnetwork
import (
"context"
"errors"
"runtime"
"strings"
"testing"
@@ -36,16 +35,6 @@ type bootstrapFixture struct {
}
func newBootstrapFixture(t *testing.T) *bootstrapFixture {
t.Helper()
return newBootstrapFixtureWith(t, func(st store.Store) store.Store { return st })
}
// newBootstrapFixtureWith hands the manager the real store as seen through
// wrap, while the fixture keeps the unwrapped store for seeding and
// assertions. It exists for cases that need something to happen between two
// of the manager's store calls — a competing claim landing mid-bootstrap —
// which a real store cannot be made to do on cue.
func newBootstrapFixtureWith(t *testing.T, wrap func(store.Store) store.Store) *bootstrapFixture {
t.Helper()
if runtime.GOOS == "windows" {
t.Skip("sqlite store not properly supported on Windows yet")
@@ -66,7 +55,7 @@ func newBootstrapFixtureWith(t *testing.T, wrap func(store.Store) store.Store) *
vendor := &stubLister{}
return &bootstrapFixture{
manager: NewManager(wrap(st), perms, accounts, nil, WithModelLister(vendor)),
manager: NewManager(st, perms, accounts, nil, WithModelLister(vendor)),
store: st,
perms: perms,
vendor: vendor,
@@ -356,279 +345,6 @@ func TestCreateSettingsRequiresPrivateCluster(t *testing.T) {
assert.Error(t, err, "no row may be left behind by a rejected bootstrap")
}
// TestCreateSettingsRejectsForeignCluster pins tenant isolation on the pin: an
// account-owned (BYOP) cluster belongs to the account that runs it and is not
// one another account may hang its gateway beneath, even though it is
// private-capable. Ownership does not lapse with the heartbeat either, so the
// refusal holds while the foreign cluster is offline.
func TestCreateSettingsRejectsForeignCluster(t *testing.T) {
ctx := context.Background()
cases := map[string]time.Time{
"live": time.Now().UTC(),
"offline": time.Now().UTC().Add(-time.Hour),
}
for name, lastSeen := range cases {
t.Run(name, func(t *testing.T) {
f := newBootstrapFixture(t)
f.seedProxyAt(t, "proxy1", "account2", "byop.account2.example.com", ptrTo(true), lastSeen)
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", "byop.account2.example.com", "")
require.Error(t, err, "another account's BYOP cluster must be rejected")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type(), "rejection must be a validation error")
assert.Contains(t, err.Error(), "not available to this account",
"the error must say the cluster is not the account's to use")
})
}
}
// TestCreateSettingsRejectsHostAnotherAccountClaims pins that ownership is
// decided before the account's own view, not after it.
//
// Two accounts holding rows for one hostname is the ambiguity the connect-time
// conflict check prevents going forward and cannot see for a row written
// before addresses were canonicalized. Deciding on the account's own view
// first would skip the ownership question exactly when the account has a row
// of its own — which is when a collision is worth catching — and the endpoint
// pinned here cannot be moved afterwards.
func TestCreateSettingsRejectsHostAnotherAccountClaims(t *testing.T) {
ctx := context.Background()
f := newBootstrapFixture(t)
// account1's own row is canonical and perfectly serviceable on its own.
f.seedProxy(t, "own", "account1", "shared.example.com", ptrTo(true))
// account2 holds a legacy, non-canonical spelling of the same host.
f.seedProxy(t, "foreign", "account2", "Shared.Example.com", ptrTo(true))
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", "shared.example.com", "")
require.Error(t, err, "a host another account also claims must be refused")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type())
assert.Contains(t, err.Error(), "not available to this account")
_, err = f.store.GetAgentNetworkSettings(ctx, store.LockingStrengthNone, "account1")
assert.Error(t, err, "no row may be left behind by a rejected bootstrap")
}
// claimingStore is a store.Store on which another account's proxy registers
// at the host being pinned in the moment the settings row is written — the
// interleaving a concurrent proxy connect produces when it passes its own
// availability check before this bootstrap's row exists, so neither side's
// pre-write check sees the other.
type claimingStore struct {
store.Store
t *testing.T
claim *proxy.Proxy
claimed bool
}
func (s *claimingStore) CreateAgentNetworkSettings(ctx context.Context, settings *types.Settings) error {
if !s.claimed {
s.claimed = true
require.NoError(s.t, s.Store.SaveProxy(ctx, s.claim), "the competing claim must land")
}
return s.Store.CreateAgentNetworkSettings(ctx, settings)
}
// foreignClaim is the competing claim the race tests let land: another
// account's embedded proxy at host.
func foreignClaim(host string) *proxy.Proxy {
return &proxy.Proxy{
ID: "foreign",
ClusterAddress: host,
Status: proxy.StatusConnected,
LastSeen: time.Now().UTC(),
AccountID: ptrTo("account2"),
Capabilities: proxy.Capabilities{Private: ptrTo(true)},
}
}
// failingStore is a store.Store that fails a named call on its nth invocation,
// for the paths where the bootstrap's own bookkeeping cannot be completed:
// an ownership re-read that cannot answer, or a withdrawal that does not go
// through.
type failingStore struct {
store.Store
failOwnershipOn int
failDelete bool
ownershipCalls int
}
func (s *failingStore) HasForeignAccountProxyAtHost(ctx context.Context, host, accountID string) (bool, error) {
s.ownershipCalls++
if s.ownershipCalls == s.failOwnershipOn {
return false, errors.New("store unavailable")
}
return s.Store.HasForeignAccountProxyAtHost(ctx, host, accountID)
}
func (s *failingStore) DeleteAgentNetworkSettings(ctx context.Context, accountID string) error {
if s.failDelete {
return errors.New("delete failed")
}
return s.Store.DeleteAgentNetworkSettings(ctx, accountID)
}
// TestCreateSettingsWithdrawsPinClaimedDuringBootstrap covers the window
// between validateGatewayCluster and the insert: a foreign proxy that claims
// the host in that window is seen by the ownership re-read after the write,
// and the pin is withdrawn rather than left standing on a cluster that will
// never serve it. The refusal reads exactly as it would have had the
// pre-write check caught the claim.
func TestCreateSettingsWithdrawsPinClaimedDuringBootstrap(t *testing.T) {
ctx := context.Background()
const host = "shared.example.com"
f := newBootstrapFixtureWith(t, func(st store.Store) store.Store {
return &claimingStore{Store: st, t: t, claim: foreignClaim(host)}
})
// A shared embedded cluster, so the pre-write validation passes on its
// own merits and only the claim landing mid-bootstrap can refuse it.
f.seedEmbeddedCluster(t, host)
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", host, "")
require.Error(t, err, "a host claimed by another account mid-bootstrap must be refused")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type())
assert.Contains(t, err.Error(), "not available to this account")
_, err = f.store.GetAgentNetworkSettings(ctx, store.LockingStrengthNone, "account1")
assert.Error(t, err, "the pin written before the claim was seen must be withdrawn")
foreign, err := f.store.HasForeignAccountProxyAtHost(ctx, host, "account1")
require.NoError(t, err)
assert.True(t, foreign, "the competing claim, having landed first, keeps the host")
}
// TestCreateSettingsSelfAddressedRejectsForeignHost pins that a self-addressed
// pin is subject to the same ownership rule as a labeled one. The hostname is
// stored as proxy_address, which is exactly what a proxy registration is
// refused on when another account holds it there — so without this check any
// account could pin an endpoint onto a host another account's proxy already
// declares and lock that proxy out on its next reconnect, owning nothing.
func TestCreateSettingsSelfAddressedRejectsForeignHost(t *testing.T) {
ctx := context.Background()
f := newBootstrapFixture(t)
f.seedProxy(t, "foreign", "account2", "gw.example.com", ptrTo(true))
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", "", "gw.example.com")
require.Error(t, err, "a hostname another account's proxy declares must be refused")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type())
assert.Contains(t, err.Error(), "not available to this account")
_, err = f.store.GetAgentNetworkSettings(ctx, store.LockingStrengthNone, "account1")
assert.Error(t, err, "no row may be left behind by a rejected bootstrap")
}
// TestCreateSettingsSelfAddressedAcceptsOwnHost is the address-first order the
// self-addressed path exists for, in both directions: a hostname no proxy has
// declared, and one the account's own proxy already declares.
func TestCreateSettingsSelfAddressedAcceptsOwnHost(t *testing.T) {
ctx := context.Background()
f := newBootstrapFixture(t)
f.seedProxy(t, "own", "account1", "gw.example.com", ptrTo(true))
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
created, err := f.createSettings(ctx, "account1", "user1", "", "gw.example.com")
require.NoError(t, err, "the account's own proxy is not a competing claim")
assert.Equal(t, "gw.example.com", created.ProxyAddress)
}
// TestCreateSettingsSelfAddressedWithdrawsPinClaimedDuringBootstrap is the
// self-addressed twin of the labeled race: the competing proxy lands as the
// row is written, the re-read sees it, and the pin is withdrawn.
func TestCreateSettingsSelfAddressedWithdrawsPinClaimedDuringBootstrap(t *testing.T) {
ctx := context.Background()
const host = "gw.example.com"
f := newBootstrapFixtureWith(t, func(st store.Store) store.Store {
return &claimingStore{Store: st, t: t, claim: foreignClaim(host)}
})
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", "", host)
require.Error(t, err, "a host claimed by another account mid-bootstrap must be refused")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type())
_, err = f.store.GetAgentNetworkSettings(ctx, store.LockingStrengthNone, "account1")
assert.Error(t, err, "the pin written before the claim was seen must be withdrawn")
}
// TestCreateSettingsWithdrawsPinWhenOwnershipRecheckFails pins fail-closed on
// the bootstrap side: a re-read that cannot answer leaves no pin behind and
// surfaces the store's error rather than a validation refusal, since nothing
// established that the cluster is somebody else's.
func TestCreateSettingsWithdrawsPinWhenOwnershipRecheckFails(t *testing.T) {
ctx := context.Background()
const host = "shared.example.com"
// The first ownership call is the pre-write check and must pass; the
// second is the re-read.
f := newBootstrapFixtureWith(t, func(st store.Store) store.Store {
return &failingStore{Store: st, failOwnershipOn: 2}
})
f.seedEmbeddedCluster(t, host)
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", host, "")
require.Error(t, err)
var sErr *status.Error
assert.False(t, errors.As(err, &sErr) && sErr.Type() == status.InvalidArgument,
"an inconclusive re-read is not a validation refusal: %v", err)
assert.ErrorContains(t, err, "store unavailable", "the store's error must be the one surfaced")
_, err = f.store.GetAgentNetworkSettings(ctx, store.LockingStrengthNone, "account1")
assert.Error(t, err, "a pin that could not be confirmed must not stand")
}
// TestCreateSettingsRefusesEvenWhenWithdrawalFails pins that a lost claim is
// reported as lost whatever happens to the compensating delete: the caller
// must not be told it holds a cluster another account's proxy declares.
func TestCreateSettingsRefusesEvenWhenWithdrawalFails(t *testing.T) {
ctx := context.Background()
const host = "shared.example.com"
f := newBootstrapFixtureWith(t, func(st store.Store) store.Store {
return &failingStore{Store: &claimingStore{Store: st, t: t, claim: foreignClaim(host)}, failDelete: true}
})
f.seedEmbeddedCluster(t, host)
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", host, "")
require.Error(t, err)
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type(), "a failed withdrawal must not turn a lost claim into a held one")
assert.Contains(t, err.Error(), "not available to this account")
}
// TestCreateSettingsAcceptsSharedClusterAlongsideOwnProxy pins the other side
// of that ordering: a shared (NetBird-operated) proxy is not foreign, so
// asking the ownership question first must not refuse the cluster most
// accounts pin to.
func TestCreateSettingsAcceptsSharedClusterAlongsideOwnProxy(t *testing.T) {
ctx := context.Background()
f := newBootstrapFixture(t)
f.seedProxy(t, "shared", "", "eu.proxy.example.com", ptrTo(true))
f.seedProxy(t, "own", "account1", "eu.proxy.example.com", ptrTo(true))
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
created, err := f.createSettings(ctx, "account1", "user1", "eu.proxy.example.com", "")
require.NoError(t, err, "a shared cluster must stay pinnable")
assert.Equal(t, "eu.proxy.example.com", created.ProxyAddress)
}
// TestCreateSettingsAcceptsOwnPrivateCluster pins the BYOP happy path: the
// account's own cluster with a connected embedded proxy is a valid pin.
func TestCreateSettingsAcceptsOwnPrivateCluster(t *testing.T) {
@@ -644,13 +360,10 @@ func TestCreateSettingsAcceptsOwnPrivateCluster(t *testing.T) {
// TestCreateSettingsMatchesClusterCasing pins that a cluster spelled with
// capitals in the store is still recognised as the same cluster the normalised
// proxy_address names. Addresses are canonicalised where they are written
// (canonicalProxyAddress on the proxy-connect path), so this is the belt to
// that braces: it covers a row written before that landed, and any future
// writer that skips it. The comparison is in memory over the account's cluster
// list, so it costs nothing at the query — the capability lookup is still
// asked under the spelling the store actually holds, which is what an exact,
// indexed match needs.
// proxy_address names, in both directions: a private cluster is accepted and a
// centralised one is refused, whatever the casing. The comparison is in memory
// over the account's cluster list; the capability lookup is still asked under
// the spelling the store actually holds, which is what an exact match needs.
func TestCreateSettingsMatchesClusterCasing(t *testing.T) {
ctx := context.Background()
@@ -664,19 +377,6 @@ func TestCreateSettingsMatchesClusterCasing(t *testing.T) {
assert.Equal(t, "eu.proxy.example.com", created.ProxyAddress)
})
t.Run("foreign cluster is still foreign", func(t *testing.T) {
f := newBootstrapFixture(t)
f.seedProxy(t, "proxy1", "account2", "BYOP.Account2.Example.com", ptrTo(true))
f.expectPermission("account1", "user1", modules.AgentNetworkSettings, operations.Create, true)
_, err := f.createSettings(ctx, "account1", "user1", "byop.account2.example.com", "")
require.Error(t, err, "another account's cluster must be refused whatever its casing")
var sErr *status.Error
require.ErrorAs(t, err, &sErr)
assert.Equal(t, status.InvalidArgument, sErr.Type())
assert.Contains(t, err.Error(), "not available to this account")
})
t.Run("non-private cluster is still refused", func(t *testing.T) {
f := newBootstrapFixture(t)
f.seedProxy(t, "proxy1", "", "Central.Example.com", ptrTo(false))
@@ -2,7 +2,6 @@ package manager
import (
"context"
"fmt"
"time"
log "github.com/sirupsen/logrus"
@@ -15,7 +14,6 @@ import (
type store interface {
SaveProxy(ctx context.Context, p *proxy.Proxy) error
DisconnectProxy(ctx context.Context, proxyID, sessionID string) error
DeleteProxy(ctx context.Context, proxyID, sessionID string) error
UpdateProxyHeartbeat(ctx context.Context, p *proxy.Proxy) error
GetActiveProxyClusterAddresses(ctx context.Context) ([]string, error)
GetActiveProxyClusterAddressesForAccount(ctx context.Context, accountID string) ([]string, error)
@@ -28,7 +26,6 @@ type store interface {
GetProxyByAccountID(ctx context.Context, accountID string) (*proxy.Proxy, error)
CountProxiesByAccountID(ctx context.Context, accountID string) (int64, error)
IsClusterAddressConflicting(ctx context.Context, clusterAddress, accountID string) (bool, error)
HasGatewayPinnedByOtherAccount(ctx context.Context, host, accountID string) (bool, error)
DeleteAccountCluster(ctx context.Context, clusterAddress, accountID string) error
}
@@ -76,12 +73,6 @@ func (m *Manager) Connect(ctx context.Context, proxyID, sessionID, clusterAddres
return nil, err
}
if accountID != nil {
if err := m.confirmClusterAddressClaim(ctx, p, *accountID); err != nil {
return nil, err
}
}
log.WithContext(ctx).WithFields(log.Fields{
"proxyID": proxyID,
"sessionID": sessionID,
@@ -92,35 +83,6 @@ func (m *Manager) Connect(ctx context.Context, proxyID, sessionID, clusterAddres
return p, nil
}
// confirmClusterAddressClaim re-reads availability once the proxy's row is
// committed and withdraws the row if the claim is lost; see
// proxy.ErrClusterAddressUnavailable for why the re-read is what closes the
// race with a concurrent claim. An inconclusive re-read refuses the connect
// but only marks the row disconnected: SaveProxy upserts on the proxy ID, so
// on a reconnect the row is a claim the account already held, and a transient
// store error must not surrender it.
func (m *Manager) confirmClusterAddressClaim(ctx context.Context, p *proxy.Proxy, accountID string) error {
available, err := m.IsClusterAddressAvailable(ctx, p.ClusterAddress, accountID)
if err != nil {
if discErr := m.store.DisconnectProxy(ctx, p.ID, p.SessionID); discErr != nil {
log.WithContext(ctx).Errorf("failed to mark proxy %s session %s disconnected after an inconclusive claim check on %s: %v",
p.ID, p.SessionID, p.ClusterAddress, discErr)
}
return fmt.Errorf("confirm claim on cluster address %s: %w", p.ClusterAddress, err)
}
if available {
return nil
}
if delErr := m.store.DeleteProxy(ctx, p.ID, p.SessionID); delErr != nil {
log.WithContext(ctx).Errorf("failed to withdraw proxy %s session %s after losing the claim on %s: %v",
p.ID, p.SessionID, p.ClusterAddress, delErr)
}
log.WithContext(ctx).Warnf("cluster address %s was claimed while proxy %s registered for account %s, withdrawing its row",
p.ClusterAddress, p.ID, accountID)
return fmt.Errorf("cluster address %s: %w", p.ClusterAddress, proxy.ErrClusterAddressUnavailable)
}
// Disconnect marks a proxy as disconnected in the database.
func (m *Manager) Disconnect(ctx context.Context, proxyID, sessionID string) error {
if err := m.store.DisconnectProxy(ctx, proxyID, sessionID); err != nil {
@@ -207,36 +169,12 @@ func (m *Manager) CountAccountProxies(ctx context.Context, accountID string) (in
return m.store.CountProxiesByAccountID(ctx, accountID)
}
// IsClusterAddressAvailable reports whether the account may claim this cluster
// address.
//
// Two kinds of claim make an address unavailable, and both are checked here so
// that no caller can consult one and forget the other. A proxy row is the
// obvious one. An agent network gateway pinned to the address by another
// account is the second: that pin is immutable and is served by whichever
// proxy declares the address, so letting a proxy from a different account take
// it strands the pin — an account-scoped proxy never receives another
// account's mappings. An account claiming an address its own gateway is pinned
// to is the intended order, not a conflict: pin first, deploy the proxy after.
func (m *Manager) IsClusterAddressAvailable(ctx context.Context, clusterAddress, accountID string) (bool, error) {
conflicting, err := m.store.IsClusterAddressConflicting(ctx, clusterAddress, accountID)
if err != nil {
return false, err
}
if conflicting {
return false, nil
}
pinned, err := m.store.HasGatewayPinnedByOtherAccount(ctx, clusterAddress, accountID)
if err != nil {
return false, err
}
if pinned {
log.WithContext(ctx).Infof("cluster address %s is pinned as another account's agent network gateway, refusing claim by account %s", clusterAddress, accountID)
return false, nil
}
return true, nil
return !conflicting, nil
}
func (m *Manager) DeleteAccountCluster(ctx context.Context, clusterAddress, accountID string) error {
@@ -1,91 +0,0 @@
package manager
import (
"context"
"runtime"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/otel/metric/noop"
agentNetworkTypes "github.com/netbirdio/netbird/management/internals/modules/agentnetwork/types"
"github.com/netbirdio/netbird/management/internals/modules/reverseproxy/proxy"
nbstore "github.com/netbirdio/netbird/management/server/store"
nbtypes "github.com/netbirdio/netbird/management/server/types"
)
// newStoreBackedManager wires the manager to a real sqlite store, for the
// cases where what matters is how the store's own queries answer the
// post-write re-read — which the function-field mock cannot say.
func newStoreBackedManager(t *testing.T) (*Manager, nbstore.Store) {
t.Helper()
if runtime.GOOS == "windows" {
t.Skip("sqlite store not properly supported on Windows yet")
}
t.Setenv("NETBIRD_STORE_ENGINE", string(nbtypes.SqliteStoreEngine))
st, cleanUp, err := nbstore.NewTestStoreFromSQL(context.Background(), "", t.TempDir())
require.NoError(t, err, "test store setup must succeed")
t.Cleanup(cleanUp)
mgr, err := NewManager(st, noop.NewMeterProvider().Meter("test"))
require.NoError(t, err)
return mgr, st
}
// TestConnect_RealStore_ConfirmsOwnClaims drives the post-write re-read
// through the real queries. Every account-scoped connect now reads its own
// just-written row back, so the whole path depends on the store excluding the
// account's own claims: its own proxy row on a reconnect, and its own gateway
// pin when the account deploys a proxy at the address it pinned first.
func TestConnect_RealStore_ConfirmsOwnClaims(t *testing.T) {
ctx := context.Background()
accountID := "account1"
const host = "byop.account1.example.com"
t.Run("a reconnect keeps the proxy's own row", func(t *testing.T) {
mgr, st := newStoreBackedManager(t)
_, err := mgr.Connect(ctx, "proxy-1", "session-1", host, "10.0.0.1", &accountID, nil)
require.NoError(t, err, "first connect must succeed")
_, err = mgr.Connect(ctx, "proxy-1", "session-2", host, "10.0.0.1", &accountID, nil)
require.NoError(t, err, "a reconnect must not be refused by the row it is replacing")
rows, err := st.GetAllProxies(ctx)
require.NoError(t, err)
require.Len(t, rows, 1, "a reconnect upserts the same row")
assert.Equal(t, "session-2", rows[0].SessionID, "the row must carry the new session")
assert.Equal(t, proxy.StatusConnected, rows[0].Status)
})
t.Run("the account's own gateway pin is not a competing claim", func(t *testing.T) {
mgr, st := newStoreBackedManager(t)
settings := agentNetworkTypes.DefaultSettings(accountID)
settings.Domain = host
settings.ProxyAddress = host
require.NoError(t, st.CreateAgentNetworkSettings(ctx, settings), "seeding the account's own pin must succeed")
_, err := mgr.Connect(ctx, "proxy-1", "session-1", host, "10.0.0.1", &accountID, nil)
require.NoError(t, err, "pin first, deploy the proxy after is the documented order")
rows, err := st.GetAllProxies(ctx)
require.NoError(t, err)
assert.Len(t, rows, 1, "the proxy's row must stand next to the account's own pin")
})
t.Run("another account's gateway pin withdraws the row", func(t *testing.T) {
mgr, st := newStoreBackedManager(t)
settings := agentNetworkTypes.DefaultSettings("account2")
settings.Domain = host
settings.ProxyAddress = host
require.NoError(t, st.CreateAgentNetworkSettings(ctx, settings), "seeding the other account's pin must succeed")
_, err := mgr.Connect(ctx, "proxy-1", "session-1", host, "10.0.0.1", &accountID, nil)
require.ErrorIs(t, err, proxy.ErrClusterAddressUnavailable)
rows, err := st.GetAllProxies(ctx)
require.NoError(t, err)
assert.Empty(t, rows, "a withdrawn registration must leave no row behind")
})
}
@@ -24,9 +24,7 @@ type mockStore struct {
getProxyByAccountIDFunc func(ctx context.Context, accountID string) (*proxy.Proxy, error)
countProxiesByAccountIDFunc func(ctx context.Context, accountID string) (int64, error)
isClusterAddressConflictingFunc func(ctx context.Context, clusterAddress, accountID string) (bool, error)
hasGatewayPinnedByOtherAccountFunc func(ctx context.Context, host, accountID string) (bool, error)
deleteAccountClusterFunc func(ctx context.Context, clusterAddress, accountID string) error
deleteProxyFunc func(ctx context.Context, proxyID, sessionID string) error
}
func (m *mockStore) SaveProxy(ctx context.Context, p *proxy.Proxy) error {
@@ -41,12 +39,6 @@ func (m *mockStore) DisconnectProxy(ctx context.Context, proxyID, sessionID stri
}
return nil
}
func (m *mockStore) DeleteProxy(ctx context.Context, proxyID, sessionID string) error {
if m.deleteProxyFunc != nil {
return m.deleteProxyFunc(ctx, proxyID, sessionID)
}
return nil
}
func (m *mockStore) UpdateProxyHeartbeat(ctx context.Context, p *proxy.Proxy) error {
if m.updateProxyHeartbeatFunc != nil {
return m.updateProxyHeartbeatFunc(ctx, p)
@@ -92,12 +84,6 @@ func (m *mockStore) IsClusterAddressConflicting(ctx context.Context, clusterAddr
}
return false, nil
}
func (m *mockStore) HasGatewayPinnedByOtherAccount(ctx context.Context, host, accountID string) (bool, error) {
if m.hasGatewayPinnedByOtherAccountFunc != nil {
return m.hasGatewayPinnedByOtherAccountFunc(ctx, host, accountID)
}
return false, nil
}
func (m *mockStore) DeleteAccountCluster(ctx context.Context, clusterAddress, accountID string) error {
if m.deleteAccountClusterFunc != nil {
return m.deleteAccountClusterFunc(ctx, clusterAddress, accountID)
@@ -352,203 +338,3 @@ func TestGetActiveClusterAddressesForAccount(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, expected, result)
}
// TestIsClusterAddressAvailableConsidersGatewayPins pins that a proxy row is
// not the only claim on an address.
//
// An agent network gateway pinned to the address by another account is
// immutable and is served by whichever proxy declares that address, so a proxy
// from a different account taking it strands the pin — the mapping paths never
// hand an account-scoped proxy another account's mappings. Refusing the later
// claimant is what makes the bootstrap-time ownership check hold over time
// rather than only at the instant it runs: without this, an address a gateway
// pinned while no proxy served it could be taken a moment, or a week, later.
func TestIsClusterAddressAvailableConsidersGatewayPins(t *testing.T) {
ctx := context.Background()
tests := []struct {
name string
conflicting bool
pinned bool
available bool
}{
{name: "free address", available: true},
{name: "claimed by a proxy", conflicting: true},
{name: "pinned by another account's gateway", pinned: true},
{name: "claimed both ways", conflicting: true, pinned: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
st := &mockStore{
isClusterAddressConflictingFunc: func(context.Context, string, string) (bool, error) {
return tt.conflicting, nil
},
hasGatewayPinnedByOtherAccountFunc: func(context.Context, string, string) (bool, error) {
return tt.pinned, nil
},
}
m, err := NewManager(st, noop.NewMeterProvider().Meter(""))
require.NoError(t, err)
available, err := m.IsClusterAddressAvailable(ctx, "gw.example.com", "account1")
require.NoError(t, err)
assert.Equal(t, tt.available, available)
})
}
}
// TestIsClusterAddressAvailableSurfacesGatewayPinError pins that a failed pin
// lookup refuses the claim rather than falling through to available: this runs
// on the proxy-connect path, where "could not tell" must not read as "yes".
func TestIsClusterAddressAvailableSurfacesGatewayPinError(t *testing.T) {
st := &mockStore{
hasGatewayPinnedByOtherAccountFunc: func(context.Context, string, string) (bool, error) {
return false, errors.New("db down")
},
}
m, err := NewManager(st, noop.NewMeterProvider().Meter(""))
require.NoError(t, err)
available, err := m.IsClusterAddressAvailable(context.Background(), "gw.example.com", "account1")
require.Error(t, err)
assert.False(t, available)
}
// TestConnect_WithdrawsClaimLostDuringRegistration covers the window between
// the connect path's availability check and the row being written: a claim
// that lands there — another account's proxy row or gateway pin — is seen by
// the re-read after the write, and the proxy's own row is withdrawn rather
// than left standing next to it. The refusal carries
// ErrClusterAddressUnavailable so the connect path reports it exactly as it
// would have had the pre-write check caught it.
func TestConnect_WithdrawsClaimLostDuringRegistration(t *testing.T) {
accountID := "acc-1"
cases := map[string]func(s *mockStore, landed *bool){
"another account pinned its gateway to the address": func(s *mockStore, landed *bool) {
s.hasGatewayPinnedByOtherAccountFunc = func(_ context.Context, _, _ string) (bool, error) { return *landed, nil }
},
"another account's proxy declared the address": func(s *mockStore, landed *bool) {
s.isClusterAddressConflictingFunc = func(_ context.Context, _, _ string) (bool, error) { return *landed, nil }
},
}
for name, arm := range cases {
t.Run(name, func(t *testing.T) {
landed := false
var withdrawn []string
s := &mockStore{
// The competing claim commits as this row is written: the
// pre-write check (not exercised here) saw nothing, the
// re-read must.
saveProxyFunc: func(_ context.Context, _ *proxy.Proxy) error { landed = true; return nil },
deleteProxyFunc: func(_ context.Context, proxyID, sessionID string) error {
withdrawn = append(withdrawn, proxyID+"/"+sessionID)
return nil
},
}
arm(s, &landed)
mgr := newTestManager(s)
p, err := mgr.Connect(context.Background(), "proxy-1", "session-1", "gw.example.com", "10.0.0.1", &accountID, nil)
require.ErrorIs(t, err, proxy.ErrClusterAddressUnavailable, "a claim lost after the write must surface as the address being unavailable")
assert.Nil(t, p, "no record may be handed back for a withdrawn registration")
assert.Equal(t, []string{"proxy-1/session-1"}, withdrawn, "exactly this session's row must be withdrawn")
})
}
}
// TestConnect_KeepsClaimWhenRecheckFails pins what an inconclusive re-read
// does: the connect is refused with the store's error, not
// ErrClusterAddressUnavailable, since nothing established that the address is
// taken — and the row is marked disconnected rather than deleted. SaveProxy
// upserts on the proxy ID, so on a reconnect that row is the claim the account
// has held since its first connect; a transient store error must not hand the
// address to whoever asks next.
func TestConnect_KeepsClaimWhenRecheckFails(t *testing.T) {
accountID := "acc-1"
var disconnected []string
s := &mockStore{
hasGatewayPinnedByOtherAccountFunc: func(_ context.Context, _, _ string) (bool, error) {
return false, errors.New("db unavailable")
},
disconnectProxyFunc: func(_ context.Context, proxyID, sessionID string) error {
disconnected = append(disconnected, proxyID+"/"+sessionID)
return nil
},
deleteProxyFunc: func(_ context.Context, proxyID, _ string) error {
t.Fatalf("an inconclusive re-read must not withdraw the row, but proxy %s was deleted", proxyID)
return nil
},
}
mgr := newTestManager(s)
_, err := mgr.Connect(context.Background(), "proxy-1", "session-1", "gw.example.com", "10.0.0.1", &accountID, nil)
require.Error(t, err)
assert.NotErrorIs(t, err, proxy.ErrClusterAddressUnavailable, "an inconclusive re-read is not a conflict")
assert.ErrorContains(t, err, "db unavailable", "the store's error must be the one surfaced")
assert.Equal(t, []string{"proxy-1/session-1"}, disconnected, "the refused session must not stay marked connected")
}
// TestConnect_RefusesEvenWhenWithdrawalFails pins that a lost claim is
// reported as lost whatever happens to the compensating delete: the caller
// must never be told it holds an address another claim already has, and the
// stale row is the reaper's problem, not a reason to lie.
func TestConnect_RefusesEvenWhenWithdrawalFails(t *testing.T) {
accountID := "acc-1"
s := &mockStore{
hasGatewayPinnedByOtherAccountFunc: func(_ context.Context, _, _ string) (bool, error) { return true, nil },
deleteProxyFunc: func(_ context.Context, _, _ string) error {
return errors.New("delete failed")
},
}
mgr := newTestManager(s)
p, err := mgr.Connect(context.Background(), "proxy-1", "session-1", "gw.example.com", "10.0.0.1", &accountID, nil)
require.ErrorIs(t, err, proxy.ErrClusterAddressUnavailable, "a failed withdrawal must not turn a lost claim into a held one")
assert.Nil(t, p)
}
// TestConnect_ConfirmedClaimKeepsRow is the common case: nothing landed in the
// window, the re-read confirms the claim, and the row stays.
func TestConnect_ConfirmedClaimKeepsRow(t *testing.T) {
accountID := "acc-1"
s := &mockStore{
deleteProxyFunc: func(_ context.Context, proxyID, _ string) error {
t.Fatalf("a confirmed claim must not be withdrawn, but proxy %s was", proxyID)
return nil
},
}
mgr := newTestManager(s)
p, err := mgr.Connect(context.Background(), "proxy-1", "session-1", "gw.example.com", "10.0.0.1", &accountID, nil)
require.NoError(t, err)
require.NotNil(t, p)
assert.Equal(t, proxy.StatusConnected, p.Status)
}
// TestConnect_SharedProxySkipsClaimRecheck pins that a shared, NetBird-operated
// proxy — no account on its token — is not subject to the claim re-read: the
// connect path never asks availability for it before the write either, and a
// shared cluster is what accounts pin their gateways to, not a claim against
// them.
func TestConnect_SharedProxySkipsClaimRecheck(t *testing.T) {
s := &mockStore{
isClusterAddressConflictingFunc: func(_ context.Context, _, _ string) (bool, error) {
t.Fatal("a shared proxy must not be checked for address conflicts")
return false, nil
},
hasGatewayPinnedByOtherAccountFunc: func(_ context.Context, _, _ string) (bool, error) {
t.Fatal("a shared proxy must not be checked against gateway pins")
return false, nil
},
deleteProxyFunc: func(_ context.Context, _, _ string) error {
t.Fatal("a shared proxy's row must not be withdrawn")
return nil
},
}
mgr := newTestManager(s)
_, err := mgr.Connect(context.Background(), "proxy-1", "session-1", "eu.proxy.netbird.io", "10.0.0.1", nil, nil)
require.NoError(t, err)
}
@@ -1,7 +1,6 @@
package proxy
import (
"errors"
"time"
)
@@ -10,25 +9,6 @@ const (
StatusDisconnected = "disconnected"
)
// ErrClusterAddressUnavailable is returned by Manager.Connect when the cluster
// address turns out to be claimed by someone else once the proxy's own row is
// written: a conflicting proxy row, or another account's agent network gateway
// pinned to the address. The row has been withdrawn by then, and the caller
// reports the address as taken exactly as if the pre-write check had caught it.
//
// Both kinds of claim are made the same way, write then re-read then withdraw,
// and the re-read is the whole mechanism. Each side's availability check and
// its write are separate autocommit statements, so two concurrent claimants
// can each pass their check with neither row committed yet. Because both write
// before they re-read, of two concurrent claims at least one re-reads after
// the other has committed and backs off; each statement sees every commit
// before it on sqlite, postgres and mysql alike. Both may back off, which
// costs a retry; neither keeps a claim the other holds. No lock spans the
// proxies and settings tables portably, and a claims table would be more
// machinery than the property needs. The gateway side of the same protocol is
// agentnetwork's confirmGatewayClusterOwnership.
var ErrClusterAddressUnavailable = errors.New("cluster address is not available")
// Capabilities describes what a proxy can handle, as reported via gRPC.
// Nil fields mean the proxy never reported this capability.
type Capabilities struct {