* [management] Return the reason when custom domain validation fails
The validate endpoint returned an empty 202 and ran the check in a
goroutine, so a failed verification only reached the management log.
Run the check in the request and return 204 or 412 with a classified
reason. Resolver errors are not forwarded.
* [management] Address review feedback on custom domain validation
Pass the request context to the lookup and store calls now that the
check runs in the request, compare the self-resolved name
case-insensitively, document the 412 like the other entries, and
check the returned error in the existing tests.
* [management] Match custom domain CNAME targets case-insensitively
DNS names are case-insensitive, so a CNAME pointing at the cluster in
different casing should validate instead of failing as a mismatch.
* [management] Add validation timeout
---------
Co-authored-by: DevBot NetBird <91386968+netbirddev@users.noreply.github.com>
Let HTTP targets inherit service authentication, bypass it, or block access. Validate and persist the action, expose it through the API, and deliver it to the proxy. Private services reject bypass.
Validation: service/manager, store, and management-to-proxy integration race tests passed; scoped lint passed.
Zone and record changes refreshed every peer in the account, and zone
create/update passed the request context to the update goroutine, so it
could be cancelled when the handler returned. They now compute affected
peers from the zone's distribution groups inside the transaction and
dispatch through ExpandAndUpdateAffected, which detaches the context.
The resolver did not know about zones, so changing a group referenced only
by a zone never pushed the zone to its added or removed members. It now
folds the distribution groups of shipped zones on whole-group changes.
* [management] Require a private proxy cluster for cluster and direct upstream targets
Cluster targets and direct upstream targets make the proxy dial the
upstream from its own host network instead of through the embedded
NetBird client. Only clusters running in private mode are meant to do
that, but the service API accepted these targets on any cluster.
Service create and update now reject such targets unless the service's
proxy cluster reports the private capability. An unreported capability
is treated as unsupported.
* [management] Require every proxy in the cluster to be private
The private capability is aggregated as any-true, so a cluster where
only one proxy runs in private mode passed the check. The mapping is
delivered to every proxy in the cluster, so the non-private ones would
serve cluster and direct upstream targets from their host network too.
Validate these targets against a unanimous aggregation instead. The
existing any-true lookup stays as is for the dashboard flags and the
agent network gateway.
Deleting an account left state behind that DeleteAccount's store
associations don't reach. The Agent Network tables outlived the account,
keeping its gateway domain claimed and its provider API keys stored. The
proxies kept serving its gateway until they next resynced. Cloud-side
state, such as managed proxy deployments, had no way to be cleaned up at
all.
Account deletion now runs registered hooks after the permission check
and before any users or data are removed. A failing hook aborts the
deletion. Agent Network registers one that tells the proxies to drop the
account's gateway mappings. The account's settings, providers, policies,
guardrails and budget rules are deleted in the account's transaction.
Consumption counters, and the access logs of deleted accounts, are left
to the background cleanup; usage records are kept.
* Generalize PKCE verifier store into SingleUseStore
* Generalize PKCE verifier store into SingleUseStore
* Extend single-use store to generate one-time retrieval codes
* Hand off proxy OIDC session via one-time code instead of URL token
* Use the single-use store in integration tests
* Read active proxy versions by cluster
* Detect proxy clusters that support session codes
* Bind OIDC session handoff mode to signed state
* Deprecate legacy OIDC session token handoff
* Remove unrelated session code test stub
* fix tests
* fix merge
* Fix session code compatibility detection
* Isolate proxy session codes in shared cache
* bump min session version
Let an embedding binary refuse DELETE /api/reverse-proxies/proxy-tokens/{id}
through an optional proxytoken.RevocationGuard passed to NewAPIHandler.
It is checked after the ownership check, so another account's token
still returns 404 without reaching the guard, and before the token is
revoked. A status error from the guard is written with util.WriteError;
any other error becomes a generic 500.
Nothing installs a guard here, so OSS behavior is unchanged.
* extract shared db conn + data repository
* extract repository interface
* protect against nested transactions
* fix mysql and db conn creation
* fix context management
* remove query warpper
* remove withContext and withLock wrapper
* remove context from function call
* remove in memory mode
* fix nested transaction handling
* use db directly
* remove leftover test
* remove pool close on error during conn creation
Deleting a custom domain released its name while services still pointed
at it, leaving them on a namespace the account no longer held.
Deletion now refuses with 412 when a service in the same account uses the
domain or a subdomain, including disabled ones. Service writes revalidate
authorization inside their transaction and hold a shared lock on the
matching registrations, so a delete racing a create cannot strand either.
The dependency lookup is account-scoped: registrations are unique by name,
so another account can hold team.example.com under example.com and its
services are authorized by its own registration.
The agent network gateway service is private: agents reach it over the WireGuard tunnel, authorised by peer identity, with the cluster as its only target. Only a reverse proxy cluster with private capabilities can serve that, reported per cluster as the `private` capability — the same `supports_private` flag the dashboard gates NetBird-only services on.
A bootstrap could pin an account to a cluster without private capabilities, leaving an immutable dead gateway. Both bootstrap paths now validate the picked cluster: one the account can see must have a connected proxy reporting the capability. Shared and account-owned clusters qualify alike. Known-ness comes from proxy rows, not heartbeat freshness, so a cluster without the capability stays refused while merely offline. A hostname no proxy has declared stays pinnable (address-first). Identity is compared case-insensitively over the account's cluster list.
An agent network bootstrap stores its cluster as proxy_address, which selects the proxy that serves the endpoint. An account-scoped proxy only receives its own account's mappings, so a pin onto a host another account's proxy declares can never be served, and the endpoint is immutable — a dead gateway until the account deletes its settings. Nothing refused that pin; the domain unique index only arbitrates between endpoints.
Both bootstrap paths now refuse, before the insert, a host that another account's proxy declares, a host another account has labeled pins beneath (self-addressed path), or a hostname that is another account's endpoint (labeled path).
Shared clusters are unaffected: shared proxies are never foreign, and labeled pins under one cluster are never asked about, so any number of accounts still pin beneath eu.proxy.netbird.io. Registration is deliberately unchanged — refusing a proxy for another account's pin would let a pin lock a tenant out after the reaper drops its rows.
Prevent unvalidated registrations from reserving domain names indefinitely.
Give pending registrations a 48-hour validation window and clean up expired entries at startup and every 60 minutes. Emit CustomDomainValidationExpired for each deletion and preserve registrations referenced by services.
Reject validation after expiry and prevent concurrent validation from recreating deleted registrations. Normalize domain names with the shared parser before registration.
Migrate existing pending registrations to receive a fresh 48-hour validation window.
Require validated custom domains when creating or updating reverse proxy services.
Propagate validation errors during updates and return HTTP 409 for duplicate domain claims.
Add regression tests for domain validation, ownership, and service creation and updates.
A bad upstream or key saved cleanly and surfaced minutes later as a failed
request or an empty model picker, with nothing pointing back at the record.
CreateProvider now spends the credential once against the vendor's model
listing. UpdateProvider does the same when the upstream, the key, the catalog
provider or the skip-TLS flag changed — only then, so renames and price edits
neither wait on a vendor nor fail because one is down. Both run before the store
write, so a rejected rotation leaves the working key where it was.
What cannot be checked still saves: no listing endpoint, no derivable Bedrock
control-plane host, a private upstream, a record skipping TLS verification.
Everything else blocks, outages included — 5xx, 429 and timeouts leave the
record unverified just as a refusal does. Refusals return 422 and carry no
status code or echoed URL.
Discovery now reads as a partial edit, so a retyped URL can be listed against
without also rotating the credential. Entries with their own listing host
(Bedrock) get their configured upstream resolved separately, since a successful
listing said nothing about it.
After #7221, guardrail allowlists built from a path-style provider's declared
model ids (Bedrock, Vertex) stopped working: the raw region/version form was
compared against the parser's canonical id, so the agent config advertised an
empty model list and requests for the allowlisted model were refused.
Make every allowlist compare provider-aware, keyed on the destination
provider's catalog id: the agent config, the policy gate, and the synthesized
proxy allowlists match an entry on both its verbatim and canonical form —
Bedrock's strip only under bedrock_api, Vertex's only under vertex_ai_api,
verbatim everywhere else, so a plain provider's suffixed entries never widen.
The router's claim compare learns the Vertex @version strip.
New e2e, realstore, and unit tests reproduce both regressions and pin the fix.
Delegating Agent Network today means handing out full account admin, and
regular users cannot see their own usage or how to connect a local tool.
Add two roles on top of the existing agent_network permission
submodules. agent_network_admin owns the whole area (providers,
policies, guardrails, budgets, usage, logs, settings) with read-only
users, groups, peers, and account info needed to build policies, and
nothing else in the account. usage_viewer is the regular User baseline
plus read on the aggregated usage and cost overview: no provider
configuration, no policies, no request-level logs, which can contain
captured prompts. billing_admin gets a proper permission-map entry with
the User baseline so role resolution stops failing with role-not-found;
its plan and invoice permissions stay enforced cloud-side.
Add the self-service endpoints behind the "My Agent Network" view,
available to every authenticated user because both answers are scoped
strictly to the caller. GET /api/agent-network/me/setup returns the
account endpoint plus the providers and models the caller's own groups
authorize, computed with the same rules the proxy enforces: policy
filtering as in policy selection, model allowlist union intersected
with declared models, orphan and disabled providers omitted. Not set up
and no access are deliberately indistinguishable, and the response
carries display metadata only. GET /api/agent-network/me/consumption
returns the caller's own user-dimension counters.
[proxy,management] Serve Bedrock model discovery from the control plane
A Bedrock provider could never answer a model-discovery request. The router
sent GET /inference-profiles to the record's upstream, which has to be
bedrock-runtime.<region> for InvokeModel to work, and that host does not
implement the operation. ListInferenceProfiles is a control-plane operation on
bedrock.<region>.amazonaws.com, and one provider record carries one upstream,
so the two hosts genuinely differ.
The route now carries a discovery host, taken from the catalog's declaration
with the region read back out of the configured upstream, and the listing — and
only the listing — goes there. Inference is untouched. A proxied or self-hosted
Bedrock endpoint gets no discovery host at all rather than a guessed one, since
inventing a host would send the operator's credential somewhere they never
configured.
Two things had to follow for the listing to be usable once it arrives. The
response filter only understood OpenAI's {"data":[{"id":…}]}, so a Bedrock
listing fell through it untouched, offering every profile in the account
whatever the policy said. And discoverableModels intersected by exact string,
so a record registering the raw profile id while a guardrail names the catalog
key intersected to nothing — bounding a working provider's listing down to
empty.
Normalisation is the third. The geography in front of a cross-region profile
was matched against a hardcoded list of four, so every profile issued under jp,
au, ca, sa or us-gov carried its prefix into the pricing key, matched no
catalog entry and metered at zero. It is now recognised by either the geography
or the vendor that follows it, so an id has to be new on both axes at once to
slip through — a live eu-central-1 listing returned "global.xai.grok-4.6" days
after the vendor list was first written.
[management] Offer a provider's live model list in the config form
Adds POST /api/agent-network/catalog/providers/models, which asks a vendor
which models an operator's own credential can actually reach, so the provider
form can offer a live list instead of only the compiled-in catalog. The catalog
goes stale, and it cannot see an account: which OpenAI models an org is
entitled to, which Bedrock inference profiles an account and region hold, which
Vertex models a project has enabled.
The endpoints, auth headers and response shapes come from probing the live APIs
(#7244); each vendor invented its own envelope and none can be guessed from the
request. Bedrock shaped the design: its listing lives on the control plane
while inference must go to the runtime host, so Discovery carries its own host
rather than reusing the record's upstream, and profile ids are taken verbatim
because the region prefix is what AWS requires at invoke time.
A caller supplies either the key they are typing or the id of a saved record
whose stored credential is reused — never both, since accepting both would run
an arbitrary credential under the identity of a record the caller may only be
permitted to read. Gated on Create rather than Read, because this spends the
operator's credential against a third party.
Management has not made outbound calls on an operator's behalf before and it
holds a credential for every provider, so every resolved address must be public
— covering loopback, RFC1918, the cloud metadata address and NetBird's own
100.64/10 range — and redirects are not followed, since a redirect moves the
request to a host the check never saw.
The vendor is authoritative for the id; the catalog stays authoritative for
pricing. A discovered model the shipped table cannot price returns
pricing_known: false so the operator must set rates rather than being
registered at a silent zero.
[proxy,management] Bound model discovery to the caller's own policies
GET /v1/models was bounded by the provider record's enumerated models, which is
the right bound only while a single policy reaches a provider. Where two teams
share one provider under different allowlists, every caller was offered the
union — each model outside their own policy being a request the guardrail
refuses a moment later. A gateway record enumerating nothing was worse: it
offered the upstream's entire catalogue however narrow the policy was.
Each route now carries one rule per authorising policy — its source groups and
the models it permits — instead of a single flattened list. At request time the
router keeps the rules whose groups intersect the caller's, unions their
models, and intersects that with what the provider serves.
nil and [] stay distinct end to end: a policy setting no allowlist reaches the
router as nil and lifts the restriction for the groups it binds, while an
enabled allowlist with no models arrives as [] and permits nothing. Collapsing
them would let a listing that should offer nothing fall open to everything.
The guardrail's own per-provider allowlist is untouched. It is a fail-closed
backstop that cannot tell who is asking, so discovery is now narrower than the
backstop rather than wider.
[proxy,management] Conform the Agent Network endpoint to the LLM gateway protocol
Reviewed the proxy against Claude Code's published gateway contract. The
transport layer already held up; fourteen gaps sat one layer up, in the model
catalog and in the non-inference endpoints clients call.
Two of them cost money. The catalog carried no claude-opus-5 or
claude-sonnet-5, so an operator could not authorise the models coding agents
default to — those requests denied as not-routable, or priced at zero where a
catch-all carried them. And gateway records pin ParserID "openai" while the
same record serves /v1/messages, so Anthropic responses were read with the
OpenAI parser, which never looks at message_start where input tokens live:
input metered as roughly zero on every stream and cost was skipped entirely.
The rest fix requests refused for structural rather than policy reasons: model
discovery denied for every account with a model allowlist, token counting
denied on Bedrock and mis-parsed on Vertex, startup probes refused and written
into the access log at every session start, and denials rendered in a shape no
LLM client parses. Two changes are additive by design — the deny body keeps
every field it had and adds the vendor's error object alongside, and body-level
identity injection is now gated on the request's dialect so it stops sending
OpenAI-shape fields into Anthropic bodies that reject them.
The end-to-end work turned up one more: the discovery filter treated any slash
in a model id as a gateway prefix, which would have dropped every self-hosted
"Qwen/..." model from the picker.
People who only ever reach private services through the reverse proxy were
invisible to activity accounting. Active users are counted from user.LastLogin
or from the LastSeen of a peer they own, and neither column was written on the
proxy paths — so a person signing in via SSO to a proxied service, or a peer
serving one over the mesh, never showed up in the 24 hour numbers.
Both writes now happen where the proxy already authenticates:
- GenerateSessionToken stamps LastLogin after the session token is signed,
the same column and the same way the dashboard and device login paths do.
- ValidateTunnelPeer stamps the calling peer's LastSeen, the column its owner
activates through.
The policy lives in a new reverseproxy/activity manager rather than in the gRPC
service, matching the module layout the other reverse proxy domains use. It
skips what can never count — service users, embedded proxy peers and WASM
clients — and throttles peer writes to once an hour, well inside the window
accounting asks about and far above the proxy's five minute tunnel cache.
The peer write is a single indexed UPDATE that touches only
peer_status_last_seen. Connected and SessionStartedAt are left alone so the
session-ownership fencing MarkPeerConnectedIfNewerSession relies on is never
disturbed, and the timestamp comes from the database clock rather than the
caller, for the same reason the other status writers take it from there. The
caller's cutoff travels into the statement's WHERE, so concurrent requests for
one peer collapse into a single write instead of each acting on its own stale
read, and a peer that was never seen — NULL last seen, since Status is an
embedded pointer — still records its first activity.
Nothing outside the reverse proxy changes behaviour: the only addition
elsewhere is the RefreshPeerLastSeen store method the manager calls.
Store the per-account gateway endpoint as {domain, proxy_address} with a
global unique index on the full hostname; dedicated = (domain ==
proxy_address). Bootstrap becomes an explicit POST carrying exactly one
of proxy_address (server allocates an adjective-noun label beneath it)
or endpoint (claimed verbatim, address-first); provider create loses its
bootstrap side effect. PUT is a full replace with every field required —
the immutable identity fields must be echoed unchanged and a mismatch is
rejected with 422. A guarded DELETE releases the endpoint: refused with
412 while providers exist or a proxy is actively serving the endpoint
hostname (matched case-insensitively); re-creating bootstraps fresh. A
self-addressed pin excludes its address from the account's cluster allow
list, and the live mapping update path now addresses the serving proxy
from the synthesized service. Existing rows are migrated on all three
store engines.
## Describe your changes
Work on the Terraform provider (terraform-provider-netbird #177–#183)
surfaced places where the agent-network API broke its own contracts or
deviated from the conventions the rest of the management API follows,
forcing client-side workarounds.
Settings reads now follow the settings-endpoint convention: GET always
answers with a JSON object. Before bootstrap it returns the defaults
with an empty cluster/subdomain/endpoint (previously 200 with a JSON
`null` body, while the spec said 404). The settings PUT can bootstrap
the account by carrying a `cluster` — previously the row could only come
into existence through the first provider create, and a settings-first
setup was impossible; a differing cluster on a bootstrapped account is
rejected instead of silently ignored. PUT remains full-state.
The provider PUT schema promised omit-preserves semantics for several
operator-editable fields that the handler never delivered (it builds the
row from the request, like every other update handler). The schema
wording now matches the shipped full-state behavior; only the api_key
(secret) and session keys stay preserved by the manager. Identity
headers are always present in provider responses so an explicitly
cleared value round-trips as an empty string.
The Go REST client gains the full agent-network surface (catalog,
providers, policies, guardrails, budget rules, settings), including a
shim translating the legacy 200+`null` settings body from older servers
into an `IsNotFound` error.
Note for reviewers: the dashboard special-cased the `null` settings
body; it needs a small follow-up for the new defaults response (in
progress).
## Describe your changes
Agent Network gates providers, policies, guardrails, budgets, usage,
access logs, and settings behind the single `agent_network` permission
module, so access is all-or-nothing: a future delegated role cannot be
scoped to a subset of the area (for example usage-only visibility).
This introduces dotted submodules (`agent_network.providers`,
`.policies`, `.guardrails`, `.budgets`, `.usage`, `.logs`, `.settings`)
and resolves grants with a cascade: exact module first, then its parent,
then the role's `AutoAllowNew` default. The agent network manager now
validates each operation against its matching submodule. `usage`
(aggregated counters, overview) is deliberately separate from `logs`
(request-level entries, which can contain captured prompts).
No role definitions change. No built-in role carries an explicit
`agent_network` entry, so every role resolves the submodules exactly as
it resolved the parent module before — pinned by a test that compares
each built-in role's answer on every submodule against its answer on
`agent_network`. Role additions that use these submodules come
separately.
## Describe your changes
The model-allowlist guardrail was merged into one account-wide union and
enforced flat on every request, ignoring which policy/group/provider
authorised it. With multiple policies — especially a mix of guardrailed
and un-guardrailed ones — this caused:
- **false-allow**: a model allowlisted for one group/provider leaked to
any caller; and
- **false-deny**: an un-guardrailed policy (intended unrestricted) was
blocked by another policy's allowlist.
Enforcement is now policy/group-aware, mirroring `llm_limit_check`:
- **Management (`SelectPolicyForRequest`) is authoritative.** It uses
the request model (already carried in
`CheckLLMPolicyLimitsRequest.model`, previously ignored) to keep only
applicable policies whose guardrails permit the model; no
allowlist-enabled guardrail = unrestricted. Denies
`llm_policy.model_blocked` when policies govern the (provider, groups)
but none permits the model.
- **Proxy `llm_guardrail` becomes a per-provider fail-closed backstop.**
The synthesiser emits an allowlist only for providers every authorising
policy restricts; the middleware keys off the resolved provider id and
keeps unknown-model fail-closed.
## Describe your changes
* [proxy] enforce model allowlist for URL-routed providers
(Bedrock/Vertex) by @mlsmaycon in
https://github.com/netbirdio/netbird/pull/6764
* [management] Remove proxy peer stale deduplication logic by @mlsmaycon
in https://github.com/netbirdio/netbird/pull/6768
## Issue ticket number and link
## Stack
<!-- branch-stack -->
### Checklist
- [ ] Is it a bug fix
- [ ] Is a typo/documentation fix
- [ ] Is a feature enhancement
- [ ] It is a refactor
- [ ] Created tests that fail without the change (if possible)
- [ ] This change does **not** modify the public API, gRPC protocols,
functionality behavior, CLI / service flags, or introduce a new feature
— **OR** I have discussed it with the NetBird team beforehand (link the
issue / Slack thread in the description). See
[CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first).
> By submitting this pull request, you confirm that you have read and
agree to the terms of the [Contributor License
Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md).
## Documentation
Select exactly one:
- [ ] I added/updated documentation for this change
- [x] Documentation is **not needed** for this change (explain why)
### Docs PR URL (required if "docs added" is checked)
Paste the PR link from https://github.com/netbirdio/docs here:
https://github.com/netbirdio/docs/pull/__
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added model-allowlist guardrails for path-routed providers, including
Bedrock and Vertex.
- Added Bedrock request support for chat interactions.
- Added guardrail management capabilities.
- **Bug Fixes**
- Requests with missing or blank model identifiers are now denied when a
model allowlist is configured, improving fail-closed protection.
- Corrected provider-specific request handling and session tracking for
Bedrock interactions.
- **Tests**
- Expanded coverage for allowlist enforcement and provider routing
scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Theodor Midtlien <theodor@midtlien.com>
Co-authored-by: blaugrau90 <61945343+blaugrau90@users.noreply.github.com>
Co-authored-by: Viktor Liu <17948409+lixmal@users.noreply.github.com>
* [management,proxy] Add per-provider skip_tls_verification for agent-network
Let agent-network providers opt into skipping upstream TLS verification for
self-hosted / internal gateways behind a private or self-signed cert.
- provider: add SkipTLSVerification (persisted via AutoMigrate) with
request/response mapping (nil on update preserves, explicit false clears).
- openapi: skip_tls_verification on the provider request + response; types
regenerated.
- synthesizer: carry the flag into the llm_router route config so it reaches
the proxy.
- proxy: llm_router sets it on the UpstreamRewrite mutation, and the reverse
proxy applies roundtrip.WithSkipTLSVerify per selected route when forwarding
upstream (the router dials per provider, so a per-target flag alone wouldn't
cover it).
- tests: synthesizer route config carries the flag, router rewrite propagates
it, and the request/response round-trip incl. update semantics.
* [e2e] Validate per-provider skip_tls_verification end to end
Add a self-signed HTTPS upstream (nginx) to the harness and a test that
provisions two providers on that same upstream — one with
skip_tls_verification=true, one false — behind one proxy + client. The
skip=true provider's chat reaches the upstream (200); the skip=false
provider's fails the TLS handshake (5xx). Same upstream, opposite outcome,
which proves the flag is honoured per provider (a single target-level flag
could not, since all of an account's providers share one synthesised
target).
* [e2e] WaitProxyPeer: require >=1 connected peer, not exact 1/1
Each proxy container registers a fresh WireGuard key and its peer is not
removed on teardown, so proxy peers from earlier tests linger in the
account as disconnected. WaitProxyPeer matched the exact string
"1/1 Connected", which failed once a second proxy-using test ran in the
same package (status "1/2"). Parse the "Peers count: X/Y Connected" line
and wait for X>=1 instead: only the live proxy can be connected, and the
caller's subsequent chat is the real end-to-end assertion. Fixes the CI
failure of TestProviderSkipTLSVerification (runs after TestProvidersMatrix).
Bethuel Mmbaga