The discovery isolation test drove a single client in the main group.
VLLMUnlistedModel's absence from that client's listing was consistent
with two different worlds: the listing being scoped to the caller's
policy, or the other team's policy never having applied at all. The
test passed either way, so it did not prove what it claimed.
Mint a setup key per group and join a second client on the other
group, reusing the running proxy. The other client must see its own
model before the main client's listing is asserted, and must not see
the main group's model — isolation is checked in both directions.
The listing was narrowed by the provider record's enumerated models, which is
the right bound only while one policy reaches a provider. Where two teams
share a provider under different allowlists, every caller was offered the
union: each model outside their own policy is a request the guardrail refuses
a moment later, which is the empty-or-wrong picker this endpoint exists to
avoid, moved one level up. A gateway record enumerating nothing was worse
still — it offered the upstream's entire catalogue however narrow the policy.
The synthesiser already knows which policies authorise a provider and which
groups each binds, so the router can answer this at request time where it
knows the caller's groups. Each route now carries one rule per authorising
policy — its source groups and the models it permits — and the listing is
bounded to the union across the rules matching the caller, intersected with
what the provider serves.
This is deliberately finer than the guardrail's own per-provider allowlist,
which stays as it is: that list is a fail-closed backstop and cannot tell who
is asking, so discovery is now narrower than the backstop rather than wider.
A policy setting no allowlist lifts the restriction for the groups it binds,
so nil and empty model lists stay distinct end to end — collapsing them would
let a listing that should offer nothing fall open to everything.
Publishing the chain before the route only helps if the chain is there.
rebuildMiddlewareChains logged its error and returned, so a failed rebuild
still fell through to AddMapping and published a route over chains that were
never installed — served with no policy enforcement and no metering, which
is the outcome the ordering change exists to prevent.
Report the error instead. The caller already unwinds a failed setup, so the
service stays unpublished rather than reachable and uncounted. An unset
middleware manager is still not an error: that is a deployment without
middleware, not a failure to install it.
A provider update added the proxy mapping and then rebuilt the middleware
chain. Between the two, the route was live with no chain behind it, and a
request that landed there was served straight through — a successful
inference that was neither routed by policy nor metered.
Rebuild first. The worst a request in the remaining window meets is the new
chain in front of the previous target, which is still counted.
The loop bounded when a new attempt could start, not how long one could
run. The chat container is capped at 90 seconds of its own and the row
lookup at another 20, so an attempt begun just inside the 180-second window
could report a repricing failure nearly two minutes after that window
closed. Run every call in the loop under a context that expires with the
deadline.
Three open review findings.
The discovery filter treated every slash in a listed id as a gateway prefix
and matched the tail against the policy. A self-hosted id carries slashes of
its own, and an upstream may scope ids per tenant, so "tenant-b/claude-sonnet-5"
matched a permitted "claude-sonnet-5" and reached the picker — a model the
policy never named, and one the guardrail denies on sight, since enforcement
compares the id as written. Strip only the namespaces a gateway is known to
prepend, taken from the first slash rather than the last.
The e2e retry loops slept between attempts without watching the context, so a
cancelled run kept retrying calls that fail instantly and spent its remaining
window sleeping between them. They now stop when the context is done.
The streamed provider's setup key outlived its test: deleting the group does
not delete the key that auto-joins it.
The failure message read repriced.InputCostUsd, and repriced is the zero
value on every path that reaches that line — so a run that gave up always
reported "last input_cost_usd=$0.000000", which reads as a row priced at
zero rather than as a row still at the old rate, or as no row at all.
Keep the last cost read and report that, saying so plainly when no row was
ever read.
The poll interval was not bounded by the window: a page that came back
without the row just before the deadline still slept a full two seconds
before the loop noticed it was out of budget, so the caller waited longer
than the window it asked for to be told nothing arrived.
Cap the wait at whatever is left of the window.
Two review findings on the e2e suite.
lookupAccessLogBySession polled under the caller's context, so one stalled
request could hold the loop open well past the 30s ingest window it exists to
enforce — and the caller would read that delay as a missing row rather than a
slow one. Each poll now expires with the window; the parent's cancellation
still applies, since the request context derives from it.
The streaming test asserted only that the total cost was positive. Input and
output are positive on their own, so a cache-read bucket that was parsed and
then never billed would have passed. Assert the sum of the three buckets: the
gap a dropped cache read leaves is 7e-6, well outside the delta.
Six behaviours had unit coverage only, either because they arrived from code
review after the end-to-end tests were written or because no request in the
suite had the shape that reaches them.
Streaming is the important one. Input tokens exist only in a stream's opening
message_start event, and reading a stream with the wrong vendor's parser
misses it — the metering bug this endpoint's protocol work fixed. Nothing in
the suite sent stream: true, so the branch never ran. The mock now serves an
SSE surface on a second listener, reporting counts that differ from its
buffered ones so a passing assertion can only mean the stream accumulator ran,
and one case drives it through a record typed for the wrong surface.
The rest need no new harness capability: the per-model lookup against the
allowlist, the read-method gate on the non-inference paths, dated ids reaching
an undated registration while a pinned build refuses a different one, the
Bedrock inference-profile lookup reaching its upstream rather than a policy
denial, and a custom dated id keeping its own price.
Sub-agent ids stay uncovered: the parser lifts them onto request metadata but
nothing persists them, so there is no queryable surface to assert against
until that half lands. Covered here only to the extent that sending the
headers leaves the request served and metered.
TestPriceChangeUpdatesRecordedCost drives requests in a loop until one is
priced at the new rate, because the price push and the proxy's chain rebuild
are async. The loop could not actually retry: it looked the row up through
findAccessLogBySession, which fails the test outright when no row lands
within 30s, so the first post-update request that produced no row ended the
run instead of yielding to the next attempt.
That is the observed failure — the nightly run has been red on this test
roughly half the time, always with "session id ...-reprice-b-... must be
recorded in an access-log row" after ~41s: container setup, one request, one
30s wait, dead.
A missing row there is expected rather than exceptional. The provider update
rebuilds the middleware chain, and a request served mid-rebuild can complete
without a resolved provider: 200 to the caller, nothing to attribute, so no
row is ever written for it. Split the polling helper into a non-fatal lookup
and keep the fail-fast wrapper for callers whose row must exist, then treat a
miss in the loop like any other not-yet-repriced iteration. Only the outer
deadline is fatal.
Shorten the per-attempt wait to 20s and raise the overall deadline to 180s so
several attempts fit where before the budget allowed barely one.
Two changes to routeClaimsModel, both about dated Anthropic ids.
Normalising the configured candidate as well as the requested model made
every dated build of a family interchangeable: a route registered against
claude-sonnet-4-5-20250101 also claimed ...-20250202, so an operator who
pinned a build deliberately would have served a different one, and with
several such routes declaration or path order decided which. Only an undated
registration now absorbs a dated request.
The per-model lookup also stamps the model its path names, so the guardrail's
allowlist — a separate and possibly narrower list than the route's — still
decides GET /v1/models/{id} rather than seeing no model at all.
The guardrail enforces its own per-provider model allowlist and fails closed
when the request names no model, which is right for a path-routed inference
request whose shape the parser could not read. GET /v1/models names no model
anywhere, so discovery still denied with model_unknown for exactly the
accounts that configured an allowlist — the case skipping the management
pre-flight was meant to fix. Only one of the two gates had been opened, and
a client reads the 403 as an empty model picker.
Exempt requests the router marked non-inference from the unknown-model
branch. A named model is still checked, so the exemption covers only the
endpoints that genuinely name nothing: the listing and the warm probe.
The discovery filter read one byte past its 1 MiB cap to detect a body too
large to rewrite, then closed the upstream body and forwarded the buffer it
had — the response reached the client truncated at exactly the cap, with
Content-Length rewritten to match so nothing looked wrong until the client
tried to parse it.
Splice the bytes already read back in front of the unread remainder and
forward the response as the upstream sent it, headers untouched.
The router classified a non-inference request by path alone, so a POST to
/v1/models/{id} — or to the listing, or the warm probe — was marked
llm.non_inference and skipped the limit check's management pre-flight, even
though such a request can carry an inference body.
Require GET or HEAD, the methods these endpoints actually use. Anything else
falls through to normal per-model routing, which routes on the body's model
under the usual pre-flight, or denies as missing-model when there is none.
Invoke had grown a branch per API surface, each repeating the same
found/unauthorised/unknown switch. Lift the shared denial arms into decide(),
move the model-less endpoints into their own method, and name the two allow
decorations (non-inference marking, Bedrock namespace stripping) so each
surface reads as the one thing it does differently.
No behaviour change.
GET /v1/models/{id} was folded into the model-less endpoints so it would
route rather than deny. Once model-less requests started skipping the
management pre-flight, that also skipped the per-model allowlist: a caller
could confirm the existence and reachability of a model the route does not
list, even though the listing beside it is bounded to that same allowlist.
Resolve the id from the path and route it through the model table like any
other per-model request, keeping model-less treatment for the listing and
the connection-warming probe only. It stays marked non-inference, since the
lookup spends no tokens. A gateway route that enumerates no models still
answers every lookup, as before.
Also record why the Bedrock inference-profile lookup is forwarded rather
than denied: those live on the AWS control plane, and forwarding reproduces
what an unproxied client with the same base URL would see.
The release-date normalizer matched a bare "-YYYYMMDD" suffix on any id.
Pricing looks every model up through it regardless of surface, and an
operator can register a custom model under any id at all, so a custom
"internal-llm-20250101" would silently inherit the rate registered for
"internal-llm".
Anchor the pattern on "claude" so it still covers the vendor-prefixed
Bedrock forms while leaving every other vendor's id untouched.
The routing and parser-selection fixes touch every provider surface, and
the unit tests only prove each side of a seam in isolation. Two suites
close that:
The provider matrix drives one request per wire shape over a single tunnel,
with a record per catalog surface behind it, and asserts the surface each
request was metered under together with the token counts that surface's own
usage block carries. A response read by the wrong provider's parser meters
zero, so a regression fails instead of passing on a coincidental non-zero.
It also covers the Bedrock and Vertex token-counting paths, the warm-up
probe, and the vendor error envelope on a refusal.
The discovery suite covers the configuration that broke: an account with a
model allowlist, where the listing carries no model and the gate failed
closed. It asserts the listing is served, that it is bounded to the
authorised model, and that inference outside the allowlist is still
refused, so the exemption cannot be read as a way around the gate.
The mock upstream grows the Anthropic, Bedrock and token-counting shapes so
one container stands in for every surface, and the client gains GET and
arbitrary-POST helpers for the endpoints that carry no chat body.
The filter treated a slash in a listing entry's id as a gateway provider
prefix and matched only the tail. Self-hosted backends serve ids that carry
a slash of their own, so every "Qwen/Qwen2.5-0.5B-Instruct" style model was
dropped from the picker even when the policy named it exactly.
Try the id as written first and fall back to the tail, so both a prefixed
id and a self-hosted one resolve.
A few client-side checks call their vendor directly instead of following
the configured base URL, so they fail on exactly the locked-down networks
Agent Network is built for while inference keeps working. Fast mode reports
a connectivity error, or reports itself disabled by the organization when
the agent holds only a proxy-issued token, and model discovery stays off
until it is turned on explicitly.
Name the variables that settle each case, and say plainly which ones
allowing direct egress does not fix.
A coding agent that spawns helpers stamps the spawned agent's id on every
request it makes, and the spawning agent's id when that helper is nested.
The parser read the session header and ignored both, so parallel agents
inside one session all attributed to the session alone and there was no way
to see which one spent the tokens.
Emit them as metadata alongside the session id. They are opaque grouping
identifiers rather than content, so they are stamped regardless of the
prompt-collection toggle. Persisting them as queryable access-log columns
is a schema change and is deliberately not part of this commit.
Discovery proxies the upstream's full list, so the picker offers every
model the shared provider key can reach and each one outside the policy is
a request the chain denies a moment later. Restricting models is the point
of the product, and the client had no way to see the restriction.
Carry the resolved route's model list on the upstream rewrite and drop the
rest from the listing response. Only a route that enumerates its models
bounds anything: a catch-all claims every model, so its list passes
through. Anything the filter cannot safely rewrite, including a compressed
or oversized body, reaches the client untouched.
Clients send HEAD /api/hello before their first inference request to open
the upstream connection early. The path carries no model, so it denied as
not-routable and each session start left a policy rejection in the access
log for a request that was never a policy question.
Treat it as a model-less endpoint. Forwarding it warms the connection the
first real request will use, which is what the probe is for.
A client resolving a configured inference profile calls
GET /inference-profiles at startup. The path carries no model and was not
recognised as non-inference, so it denied as not-routable and wrote a
policy rejection into the access log on every session start, which is the
log operators read to find real policy problems.
Recognise the path and match it against a Bedrock provider specifically:
sending it to whichever provider happened to be authorised first would
rewrite it to an upstream that 404s it. The optional gateway namespace is
stripped the same way the runtime paths strip it.
Vertex hangs token counting off the model as its own path segment, and the
parser split the tail on the final colon alone. A count-tokens request
therefore reported its model as "claude-sonnet-5/count-tokens", which no
route claims, so the request denied as not-routable and the access log
recorded a model that does not exist.
Stop at the first "/" after the model id so the method segment stays out of
it, leaving the client free to price its context against the dedicated
endpoint instead of the billable inference one.
Both the request parser and the router enumerated Bedrock actions without
count-tokens, so the path carried no model and the request denied as
not-routable. Nothing breaks outright, because the client falls back to
counting context through the inference endpoint, but that fallback is
billable and the dedicated endpoint exists to avoid exactly that.
The action carries a model in the path and returns no usage, so it routes
like any other Bedrock action and meters to zero.
A budget stop, a blocked model or an unroutable model all rendered as the
NetBird deny envelope alone. LLM clients only parse their own provider's
error shape, so the reason never reached the user: Claude Code showed an
unexplained API error where it could have shown the policy message.
Carry the resolved surface on the deny reason and add the vendor's error
object next to the existing fields. The body stays a superset of what it
was, so anything reading code, message, details or middleware is
unaffected. Status codes are unchanged here: mapping window caps to 429
needs the window reset plumbed through the limits response before a
correct Retry-After can be sent.
Gateway records enable body-level identity so LiteLLM's tag-budget check
can read it, and the injector wrote "user" and "metadata.tags" into every
JSON object regardless of dialect. Claude Code reaches those same records
on /v1/messages, where "user" is not a permitted top-level field and
metadata accepts only "user_id", so the upstream rejected the request with
a 400 naming a field the client never sent. Rewriting the body also
changed the bytes a gateway-side prompt cache keys on.
Gate the body write on the surface llm_request_parser resolved from the
path. Header stamping is untouched, so spend tracking and per-end-user
budgets keep working on the surfaces that lose the body path.
shared/llm normalizes Bedrock and Vertex model ids so both sides of the
routing and pricing contract compare equal, but nothing did the same for a
first-party Anthropic id. A client pinning "claude-sonnet-4-5-20250929"
against a record registered as "claude-sonnet-4-5" denied as not-routable,
and where a catch-all route carried it through, the price lookup missed and
the request recorded no cost.
Add NormalizeAnthropicModel beside the existing two and consult it after an
exact match fails in the router's claim check, the pricing table, and the
per-record price map. Exact matches still win, so an operator who registers
two dated releases of the same family keeps them distinct.
GET /v1/models carries no model, and management's per-model allowlist
fails closed on an undetermined one, so gateway model discovery denied
with model_blocked for every account that enables a model allowlist. The
client treats a failed discovery as silent and falls back to its built-in
list, so the operator sees an empty picker with no error to chase.
The router already classifies these paths and authorises the route against
the caller's groups before allowing them, so mark them non-inference there
and let the limits gate skip a pre-flight that has no model to evaluate
and no tokens to book. The marker comes from the router's own path
classification, never from client input, so an inference request cannot
set it to escape the allowlist.
Gateway catalog entries pin provider_id "openai", and the same record
serves Claude Code on /v1/messages. The parser preferred the pinned id
over the path, so an Anthropic body was read with the OpenAI parser: on a
streaming response the input tokens ride message_start nested under
message, which that parser never reads, so input counted as zero. Both
cache buckets were dropped, and pricing resolved against the openai
surface where no claude-* model exists, skipping cost entirely.
Detect from the path first and keep provider_id as the fallback for
upstreams whose path carries no surface. The Kimi entry already leaves
ParserID empty to work around this; the fallback ordering makes that
unnecessary.
Claude Code resolves to Opus 5 and Sonnet 5 by default, and neither was
selectable on a provider record. An operator building a record from the
catalog could not authorise the client's own default, so llm_router denied
those requests as model_not_routable. Opus 5 carried a supplemental pricing
row that priced gateway traffic but never reached the dashboard; Sonnet 5
was absent everywhere, so a request that did route through a catch-all
gateway recorded zero cost and under-counted every budget it should have
charged.
Add both to the Anthropic, Bedrock and Vertex lineups at the published
rates, and drop the supplemental rows now that the catalog carries them.
* [misc] Let gomobile init fetch the toolchain gobind needs
The previous commit's CI run confirmed the failure on a comment-only diff
off main, so the cause is not any branch's changes:
Android / Build failure
iOS / Build failure
`gomobile init` re-installs gobind from x/mobile@latest whatever gomobile is
pinned to, and setup-go sets GOTOOLCHAIN=local, so the install dies the
moment @latest declares a newer Go than go.mod does:
gomobile: go install golang.org/x/mobile/cmd/gobind@latest failed: exit status 1
go: golang.org/x/mobile@v0.0.0-20260821190718-4776eadac327
requires go >= 1.26.0 (running go 1.25.12; GOTOOLCHAIN=local)
GOTOOLCHAIN=auto on that step alone lets the install fetch what it asks for.
Scoped to the step deliberately: the repo's Go version and every build below
it stay on go.mod's toolchain, so this buys the mobile jobs nothing except
the ability to run gobind.
Pinning gobind next to gomobile does not work — init re-installs @latest
regardless. A durable fix is to stop `init` reaching the network at all, or
to track x/mobile's Go requirement in go.mod; both are larger changes than a
red CI warrants right now.
The updater runs as LocalSystem and started netbird-ui via
CreateProcessAsUser with a nil environment, so the UI inherited the
SYSTEM environment (USERPROFILE, APPDATA pointing at systemprofile)
while running under the user's token. The WebView2-based UI exits
immediately in that state, so the UI never came back after an update.
Build the environment from the user's token with CreateEnvironmentBlock
and pass it to CreateProcessAsUser.
Android never reported its local network interfaces, so PeerNetworkRange posture checks could not be evaluated: NetworkAddresses always arrived empty.
net.Interfaces() is unusable on Android 11+ (SELinux blocks netlink), so the addresses are parsed from the interface description the host app already provides via stdnet.ExternalIFaceDiscover. The MAC filter is skipped, mirroring #5906
for iOS, since Android does not expose MACs either and nothing reads Mac server side.
Stop the UI before a silent Windows update and suppress the installer reboot
On silent MSI updates msiexec could reboot the machine on its own. The running UI holds a lock on its own exe, and since msiexec runs as LocalSystem it cannot close the interactive user's UI via Restart Manager, so the MSI scheduled the
file replacement for the next reboot and marked the install restart-required.
Terminate netbird-ui.exe before launching the installer and wait until its image file is released; the existing deferred restart brings it back after the install on every exit path
Run msiexec with /norestart REBOOT=ReallySuppress so it never reboots on its own
Treat exit codes 3010/1641 as success with a warning instead of a failure
---------
Co-authored-by: Viktor Liu <viktor@netbird.io>
* [client] Clear stale installer result before starting update
The installer result file could survive a previous update attempt (e.g.
when the updater wrote it after the restarted daemon already ran its
startup check). A new install attempt left the old file in place, so the
GUI progress window's first GetInstallerResult poll read the outdated
result: a stale success made the GUI quit mid-install, which cancelled
the TriggerUpdate context and aborted the artifact verification; a stale
error surfaced a bogus failure dialog for a succeeding update.
Remove any leftover result file at the start of RunInstallation, before
the download begins, so result watchers only see the current attempt's
outcome.
* [client] Align stale-result warning with log message style
On network changes the client restarted the whole engine. That is heavy-handed and slow: it tears down working state to recover from a transition the engine could handle itself. This replaces the restart with proper network event handling.
Suspend the retry loops while no network is available. Instead of burning through backoff intervals against an unreachable network, the reconnection loops park until the OS reports a usable network again.
Reconnect immediately on a network switch. When the OS hands us a new network, connections bound to the old one are swept and re-dialed right away, rather than waiting for a timeout to notice they are dead.
Adds an SSHClient gomobile binding so the Android app can run an SSH session over the tunnel with a PTY, exposed through a listener interface for the in-app terminal.
Server type is auto-detected from the SSH banner, which selects the auth path: JWT device-code flow, NetBird key, or a regular server (NetBird key first, then password). Host keys are verified against the peer registry for NetBird servers and trust-on-first-use for regular ones.
