Runs beside the other live discovery tests and takes its credential from
the same AWS_BEARER_TOKEN_BEDROCK the suite already sources, so it needs no
setup of its own.
Reads the listing three ways — one unparameterised GET as Fetch issues it,
the same call followed through nextToken, and Fetch itself — then breaks the
result down by status, type, geography and vendor, counts distinct models
after normalization, and separates the ones the catalog can price from the
ones it cannot.
Not for merge.
Two things the first pass could not do.
An environment may carry SigV4 credentials rather than a Bedrock API key,
and skipping there wastes the only account within reach. It now signs with
the default credential chain when no bearer token is set, and records which
mode it used. Fetch is bearer-only, so its step is skipped under SigV4 —
itself worth knowing, since it means a record holding an access key cannot
discover at all.
A non-200 printed nothing but its status. The body is the whole point of a
failure here: an IAM denial names the action it refused, which is a
different fix from a credential AWS does not recognise.
Not for merge. Discovery reports 100+ models for an account whose console
shows 38 in the same region, and the production path cannot explain it:
parseListing keeps an id, a name and a status and drops the rest of every
summary, so the type, the geography and whether a nextToken was present
never reach a log line.
Reads the listing three ways — one unparameterised GET as Fetch issues it,
the same call followed through nextToken, and Fetch itself — then breaks
the result down by status, type, geography and vendor, counts distinct
models after normalization, and lists which of those the catalog can
price.
Carries its own build tag, so no ordinary test run or CI job compiles it,
and needs only the token: no docker, no management server.
The cross-region inference-profile prefix was matched against a list of
four — us, eu, apac, global — so a profile issued under any other
geography kept its prefix through normalization. That form matches no
catalog key, which cost more than a blank price column:
- discovery returned those models unpriced, so a real account's listing
came back almost entirely at $0
- the cost meter keys its table by the same normalized id, and operators
are told to register a Bedrock id exactly as AWS issues it, region
prefix included — so the default entry never resolved and every cache
bucket, and any model priced only by catalog defaults, metered free
Identify the geography by what follows it instead: a leading segment is a
geography when a known Bedrock vendor namespace comes next. New
geographies then need no change at all, and a vendor missing from the map
fails safe by keeping the prefix — the behaviour of the list this
replaces. Over-stripping is the direction that must not happen, since the
normalized id also decides which route may claim a model.
Covered at all three seams the id passes through: the normalizer, the cost
meter config the proxy bills from, and the discovery listing the dashboard
renders. Each test fails against the old four-geography list.
GetInferenceProfile was routed by provider type alone, so any caller with a
Bedrock route could read the full configuration of every profile in the
account — name, ARN, and underlying models — including profiles its policy
never named. The listing beside it is bounded on the way back, but a detail
lookup answers with a single object no filter inspects, so nothing narrowed
it.
Authorise the named profile like any other per-model request, as the
/v1/models/{id} lookup already is. The identifier is normalised first: a
record may register the raw profile id AWS issues or the catalog key it
reduces to, and either spelling must resolve. The listing itself names no
profile and stays model-less.
A discovery failure in the live e2e rendered the vendor's response into the
assertion message. When the vendor is Bedrock, that response is an AWS
refusal naming the resource it refused, and the name is an ARN carrying the
12-digit account id — into a job log anyone who can see the run can read.
The earlier change kept the body out of the success log and left the failure
paths quoting it, which is the path that actually carries the refusal.
Report the response's size and top-level keys instead. That is what the
failure is diagnosed from anyway: which envelope arrived, not what was in it.
The discovery e2e echoed each vendor's whole response into the log. A
Bedrock listing embeds inference-profile ARNs, and an ARN carries the
12-digit AWS account id — so every run published one to a log anyone who
can see the run can read. Vertex project-scoped responses echo the
project id the same way.
Log the status instead. The ids line that follows is the finding, and
the assertion messages still carry the body, but those render only on a
failure that needs diagnosing.
A dispatch can now narrow the run to a single package instead of paying
the sixteen minutes the container suite costs, which is the difference
between iterating on one test in under a minute and not iterating on it.
The pattern reaches the shell through an env var rather than being
interpolated into the run script, since a dispatch input landing
directly in a shell command is a script-injection seam however trusted
the dispatcher is.
The live run proved the routing change: Bedrock answered
GET /inference-profiles with 200 and real inference profiles from the
control plane. The test then failed anyway, because its own id extractor
only knew {"data":[{"id":…}]} and reported the response as 'not a
listing' — the one shape the proxy had just been taught to filter.
Read both envelopes here, for the same reason the filter reads both: the
two have to stay in step, or this test contradicts the code it covers.
A Bedrock provider could never answer a discovery request. The router
routed GET /inference-profiles to the record's upstream, which has to be
bedrock-runtime.<region> for InvokeModel to work, and that host does not
implement the operation — AWS answers <UnknownOperationException/>.
ListInferenceProfiles lives on the control plane at bedrock.<region>.
Give the route a discovery host, taken from the catalog's declaration
with the region read back out of the configured upstream, and send the
listing — and only the listing — there. Inference is untouched, and a
proxied or self-hosted Bedrock endpoint gets no discovery host at all
rather than a guessed one, since inventing a host would send the
operator's credential somewhere they never configured.
Two things had to follow for the listing to be usable once it arrives.
The response filter only understood OpenAI's {data:[{id:…}]}, so a
Bedrock listing was forwarded whole — offering every profile in the
account whatever the policy said. It now recognises the
inferenceProfileSummaries envelope, and matches a listing id against the
record's models after stripping the region prefix and version suffix, so
the two spellings of one model line up.
The policy bound had the same problem from the other side: it intersected
by exact string, so a record registering the raw profile id while a
guardrail names the catalog key intersected to nothing and would have
bounded a working provider's listing down to empty. routeClaimsModel
already normalises the candidate for this reason; the bound now agrees
with it.
The live discovery e2e flips from asserting the 404 to asserting a real
filtered listing. The mock upstream cannot cover any of this: it answers
/inference-profiles on the same listener as everything else, so a
mock-based test passes whichever host the request went to.
The endpoint reported pricing_known and then made the operator find the
price themselves. The dashboard has nothing to prefill a model row with, so
a discovered model either arrived at zero — silently metering every request
against it as free — or had to be priced by hand against a table NetBird
already ships.
Each model now carries the rates it would actually be billed at.
Rates come from the live default pricing table rather than the compiled-in
catalog, because that is the table the synthesiser ships to the proxy: an
operator running a defaults_llm_pricing.yaml would otherwise be shown one
price in the form and charged another. It is the same lookup the catalog
endpoint prefills from, so a model reached by either route prices
identically — pinned by TestDiscoveredRatesMatchTheCatalogEndpoint, since
the two are separate call paths that would otherwise drift.
pricing_known now derives from that same lookup instead of a second pass
over the compiled catalog, so "we can price this" and "here is the price"
can no longer disagree.
input_per_1k and output_per_1k are required and sent even at zero: an
unpriced model is offered at zero and flagged rather than withheld — the
vendor says the credential can reach it, and hiding it would hide a model
the operator genuinely has. The cache rates stay absent when unset, matching
the catalog response, because a zero there reads as "free" rather than "not
applicable".
Five smaller points from the same review as the four already fixed.
**The no-redirect policy had no test.** Every Fetch test injects an
HTTPClient, which bypasses httpClient() and therefore the policy entirely,
so nothing asserted that a 302 is refused — and the policy is a security
control: a redirect moves the request to a host checkPublicHost never
resolved. TestRedirectsAreNotFollowed drives the real constructor against an
httptest server that redirects to the cloud metadata address, and asserts
exactly one request leaves the client.
**An injected client silently lost that policy.** Production is safe today
only because NewManager passes a nil HTTPClient; any future non-test
injection would have dropped the guarantee with no signal. An injected
client that states no policy now inherits ours.
Implemented by copying the client rather than assigning into it. Writing
c.HTTPClient.CheckRedirect from httpClient() would mutate a struct shared by
every request goroutine for the process's lifetime — a data race, and the
exact hazard the same review's next point warns about. The copy shares the
Transport, which is safe for concurrent use by design.
**Two comments described things that were not true.** The manager's doc
claimed reading a stored credential "is permission-checked as one"; there is
no per-record Read check, just the single Create check, which covers it
because Create is stronger and the lookup is account-scoped. Said that
instead. The modelDiscovery field now records that it is shared across
requests and must stay read-only after construction.
**The handler test asserted neither the label contract nor the upstream.**
The response omits label entirely when a vendor supplies none, and the
dashboard falls back to the id on absence — an empty string would render a
blank row. The fixture had no label-less model to prove it with (the review
described one, but both existing entries carry labels), so this adds one.
The upstream assertion matters because Bedrock's region is read back out of
it. Also fixes the "blank catalog provide" subtest name.
Four points from the review of #7246, all confirmed against the code.
**regionFromUpstream panicked on Bedrock's regionless endpoint.** The
template is "bedrock-runtime.<region>.amazonaws.com", so the two fixed
halves are "bedrock-runtime." and ".amazonaws.com". The regionless host
"bedrock-runtime.amazonaws.com" carries both at once, with the halves
overlapping rather than sandwiching a region — it satisfied HasPrefix and
HasSuffix, then sliced host[16:15]:
panic: runtime error: slice bounds out of range [16:15]
That is reachable from any operator who types that host into upstream_url on
a Bedrock record. The length check makes the overlap read as "no region
here", which is what it is.
**A DNS-rebinding window sat between the guard and the dial.**
checkPublicHost resolved the host and the transport resolved it again to
dial, and the name's owner picks both answers. Public to the first lookup,
127.0.0.1 to the second, and the request reached loopback carrying the
operator's provider credential. The dialer now re-checks at the socket
through net.Dialer.Control, which runs post-resolution and pre-connect for
each address tried, so it sees what the second lookup actually returned.
The transport is cloned from http.DefaultTransport to keep its proxy and
TLS behaviour, and shared package-wide so the connection pool survives.
**Caller-input failures answered 500.** An unknown provider, an unusable
upstream, a region that cannot be read and a missing key are all reachable
from a well-formed request with a bad field value, and the OpenAPI document
already declares 400 for this endpoint. They now carry ErrInvalidRequest and
the handler branches on the sentinel rather than on message text. The
non-public-address refusal is included: that is the caller's own URL.
**The catalog id was trimmed for the emptiness test and then discarded.** A
padded " openai_api " cleared the check and reached catalog.Lookup with its
spaces, so the operator was told their provider was unknown.
Adds POST /api/agent-network/catalog/providers/models, so the provider
form can offer the models an operator's own credential can reach instead
of only the compiled-in catalog.
A caller names a catalog provider and supplies either the key they are
typing (the record does not exist yet) or the id of a saved record whose
stored credential should be reused — which lets the dashboard refresh a
list without ever holding the key. The two are mutually exclusive:
accepting both would run an arbitrary credential under the identity of a
record the caller may only be permitted to read. When a record id is
given, the catalog id and upstream come from the record too, so the
credential cannot be aimed at a different vendor's endpoint.
Gated on Create rather than Read. This spends the operator's credential
against a third party, which is not something a read-only role should be
able to make the server do.
A provider with no listing endpoint answers 422 rather than 500: the
caller falls back to the catalog's own models on that outcome, so it has
to be distinguishable from a failure.
The region is read back out of the configured upstream by matching it
against the catalog's host template, since a provider record has no
region field and the operator already encoded one when they set up
inference. An upstream matching no template is refused rather than
guessed at — a wrong region would dial another account's endpoint.
The catalog is the only source of models an operator can pick from, and
it cannot know two things that matter. It goes stale — its entries carry
comments recording which models a vendor retired on which date — and it
cannot see an account: which OpenAI models an org is entitled to, which
Bedrock inference profiles a given account and region hold, which Vertex
models a project has enabled.
Add a client that asks the vendor directly, with the endpoint, auth
header and response shape all declared by the catalog rather than
supplied by the caller. The four shapes come from probing the live APIs
(see the discovery e2e); each vendor invented its own envelope and none
can be guessed from the request.
Bedrock is the case that shaped the design. Its listing lives on the
control plane while inference must go to the runtime host, so Discovery
carries its own host. The ids it returns are region-prefixed and are
taken verbatim, because that prefix is what AWS requires and it cannot
be derived from the configured region — an eu-central-1 account holds
global.* profiles alongside its eu.* ones.
The vendor is authoritative for the id; the catalog stays authoritative
for pricing. A discovered model the shipped table cannot price is
reported as such, so it cannot be registered at a silent zero rate.
Management has not made outbound calls on an operator's behalf before,
and it holds a credential for every provider, so the host is checked
before dialing: every resolved address must be public, which covers the
cloud metadata address and NetBird's own overlay range, and redirects
are not followed since they would move the request to a host the check
never saw.
Review point on #7239: the two setup checks in joinClient reported through
t.Fatalf while every other check in the suite goes through require. Same
outcome, one less shape to read.
The WaitProxyPeer check stays behind an if rather than being passed straight
to require.NoError: the message interpolates the proxy container's whole log,
and require evaluates its arguments before it knows the assertion passed. As
written the fetch happens only on the failure it exists to explain.
ResolveProxyIP exists to wake the lazy proxy peer, and it retried only
curl exit 6 — DNS. The wake-up attempt that arrives before WireGuard has
brought the tunnel up fails with exit 7 instead, and that returned
immediately:
no HTTP response from vast-azalea.netbird.local: exit status 7
(curl: (7) Failed to connect ... after 0 ms)
So the one function whose job is to tolerate a not-yet-ready endpoint
failed on the readiness state it was written for, one second after the
client container reported ready. Retry both exit codes within the same
window; anything else would still be failing when the window closed and
still fails immediately.
Raise the access-log ingest window to 60s for the same reason. The proxy
streams each entry with a 10s send timeout of its own, so 30s left
barely three attempts of headroom before a test that had already got its
200 was failed for a row still in flight.
The first live run answered the question the mock could not. OpenAI and
Anthropic both filter correctly against real catalogues — Anthropic's
dated claude-haiku-4-5-20251001 survives a record registering the
undated id, and OpenAI's listing comes back as the single model the
guardrail permits. Vertex is refused by the proxy, as intended.
Bedrock is the one that was wrong, and wrong about something worth
recording: GET /inference-profiles reaches AWS and AWS answers
<UnknownOperationException/>. ListInferenceProfiles is a control-plane
operation on bedrock.<region>.amazonaws.com; a provider record carries a
single upstream and it must be the runtime host for InvokeModel to work,
so no Bedrock record can serve a listing as the model stands. The mock
serves that path on the same listener as everything else, which is
exactly why this went unnoticed.
Replace the routed/filtered pair with an explicit outcome, since the
three cases are different contracts rather than degrees of success, and
tell apart 'the proxy refused' from 'the vendor refused' by whether the
body names a middleware — no upstream error body does. The two
non-listing outcomes now issue a single request instead of retrying for
the full window waiting on a status that is never coming, which is where
92 of the failing run's 136 seconds went.
The mock upstream advertises ids we chose, so a listing narrowing to the
ones we authorised is arithmetic we controlled both sides of. It cannot
show the filter surviving a real catalogue: ids we never enumerated,
dated builds whose suffix the vendor picks, surfaces that answer a
listing request with something that is not a listing.
Cover the four surfaces against their real endpoints, each gated on its
own credential so a partial key set still yields partial coverage:
- OpenAI enumerates two real models and the policy permits one, so
both bounds are observable at once against a catalogue of dozens.
- Anthropic returns dated build ids while the record registers the
undated one, which exercises date-normalisation on ids the vendor
chose. This is also the surface Claude Code actually calls.
- Bedrock lists inference profiles rather than models; the request is
routed but not model-bounded, since filtering keys on /v1/models.
- Vertex serves no listing at all, so discovery must be refused rather
than rewritten onto an upstream that would 404 it.
One proxy serves every case, with a group, policy and client per
provider: a model-less request matches exactly one route, so two
providers authorised for the same caller would leave one untested.
Every response is logged before anything is asserted on it. A live
catalogue is the one input the suite does not control, so a failure has
to arrive carrying the response that caused it.
The discovery isolation test drove a single client in the main group.
VLLMUnlistedModel's absence from that client's listing was consistent
with two different worlds: the listing being scoped to the caller's
policy, or the other team's policy never having applied at all. The
test passed either way, so it did not prove what it claimed.
Mint a setup key per group and join a second client on the other
group, reusing the running proxy. The other client must see its own
model before the main client's listing is asserted, and must not see
the main group's model — isolation is checked in both directions.
The listing was narrowed by the provider record's enumerated models, which is
the right bound only while one policy reaches a provider. Where two teams
share a provider under different allowlists, every caller was offered the
union: each model outside their own policy is a request the guardrail refuses
a moment later, which is the empty-or-wrong picker this endpoint exists to
avoid, moved one level up. A gateway record enumerating nothing was worse
still — it offered the upstream's entire catalogue however narrow the policy.
The synthesiser already knows which policies authorise a provider and which
groups each binds, so the router can answer this at request time where it
knows the caller's groups. Each route now carries one rule per authorising
policy — its source groups and the models it permits — and the listing is
bounded to the union across the rules matching the caller, intersected with
what the provider serves.
This is deliberately finer than the guardrail's own per-provider allowlist,
which stays as it is: that list is a fail-closed backstop and cannot tell who
is asking, so discovery is now narrower than the backstop rather than wider.
A policy setting no allowlist lifts the restriction for the groups it binds,
so nil and empty model lists stay distinct end to end — collapsing them would
let a listing that should offer nothing fall open to everything.
Publishing the chain before the route only helps if the chain is there.
rebuildMiddlewareChains logged its error and returned, so a failed rebuild
still fell through to AddMapping and published a route over chains that were
never installed — served with no policy enforcement and no metering, which
is the outcome the ordering change exists to prevent.
Report the error instead. The caller already unwinds a failed setup, so the
service stays unpublished rather than reachable and uncounted. An unset
middleware manager is still not an error: that is a deployment without
middleware, not a failure to install it.
A provider update added the proxy mapping and then rebuilt the middleware
chain. Between the two, the route was live with no chain behind it, and a
request that landed there was served straight through — a successful
inference that was neither routed by policy nor metered.
Rebuild first. The worst a request in the remaining window meets is the new
chain in front of the previous target, which is still counted.
The loop bounded when a new attempt could start, not how long one could
run. The chat container is capped at 90 seconds of its own and the row
lookup at another 20, so an attempt begun just inside the 180-second window
could report a repricing failure nearly two minutes after that window
closed. Run every call in the loop under a context that expires with the
deadline.
Three open review findings.
The discovery filter treated every slash in a listed id as a gateway prefix
and matched the tail against the policy. A self-hosted id carries slashes of
its own, and an upstream may scope ids per tenant, so "tenant-b/claude-sonnet-5"
matched a permitted "claude-sonnet-5" and reached the picker — a model the
policy never named, and one the guardrail denies on sight, since enforcement
compares the id as written. Strip only the namespaces a gateway is known to
prepend, taken from the first slash rather than the last.
The e2e retry loops slept between attempts without watching the context, so a
cancelled run kept retrying calls that fail instantly and spent its remaining
window sleeping between them. They now stop when the context is done.
The streamed provider's setup key outlived its test: deleting the group does
not delete the key that auto-joins it.
The failure message read repriced.InputCostUsd, and repriced is the zero
value on every path that reaches that line — so a run that gave up always
reported "last input_cost_usd=$0.000000", which reads as a row priced at
zero rather than as a row still at the old rate, or as no row at all.
Keep the last cost read and report that, saying so plainly when no row was
ever read.
The poll interval was not bounded by the window: a page that came back
without the row just before the deadline still slept a full two seconds
before the loop noticed it was out of budget, so the caller waited longer
than the window it asked for to be told nothing arrived.
Cap the wait at whatever is left of the window.
Two review findings on the e2e suite.
lookupAccessLogBySession polled under the caller's context, so one stalled
request could hold the loop open well past the 30s ingest window it exists to
enforce — and the caller would read that delay as a missing row rather than a
slow one. Each poll now expires with the window; the parent's cancellation
still applies, since the request context derives from it.
The streaming test asserted only that the total cost was positive. Input and
output are positive on their own, so a cache-read bucket that was parsed and
then never billed would have passed. Assert the sum of the three buckets: the
gap a dropped cache read leaves is 7e-6, well outside the delta.
Six behaviours had unit coverage only, either because they arrived from code
review after the end-to-end tests were written or because no request in the
suite had the shape that reaches them.
Streaming is the important one. Input tokens exist only in a stream's opening
message_start event, and reading a stream with the wrong vendor's parser
misses it — the metering bug this endpoint's protocol work fixed. Nothing in
the suite sent stream: true, so the branch never ran. The mock now serves an
SSE surface on a second listener, reporting counts that differ from its
buffered ones so a passing assertion can only mean the stream accumulator ran,
and one case drives it through a record typed for the wrong surface.
The rest need no new harness capability: the per-model lookup against the
allowlist, the read-method gate on the non-inference paths, dated ids reaching
an undated registration while a pinned build refuses a different one, the
Bedrock inference-profile lookup reaching its upstream rather than a policy
denial, and a custom dated id keeping its own price.
Sub-agent ids stay uncovered: the parser lifts them onto request metadata but
nothing persists them, so there is no queryable surface to assert against
until that half lands. Covered here only to the extent that sending the
headers leaves the request served and metered.
TestPriceChangeUpdatesRecordedCost drives requests in a loop until one is
priced at the new rate, because the price push and the proxy's chain rebuild
are async. The loop could not actually retry: it looked the row up through
findAccessLogBySession, which fails the test outright when no row lands
within 30s, so the first post-update request that produced no row ended the
run instead of yielding to the next attempt.
That is the observed failure — the nightly run has been red on this test
roughly half the time, always with "session id ...-reprice-b-... must be
recorded in an access-log row" after ~41s: container setup, one request, one
30s wait, dead.
A missing row there is expected rather than exceptional. The provider update
rebuilds the middleware chain, and a request served mid-rebuild can complete
without a resolved provider: 200 to the caller, nothing to attribute, so no
row is ever written for it. Split the polling helper into a non-fatal lookup
and keep the fail-fast wrapper for callers whose row must exist, then treat a
miss in the loop like any other not-yet-repriced iteration. Only the outer
deadline is fatal.
Shorten the per-attempt wait to 20s and raise the overall deadline to 180s so
several attempts fit where before the budget allowed barely one.
Two changes to routeClaimsModel, both about dated Anthropic ids.
Normalising the configured candidate as well as the requested model made
every dated build of a family interchangeable: a route registered against
claude-sonnet-4-5-20250101 also claimed ...-20250202, so an operator who
pinned a build deliberately would have served a different one, and with
several such routes declaration or path order decided which. Only an undated
registration now absorbs a dated request.
The per-model lookup also stamps the model its path names, so the guardrail's
allowlist — a separate and possibly narrower list than the route's — still
decides GET /v1/models/{id} rather than seeing no model at all.
The guardrail enforces its own per-provider model allowlist and fails closed
when the request names no model, which is right for a path-routed inference
request whose shape the parser could not read. GET /v1/models names no model
anywhere, so discovery still denied with model_unknown for exactly the
accounts that configured an allowlist — the case skipping the management
pre-flight was meant to fix. Only one of the two gates had been opened, and
a client reads the 403 as an empty model picker.
Exempt requests the router marked non-inference from the unknown-model
branch. A named model is still checked, so the exemption covers only the
endpoints that genuinely name nothing: the listing and the warm probe.
The discovery filter read one byte past its 1 MiB cap to detect a body too
large to rewrite, then closed the upstream body and forwarded the buffer it
had — the response reached the client truncated at exactly the cap, with
Content-Length rewritten to match so nothing looked wrong until the client
tried to parse it.
Splice the bytes already read back in front of the unread remainder and
forward the response as the upstream sent it, headers untouched.
The router classified a non-inference request by path alone, so a POST to
/v1/models/{id} — or to the listing, or the warm probe — was marked
llm.non_inference and skipped the limit check's management pre-flight, even
though such a request can carry an inference body.
Require GET or HEAD, the methods these endpoints actually use. Anything else
falls through to normal per-model routing, which routes on the body's model
under the usual pre-flight, or denies as missing-model when there is none.
Invoke had grown a branch per API surface, each repeating the same
found/unauthorised/unknown switch. Lift the shared denial arms into decide(),
move the model-less endpoints into their own method, and name the two allow
decorations (non-inference marking, Bedrock namespace stripping) so each
surface reads as the one thing it does differently.
No behaviour change.
GET /v1/models/{id} was folded into the model-less endpoints so it would
route rather than deny. Once model-less requests started skipping the
management pre-flight, that also skipped the per-model allowlist: a caller
could confirm the existence and reachability of a model the route does not
list, even though the listing beside it is bounded to that same allowlist.
Resolve the id from the path and route it through the model table like any
other per-model request, keeping model-less treatment for the listing and
the connection-warming probe only. It stays marked non-inference, since the
lookup spends no tokens. A gateway route that enumerates no models still
answers every lookup, as before.
Also record why the Bedrock inference-profile lookup is forwarded rather
than denied: those live on the AWS control plane, and forwarding reproduces
what an unproxied client with the same base URL would see.
The release-date normalizer matched a bare "-YYYYMMDD" suffix on any id.
Pricing looks every model up through it regardless of surface, and an
operator can register a custom model under any id at all, so a custom
"internal-llm-20250101" would silently inherit the rate registered for
"internal-llm".
Anchor the pattern on "claude" so it still covers the vendor-prefixed
Bedrock forms while leaving every other vendor's id untouched.
The routing and parser-selection fixes touch every provider surface, and
the unit tests only prove each side of a seam in isolation. Two suites
close that:
The provider matrix drives one request per wire shape over a single tunnel,
with a record per catalog surface behind it, and asserts the surface each
request was metered under together with the token counts that surface's own
usage block carries. A response read by the wrong provider's parser meters
zero, so a regression fails instead of passing on a coincidental non-zero.
It also covers the Bedrock and Vertex token-counting paths, the warm-up
probe, and the vendor error envelope on a refusal.
The discovery suite covers the configuration that broke: an account with a
model allowlist, where the listing carries no model and the gate failed
closed. It asserts the listing is served, that it is bounded to the
authorised model, and that inference outside the allowlist is still
refused, so the exemption cannot be read as a way around the gate.
The mock upstream grows the Anthropic, Bedrock and token-counting shapes so
one container stands in for every surface, and the client gains GET and
arbitrary-POST helpers for the endpoints that carry no chat body.
The filter treated a slash in a listing entry's id as a gateway provider
prefix and matched only the tail. Self-hosted backends serve ids that carry
a slash of their own, so every "Qwen/Qwen2.5-0.5B-Instruct" style model was
dropped from the picker even when the policy named it exactly.
Try the id as written first and fall back to the tail, so both a prefixed
id and a self-hosted one resolve.
A few client-side checks call their vendor directly instead of following
the configured base URL, so they fail on exactly the locked-down networks
Agent Network is built for while inference keeps working. Fast mode reports
a connectivity error, or reports itself disabled by the organization when
the agent holds only a proxy-issued token, and model discovery stays off
until it is turned on explicitly.
Name the variables that settle each case, and say plainly which ones
allowing direct egress does not fix.
A coding agent that spawns helpers stamps the spawned agent's id on every
request it makes, and the spawning agent's id when that helper is nested.
The parser read the session header and ignored both, so parallel agents
inside one session all attributed to the session alone and there was no way
to see which one spent the tokens.
Emit them as metadata alongside the session id. They are opaque grouping
identifiers rather than content, so they are stamped regardless of the
prompt-collection toggle. Persisting them as queryable access-log columns
is a schema change and is deliberately not part of this commit.
Discovery proxies the upstream's full list, so the picker offers every
model the shared provider key can reach and each one outside the policy is
a request the chain denies a moment later. Restricting models is the point
of the product, and the client had no way to see the restriction.
Carry the resolved route's model list on the upstream rewrite and drop the
rest from the listing response. Only a route that enumerates its models
bounds anything: a catch-all claims every model, so its list passes
through. Anything the filter cannot safely rewrite, including a compressed
or oversized body, reaches the client untouched.
Clients send HEAD /api/hello before their first inference request to open
the upstream connection early. The path carries no model, so it denied as
not-routable and each session start left a policy rejection in the access
log for a request that was never a policy question.
Treat it as a model-less endpoint. Forwarding it warms the connection the
first real request will use, which is what the probe is for.
A client resolving a configured inference profile calls
GET /inference-profiles at startup. The path carries no model and was not
recognised as non-inference, so it denied as not-routable and wrote a
policy rejection into the access log on every session start, which is the
log operators read to find real policy problems.
Recognise the path and match it against a Bedrock provider specifically:
sending it to whichever provider happened to be authorised first would
rewrite it to an upstream that 404s it. The optional gateway namespace is
stripped the same way the runtime paths strip it.
Vertex hangs token counting off the model as its own path segment, and the
parser split the tail on the final colon alone. A count-tokens request
therefore reported its model as "claude-sonnet-5/count-tokens", which no
route claims, so the request denied as not-routable and the access log
recorded a model that does not exist.
Stop at the first "/" after the model id so the method segment stays out of
it, leaving the client free to price its context against the dedicated
endpoint instead of the billable inference one.
Both the request parser and the router enumerated Bedrock actions without
count-tokens, so the path carried no model and the request denied as
not-routable. Nothing breaks outright, because the client falls back to
counting context through the inference endpoint, but that fallback is
billable and the dedicated endpoint exists to avoid exactly that.
The action carries a model in the path and returns no usage, so it routes
like any other Bedrock action and meters to zero.
A budget stop, a blocked model or an unroutable model all rendered as the
NetBird deny envelope alone. LLM clients only parse their own provider's
error shape, so the reason never reached the user: Claude Code showed an
unexplained API error where it could have shown the policy message.
Carry the resolved surface on the deny reason and add the vendor's error
object next to the existing fields. The body stays a superset of what it
was, so anything reading code, message, details or middleware is
unaffected. Status codes are unchanged here: mapping window caps to 429
needs the window reset plumbed through the limits response before a
correct Retry-After can be sent.
Gateway records enable body-level identity so LiteLLM's tag-budget check
can read it, and the injector wrote "user" and "metadata.tags" into every
JSON object regardless of dialect. Claude Code reaches those same records
on /v1/messages, where "user" is not a permitted top-level field and
metadata accepts only "user_id", so the upstream rejected the request with
a 400 naming a field the client never sent. Rewriting the body also
changed the bytes a gateway-side prompt cache keys on.
Gate the body write on the surface llm_request_parser resolved from the
path. Header stamping is untouched, so spend tracking and per-end-user
budgets keep working on the surfaces that lose the body path.
shared/llm normalizes Bedrock and Vertex model ids so both sides of the
routing and pricing contract compare equal, but nothing did the same for a
first-party Anthropic id. A client pinning "claude-sonnet-4-5-20250929"
against a record registered as "claude-sonnet-4-5" denied as not-routable,
and where a catch-all route carried it through, the price lookup missed and
the request recorded no cost.
Add NormalizeAnthropicModel beside the existing two and consult it after an
exact match fails in the router's claim check, the pricing table, and the
per-record price map. Exact matches still win, so an operator who registers
two dated releases of the same family keeps them distinct.
