A discovery failure in the live e2e rendered the vendor's response into the
assertion message. When the vendor is Bedrock, that response is an AWS
refusal naming the resource it refused, and the name is an ARN carrying the
12-digit account id — into a job log anyone who can see the run can read.
The earlier change kept the body out of the success log and left the failure
paths quoting it, which is the path that actually carries the refusal.
Report the response's size and top-level keys instead. That is what the
failure is diagnosed from anyway: which envelope arrived, not what was in it.
The discovery e2e echoed each vendor's whole response into the log. A
Bedrock listing embeds inference-profile ARNs, and an ARN carries the
12-digit AWS account id — so every run published one to a log anyone who
can see the run can read. Vertex project-scoped responses echo the
project id the same way.
Log the status instead. The ids line that follows is the finding, and
the assertion messages still carry the body, but those render only on a
failure that needs diagnosing.
The live run proved the routing change: Bedrock answered
GET /inference-profiles with 200 and real inference profiles from the
control plane. The test then failed anyway, because its own id extractor
only knew {"data":[{"id":…}]} and reported the response as 'not a
listing' — the one shape the proxy had just been taught to filter.
Read both envelopes here, for the same reason the filter reads both: the
two have to stay in step, or this test contradicts the code it covers.
A Bedrock provider could never answer a discovery request. The router
routed GET /inference-profiles to the record's upstream, which has to be
bedrock-runtime.<region> for InvokeModel to work, and that host does not
implement the operation — AWS answers <UnknownOperationException/>.
ListInferenceProfiles lives on the control plane at bedrock.<region>.
Give the route a discovery host, taken from the catalog's declaration
with the region read back out of the configured upstream, and send the
listing — and only the listing — there. Inference is untouched, and a
proxied or self-hosted Bedrock endpoint gets no discovery host at all
rather than a guessed one, since inventing a host would send the
operator's credential somewhere they never configured.
Two things had to follow for the listing to be usable once it arrives.
The response filter only understood OpenAI's {data:[{id:…}]}, so a
Bedrock listing was forwarded whole — offering every profile in the
account whatever the policy said. It now recognises the
inferenceProfileSummaries envelope, and matches a listing id against the
record's models after stripping the region prefix and version suffix, so
the two spellings of one model line up.
The policy bound had the same problem from the other side: it intersected
by exact string, so a record registering the raw profile id while a
guardrail names the catalog key intersected to nothing and would have
bounded a working provider's listing down to empty. routeClaimsModel
already normalises the candidate for this reason; the bound now agrees
with it.
The live discovery e2e flips from asserting the 404 to asserting a real
filtered listing. The mock upstream cannot cover any of this: it answers
/inference-profiles on the same listener as everything else, so a
mock-based test passes whichever host the request went to.
The first live run answered the question the mock could not. OpenAI and
Anthropic both filter correctly against real catalogues — Anthropic's
dated claude-haiku-4-5-20251001 survives a record registering the
undated id, and OpenAI's listing comes back as the single model the
guardrail permits. Vertex is refused by the proxy, as intended.
Bedrock is the one that was wrong, and wrong about something worth
recording: GET /inference-profiles reaches AWS and AWS answers
<UnknownOperationException/>. ListInferenceProfiles is a control-plane
operation on bedrock.<region>.amazonaws.com; a provider record carries a
single upstream and it must be the runtime host for InvokeModel to work,
so no Bedrock record can serve a listing as the model stands. The mock
serves that path on the same listener as everything else, which is
exactly why this went unnoticed.
Replace the routed/filtered pair with an explicit outcome, since the
three cases are different contracts rather than degrees of success, and
tell apart 'the proxy refused' from 'the vendor refused' by whether the
body names a middleware — no upstream error body does. The two
non-listing outcomes now issue a single request instead of retrying for
the full window waiting on a status that is never coming, which is where
92 of the failing run's 136 seconds went.
The mock upstream advertises ids we chose, so a listing narrowing to the
ones we authorised is arithmetic we controlled both sides of. It cannot
show the filter surviving a real catalogue: ids we never enumerated,
dated builds whose suffix the vendor picks, surfaces that answer a
listing request with something that is not a listing.
Cover the four surfaces against their real endpoints, each gated on its
own credential so a partial key set still yields partial coverage:
- OpenAI enumerates two real models and the policy permits one, so
both bounds are observable at once against a catalogue of dozens.
- Anthropic returns dated build ids while the record registers the
undated one, which exercises date-normalisation on ids the vendor
chose. This is also the surface Claude Code actually calls.
- Bedrock lists inference profiles rather than models; the request is
routed but not model-bounded, since filtering keys on /v1/models.
- Vertex serves no listing at all, so discovery must be refused rather
than rewritten onto an upstream that would 404 it.
One proxy serves every case, with a group, policy and client per
provider: a model-less request matches exactly one route, so two
providers authorised for the same caller would leave one untested.
Every response is logged before anything is asserted on it. A live
catalogue is the one input the suite does not control, so a failure has
to arrive carrying the response that caused it.
mlsmaycon