[proxy,management] Conform the Agent Network endpoint to the LLM gateway protocol (#7154)

[proxy,management] Conform the Agent Network endpoint to the LLM gateway protocol

Reviewed the proxy against Claude Code's published gateway contract. The
transport layer already held up; fourteen gaps sat one layer up, in the model
catalog and in the non-inference endpoints clients call.

Two of them cost money. The catalog carried no claude-opus-5 or
claude-sonnet-5, so an operator could not authorise the models coding agents
default to — those requests denied as not-routable, or priced at zero where a
catch-all carried them. And gateway records pin ParserID "openai" while the
same record serves /v1/messages, so Anthropic responses were read with the
OpenAI parser, which never looks at message_start where input tokens live:
input metered as roughly zero on every stream and cost was skipped entirely.

The rest fix requests refused for structural rather than policy reasons: model
discovery denied for every account with a model allowlist, token counting
denied on Bedrock and mis-parsed on Vertex, startup probes refused and written
into the access log at every session start, and denials rendered in a shape no
LLM client parses. Two changes are additive by design — the deny body keeps
every field it had and adds the vendor's error object alongside, and body-level
identity injection is now gated on the request's dialect so it stops sending
OpenAI-shape fields into Anthropic bodies that reject them.

The end-to-end work turned up one more: the discovery filter treated any slash
in a model id as a gateway prefix, which would have dropped every self-hosted
"Qwen/..." model from the picker.
This commit is contained in:
Maycon Santos
2026-08-23 20:02:33 +02:00
committed by GitHub
parent ee253feddf
commit 766fcae3f8
39 changed files with 3094 additions and 154 deletions
@@ -224,3 +224,35 @@ func TestMetadataKeys_Allowlist(t *testing.T) {
}
assert.ElementsMatch(t, want, keys)
}
// TestInvoke_NonInferenceSkipsPreflight covers gateway model discovery:
// GET /v1/models carries no model, and management's per-model allowlist
// fails closed on an empty one, so a pre-flight would deny discovery for
// exactly the accounts that use the model allowlist. The router marks the
// request non-inference after authorising the route, and the gate must
// then allow without calling management at all.
func TestInvoke_NonInferenceSkipsPreflight(t *testing.T) {
mgmt := &fakeMgmt{
checkResp: &proto.CheckLLMPolicyLimitsResponse{
Decision: "deny",
DenyCode: "llm_policy.model_blocked",
},
}
m := New(mgmt, nil)
out := runInvoke(t, m, &middleware.Input{
AccountID: "acc-1",
UserID: "user-bob",
UserGroups: []string{"grp-engineers"},
Metadata: []middleware.KV{
{Key: middleware.KeyLLMResolvedProviderID, Value: "prov-1"},
{Key: middleware.KeyLLMNonInference, Value: "true"},
},
})
assert.Equal(t, middleware.DecisionAllow, out.Decision, "model-less endpoints must not be gated on a model")
assert.Nil(t, mgmt.checkReq, "no pre-flight may be sent for a non-inference request")
assert.Empty(t, lookupKV(out.Metadata, middleware.KeyLLMSelectedPolicyID),
"no policy is attributed when nothing was metered")
}