[proxy,management] Conform the Agent Network endpoint to the LLM gateway protocol (#7154)

[proxy,management] Conform the Agent Network endpoint to the LLM gateway protocol

Reviewed the proxy against Claude Code's published gateway contract. The
transport layer already held up; fourteen gaps sat one layer up, in the model
catalog and in the non-inference endpoints clients call.

Two of them cost money. The catalog carried no claude-opus-5 or
claude-sonnet-5, so an operator could not authorise the models coding agents
default to — those requests denied as not-routable, or priced at zero where a
catch-all carried them. And gateway records pin ParserID "openai" while the
same record serves /v1/messages, so Anthropic responses were read with the
OpenAI parser, which never looks at message_start where input tokens live:
input metered as roughly zero on every stream and cost was skipped entirely.

The rest fix requests refused for structural rather than policy reasons: model
discovery denied for every account with a model allowlist, token counting
denied on Bedrock and mis-parsed on Vertex, startup probes refused and written
into the access log at every session start, and denials rendered in a shape no
LLM client parses. Two changes are additive by design — the deny body keeps
every field it had and adds the vendor's error object alongside, and body-level
identity injection is now gated on the request's dialect so it stops sending
OpenAI-shape fields into Anthropic bodies that reject them.

The end-to-end work turned up one more: the discovery filter treated any slash
in a model id as a gateway prefix, which would have dropped every self-hosted
"Qwen/..." model from the picker.
This commit is contained in:
Maycon Santos
2026-08-23 20:02:33 +02:00
committed by GitHub
parent ee253feddf
commit 766fcae3f8
39 changed files with 3094 additions and 154 deletions

View File

@@ -2074,9 +2074,17 @@ func (s *Server) updateMapping(ctx context.Context, mapping *proto.ProxyMapping)
return fmt.Errorf("auth setup for domain %s: %w", mapping.GetDomain(), err)
}
m := s.protoToMapping(ctx, mapping)
s.proxy.AddMapping(m)
// The chain is published before the route that leads to it. A request
// arriving at a target whose chain has not been rebuilt yet is served
// straight through, so a provider update that added the route first left a
// window in which an inference could complete unrouted and unmetered.
// Rebuilding first inverts that: the worst a request in the window meets is
// the new chain in front of the previous target, which is still counted.
if err := s.rebuildMiddlewareChains(svcID, m); err != nil {
return err
}
s.meter.AddMapping(m)
s.rebuildMiddlewareChains(svcID, m)
s.proxy.AddMapping(m)
return nil
}
@@ -2114,15 +2122,21 @@ func (s *Server) initMiddlewareManager(ctx context.Context) error {
}
// rebuildMiddlewareChains converts m into per-path bindings and calls
// Manager.Rebuild. Short-circuits when the middleware manager is unset.
func (s *Server) rebuildMiddlewareChains(svcID types.ServiceID, m proxy.Mapping) {
// Manager.Rebuild. Short-circuits when the middleware manager is unset, which
// is a deployment without middleware rather than a failure to install it.
//
// A rebuild that fails is reported rather than logged: the caller publishes
// the route once this returns, and a route published over chains that were
// not installed serves requests with no policy enforcement and no metering.
func (s *Server) rebuildMiddlewareChains(svcID types.ServiceID, m proxy.Mapping) error {
if s.middlewareManager == nil {
return
return nil
}
bindings := buildMiddlewareBindings(svcID, m)
if err := s.middlewareManager.Rebuild(string(svcID), bindings); err != nil {
s.Logger.WithError(err).WithField("service_id", svcID).Error("failed to rebuild middleware chains")
return fmt.Errorf("rebuild middleware chains for service %s: %w", svcID, err)
}
return nil
}
// isLiveService reports whether svcID is currently present in the live