[management,proxy] Agent network: per-account LLM gateway (policy, metering, multi-provider) (#6555)

* [agent-network] Shared proto, OpenAPI schema, and generated types

* [agent-network] Management: store, manager, synthesizer, policy engine, provider catalog, HTTP/gRPC API

Adds the account-scoped agent-network module: provider/policy/budget CRUD and
store, the reverse-proxy service synthesizer, policy selection + limit
enforcement, the provider catalog (incl. Vertex AI and AWS Bedrock entries),
and the management HTTP + proxy gRPC surfaces.

* [management] Fix agent-network proxy-peer fan-out on affected-peer recompute

The affected-peers resolver loaded only persisted reverse-proxy services, but
agent-network services are synthesized on demand and never persisted. As a
result the embedded proxy peer was never folded into the affected set when a
client's group changed, so the proxy received no network-map update for a newly
authorised client and rejected its handshake until a full resync (restart).

loadProxyServices now merges the synthesized agent-network services (injected
via a registration hook to avoid an import cycle), so proxy peers learn newly
authorised clients immediately.

* [proxy] Reverse-proxy middleware framework, chain, and request plumbing

The per-target middleware chain (slots, dispatcher, mutation gate, metadata
merger), body capture, access-log terminal sink, and the proxy wiring that
builds + runs chains for synthesized agent-network services.

* [proxy] LLM parsers, pricing, and builtin middlewares (OpenAI, Anthropic, Vertex AI, AWS Bedrock)

Request/response parsers and SSE/event-stream metering, the embedded pricing
table, and the builtin middleware set: request parser, router, policy
limit-check/record, cost meter, guardrail, identity inject, response parser.
Includes the path-routed providers — Google Vertex AI (keyfile:: service-account
OAuth minting) and AWS Bedrock (bearer auth, invoke/converse/streaming, optional
/bedrock prefix) — plus the Models allowlist and unmeterable-publisher deny.

* [proxy] IPv6 in-place apply and TCP accept-loop hardening on netstack listeners

* [agent-network] End-to-end test suite, module docs, and deployment preset

* [agent-network] Fix codespell typos and exclude false positives

- labelgen word pool: vermillion -> vermilion, racoon -> raccoon.
- codespell ignore list: add flate (Go compress/flate package), recordin
  (a test-local identifier), and unparseable (a valid alternative spelling used
  consistently across identifiers + a metadata-value constant).

* [management] Set LastSeen on injected proxy peer in realstack test (MySQL strict-mode)

The injected embedded proxy peer had a PeerStatus with a zero LastSeen, which
serializes to '0000-00-00' and is rejected by MySQL in strict mode (SQLite
tolerates it). Set LastSeen to a valid time so SaveAccount succeeds on both
engines.

* [agent-network] Remove e2e shell-script suite from this branch

The end-to-end shell scripts under scripts/e2e/ are maintained in a separate
testing suite and are not part of this change set.

* [agent-network] Polish module docs: remove internal review scaffolding, fix links, verify diagrams

Strip PR-review framing, commit references, absolute paths, and stale internal
references from the agent-network module docs; fix broken relative links; verify
all diagrams against the current architecture. Remove the internal AI-reviewer
prompt file.

* [management] Refine session expiration handling to support 3-state encoding for SSO deadlines

* [agent-network] Relocate agentnetwork package to internals/modules

Move management/server/agentnetwork (and its catalog/, labelgen/, types/
subpackages) to management/internals/modules/agentnetwork, alongside the
reverse-proxy module, and rewrite all importers. Pure relocation: package names,
the synthesizer + affectedpeers registration hook, and store access (shared
store.Store) are unchanged, so no import cycle is introduced (affectedpeers
still depends only on the agentnetwork/types leaf).

* [agent-network] Co-locate HTTP handlers in the module (RegisterEndpoints)

Move the agent-network HTTP handlers from server/http/handlers/agentnetwork into
the module at internals/modules/agentnetwork/handlers (package handlers) and
rename the entrypoint AddEndpoints -> RegisterEndpoints, matching the
reverse-proxy module convention. Wiring in http/handler.go updated accordingly.
This commit is contained in:
Maycon Santos
2026-06-27 13:41:00 +02:00
committed by GitHub
parent 615631567a
commit b416063bcc
187 changed files with 36835 additions and 660 deletions
@@ -0,0 +1,108 @@
package llm_identity_inject
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"github.com/netbirdio/netbird/proxy/internal/middleware"
"github.com/netbirdio/netbird/proxy/internal/middleware/builtin"
)
// ProviderInjection describes one resolved provider's injection rule.
// Identity stamping uses one of HeaderPair / JSONMetadata; ExtraHeaders
// is independent — each entry is a static (operator-configured) header
// stamped on every matching request with anti-spoof. A rule with no
// shape AND no extras is dropped at New() time as a no-op.
type ProviderInjection struct {
// ProviderID is the resolved provider id — matches the value
// llm_router stamps under KeyLLMResolvedProviderID.
ProviderID string `json:"provider_id"`
// HeaderPair is the LiteLLM-style wire convention: separate
// headers for end-user id and tags CSV.
HeaderPair *HeaderPairRule `json:"header_pair,omitempty"`
// JSONMetadata is the Portkey-style wire convention: a single
// header carrying a JSON object keyed by reserved field names.
JSONMetadata *JSONMetadataRule `json:"json_metadata,omitempty"`
// ExtraHeaders is an operator-configured list of static headers
// (e.g. "x-portkey-config: pc-...") that the middleware stamps
// on every matching request. The synth pre-resolves the values
// from the provider record's ExtraValues map; the middleware
// just emits them. Each name is also added to HeadersRemove for
// anti-spoof so a client can't smuggle their own value.
ExtraHeaders []ExtraHeaderKV `json:"extra_headers,omitempty"`
}
// ExtraHeaderKV is one static header entry the middleware stamps as-is.
type ExtraHeaderKV struct {
Name string `json:"name"`
Value string `json:"value"`
}
// HeaderPairRule emits identity through dedicated per-dimension
// headers. The two *InBody flags layer body-level identity on top: when
// TagsInBody is set the middleware also writes the tag list into the
// request body's metadata.tags array (required for LiteLLM tag-budget
// enforcement, which only inspects the body); when EndUserIDInBody is
// set the display identity is also written into the body's top-level
// "user" field (the OpenAI-standard end-user identifier — defense-in-
// depth and anti-spoof on top of the header path).
type HeaderPairRule struct {
EndUserIDHeader string `json:"end_user_id_header,omitempty"`
TagsHeader string `json:"tags_header,omitempty"`
TagsInBody bool `json:"tags_in_body,omitempty"`
EndUserIDInBody bool `json:"end_user_id_in_body,omitempty"`
}
// JSONMetadataRule emits identity through a single JSON-object header.
// Empty UserKey/GroupsKey skip that dimension at emit time. When
// MaxValueLength > 0 each emitted JSON value is truncated to that many
// bytes — Portkey enforces 128 chars per value.
type JSONMetadataRule struct {
Header string `json:"header"`
UserKey string `json:"user_key,omitempty"`
GroupsKey string `json:"groups_key,omitempty"`
MaxValueLength int `json:"max_value_length,omitempty"`
}
// Config is the on-wire configuration accepted by the factory. An
// empty Providers slice yields a no-op middleware (every resolved
// provider passes through unchanged).
type Config struct {
Providers []ProviderInjection `json:"providers"`
}
// Factory builds llm_identity_inject instances from raw config bytes.
type Factory struct{}
// ID returns the registry identifier.
func (Factory) ID() string { return ID }
// New constructs a middleware instance. Empty, null, and {} configs
// yield a no-op middleware. Non-empty payloads must parse cleanly so
// misconfigurations surface at chain build time.
func (Factory) New(rawConfig []byte) (middleware.Middleware, error) {
cfg := Config{}
if !isEmptyJSON(rawConfig) {
if err := json.Unmarshal(rawConfig, &cfg); err != nil {
return nil, fmt.Errorf("decode config: %w", err)
}
}
return New(cfg), nil
}
// isEmptyJSON reports whether the payload is whitespace, null, or an
// empty object/array.
func isEmptyJSON(raw []byte) bool {
trimmed := strings.TrimSpace(string(bytes.TrimSpace(raw)))
switch trimmed {
case "", "null", "{}", "[]":
return true
}
return false
}
func init() {
builtin.Register(Factory{})
}
@@ -0,0 +1,440 @@
// Package llm_identity_inject implements the SlotOnRequest middleware
// that stamps the caller's NetBird identity onto upstream LLM-gateway
// requests. It runs after llm_router (which resolves the provider) and
// looks up the resolved provider id against a per-account injection
// table built by the synthesiser from the catalog's IdentityInjection
// metadata.
//
// Two wire shapes are supported, dispatched per-rule:
//
// - HeaderPair (LiteLLM-style): separate end-user-id and tags
// headers; tags emitted as a CSV value.
// - JSONMetadata (Portkey-style): one header carrying a JSON
// object with reserved keys for user / groups; per-value byte
// length capped when the rule sets MaxValueLength.
//
// In both cases, identity comes from Input.UserEmail (peer-attached
// user's email or peer.Name fallback) and groups come from the
// authorising-groups intersection llm_router emitted (with
// id→display-name translation via Input.UserGroups / UserGroupNames
// positional pairing). HeadersRemove runs before HeadersAdd in the
// framework, so a client can never spoof identity by stamping these
// headers themselves.
package llm_identity_inject
import (
"context"
"encoding/json"
"sort"
"strings"
"github.com/netbirdio/netbird/proxy/internal/middleware"
)
// ID is the registry identifier for this middleware.
const ID = "llm_identity_inject"
// Version is reported via Middleware.Version().
const Version = "1.0.0"
// Middleware stamps NetBird identity onto upstream requests for the
// configured set of resolved providers.
type Middleware struct {
cfg Config
byID map[string]ProviderInjection
}
// New constructs a Middleware from the supplied configuration. A nil
// or empty Providers slice yields a no-op middleware.
func New(cfg Config) *Middleware {
byID := make(map[string]ProviderInjection, len(cfg.Providers))
for _, p := range cfg.Providers {
if p.ProviderID == "" {
continue
}
// Drop entries that wouldn't inject anything — keeps the
// runtime check tight. Also drop entries that set both
// shapes (configuration error; refuse to guess which wins).
// Extras alone are enough to keep the rule alive even if
// neither identity shape is set.
hasExtras := false
for _, e := range p.ExtraHeaders {
if e.Name != "" && e.Value != "" {
hasExtras = true
break
}
}
switch {
case p.HeaderPair != nil && p.JSONMetadata != nil:
continue
case p.HeaderPair != nil:
if p.HeaderPair.EndUserIDHeader == "" && p.HeaderPair.TagsHeader == "" && !p.HeaderPair.TagsInBody && !p.HeaderPair.EndUserIDInBody && !hasExtras {
continue
}
case p.JSONMetadata != nil:
if p.JSONMetadata.Header == "" {
continue
}
if p.JSONMetadata.UserKey == "" && p.JSONMetadata.GroupsKey == "" && !hasExtras {
continue
}
default:
if !hasExtras {
continue
}
}
byID[p.ProviderID] = p
}
return &Middleware{cfg: cfg, byID: byID}
}
// ID returns the registry identifier.
func (m *Middleware) ID() string { return ID }
// Version returns the implementation version.
func (m *Middleware) Version() string { return Version }
// Slot reports the chain slot the middleware lives in.
func (m *Middleware) Slot() middleware.Slot { return middleware.SlotOnRequest }
// AcceptedContentTypes returns nil — this middleware reads only
// metadata and identity fields on the Input envelope.
func (m *Middleware) AcceptedContentTypes() []string { return nil }
// MetadataKeys is empty: the middleware emits no metadata. Identity
// stamping is a header-only operation.
func (m *Middleware) MetadataKeys() []string { return nil }
// MutationsSupported reports that the middleware emits header
// mutations on the Output envelope.
func (m *Middleware) MutationsSupported() bool { return true }
// Close releases resources owned by the middleware. Stateless, so
// this is a no-op.
func (m *Middleware) Close() error { return nil }
// Invoke stamps identity headers when the resolved provider has an
// injection rule. Always Allow.
func (m *Middleware) Invoke(_ context.Context, in *middleware.Input) (*middleware.Output, error) {
out := &middleware.Output{Decision: middleware.DecisionAllow}
if len(m.byID) == 0 || in == nil {
return out, nil
}
resolved, ok := lookupMetadata(in.Metadata, middleware.KeyLLMResolvedProviderID)
if !ok || resolved == "" {
return out, nil
}
rule, ok := m.byID[resolved]
if !ok {
return out, nil
}
var mutations *middleware.Mutations
switch {
case rule.HeaderPair != nil:
mutations = applyHeaderPair(rule.HeaderPair, in)
case rule.JSONMetadata != nil:
mutations = applyJSONMetadata(rule.JSONMetadata, in)
}
// ExtraHeaders are independent of the identity shape. Stamp each
// non-empty entry with anti-spoof: Remove first (frame strips it
// before our Add lands) so a client can't smuggle a value, then
// Add our trusted one.
if len(rule.ExtraHeaders) > 0 {
if mutations == nil {
mutations = &middleware.Mutations{}
}
for _, h := range rule.ExtraHeaders {
if h.Name == "" || h.Value == "" {
continue
}
mutations.HeadersRemove = append(mutations.HeadersRemove, h.Name)
mutations.HeadersAdd = append(mutations.HeadersAdd, middleware.KV{
Key: h.Name,
Value: h.Value,
})
}
}
if mutations == nil || (len(mutations.HeadersAdd) == 0 && len(mutations.HeadersRemove) == 0 && len(mutations.BodyReplace) == 0) {
return out, nil
}
out.Mutations = mutations
return out, nil
}
// applyHeaderPair builds the LiteLLM-style mutations: separate per-
// dimension headers, with anti-spoof Removes paired with trusted Adds.
func applyHeaderPair(rule *HeaderPairRule, in *middleware.Input) *middleware.Mutations {
mutations := &middleware.Mutations{}
if rule.EndUserIDHeader != "" {
mutations.HeadersRemove = append(mutations.HeadersRemove, rule.EndUserIDHeader)
// Prefer the email when the auth path carried it: gateways
// like LiteLLM key per-user budgets and dashboards on a
// human-readable identifier; the user_id is an opaque
// management-server primary key. Fall back to user_id when
// no email is available (non-OIDC schemes, legacy JWTs).
if identity := identityFor(in); identity != "" {
mutations.HeadersAdd = append(mutations.HeadersAdd, middleware.KV{
Key: rule.EndUserIDHeader,
Value: identity,
})
}
}
if rule.TagsHeader != "" {
mutations.HeadersRemove = append(mutations.HeadersRemove, rule.TagsHeader)
if csv := authorisingTagsCSV(in); csv != "" {
mutations.HeadersAdd = append(mutations.HeadersAdd, middleware.KV{
Key: rule.TagsHeader,
Value: csv,
})
}
}
if rule.TagsInBody || rule.EndUserIDInBody {
// Body-level identity unlocks gateway behaviour the header
// path can't reach (LiteLLM's _tag_max_budget_check only
// inspects the body; OpenAI direct only reads the body's
// "user" field for attribution). The header path stays
// intact, so we still get attribution + per-end-user budget
// gating when body inject can't run (truncated body,
// non-JSON, hostile metadata shape).
var bodyTags []string
if rule.TagsInBody {
bodyTags = authorisingTagsSlice(in)
}
var bodyUser string
if rule.EndUserIDInBody {
bodyUser = identityFor(in)
}
if newBody, ok := injectIntoBody(in, bodyTags, bodyUser); ok {
mutations.BodyReplace = newBody
}
}
return mutations
}
// injectIntoBody parses the request body and writes the supplied
// identity dimensions into it. Tags land at metadata.tags (creating
// the metadata object when absent); the user identity lands at the
// top-level "user" field (OpenAI-standard end-user identifier).
// Returns the re-marshaled body and ok=true when at least one field
// was written. Returns ok=false (no mutation) when:
//
// - both inputs are empty (nothing to write);
// - the body is empty or truncated (we don't have the full document
// to safely round-trip);
// - the body isn't a JSON object (skip silently — this middleware
// only knows how to inject into OpenAI-compatible JSON payloads).
//
// A non-object existing `metadata` field skips the tag write but
// still allows the user write to land — we don't clobber the client's
// non-object metadata, but the orthogonal user field is fair game.
// The header path emission still runs in skip cases, so spend tracking
// + header-resolved end-user budgets continue to work without body-
// level enforcement.
func injectIntoBody(in *middleware.Input, tags []string, userID string) ([]byte, bool) {
wantTags := len(tags) > 0
wantUser := userID != ""
if !wantTags && !wantUser {
return nil, false
}
if in == nil || len(in.Body) == 0 || in.BodyTruncated {
return nil, false
}
var doc map[string]any
if err := json.Unmarshal(in.Body, &doc); err != nil {
return nil, false
}
injected := false
if wantTags {
var meta map[string]any
if existing, ok := doc["metadata"]; ok {
if typed, isObject := existing.(map[string]any); isObject {
meta = typed
}
// non-object metadata: leave it; tags go unwritten so we
// don't clobber the client's value. Header fallback covers
// spend tracking.
} else {
meta = map[string]any{}
}
if meta != nil {
meta["tags"] = tags
doc["metadata"] = meta
injected = true
}
}
if wantUser {
// Anti-spoof: overwrite any client-supplied "user" so the
// gateway only sees our trusted identity.
doc["user"] = userID
injected = true
}
if !injected {
return nil, false
}
out, err := json.Marshal(doc)
if err != nil {
return nil, false
}
return out, true
}
// applyJSONMetadata builds the Portkey-style mutations: a single header
// carrying a JSON object keyed by the rule's reserved field names. Per-
// value byte length is capped at MaxValueLength when set (Portkey
// enforces 128 chars).
func applyJSONMetadata(rule *JSONMetadataRule, in *middleware.Input) *middleware.Mutations {
mutations := &middleware.Mutations{}
mutations.HeadersRemove = append(mutations.HeadersRemove, rule.Header)
payload := map[string]string{}
if rule.UserKey != "" {
if identity := identityFor(in); identity != "" {
payload[rule.UserKey] = truncate(identity, rule.MaxValueLength)
}
}
if rule.GroupsKey != "" {
if csv := authorisingTagsCSV(in); csv != "" {
payload[rule.GroupsKey] = truncate(csv, rule.MaxValueLength)
}
}
if len(payload) == 0 {
return mutations
}
raw, err := json.Marshal(payload)
if err != nil {
return mutations
}
mutations.HeadersAdd = append(mutations.HeadersAdd, middleware.KV{
Key: rule.Header,
Value: string(raw),
})
return mutations
}
// identityFor returns the caller's display identity. UserEmail wins
// (carries the user email when peer-attached, peer.Name otherwise);
// UserID falls in only as a defensive last resort.
func identityFor(in *middleware.Input) string {
if in.UserEmail != "" {
return in.UserEmail
}
return in.UserID
}
// authorisingTagsSlice returns the sorted, deduplicated slice of group
// display names the request was authorised under. Prefers the per-
// request authorising groups emitted by llm_router (intersection of the
// caller's UserGroups with the resolved route's AllowedGroupIDs) so the
// tags carry only the groups that actually authorise THIS request, not
// every group the peer happens to be in. Falls back to the full
// UserGroups when the router metadata key is absent.
func authorisingTagsSlice(in *middleware.Input) []string {
ids := tagsIDsFromAuthorising(in.Metadata)
if len(ids) == 0 {
ids = in.UserGroups
}
return tagsNamedSlice(ids, in.UserGroups, in.UserGroupNames)
}
// authorisingTagsCSV is a convenience wrapper that joins
// authorisingTagsSlice with commas for HeaderPair-style emission.
func authorisingTagsCSV(in *middleware.Input) string {
return strings.Join(authorisingTagsSlice(in), ",")
}
// truncate caps s to maxBytes bytes when maxBytes > 0. No-op when
// maxBytes <= 0 or s already fits. Truncation is byte-wise — sufficient
// for Portkey's 128-char ASCII limit. UTF-8 sequences could in theory
// be split, but the gateway treats the value as opaque bytes.
func truncate(s string, maxBytes int) string {
if maxBytes <= 0 || len(s) <= maxBytes {
return s
}
return s[:maxBytes]
}
// tagsIDsFromAuthorising reads llm_router's authorising-groups metadata
// (a CSV of group ids) and returns the parsed slice. Returns nil when
// the key is absent or empty so the caller can fall back to the full
// UserGroups.
func tagsIDsFromAuthorising(meta []middleware.KV) []string {
v, ok := lookupMetadata(meta, middleware.KeyLLMAuthorisingGroups)
if !ok {
return nil
}
v = strings.TrimSpace(v)
if v == "" {
return nil
}
parts := strings.Split(v, ",")
out := make([]string, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p != "" {
out = append(out, p)
}
}
if len(out) == 0 {
return nil
}
return out
}
// tagsNamedSlice returns the sorted, deduplicated list of group display
// names. ids carries the canonical group identifiers to emit;
// userGroups + userGroupNames provide the positional id→name
// translation table from the Input envelope. When a name is missing
// for a given id (slice shorter than userGroups, or id absent from the
// table), the id is used verbatim so the tag still attributes
// correctly. Sorted so the same caller produces the same header value
// across requests (helps gateway-side cache hits and log correlation).
func tagsNamedSlice(ids, userGroups, userGroupNames []string) []string {
if len(ids) == 0 {
return nil
}
idToName := make(map[string]string, len(userGroups))
for i, id := range userGroups {
if i < len(userGroupNames) {
idToName[id] = userGroupNames[i]
}
}
seen := make(map[string]struct{}, len(ids))
out := make([]string, 0, len(ids))
for _, id := range ids {
id = strings.TrimSpace(id)
if id == "" {
continue
}
tag := idToName[id]
if tag == "" {
tag = id
}
if _, dup := seen[tag]; dup {
continue
}
seen[tag] = struct{}{}
out = append(out, tag)
}
if len(out) == 0 {
return nil
}
sort.Strings(out)
return out
}
// lookupMetadata returns the value for key plus a presence flag.
func lookupMetadata(meta []middleware.KV, key string) (string, bool) {
for _, kv := range meta {
if kv.Key == key {
return kv.Value, true
}
}
return "", false
}
@@ -0,0 +1,666 @@
package llm_identity_inject
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/netbirdio/netbird/proxy/internal/middleware"
)
const (
litellmProvider = "ainp_litellm-test"
portkeyProvider = "ainp_portkey-test"
)
func newInput(resolvedProvider, userID string, groups []string) *middleware.Input {
return &middleware.Input{
Slot: middleware.SlotOnRequest,
AccountID: "acct-test",
UserID: userID,
UserGroups: groups,
SourceIP: "100.64.0.5",
RequestID: "req-1",
Metadata: []middleware.KV{
{Key: middleware.KeyLLMResolvedProviderID, Value: resolvedProvider},
},
}
}
func liteLLMRule() ProviderInjection {
return ProviderInjection{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
TagsHeader: "x-litellm-tags",
},
}
}
func TestMiddlewareIdentity(t *testing.T) {
mw := New(Config{})
assert.Equal(t, ID, mw.ID())
assert.Equal(t, Version, mw.Version())
assert.Equal(t, middleware.SlotOnRequest, mw.Slot())
assert.True(t, mw.MutationsSupported())
assert.Empty(t, mw.MetadataKeys(), "middleware emits no metadata")
assert.Nil(t, mw.AcceptedContentTypes())
require.NoError(t, mw.Close())
}
func TestInject_MatchedProvider_StampsHeaders(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng", "grp-it"})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
assert.Equal(t, middleware.DecisionAllow, out.Decision)
require.NotNil(t, out.Mutations)
// Strips the same headers we're about to add (anti-spoof).
assert.ElementsMatch(t,
[]string{"x-litellm-end-user-id", "x-litellm-tags"},
out.Mutations.HeadersRemove,
"every injected header must also appear in HeadersRemove so client-supplied values are wiped before our trusted values land")
added := map[string]string{}
for _, kv := range out.Mutations.HeadersAdd {
added[kv.Key] = kv.Value
}
assert.Equal(t, "alice", added["x-litellm-end-user-id"])
assert.Equal(t, "grp-eng,grp-it", added["x-litellm-tags"], "tags CSV must be sorted")
}
func TestInject_UnmatchedProvider_NoMutations(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput("ainp_some-other-provider", "alice", []string{"grp-eng"})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
assert.Equal(t, middleware.DecisionAllow, out.Decision)
assert.Nil(t, out.Mutations, "non-LiteLLM resolved provider must produce no mutations")
}
func TestInject_NoResolvedProvider_NoMutations(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := &middleware.Input{Slot: middleware.SlotOnRequest, UserID: "alice"}
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
assert.Nil(t, out.Mutations,
"missing llm.resolved_provider_id metadata means the router didn't run; never stamp identity blindly")
}
func TestInject_PartialRule_StampsOnlyConfiguredHeaders(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
// TagsHeader intentionally empty.
},
}}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
assert.Equal(t, []string{"x-litellm-end-user-id"}, out.Mutations.HeadersRemove,
"only configured header should be stripped")
require.Len(t, out.Mutations.HeadersAdd, 1)
assert.Equal(t, "x-litellm-end-user-id", out.Mutations.HeadersAdd[0].Key)
assert.Equal(t, "alice", out.Mutations.HeadersAdd[0].Value)
}
func TestInject_EmptyIdentity_StripsButDoesNotAdd(t *testing.T) {
// Caller has no UserID and no groups. We still strip the headers
// (so the client can't inject identity) but we don't add empty
// values that would mislead the gateway.
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput(litellmProvider, "", nil)
in.AccountID = ""
in.SourceIP = ""
in.RequestID = ""
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
assert.ElementsMatch(t,
[]string{"x-litellm-end-user-id", "x-litellm-tags"},
out.Mutations.HeadersRemove,
"identity headers must be stripped even when we don't have values to add — anti-spoof")
assert.Empty(t, out.Mutations.HeadersAdd,
"no NetBird identity available; do not stamp empty / misleading values")
}
func TestInject_TagsCSV_DedupesAndSorts(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput(litellmProvider, "alice", []string{"grp-zzz", "grp-aaa", "grp-zzz", "", " "})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
for _, kv := range out.Mutations.HeadersAdd {
if kv.Key == "x-litellm-tags" {
assert.Equal(t, "grp-aaa,grp-zzz", kv.Value,
"tags CSV must dedupe, drop empty, and sort")
return
}
}
t.Fatalf("expected x-litellm-tags in HeadersAdd; got %v", out.Mutations.HeadersAdd)
}
func TestFactory_RejectsBadJSON(t *testing.T) {
_, err := Factory{}.New([]byte("{not json"))
require.Error(t, err)
}
func TestFactory_AcceptsEmptyShapes(t *testing.T) {
for _, raw := range [][]byte{nil, []byte(""), []byte(" "), []byte("null"), []byte("{}"), []byte("[]")} {
mw, err := Factory{}.New(raw)
require.NoError(t, err)
require.NotNil(t, mw)
out, ierr := mw.Invoke(context.Background(),
newInput(litellmProvider, "alice", []string{"grp-eng"}))
require.NoError(t, ierr)
assert.Equal(t, middleware.DecisionAllow, out.Decision)
assert.Nil(t, out.Mutations,
"empty config means no providers to inject for; every resolved provider passes through")
}
}
func TestFactory_DropsInjectionRuleWithEmptyHeaders(t *testing.T) {
mw, err := Factory{}.New([]byte(`{"providers":[{"provider_id":"x"}]}`))
require.NoError(t, err)
out, ierr := mw.Invoke(context.Background(), newInput("x", "alice", []string{"grp-eng"}))
require.NoError(t, ierr)
assert.Nil(t, out.Mutations,
"a rule with no header names is functionally a no-op and must be dropped at New() time")
}
// TestInject_TagsFromAuthorisingMetadata pins that when llm_router has
// emitted llm.authorising_groups, the inject middleware uses THAT
// (the per-request authorising intersection) for the tags header — not
// the full UserGroups, which can include groups unrelated to this
// request's routing.
func TestInject_TagsFromAuthorisingMetadata(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng", "grp-it", "grp-oncall"})
in.Metadata = append(in.Metadata, middleware.KV{
Key: middleware.KeyLLMAuthorisingGroups,
Value: "grp-eng",
})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
for _, kv := range out.Mutations.HeadersAdd {
if kv.Key == "x-litellm-tags" {
assert.Equal(t, "grp-eng", kv.Value,
"tags must come from llm.authorising_groups, not the full UserGroups; unrelated peer groups must not leak")
return
}
}
t.Fatalf("expected x-litellm-tags in HeadersAdd; got %v", out.Mutations.HeadersAdd)
}
// TestInject_TagsFallsBackToUserGroups pins the defensive fallback: if
// llm_router didn't emit authorising-groups metadata (chain
// misconfiguration) the middleware uses UserGroups so identity is
// still stamped, just over-broad.
func TestInject_TagsFallsBackToUserGroups(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRule()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng", "grp-it"})
// No llm.authorising_groups metadata.
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
for _, kv := range out.Mutations.HeadersAdd {
if kv.Key == "x-litellm-tags" {
assert.Equal(t, "grp-eng,grp-it", kv.Value,
"absent metadata must fall back to the full UserGroups CSV")
return
}
}
t.Fatalf("expected x-litellm-tags in HeadersAdd; got %v", out.Mutations.HeadersAdd)
}
// portkeyRule is the JSONMetadata-shape analogue of liteLLMRule: a
// single x-portkey-metadata header carrying _user and groups, with
// Portkey's 128-byte per-value cap.
func portkeyRule() ProviderInjection {
return ProviderInjection{
ProviderID: portkeyProvider,
JSONMetadata: &JSONMetadataRule{
Header: "x-portkey-metadata",
UserKey: "_user",
GroupsKey: "groups",
MaxValueLength: 128,
},
}
}
// TestInject_JSONMetadata_StampsHeader pins the Portkey-style emission:
// one header carrying a JSON envelope with reserved keys for user
// identity and groups CSV.
func TestInject_JSONMetadata_StampsHeader(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{portkeyRule()}})
in := newInput(portkeyProvider, "alice", []string{"grp-eng", "grp-it"})
in.UserEmail = "alice@example.com"
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out)
require.NotNil(t, out.Mutations)
assert.Equal(t, []string{"x-portkey-metadata"}, out.Mutations.HeadersRemove,
"the JSON header must be stripped before we add our trusted value")
require.Len(t, out.Mutations.HeadersAdd, 1)
added := out.Mutations.HeadersAdd[0]
assert.Equal(t, "x-portkey-metadata", added.Key)
var payload map[string]string
require.NoError(t, json.Unmarshal([]byte(added.Value), &payload))
assert.Equal(t, "alice@example.com", payload["_user"],
"_user reserved key carries the display identity (UserEmail)")
assert.Equal(t, "grp-eng,grp-it", payload["groups"],
"groups key carries the sorted CSV of group display names")
}
// TestInject_JSONMetadata_TruncatesValues pins the per-value byte cap.
// Portkey rejects metadata values longer than 128 chars; oversized
// values are truncated rather than failing the request.
func TestInject_JSONMetadata_TruncatesValues(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{portkeyRule()}})
in := newInput(portkeyProvider, "alice", []string{"grp-eng"})
in.UserEmail = strings.Repeat("a", 200) + "@example.com"
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
require.Len(t, out.Mutations.HeadersAdd, 1)
var payload map[string]string
require.NoError(t, json.Unmarshal([]byte(out.Mutations.HeadersAdd[0].Value), &payload))
assert.Len(t, payload["_user"], 128,
"per-value byte length must be capped at MaxValueLength")
}
// TestInject_JSONMetadata_EmptyIdentity_StripsButDoesNotAdd verifies the
// anti-spoof Remove still fires when there's nothing to stamp.
func TestInject_JSONMetadata_EmptyIdentity_StripsButDoesNotAdd(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{portkeyRule()}})
in := newInput(portkeyProvider, "", nil)
in.UserEmail = ""
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
assert.Equal(t, []string{"x-portkey-metadata"}, out.Mutations.HeadersRemove,
"strip even with no payload — client can't smuggle identity headers")
assert.Empty(t, out.Mutations.HeadersAdd,
"no NetBird identity available; do not stamp empty / misleading values")
}
// TestFactory_RejectsRuleWithBothShapes pins the configuration-error
// guard: a rule that sets both HeaderPair and JSONMetadata is dropped
// at New() time rather than guessing which wins.
func TestFactory_RejectsRuleWithBothShapes(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
},
JSONMetadata: &JSONMetadataRule{
Header: "x-portkey-metadata",
UserKey: "_user",
},
}}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
assert.Nil(t, out.Mutations,
"a rule that sets both shapes is ambiguous and must be dropped at New() time")
}
// liteLLMRuleWithBody is the LiteLLM-style rule with body tag injection
// enabled (matches the catalog default).
func liteLLMRuleWithBody() ProviderInjection {
return ProviderInjection{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
TagsHeader: "x-litellm-tags",
TagsInBody: true,
},
}
}
// TestInject_BodyTags_AddsMetadataTags pins the body-inject path that
// LiteLLM's _tag_max_budget_check requires. With TagsInBody set, the
// middleware writes the authorising-groups slice into
// request.metadata.tags (in addition to the header).
func TestInject_BodyTags_AddsMetadataTags(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleWithBody()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng", "grp-sre"})
in.Body = []byte(`{"model":"gpt-4o-mini","messages":[]}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
require.NotEmpty(t, out.Mutations.BodyReplace, "body must be rewritten when TagsInBody is set")
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
meta, ok := doc["metadata"].(map[string]any)
require.True(t, ok, "metadata must be an object")
tags, ok := meta["tags"].([]any)
require.True(t, ok, "metadata.tags must be a JSON array")
got := make([]string, 0, len(tags))
for _, t := range tags {
s, _ := t.(string)
got = append(got, s)
}
assert.Equal(t, []string{"grp-eng", "grp-sre"}, got,
"metadata.tags must carry the sorted authorising-groups slice")
assert.Equal(t, "gpt-4o-mini", doc["model"],
"the rest of the body must be preserved verbatim")
}
// TestInject_BodyTags_PreservesExistingMetadata pins that an existing
// metadata object on the request is merged with our tags rather than
// clobbered — clients sometimes set metadata fields the proxy
// shouldn't blow away (jobID, taskName, etc.).
func TestInject_BodyTags_PreservesExistingMetadata(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleWithBody()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.Body = []byte(`{"model":"gpt-4o-mini","metadata":{"jobID":"j-42","tags":["should-be-replaced"]}}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotEmpty(t, out.Mutations.BodyReplace)
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
meta := doc["metadata"].(map[string]any)
assert.Equal(t, "j-42", meta["jobID"],
"client-supplied metadata fields outside `tags` must survive")
tags := meta["tags"].([]any)
require.Len(t, tags, 1)
assert.Equal(t, "grp-eng", tags[0],
"our tags overwrite any client-supplied metadata.tags so spoofing is impossible")
}
// TestInject_BodyTags_SkipsHostileMetadataShape pins the defensive
// refusal: when the request body has a non-object metadata field
// (string/number/array), we don't inject — header path still emits.
func TestInject_BodyTags_SkipsHostileMetadataShape(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleWithBody()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.Body = []byte(`{"model":"gpt-4o-mini","metadata":"not-an-object"}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
assert.Empty(t, out.Mutations.BodyReplace,
"non-object metadata must skip body inject (don't clobber)")
for _, kv := range out.Mutations.HeadersAdd {
if kv.Key == "x-litellm-tags" {
assert.Equal(t, "grp-eng", kv.Value,
"header path must still emit so spend tracking keeps working")
return
}
}
t.Fatalf("expected x-litellm-tags header even when body inject was skipped")
}
// TestInject_BodyTags_SkipsTruncatedBody pins that we don't blindly
// rewrite a body we don't have in full. The header path still runs.
func TestInject_BodyTags_SkipsTruncatedBody(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleWithBody()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.Body = []byte(`{"model":"gpt-4o-mini","messages":[]}`)
in.BodyTruncated = true
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
assert.Empty(t, out.Mutations.BodyReplace,
"truncated body must skip body inject — re-marshaling would corrupt the request")
}
// TestInject_BodyTags_SkipsNonJSONBody pins graceful behavior when the
// body isn't JSON (e.g. a streaming binary or form upload sneaking
// through the LLM chain). Header path still runs.
func TestInject_BodyTags_SkipsNonJSONBody(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleWithBody()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.Body = []byte(`not even close to json`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
assert.Empty(t, out.Mutations.BodyReplace,
"non-JSON body must skip body inject silently")
}
// liteLLMRuleFull mirrors the catalog default: header path + body
// metadata.tags (groups) + body user (end-user id).
func liteLLMRuleFull() ProviderInjection {
return ProviderInjection{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
TagsHeader: "x-litellm-tags",
TagsInBody: true,
EndUserIDInBody: true,
},
}
}
// TestInject_BodyUser_WritesTopLevelUser pins the EndUserIDInBody path
// alone: body's top-level "user" field carries the display identity.
// Tags-in-body is OFF here so we isolate the user write.
func TestInject_BodyUser_WritesTopLevelUser(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: litellmProvider,
HeaderPair: &HeaderPairRule{
EndUserIDHeader: "x-litellm-end-user-id",
EndUserIDInBody: true,
},
}}})
in := newInput(litellmProvider, "alice", nil)
in.UserEmail = "alice@example.com"
in.Body = []byte(`{"model":"gpt-4o-mini","messages":[]}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
require.NotEmpty(t, out.Mutations.BodyReplace)
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
assert.Equal(t, "alice@example.com", doc["user"],
"body's top-level user field must carry the display identity")
_, hasMeta := doc["metadata"]
assert.False(t, hasMeta, "TagsInBody is off; metadata must not be added")
}
// TestInject_BodyUser_OverwritesClientSupplied pins anti-spoof: a
// client-supplied "user" in the body is overwritten so the gateway
// only sees our trusted identity.
func TestInject_BodyUser_OverwritesClientSupplied(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleFull()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.UserEmail = "alice@example.com"
in.Body = []byte(`{"model":"gpt-4o-mini","user":"ceo@company.com"}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotEmpty(t, out.Mutations.BodyReplace)
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
assert.Equal(t, "alice@example.com", doc["user"],
"client-supplied user must be overwritten with the trusted identity")
}
// TestInject_BodyCombined_TagsAndUser pins that with both flags on,
// the body carries both metadata.tags AND top-level user, and the
// header path still emits.
func TestInject_BodyCombined_TagsAndUser(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleFull()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng", "grp-sre"})
in.UserEmail = "alice@example.com"
in.Body = []byte(`{"model":"gpt-4o-mini"}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotEmpty(t, out.Mutations.BodyReplace)
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
assert.Equal(t, "alice@example.com", doc["user"])
meta := doc["metadata"].(map[string]any)
tags := meta["tags"].([]any)
require.Len(t, tags, 2)
assert.Equal(t, "grp-eng", tags[0])
assert.Equal(t, "grp-sre", tags[1])
// Header path still emits — header end-user-id is the primary
// path for LiteLLM's resolver, body is defense-in-depth.
added := map[string]string{}
for _, kv := range out.Mutations.HeadersAdd {
added[kv.Key] = kv.Value
}
assert.Equal(t, "alice@example.com", added["x-litellm-end-user-id"])
assert.Equal(t, "grp-eng,grp-sre", added["x-litellm-tags"])
}
// TestInject_BodyCombined_HostileMetadataKeepsUser pins the partial-
// success path: a hostile (non-object) metadata field skips the tag
// write but still allows the orthogonal user write to land.
func TestInject_BodyCombined_HostileMetadataKeepsUser(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{liteLLMRuleFull()}})
in := newInput(litellmProvider, "alice", []string{"grp-eng"})
in.UserEmail = "alice@example.com"
in.Body = []byte(`{"model":"gpt-4o-mini","metadata":"not-an-object"}`)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotEmpty(t, out.Mutations.BodyReplace,
"user write must still go through even when metadata is hostile")
var doc map[string]any
require.NoError(t, json.Unmarshal(out.Mutations.BodyReplace, &doc))
assert.Equal(t, "alice@example.com", doc["user"])
assert.Equal(t, "not-an-object", doc["metadata"],
"hostile metadata must be left untouched, not clobbered")
}
// TestInject_ExtraHeaders_Stamped pins the extras path: with a
// per-provider ExtraHeader configured (e.g. Portkey config id), the
// middleware stamps it on every matching request and adds the same
// name to HeadersRemove for anti-spoof.
func TestInject_ExtraHeaders_Stamped(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: portkeyProvider,
JSONMetadata: &JSONMetadataRule{
Header: "x-portkey-metadata",
UserKey: "_user",
GroupsKey: "groups",
},
ExtraHeaders: []ExtraHeaderKV{
{Name: "x-portkey-config", Value: "pc-prod-3f2a"},
},
}}})
in := newInput(portkeyProvider, "alice", []string{"grp-eng"})
in.UserEmail = "alice@example.com"
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
assert.Contains(t, out.Mutations.HeadersRemove, "x-portkey-config",
"extras must be stripped before stamping for anti-spoof")
added := map[string]string{}
for _, kv := range out.Mutations.HeadersAdd {
added[kv.Key] = kv.Value
}
assert.Equal(t, "pc-prod-3f2a", added["x-portkey-config"],
"extras must carry the operator-configured value verbatim")
// Identity-stamping shape (JSONMetadata header) still emitted.
assert.Contains(t, added, "x-portkey-metadata",
"extras and identity stamping are independent — both must land")
}
// TestInject_ExtraHeaders_OnlyRule pins that an extras-only rule
// (no HeaderPair, no JSONMetadata) survives New() and stamps the
// extras anyway. Useful for hypothetical gateways that need a static
// routing header but no NetBird identity stamping.
func TestInject_ExtraHeaders_OnlyRule(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: "ainp_extras-only",
ExtraHeaders: []ExtraHeaderKV{
{Name: "x-routing-key", Value: "rk-1"},
},
}}})
in := newInput("ainp_extras-only", "alice", nil)
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations,
"extras alone keep the rule alive — middleware must emit them")
added := map[string]string{}
for _, kv := range out.Mutations.HeadersAdd {
added[kv.Key] = kv.Value
}
assert.Equal(t, "rk-1", added["x-routing-key"])
}
// TestInject_ExtraHeaders_EmptyValueSkipped pins that empty values are
// dropped silently (the synth would normally not send them, but the
// middleware is defensive).
func TestInject_ExtraHeaders_EmptyValueSkipped(t *testing.T) {
mw := New(Config{Providers: []ProviderInjection{{
ProviderID: portkeyProvider,
JSONMetadata: &JSONMetadataRule{
Header: "x-portkey-metadata",
UserKey: "_user",
},
ExtraHeaders: []ExtraHeaderKV{
{Name: "x-portkey-config", Value: ""},
},
}}})
in := newInput(portkeyProvider, "alice", []string{"grp-eng"})
in.UserEmail = "alice@example.com"
out, err := mw.Invoke(context.Background(), in)
require.NoError(t, err)
require.NotNil(t, out.Mutations)
assert.NotContains(t, out.Mutations.HeadersRemove, "x-portkey-config",
"empty extra value must not even strip the header")
for _, kv := range out.Mutations.HeadersAdd {
assert.NotEqual(t, "x-portkey-config", kv.Key,
"empty extra value must not be stamped")
}
}