[management,proxy,client] 0.74.0 version (#6563)

* [management,proxy] Agent network: per-account LLM gateway (policy, metering, multi-provider) (#6555)

* [agent-network] Shared proto, OpenAPI schema, and generated types

* [agent-network] Management: store, manager, synthesizer, policy engine, provider catalog, HTTP/gRPC API

Adds the account-scoped agent-network module: provider/policy/budget CRUD and
store, the reverse-proxy service synthesizer, policy selection + limit
enforcement, the provider catalog (incl. Vertex AI and AWS Bedrock entries),
and the management HTTP + proxy gRPC surfaces.

* [management] Fix agent-network proxy-peer fan-out on affected-peer recompute

The affected-peers resolver loaded only persisted reverse-proxy services, but
agent-network services are synthesized on demand and never persisted. As a
result the embedded proxy peer was never folded into the affected set when a
client's group changed, so the proxy received no network-map update for a newly
authorised client and rejected its handshake until a full resync (restart).

loadProxyServices now merges the synthesized agent-network services (injected
via a registration hook to avoid an import cycle), so proxy peers learn newly
authorised clients immediately.

* [proxy] Reverse-proxy middleware framework, chain, and request plumbing

The per-target middleware chain (slots, dispatcher, mutation gate, metadata
merger), body capture, access-log terminal sink, and the proxy wiring that
builds + runs chains for synthesized agent-network services.

* [proxy] LLM parsers, pricing, and builtin middlewares (OpenAI, Anthropic, Vertex AI, AWS Bedrock)

Request/response parsers and SSE/event-stream metering, the embedded pricing
table, and the builtin middleware set: request parser, router, policy
limit-check/record, cost meter, guardrail, identity inject, response parser.
Includes the path-routed providers — Google Vertex AI (keyfile:: service-account
OAuth minting) and AWS Bedrock (bearer auth, invoke/converse/streaming, optional
/bedrock prefix) — plus the Models allowlist and unmeterable-publisher deny.

* [proxy] IPv6 in-place apply and TCP accept-loop hardening on netstack listeners

* [agent-network] End-to-end test suite, module docs, and deployment preset

* [agent-network] Fix codespell typos and exclude false positives

- labelgen word pool: vermillion -> vermilion, racoon -> raccoon.
- codespell ignore list: add flate (Go compress/flate package), recordin
  (a test-local identifier), and unparseable (a valid alternative spelling used
  consistently across identifiers + a metadata-value constant).

* [management] Set LastSeen on injected proxy peer in realstack test (MySQL strict-mode)

The injected embedded proxy peer had a PeerStatus with a zero LastSeen, which
serializes to '0000-00-00' and is rejected by MySQL in strict mode (SQLite
tolerates it). Set LastSeen to a valid time so SaveAccount succeeds on both
engines.

* [agent-network] Remove e2e shell-script suite from this branch

The end-to-end shell scripts under scripts/e2e/ are maintained in a separate
testing suite and are not part of this change set.

* [agent-network] Polish module docs: remove internal review scaffolding, fix links, verify diagrams

Strip PR-review framing, commit references, absolute paths, and stale internal
references from the agent-network module docs; fix broken relative links; verify
all diagrams against the current architecture. Remove the internal AI-reviewer
prompt file.

* [management] Refine session expiration handling to support 3-state encoding for SSO deadlines

* [agent-network] Relocate agentnetwork package to internals/modules

Move management/server/agentnetwork (and its catalog/, labelgen/, types/
subpackages) to management/internals/modules/agentnetwork, alongside the
reverse-proxy module, and rewrite all importers. Pure relocation: package names,
the synthesizer + affectedpeers registration hook, and store access (shared
store.Store) are unchanged, so no import cycle is introduced (affectedpeers
still depends only on the agentnetwork/types leaf).

* [agent-network] Co-locate HTTP handlers in the module (RegisterEndpoints)

Move the agent-network HTTP handlers from server/http/handlers/agentnetwork into
the module at internals/modules/agentnetwork/handlers (package handlers) and
rename the entrypoint AddEndpoints -> RegisterEndpoints, matching the
reverse-proxy module convention. Wiring in http/handler.go updated accordingly.

* Update getting started to point to rc when agent network enabled

* Add a reference to a commercial license

* Fix docs localhost link

* Fix docs localhost link

* Add private services domain note

* [management] Add agent-network telemetry metrics (#6561)

Surface agent-network adoption and usage in the self-hosted metrics
worker: distinct accounts, providers, policies, budget rules, accounts
with log collection enabled, and aggregated input/output tokens plus
cost.

Tokens and cost are summed from agent_network_request_usage (the
always-written per-request ledger) so the figures are accurate
regardless of the log-collection toggle and carry no double-counting.
All values come from a handful of indexed aggregate queries run only on
the worker's periodic tick.

Adds store.AgentNetworkMetrics with GetAgentNetworkMetrics on the Store
interface, the SqlStore implementation, and a zero-valued FileStore stub.

* Update NetBird server and proxy image versions to 0.74.0-rc.2

* [management,proxy] Reduce agent-network cognitive complexity (#6566)

Address the SonarCloud quality-gate findings in new agent-network code
by extracting focused helpers. No behavior change.

- synthesizer.go: split buildIdentityInjectConfigJSON into per-shape
  rule builders; extract mergeGuardrail from mergeGuardrails to cut
  nesting depth.
- llm_identity_inject: extract injectionEmitsAnything validation
  predicate from New.
- llm_response_parser/streaming.go: extract applyOpenAIStreamUsage and
  applyAnthropicStreamUsage (via a named anthropicStreamUsage type) and
  simplify the OpenAI scanner loop.
- reverseproxy.go: decompose ServeHTTP into serveRouteError,
  buildTargetContext, serveDirect, serveWithChain, captureRequestForChain,
  serveDeny, newResponseWriter, observeResponse, and forwardUpstream,
  preserving the defer ordering so response observation still reads the
  captured writer before it is released.

* [management] Move agent-network access-log ingest into the agentnetwork module (#6568)

The agent-network access-log ingest path (metaKey wire contract, flatten,
usage derivation, and the dual-write of the usage ledger + settings-gated
full row) lived in the reverseproxy accesslogs manager, even though the
agentnetwork module already owns the rest of that domain — types, read
(ListAccessLogs / GetUsageOverview), the budget-counter writes, and
retention cleanup.

Move it next to the rest: a stateless agentnetwork.IngestAccessLog(ctx,
store, entry) that the reverseproxy SaveAccessLog delegates to when the
entry is agent-network. Removes the agentNetworkTypes import from the
reverseproxy manager. No behavior change; the write/read table separation
is unchanged.

Adds real-store coverage for the disable->enable log-collection toggle
(usage ledger always written, full row gated) plus the metadata parse and
group-dedup helpers, which previously had no dedicated tests.

* Add session view support in the access log

* [management,proxy] Container-based agent-network e2e harness (#6577)

* [e2e] Add container-based agent-network e2e harness (Pillar 1)

Introduce a self-contained, OIDC-free e2e harness that stands up NetBird
in containers, so suites no longer depend on the hand-maintained Tilt
stack or a real IdP.

- harness brings up the combined server (management + signal + relay +
  STUN + embedded IdP) in a single container built from
  combined/Dockerfile.multistage, and mints an admin PAT through the
  unauthenticated /api/setup bootstrap (NB_SETUP_PAT_ENABLED). API access
  goes through the existing shared/management/client/rest typed client.
- the image is built via the docker CLI (BuildKit) so the Dockerfile's
  cache mounts are honored; testcontainers then runs the tagged image.
- everything is behind the `e2e` build tag so normal builds and unit
  tests never pull in testcontainers.

Adds BuildKit cache mounts to combined/Dockerfile.multistage so source
changes recompile incrementally rather than from scratch.

Pillar 1 proven by TestCombinedBootstrap: server builds, boots, mints a
PAT, and the PAT authenticates a real management API call.

* [e2e] Add management-side agent-network scenarios (Pillar 2)

Port the API-driven agent-network scenarios from the bash suites to Go,
sharing one combined server per package run (TestMain) with each test
owning its resource cleanup. Drives the /api/agent-network/* endpoints
through the shared REST client's NewRequest primitive with the generated
api types.

Scenarios:
- provider lifecycle (create/get/list/delete + 404 after delete)
- provider validation (missing api_key, unknown catalog id → 4xx)
- settings collection-toggle round-trip with cluster/subdomain immutability
- policy window floor (reject <60s enabled limit, accept at 60s)
- consumption read endpoint returns an array

All deterministic and dependency-free (dummy provider keys; no upstream
calls), so they run headless in CI.

* [e2e] Add live chat-through-proxy scenario (Pillar 3)

Stand up the full agent-network data path in containers and drive a real
chat-completion through the gateway:

- harness: a shared docker network (combined server reachable by alias),
  a proxy container built from the published reverse-proxy image
  (NB_PROXY_PRIVATE, NB_PROXY_ALLOW_INSECURE, NB_RELAY_TRANSPORT=ws to match
  the combined server's WS-multiplexed relay) with a generated self-signed
  wildcard cert, and a netbird client container that joins via a setup key.
- the combined image, proxy image, and client image default to the
  published rc.2 releases (overridable via NB_E2E_*_IMAGE; a bare local tag
  is built from source instead). Geolocation download is disabled so the
  server starts without external fetches.
- one shared domain is used for the management exposed address, the proxy
  domain, and the agent-network cluster; the proxy token is minted via the
  server CLI (global) to match the manual install.

TestChatCompletionThroughProxy provisions provider+policy+group+setup key,
runs proxy+client, drives an OpenAI chat-completion through the tunnel, and
asserts a 200 plus the ingested access-log row. Requires OPENAI_TOKEN
(skips otherwise). The provider must be created with enabled=true explicitly
— the create default is false despite the API doc.

* [e2e] Run the live chat scenario across a provider matrix

Replace the single-provider chat test with a data-driven matrix that runs
the same scenario through every provider whose credentials are present in
the environment (keys/URLs sourced from ~/.llm-keys locally, Actions
secrets in CI):

- OpenAI (chat), Anthropic (messages), Vercel, OpenRouter, Cloudflare
  (OpenAI-compatible gateways), and Bedrock (path-routed, bearer, via the
  messages shape) — covering both wire shapes and the gateway routing.
- all providers are created enabled with a unique model string so the
  proxy's connect-time snapshot carries them all and model->provider
  routing is unambiguous (provider toggles after connect don't reconcile
  to a connected proxy).
- the client supports both wire shapes (/v1/chat/completions and
  /v1/messages); Cloudflare gets the openai provider segment appended to
  its gateway URL.

Each provider must return 200 through the tunnel and produce an ingested
access-log row. Vertex is intentionally excluded from the uniform matrix:
it needs a bespoke rawPredict request shape rather than the shared
chat/messages path, so it warrants a dedicated scenario.

* [ci] Add manual workflow for the agent-network e2e suite

The e2e suite (build tag `e2e`) stands up the combined server + proxy +
client in Docker and drives live chat-completions, so it is slow and needs
provider credentials. Gate it out of normal CI (it already is, via the
build tag) and run it on demand via workflow_dispatch. Provider scenarios
skip when their secret is unset, so it degrades gracefully.

* [e2e] Add Vertex to the provider matrix; run e2e on ubuntu-latest

Vertex (Anthropic-on-Vertex) doesn't share the chat/messages wire shapes:
the model travels in a rawPredict path and the proxy mints the service
account's OAuth token. Add a Vertex client method that posts
/v1/projects/<project>/locations/<region>/publishers/anthropic/models/<model>:rawPredict
with the Vertex anthropic_version body, and wire it into the matrix as a
path-routed provider (created without a models array). It is keyed off
GOOGLE_VERTEX_SA_BASE64 + GOOGLE_VERTEX_PROJECT (region defaults to
"global", model to a pinned claude snapshot, both overridable).

Also bump the e2e workflow runner to ubuntu-latest and add the Vertex
secrets.

* Add docker/docker and docker/go-connections as direct dependencies in go.mod

* [ci] Trigger agent-network e2e workflow on push to main and pull requests

* [e2e] Fix proxy cert permission denied on Linux CI runners

The proxy bind-mounts a temp dir of self-signed certs. MkdirTemp creates
it 0700 and the key was 0600, which Docker Desktop on macOS ignores but a
non-root proxy container on Linux runners cannot traverse/read, so the
cert watcher failed with "open /certs/tls.crt: permission denied" and the
container exited. Widen the cert dir to 0755 and write the throwaway key
0644 so the proxy uid can read the bind-mounted material.

* [e2e] Build images from source by default instead of pulling rc.2

The agent-network code under test lives in this branch, so the e2e should
exercise it rather than a frozen published release. Flip the harness
default: combined/proxy/client are now built from their in-repo
Dockerfiles (combined/Dockerfile.multistage, proxy/Dockerfile.multistage,
e2e/harness/Dockerfile.client) under local tags. Pulling a published image
stays available by setting NB_E2E_*_IMAGE to a registry reference.

Builds now go through buildx --load so the Dockerfile cache mounts are
honored and the result is loaded for testcontainers. The CI workflow adds
a container-driver builder and a local layer cache (NB_E2E_BUILDX_CACHE)
persisted via actions/cache, which caches the base/apt/dep-download layers
across runs. The Go compile still re-runs each time, as BuildKit mount
caches cannot be exported to the GitHub cache.

* [e2e] Cover real providers in lifecycle + assert real consumption metering

- TestProviderLifecycle now runs per available real provider (create → get →
  list → delete → 404) instead of a single dummy provider, exercising each
  catalog's create and field round-trip. Create is offline, so it stays fast
  and burns no provider quota; falls back to a synthetic OpenAI provider when
  no keys are set.
- TestProvidersMatrix attaches a token limit (high caps, 60s window) to its
  policy, which switches on usage metering, and asserts consumption rows are
  recorded with positive token counts after the live traffic. Consumption is
  account-scoped (keyed by source group / user and window, not per provider),
  so the assertion is aggregate.
- TestProviderValidation gains invalid-upstream and blank-name cases. Create
  validation is uniform across catalogs (no per-provider required-field rules),
  so per-provider rejection cases would be redundant.

* [e2e] Assert session id propagates per provider

Each matrix request now sends a unique session id as the universal
x-session-id header and asserts it round-trips into that provider's
access-log row. This guards the session-grouping contract end to end for
every provider (header extraction runs in llm_request_parser ahead of the
parser-specific body extraction, so it is provider-agnostic).

* [e2e] Drop accidentally committed sync-phases dashboard

netbird-sync-phases.json was swept into the Pillar 1 commit by a broad
git add; it belongs to the unrelated sync-phases metrics work, not this
e2e harness. Remove it from the branch so the PR diff is scoped to the
e2e changes.

* [e2e] Revert accidentally committed sync-phase ingest spec

The netbird_sync_phase measurement spec in metrics ingest was swept into
the Pillar 1 commit; it belongs to the unrelated sync-phases metrics work,
not this e2e harness. Its emission side never landed here, so the spec was
orphaned anyway. Restore ingest/main.go to its origin/main state.

* Fix golint issues

* Fix sonar

* Add access log session test

* Fix access log tests

---------

Co-authored-by: braginini <bangvalo@gmail.com>
Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
This commit is contained in:
Maycon Santos
2026-07-01 12:45:14 +02:00
committed by GitHub
co-authored by braginini Zoltan Papp
parent 3be90f06b2
commit 92a66cdd20
208 changed files with 39957 additions and 688 deletions
@@ -0,0 +1,321 @@
package proxy_test
import (
"context"
"net"
"net/http"
"net/http/httptest"
"net/url"
"runtime"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/test/bufconn"
rpservice "github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service"
mgmtgrpc "github.com/netbirdio/netbird/management/internals/shared/grpc"
"github.com/netbirdio/netbird/management/internals/modules/agentnetwork"
agentNetworkTypes "github.com/netbirdio/netbird/management/internals/modules/agentnetwork/types"
"github.com/netbirdio/netbird/management/server/store"
"github.com/netbirdio/netbird/proxy/internal/middleware"
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
mwbuiltin "github.com/netbirdio/netbird/proxy/internal/middleware/builtin"
// Side-effect imports register every builtin middleware factory.
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/cost_meter"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_guardrail"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_identity_inject"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_limit_check"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_limit_record"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_request_parser"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_response_parser"
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_router"
"github.com/netbirdio/netbird/proxy/internal/proxy"
nbproxytypes "github.com/netbirdio/netbird/proxy/internal/types"
"github.com/netbirdio/netbird/shared/management/proto"
log "github.com/sirupsen/logrus"
)
// TestReverseProxy_AgentNetworkRequest_FullChain is the self-contained Go
// replacement for the bash 50 + 51 legs. It drives a real agent-network
// request through proxy.ReverseProxy.ServeHTTP with the actual middleware
// chain the synthesizer produces, against an in-process management gRPC and a
// httptest fake upstream — no tilt, no docker, no real LLM provider, no
// WireGuard tunnel. The test guarantees:
//
// 1. The reverse proxy's response-leg input construction copies UserGroups
// onto respInput so llm_limit_record sends a non-empty group_ids field
// on RecordLLMUsage. This is the exact bug class that motivated the
// reverseproxy.go fix — its regression would land the request OK but
// leave consumption at zero, defeating any group-targeted budget rule.
// 2. With settings.RedactPii=true the parsers ship redacted text on both
// llm.request_prompt_raw and llm.response_completion — proving the
// end-to-end wiring (synth → proto → spec → parser config) carries the
// toggle through to runtime emission.
// 3. The full chain (request + response + recorder) runs against a real
// management stack and the consumption row for the bound group dim
// increments.
//
// If any of those three guarantees regresses, this single test fails.
func TestReverseProxy_AgentNetworkRequest_FullChain(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("sqlite store not supported on Windows")
}
const (
testAccountID = "acct-fullchain-1"
testAdminUser = "user-admin-1"
adminGroupID = "grp-admins"
providerID = "prov-openai-test"
cluster = "test.proxy.local"
subdomain = "fullchain"
)
testLogger := log.New()
testLogger.SetLevel(log.PanicLevel) // keep test output clean
ctx := context.Background()
// ---- 1. Fake upstream that returns OpenAI-shaped JSON with PII in the
// completion. The reverse proxy's chain will redact this when the synth
// stamps redact_pii=true on the response parser config.
completion := "Sample record: Alice Johnson alice.johnson@example.com SSN 123-45-6789 phone (202) 555-0147 also Bob 202/555/0108"
upstreamBody := []byte(`{"id":"x","model":"gpt-5.4","choices":[{"message":{"role":"assistant","content":"` + completion + `"}}],"usage":{"prompt_tokens":12,"completion_tokens":40,"total_tokens":52}}`)
var upstreamHits atomic.Int64
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
upstreamHits.Add(1)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(upstreamBody)
}))
t.Cleanup(upstream.Close)
upstreamHost := strings.TrimPrefix(upstream.URL, "http://")
// ---- 2. In-process management gRPC server (bufconn) backed by a real
// sqlite store + real agentnetwork.Manager. The proxy's middlewares talk
// to this client.
st, cleanup, err := store.NewTestStoreFromSQL(ctx, "", t.TempDir())
require.NoError(t, err, "real sqlite test store must come up")
t.Cleanup(cleanup)
anMgr := agentnetwork.NewManager(st, nil, nil, nil)
server := &mgmtgrpc.ProxyServiceServer{}
server.SetAgentNetworkLimitsService(anMgr)
lis := bufconn.Listen(1024 * 1024)
srv := grpc.NewServer()
proto.RegisterProxyServiceServer(srv, server)
go func() { _ = srv.Serve(lis) }()
t.Cleanup(srv.Stop)
conn, err := grpc.NewClient("passthrough:///bufnet",
grpc.WithContextDialer(func(_ context.Context, _ string) (net.Conn, error) { return lis.Dial() }),
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
require.NoError(t, err)
t.Cleanup(func() { _ = conn.Close() })
mgmtClient := proto.NewProxyServiceClient(conn)
// ---- 3. Seed account state: settings (redact + capture on), provider
// whose upstream URL points at our fake server, policy (catch-all-allow
// over the Admins group → window=0 path), and a generous budget rule
// targeting Admins so the curl succeeds and we can prove the counter
// increments on the response leg.
require.NoError(t, st.SaveAgentNetworkSettings(ctx, &agentNetworkTypes.Settings{
AccountID: testAccountID,
Cluster: cluster,
Subdomain: subdomain,
EnablePromptCollection: true,
EnableLogCollection: true,
RedactPii: true,
}))
require.NoError(t, st.SaveAgentNetworkProvider(ctx, &agentNetworkTypes.Provider{
ID: providerID,
AccountID: testAccountID,
ProviderID: "openai_api",
Name: "openai-fullchain-test",
UpstreamURL: upstream.URL, // router rewrites to this
APIKey: "sk-test",
Enabled: true,
Models: []agentNetworkTypes.ProviderModel{{ID: "gpt-5.4"}},
SessionPrivateKey: "priv",
SessionPublicKey: "pub",
}))
require.NoError(t, st.SaveAgentNetworkPolicy(ctx, &agentNetworkTypes.Policy{
ID: "ainpol-fullchain",
AccountID: testAccountID,
Name: "admins-openai",
Enabled: true,
SourceGroups: []string{adminGroupID},
DestinationProviderIDs: []string{providerID},
// No token / budget caps → effectiveWindowSeconds=0 → exercises the
// catch-all-allow path that the GC-2 record-on-window=0 fix targets.
}))
require.NoError(t, st.SaveAgentNetworkBudgetRule(ctx, &agentNetworkTypes.AccountBudgetRule{
ID: "ainbud-admins-fullchain",
AccountID: testAccountID,
Name: "admins-monthly",
Enabled: true,
TargetGroups: []string{adminGroupID},
Limits: agentNetworkTypes.PolicyLimits{
TokenLimit: agentNetworkTypes.PolicyTokenLimit{Enabled: true, GroupCap: 1_000_000, UserCap: 1_000_000, WindowSeconds: 60},
},
}))
// ---- 4. Synth the service. This produces the exact middleware chain
// configuration the production reconcile path ships to the proxy.
services, err := agentnetwork.SynthesizeServices(ctx, st, testAccountID)
require.NoError(t, err)
require.Len(t, services, 1, "exactly one synth service expected")
synthSvc := services[0]
require.NotEmpty(t, synthSvc.Targets, "synth target must exist")
// ---- 5. Wire the middleware framework — same registry the proxy uses
// in production, configured with our bufconn-backed management client.
mwbuiltin.Configure(ctx, t.TempDir(), nil, testLogger, mgmtClient)
registry := mwbuiltin.DefaultRegistry()
mwMetrics, err := middleware.NewMetrics(nil)
require.NoError(t, err)
mwMgr := middleware.NewManager(0, mwMetrics, testLogger)
mwMgr.SetResolver(middleware.NewResolver(registry))
// Convert the synth's rpservice.MiddlewareConfig list into proxy
// middleware.Spec values. Mirrors the proto→Spec translation server.go
// does at runtime; kept inline here so the test isn't coupled to the
// proxy server's private translateMiddlewareConfig helper.
specs := make([]middleware.Spec, 0, len(synthSvc.Targets[0].Options.Middlewares))
for _, mw := range synthSvc.Targets[0].Options.Middlewares {
var slot middleware.Slot
switch mw.Slot {
case rpservice.MiddlewareSlotOnRequest:
slot = middleware.SlotOnRequest
case rpservice.MiddlewareSlotOnResponse:
slot = middleware.SlotOnResponse
case rpservice.MiddlewareSlotTerminal:
slot = middleware.SlotTerminal
default:
t.Fatalf("unknown middleware slot %q on %s", mw.Slot, mw.ID)
}
specs = append(specs, middleware.Spec{
ID: mw.ID,
Slot: slot,
Enabled: mw.Enabled,
FailMode: middleware.FailOpen,
Timeout: middleware.DefaultTimeout,
RawConfig: append([]byte(nil), mw.ConfigJSON...),
CanMutate: mw.CanMutate,
})
}
serviceIDStr := synthSvc.ID
require.NoError(t, mwMgr.Rebuild(serviceIDStr, []middleware.PathTargetBinding{{
ServiceID: serviceIDStr,
PathID: "/",
Specs: specs,
}}))
// ---- 6. Build the reverse proxy, with a mapping whose target URL goes
// straight to the fake upstream (the router middleware rewriting upstream
// from the synth's noop placeholder isn't needed when we own the mapping
// in-process — point the target at the fake URL directly so the body
// arrives at the upstream the synth would have routed to).
upstreamURL, err := url.Parse(upstream.URL)
require.NoError(t, err)
rp := proxy.NewReverseProxy(http.DefaultTransport, "auto", nil, testLogger, proxy.WithMiddlewareManager(mwMgr))
rp.AddMapping(proxy.Mapping{
ID: nbproxytypes.ServiceID(serviceIDStr),
AccountID: nbproxytypes.AccountID(testAccountID),
Host: synthSvc.Domain,
Paths: map[string]*proxy.PathTarget{
"/": {
URL: upstreamURL,
DirectUpstream: true,
AgentNetwork: true,
Middlewares: specs,
CaptureConfig: &bodytap.Config{
MaxRequestBytes: 1 << 20,
MaxResponseBytes: 1 << 20,
ContentTypes: []string{"application/json", "text/event-stream"},
},
},
},
})
// ---- 7. Send a request with the auth-stamped CapturedData (mimicking
// what the tunnel-peer auth middleware does at the edge of the proxy).
reqBody := `{"model":"gpt-5.4","client_metadata":{"session_id":"sess-fullchain-1"},"messages":[{"role":"user","content":"contact alice.johnson@example.com SSN 987-65-4321 phone (202)555-0156"}]}`
req := httptest.NewRequest("POST", "https://"+synthSvc.Domain+"/v1/chat/completions", strings.NewReader(reqBody))
req.Host = synthSvc.Domain
req.Header.Set("Content-Type", "application/json")
cd := proxy.NewCapturedData("test-request-1")
cd.SetServiceID(nbproxytypes.ServiceID(serviceIDStr))
cd.SetAccountID(nbproxytypes.AccountID(testAccountID))
cd.SetUserID(testAdminUser)
cd.SetUserGroups([]string{adminGroupID})
cd.SetAuthMethod("tunnel_peer")
req = req.WithContext(proxy.WithCapturedData(req.Context(), cd))
w := httptest.NewRecorder()
rp.ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code, "upstream call must succeed end-to-end; body=%s", w.Body.String())
assert.GreaterOrEqual(t, upstreamHits.Load(), int64(1), "fake upstream must have been hit")
// ---- 8. Assertions — the three guarantees this test exists for.
// 8a. The reverseproxy.go respInput construction carried UserGroups
// into the response-leg middleware chain, so llm_limit_record sent a
// non-empty group_ids on RecordLLMUsage. Verifying via the management
// store directly bypasses the manager's permission gate (which is nil
// in this test) — we want to confirm the row landed, not who saw it.
require.Eventually(t, func() bool {
rows, lerr := st.ListAgentNetworkConsumption(ctx, store.LockingStrengthNone, testAccountID)
if lerr != nil {
return false
}
for _, r := range rows {
if r.DimensionKind == agentNetworkTypes.DimensionGroup &&
r.DimensionID == adminGroupID &&
r.WindowSeconds == 60 &&
r.TokensInput+r.TokensOutput > 0 {
return true
}
}
return false
}, 5*time.Second, 50*time.Millisecond,
"Admins group consumption row must increment via the response leg — if this fails the proxy's respInput dropped UserGroups again or the parser/recorder wiring is broken")
// 8b. Both the captured prompt and the captured completion are
// redacted — proves the synth threads redact_pii=true into BOTH parser
// configs and the parsers honour it at emission time.
md := cd.GetMetadata()
promptRaw := md["llm.request_prompt_raw"]
completionMeta := md["llm.response_completion"]
// 8a-bis. The session id from client_metadata.session_id flows through
// the request parser into the captured metadata, so the access-log /
// usage rows can group this request with the rest of its conversation.
assert.Equal(t, "sess-fullchain-1", md["llm.session_id"],
"session id must be extracted from client_metadata.session_id and carried through the chain")
assert.NotEmpty(t, promptRaw, "llm.request_prompt_raw must be present in captured metadata")
assert.Contains(t, promptRaw, "[REDACTED:", "captured raw prompt must carry redaction markers")
assert.NotContains(t, promptRaw, "alice.johnson@example.com", "raw email must NOT survive in prompt_raw")
assert.NotContains(t, promptRaw, "987-65-4321", "raw SSN must NOT survive in prompt_raw")
assert.NotContains(t, promptRaw, "(202)555-0156", "raw paren-no-space phone must NOT survive in prompt_raw")
assert.NotEmpty(t, completionMeta, "llm.response_completion must be present in captured metadata")
assert.Contains(t, completionMeta, "[REDACTED:", "captured completion must carry redaction markers")
assert.NotContains(t, completionMeta, "alice.johnson@example.com", "raw email must NOT survive in completion")
assert.NotContains(t, completionMeta, "123-45-6789", "raw SSN must NOT survive in completion")
assert.NotContains(t, completionMeta, "(202) 555-0147", "raw paren+space phone must NOT survive in completion")
assert.NotContains(t, completionMeta, "202/555/0108", "raw slash phone must NOT survive in completion")
_ = upstreamHost // kept for future header-inspection assertions if needed
}
+40 -3
View File
@@ -58,9 +58,11 @@ type CapturedData struct {
// the JWT's group_names claim or from ValidateSession/Tunnel
// responses. Slice may be shorter than userGroups for tokens minted
// before names were resolvable.
userGroupNames []string
authMethod string
metadata map[string]string
userGroupNames []string
authMethod string
metadata map[string]string
agentNetwork bool
suppressAccessLog bool
}
// NewCapturedData creates a CapturedData with the given request ID.
@@ -178,6 +180,41 @@ func (c *CapturedData) SetUserGroups(groups []string) {
c.userGroups = append(c.userGroups[:0], groups...)
}
// SetAgentNetwork records whether the request hit a synthesised
// agent-network target. The terminal access-log middleware stamps the
// flag onto the proto so management can distinguish synthetic traffic.
func (c *CapturedData) SetAgentNetwork(b bool) {
c.mu.Lock()
defer c.mu.Unlock()
c.agentNetwork = b
}
// GetAgentNetwork reports whether the request matched a synthesised
// agent-network target.
func (c *CapturedData) GetAgentNetwork() bool {
c.mu.RLock()
defer c.mu.RUnlock()
return c.agentNetwork
}
// SetSuppressAccessLog records whether the per-request access-log emission
// must be skipped for this request. Stamped from the matched target's
// DisableAccessLog flag so the access-log middleware can short-circuit
// log delivery for opted-out agent-network targets.
func (c *CapturedData) SetSuppressAccessLog(b bool) {
c.mu.Lock()
defer c.mu.Unlock()
c.suppressAccessLog = b
}
// GetSuppressAccessLog reports whether access-log emission has been
// suppressed for this request.
func (c *CapturedData) GetSuppressAccessLog() bool {
c.mu.RLock()
defer c.mu.RUnlock()
return c.suppressAccessLog
}
// GetUserGroups returns a copy of the authenticated user's group
// memberships.
func (c *CapturedData) GetUserGroups() []string {
+424 -32
View File
@@ -2,6 +2,7 @@ package proxy
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
@@ -11,10 +12,13 @@ import (
"net/url"
"strings"
"sync"
"time"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/proxy/auth"
"github.com/netbirdio/netbird/proxy/internal/middleware"
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
"github.com/netbirdio/netbird/proxy/internal/roundtrip"
"github.com/netbirdio/netbird/proxy/internal/types"
"github.com/netbirdio/netbird/proxy/web"
@@ -32,6 +36,25 @@ type ReverseProxy struct {
mappingsMux sync.RWMutex
mappings map[string]Mapping
logger *log.Logger
// middlewareManager, when non-nil, drives per-target middleware
// dispatch. A nil manager (or an empty chain for the resolved
// target) keeps the reverse-proxy hot path on the no-capture fast
// path with no middleware overhead.
middlewareManager *middleware.Manager
}
// Option configures optional ReverseProxy behavior. Options exist so the core
// constructor signature stays stable across additive features.
type Option func(*ReverseProxy)
// WithMiddlewareManager attaches a middleware manager to the reverse
// proxy. When the manager is nil or returns an empty chain for the
// target, the request follows the fast path with no middleware
// overhead.
func WithMiddlewareManager(m *middleware.Manager) Option {
return func(p *ReverseProxy) {
p.middlewareManager = m
}
}
// NewReverseProxy configures a new NetBird ReverseProxy.
@@ -40,29 +63,28 @@ type ReverseProxy struct {
// between requested URLs and targets.
// The internal mappings can be modified using the AddMapping
// and RemoveMapping functions.
func NewReverseProxy(transport http.RoundTripper, forwardedProto string, trustedProxies []netip.Prefix, logger *log.Logger) *ReverseProxy {
func NewReverseProxy(transport http.RoundTripper, forwardedProto string, trustedProxies []netip.Prefix, logger *log.Logger, opts ...Option) *ReverseProxy {
if logger == nil {
logger = log.StandardLogger()
}
return &ReverseProxy{
p := &ReverseProxy{
transport: transport,
forwardedProto: forwardedProto,
trustedProxies: trustedProxies,
mappings: make(map[string]Mapping),
logger: logger,
}
for _, opt := range opts {
opt(p)
}
return p
}
func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
result, exists := p.findTargetForRequest(r)
if !exists {
if cd := CapturedDataFromContext(r.Context()); cd != nil {
cd.SetOrigin(OriginNoRoute)
}
requestID := getRequestID(r)
web.ServeErrorPage(w, r, http.StatusNotFound, "Service Not Found",
"The requested service could not be found. Please check the URL, try refreshing, or check if the peer is running. If that doesn't work, see our documentation for help.",
requestID, web.ErrorStatus{Proxy: true, Destination: false})
p.serveRouteError(w, r, http.StatusNotFound, "Service Not Found",
"The requested service could not be found. Please check the URL, try refreshing, or check if the peer is running. If that doesn't work, see our documentation for help.")
return
}
@@ -72,38 +94,23 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// with 421 (Misdirected Request) so the caller sees an explicit
// error instead of silently doubling tunnel traffic.
if p.isSelfTargetLoop(r, result.target.URL) {
if cd := CapturedDataFromContext(r.Context()); cd != nil {
cd.SetOrigin(OriginNoRoute)
}
requestID := getRequestID(r)
web.ServeErrorPage(w, r, http.StatusMisdirectedRequest, "Loop Detected",
"This peer is the target of the requested service. Reach the backend directly instead of dialing the public service URL from the same machine.",
requestID, web.ErrorStatus{Proxy: true, Destination: false})
p.serveRouteError(w, r, http.StatusMisdirectedRequest, "Loop Detected",
"This peer is the target of the requested service. Reach the backend directly instead of dialing the public service URL from the same machine.")
return
}
ctx := r.Context()
// Set the account ID in the context for the roundtripper to use.
ctx = roundtrip.WithAccountID(ctx, result.accountID)
pt := result.target
ctx := p.buildTargetContext(r.Context(), result)
// Populate captured data if it exists (allows middleware to read after handler completes).
// This solves the problem of passing data UP the middleware chain: we put a mutable struct
// pointer in the context, and mutate the struct here so outer middleware can read it.
if capturedData := CapturedDataFromContext(ctx); capturedData != nil {
capturedData := CapturedDataFromContext(ctx)
if capturedData != nil {
capturedData.SetServiceID(result.serviceID)
capturedData.SetAccountID(result.accountID)
}
pt := result.target
if pt.SkipTLSVerify {
ctx = roundtrip.WithSkipTLSVerify(ctx)
}
if pt.RequestTimeout > 0 {
ctx = types.WithDialTimeout(ctx, pt.RequestTimeout)
}
if pt.DirectUpstream {
ctx = roundtrip.WithDirectUpstream(ctx)
capturedData.SetAgentNetwork(result.target != nil && result.target.AgentNetwork)
capturedData.SetSuppressAccessLog(result.target != nil && result.target.DisableAccessLog)
}
rewriteMatchedPath := result.matchedPath
@@ -111,6 +118,45 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rewriteMatchedPath = ""
}
chain := p.resolveChain(result)
if chain == nil || chain.Empty() {
p.serveDirect(w, r, ctx, result, rewriteMatchedPath)
return
}
p.serveWithChain(w, r, ctx, result, chain, rewriteMatchedPath, capturedData)
}
// serveRouteError marks the request as un-routed on any captured-data
// context and renders the proxy error page.
func (p *ReverseProxy) serveRouteError(w http.ResponseWriter, r *http.Request, status int, title, message string) {
if cd := CapturedDataFromContext(r.Context()); cd != nil {
cd.SetOrigin(OriginNoRoute)
}
web.ServeErrorPage(w, r, status, title, message, getRequestID(r),
web.ErrorStatus{Proxy: true, Destination: false})
}
// buildTargetContext layers the per-target roundtrip flags (account id,
// TLS-verify skip, direct upstream, dial timeout) onto the request context.
func (p *ReverseProxy) buildTargetContext(ctx context.Context, result targetResult) context.Context {
pt := result.target
ctx = roundtrip.WithAccountID(ctx, result.accountID)
if pt.SkipTLSVerify {
ctx = roundtrip.WithSkipTLSVerify(ctx)
}
if pt.DirectUpstream {
ctx = roundtrip.WithDirectUpstream(ctx)
}
if pt.RequestTimeout > 0 {
ctx = types.WithDialTimeout(ctx, pt.RequestTimeout)
}
return ctx
}
// serveDirect forwards the request without a middleware chain — the common
// path for plain reverse-proxy targets.
func (p *ReverseProxy) serveDirect(w http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, rewriteMatchedPath string) {
pt := result.target
rp := &httputil.ReverseProxy{
Rewrite: p.rewriteFunc(pt.URL, rewriteMatchedPath, result.passHostHeader, pt.PathRewrite, pt.CustomHeaders, result.stripAuthHeaders),
Transport: p.transport,
@@ -123,6 +169,344 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rp.ServeHTTP(w, r.WithContext(ctx))
}
// serveWithChain runs the per-target middleware chain around the upstream
// request: request-leg capture and authorisation, then (on allow) the
// upstream forward with response/terminal observation deferred so it reads
// the captured response before the writer is released.
func (p *ReverseProxy) serveWithChain(w http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, chain *middleware.Chain, rewriteMatchedPath string, capturedData *CapturedData) {
middlewareIDs := chain.IDs()
p.logger.Debugf("middleware chain matched: service=%s path=%s middlewares=%v", result.serviceID, result.matchedPath, middlewareIDs)
capturedBody, truncated, originalSize, releaseBudget := p.captureRequestForChain(ctx, r, result, capturedData)
defer releaseBudget()
acc := middleware.NewAccumulator(middleware.MaxRequestMetadataBytes)
reqInput := buildRequestInput(r, result, capturedData, capturedBody, truncated, originalSize)
denyOutput, requestMeta, upstreamRewrite, _ := chain.RunRequest(ctx, r, reqInput, acc)
if capturedData != nil {
for _, kv := range requestMeta {
capturedData.SetMetadata(kv.Key, kv.Value)
}
}
if denyOutput != nil {
p.serveDeny(w, denyOutput, result, middlewareIDs)
return
}
respWriter, capturingWriter := p.newResponseWriter(ctx, w, result, capturedData)
if capturingWriter != nil {
defer capturingWriter.Release()
defer p.observeResponse(ctx, chain, acc, reqInput, requestMeta, capturingWriter, w, capturedData, result, middlewareIDs)
}
p.forwardUpstream(respWriter, r, ctx, result, rewriteMatchedPath, upstreamRewrite)
}
// captureRequestForChain copies the request body for inspection by the
// chain, records any capture bypass, and applies agent-network routing
// recovery for oversized bodies. The returned release frees the capture
// budget and must be deferred by the caller.
func (p *ReverseProxy) captureRequestForChain(ctx context.Context, r *http.Request, result targetResult, capturedData *CapturedData) ([]byte, bool, int64, func()) {
pt := result.target
capturedBody, truncated, originalSize, bypass, releaseBudget, captureErr := bodytap.CaptureRequest(r, pt.CaptureConfig, p.middlewareManager.Budget())
if captureErr != nil {
p.logger.Debugf("middleware request body capture error: %v", captureErr)
}
if bypass != "" {
if capturedData != nil {
capturedData.SetMetadata("mw.capture.bypass_reason", bypass)
}
p.middlewareManager.Metrics().IncCaptureBypass(ctx, string(result.serviceID), bypass)
}
// Routing recovery for oversized agent-network requests: when the body
// exceeded the capture cap (bypassed or truncated), the captured copy
// can't be parsed for the model, so llm_router would deny with
// model_not_routable. Scan the full stream for just the routing fields
// and hand the request parser a minimal stub so routing succeeds; the
// prompt stays uncaptured and the upstream still gets the full body.
if pt.AgentNetwork && (truncated || capturedBody == nil) {
if model, stream, ok := bodytap.ScanRoutingFields(r, bodytap.MaxRoutingScanBytes); ok {
capturedBody = buildRoutingStub(model, stream)
truncated = false
p.logger.Debugf("agent-network routing recovery: extracted model=%s stream=%t from oversized request body (service=%s)", model, stream, result.serviceID)
}
}
return capturedBody, truncated, originalSize, releaseBudget
}
// serveDeny renders the chain's deny response. Policy/budget/routing/guardrail
// denials are expected runtime outcomes and can be high-volume under
// misconfigured or hostile clients; per-request detail stays at Debug and
// metrics/access logs carry the signal at scale.
func (p *ReverseProxy) serveDeny(w http.ResponseWriter, denyOutput *middleware.Output, result targetResult, middlewareIDs []string) {
middlewareID := "middleware"
if denyOutput.DenyReason != nil && denyOutput.DenyReason.Code != "" {
middlewareID = denyOutput.DenyReason.Code
}
p.logger.Debugf("middleware chain denied request: service=%s path=%s middlewares=%v reason=%s status=%d",
result.serviceID, result.matchedPath, middlewareIDs, middlewareID, denyOutput.DenyStatus)
middleware.RenderDenyResponse(w, middlewareID, denyOutput.DenyReason, denyOutput.DenyStatus)
}
// newResponseWriter returns the writer the upstream forward should use. When
// response capture is enabled and not bypassed it wraps w in a capturing
// writer (also returned so the caller can release it and feed the response
// leg); otherwise the capturing writer is nil and w is used directly.
func (p *ReverseProxy) newResponseWriter(ctx context.Context, w http.ResponseWriter, result targetResult, capturedData *CapturedData) (http.ResponseWriter, *bodytap.CapturingResponseWriter) {
pt := result.target
if pt.CaptureConfig == nil || pt.CaptureConfig.MaxResponseBytes <= 0 {
return w, nil
}
capturingWriter := bodytap.NewCapturingResponseWriter(w, pt.CaptureConfig.MaxResponseBytes, p.middlewareManager.Budget())
if capturingWriter.Bypassed() {
if capturedData != nil {
capturedData.SetMetadata("mw.capture.bypass_reason", capturingWriter.BypassReason())
}
p.middlewareManager.Metrics().IncCaptureBypass(ctx, string(result.serviceID), capturingWriter.BypassReason())
capturingWriter.Release()
return w, nil
}
return capturingWriter, capturingWriter
}
// observeResponse runs the response and terminal middleware slots after the
// body has been forwarded. It is deferred by serveWithChain so it reads the
// captured response before the writer is released.
func (p *ReverseProxy) observeResponse(ctx context.Context, chain *middleware.Chain, acc *middleware.Accumulator, reqInput *middleware.Input, requestMeta []middleware.KV, capturingWriter *bodytap.CapturingResponseWriter, w http.ResponseWriter, capturedData *CapturedData, result targetResult, middlewareIDs []string) {
respInput := &middleware.Input{
Slot: middleware.SlotOnResponse,
RequestID: reqInput.RequestID,
TargetID: reqInput.TargetID,
Method: reqInput.Method,
URL: reqInput.URL,
Headers: reqInput.Headers,
Status: capturingWriter.Status(),
RespHeaders: headerToKV(w.Header()),
RespBody: capturingWriter.Body(),
RespBodyTruncated: capturingWriter.Truncated(),
OriginalRespSize: capturingWriter.BytesWritten(),
ServiceID: reqInput.ServiceID,
AccountID: reqInput.AccountID,
UserID: reqInput.UserID,
// UserEmail / UserGroups / UserGroupNames must flow into the
// response leg too — llm_limit_record needs UserGroups to send
// group_ids on RecordLLMUsage so management's account-budget
// fan-out can match group-targeted rules; identity-stamping and
// any future response-side authorisation also depend on these.
UserEmail: reqInput.UserEmail,
UserGroups: reqInput.UserGroups,
UserGroupNames: reqInput.UserGroupNames,
AuthMethod: reqInput.AuthMethod,
SourceIP: reqInput.SourceIP,
Metadata: requestMeta,
AgentNetwork: reqInput.AgentNetwork,
}
// The response/terminal phase runs after the body is forwarded, so
// a streaming client (e.g. Codex) has usually disconnected by now,
// cancelling r.Context(). These middlewares only observe and record
// (token/cost metering, usage recording) and must still complete —
// otherwise the dispatcher short-circuits each to fail-mode and the
// usage is silently lost. Detach from client cancellation, keep ctx
// values, and bound the work.
obsCtx, obsCancel := context.WithTimeout(context.WithoutCancel(ctx), observabilityPhaseTimeout)
defer obsCancel()
respMeta := chain.RunResponse(obsCtx, respInput, acc)
if capturedData != nil {
for _, kv := range respMeta {
capturedData.SetMetadata(kv.Key, kv.Value)
}
}
// Terminal slot sees the merged metadata bag from request and
// response phases.
mergedMeta := append(append([]middleware.KV(nil), requestMeta...), respMeta...)
termInput := *respInput
termInput.Slot = middleware.SlotTerminal
termInput.Metadata = mergedMeta
termMeta := chain.RunTerminal(obsCtx, &termInput, acc)
if capturedData != nil {
for _, kv := range termMeta {
capturedData.SetMetadata(kv.Key, kv.Value)
}
}
p.logger.Debugf("middleware chain ran: service=%s path=%s middlewares=%v status=%d req_meta=%d resp_meta=%d term_meta=%d",
result.serviceID, result.matchedPath, middlewareIDs, capturingWriter.Status(), len(requestMeta), len(respMeta), len(termMeta))
}
// forwardUpstream applies any middleware-emitted upstream rewrite and proxies
// the request to the effective upstream URL.
func (p *ReverseProxy) forwardUpstream(respWriter http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, rewriteMatchedPath string, upstreamRewrite *middleware.UpstreamRewrite) {
pt := result.target
effectiveURL := applyUpstreamRewrite(pt.URL, upstreamRewrite)
if upstreamRewrite != nil {
r.Host = effectiveURL.Host
applyUpstreamHeaders(r, upstreamRewrite)
stripUpstreamPathPrefix(r, upstreamRewrite.StripPathPrefix)
}
rp := &httputil.ReverseProxy{
Rewrite: p.rewriteFunc(effectiveURL, rewriteMatchedPath, result.passHostHeader, pt.PathRewrite, pt.CustomHeaders, result.stripAuthHeaders),
Transport: p.transport,
FlushInterval: -1,
ErrorHandler: p.proxyErrorHandler,
}
if result.rewriteRedirects {
rp.ModifyResponse = p.rewriteLocationFunc(effectiveURL, rewriteMatchedPath, r) //nolint:bodyclose
}
rp.ServeHTTP(respWriter, r.WithContext(ctx))
}
// buildRoutingStub returns a minimal JSON request body carrying only the
// model and stream fields. It feeds the LLM request parser when the real
// body was too large to capture: the parser emits llm.model / llm.stream
// so llm_router can route, while ExtractPrompt on the stub yields nothing
// — no prompt is captured for oversized requests.
func buildRoutingStub(model string, stream bool) []byte {
b, err := json.Marshal(map[string]any{"model": model, "stream": stream})
if err != nil {
return nil
}
return b
}
// applyUpstreamRewrite returns the effective upstream URL after
// applying a middleware-emitted rewrite. When rewrite is nil or
// incomplete, the original target is returned unchanged. The original
// URL is never mutated; a clone is returned when a rewrite applies.
//
// Rewrite Path semantics: when non-empty, replaces the cloned URL's
// path entirely. httputil.ProxyRequest.SetURL then joins target.Path
// with the agent's request path, so an operator-configured upstream
// path like "/v1/{account}/{gateway}/compat" gets prepended to
// "/chat/completions" yielding the full Cloudflare-shaped path.
// Empty rewrite.Path preserves the original target's path (the
// historical, non-agent-network behavior).
func applyUpstreamRewrite(orig *url.URL, rewrite *middleware.UpstreamRewrite) *url.URL {
if rewrite == nil || orig == nil {
return orig
}
if rewrite.Scheme == "" || rewrite.Host == "" {
return orig
}
cloned := *orig
cloned.Scheme = rewrite.Scheme
cloned.Host = rewrite.Host
if rewrite.Path != "" {
cloned.Path = rewrite.Path
cloned.RawPath = ""
}
return &cloned
}
// stripUpstreamPathPrefix removes a gateway-namespace prefix (e.g. "/bedrock")
// from the request path before it is forwarded, so the upstream receives its
// native path. The chain has already run by this point, so metering/logging
// keep the original client path; only the outbound path is rewritten. RawPath
// is cleared so the escaped form is recomputed from the trimmed Path.
func stripUpstreamPathPrefix(r *http.Request, prefix string) {
if r == nil || r.URL == nil || prefix == "" {
return
}
if !strings.HasPrefix(r.URL.Path, prefix+"/") && r.URL.Path != prefix {
return
}
r.URL.Path = strings.TrimPrefix(r.URL.Path, prefix)
if r.URL.Path == "" {
r.URL.Path = "/"
}
r.URL.RawPath = ""
}
// applyUpstreamHeaders strips the headers the rewrite asks for and
// injects the resolved auth header on the in-flight request. It is
// the proxy-trusted counterpart to chain.applyMutations: regular
// middleware HeadersAdd/HeadersRemove pass through the framework
// denylist (which blocks Authorization, Cookie, etc.), but the
// router middleware needs to replace Authorization on the upstream
// request as a first-class operation. AuthHeader/StripHeaders ride
// on UpstreamRewrite so only the proxy's upstream-build path
// unpacks them — middlewares can't smuggle these in via the
// regular mutation surface.
func applyUpstreamHeaders(r *http.Request, rewrite *middleware.UpstreamRewrite) {
if r == nil || rewrite == nil {
return
}
for _, name := range rewrite.StripHeaders {
if name == "" {
continue
}
r.Header.Del(name)
}
if rewrite.AuthHeader != nil && rewrite.AuthHeader.Name != "" {
r.Header.Set(rewrite.AuthHeader.Name, rewrite.AuthHeader.Value)
}
}
// resolveChain returns the middleware chain registered for the
// resolved target, or nil when middleware is disabled for the proxy
// or the target.
func (p *ReverseProxy) resolveChain(result targetResult) *middleware.Chain {
if p.middlewareManager == nil {
return nil
}
return p.middlewareManager.ChainFor(string(result.serviceID), result.matchedPath)
}
// buildRequestInput gathers the per-request fields the middleware
// chain needs. Body and captured metadata are passed in; the rest are
// copied from the request and CapturedData.
func buildRequestInput(r *http.Request, result targetResult, cd *CapturedData, body []byte, truncated bool, originalSize int64) *middleware.Input {
in := &middleware.Input{
Slot: middleware.SlotOnRequest,
TargetID: result.matchedPath,
Method: r.Method,
URL: r.URL.String(),
Headers: headerToKV(r.Header),
Body: body,
BodyTruncated: truncated,
OriginalBodySize: originalSize,
ServiceID: string(result.serviceID),
AccountID: string(result.accountID),
AgentNetwork: result.target != nil && result.target.AgentNetwork,
}
if cd != nil {
in.RequestID = cd.GetRequestID()
in.UserID = cd.GetUserID()
in.UserEmail = cd.GetUserEmail()
in.UserGroups = cd.GetUserGroups()
in.UserGroupNames = cd.GetUserGroupNames()
in.AuthMethod = cd.GetAuthMethod()
if ip := cd.GetClientIP(); ip.IsValid() {
in.SourceIP = ip.String()
}
}
return in
}
// headerToKV flattens an http.Header into the KV slice shape expected
// by the middleware envelope, preserving value order under the same
// key.
func headerToKV(h http.Header) []middleware.KV {
if len(h) == 0 {
return nil
}
total := 0
for _, v := range h {
total += len(v)
}
out := make([]middleware.KV, 0, total)
for k, vs := range h {
for _, v := range vs {
out = append(out, middleware.KV{Key: k, Value: v})
}
}
return out
}
// isSelfTargetLoop reports whether an overlay-origin request is about to
// be forwarded back to the very peer that initiated it. The detection
// is intentionally narrow: it only fires when the request arrived on
@@ -486,6 +870,14 @@ const (
// comma or any non-printable byte are dropped at stamp time so the
// list is unambiguously splittable by consumers.
headerNetBirdGroups = "X-NetBird-Groups"
// observabilityPhaseTimeout bounds the detached response/terminal
// metering phase. It runs after the client connection (and its context)
// may be gone, so it can't borrow the request deadline; this ceiling
// keeps a slow management round-trip (RecordLLMUsage) from pinning the
// handler goroutine indefinitely while still allowing each middleware
// its own per-invoke timeout.
observabilityPhaseTimeout = 30 * time.Second
)
// isHeaderValueSafe reports whether v is a valid RFC 7230 field-value:
+43
View File
@@ -19,6 +19,7 @@ import (
"github.com/stretchr/testify/require"
"github.com/netbirdio/netbird/proxy/auth"
"github.com/netbirdio/netbird/proxy/internal/middleware"
"github.com/netbirdio/netbird/proxy/internal/roundtrip"
"github.com/netbirdio/netbird/proxy/internal/types"
"github.com/netbirdio/netbird/proxy/web"
@@ -1407,3 +1408,45 @@ func TestStampNetBirdIdentity_CapturedDataPresentButEmpty(t *testing.T) {
assert.Empty(t, pr.Out.Header.Get(headerNetBirdGroups),
"X-NetBird-Groups must be stripped when CapturedData has no groups")
}
// TestBuildRequestInput_PropagatesIdentityAndGroups locks the final wiring link
// between auth and the middleware chain: CapturedData identity (user, groups,
// auth method, client IP) and the target's AgentNetwork flag must land on the
// middleware Input the chain runs against. If UserGroups stops flowing here,
// llm_router denies every request with no_authorised_provider.
func TestBuildRequestInput_PropagatesIdentityAndGroups(t *testing.T) {
cd := NewCapturedData("req-123")
cd.SetUserID("user-1")
cd.SetUserEmail("user@example.com")
cd.SetUserGroups([]string{"grp-admins", "grp-users"})
cd.SetUserGroupNames([]string{"Admins", "Users"})
cd.SetAuthMethod("oidc")
cd.SetClientIP(netip.MustParseAddr("100.90.1.14"))
r := httptest.NewRequest(http.MethodPost, "http://agent.example.com/v1/chat/completions", nil)
r.Header.Set("Content-Type", "application/json")
result := targetResult{
target: &PathTarget{AgentNetwork: true},
matchedPath: "/",
serviceID: types.ServiceID("svc-1"),
accountID: types.AccountID("acct-1"),
}
body := []byte(`{"model":"gpt-5.4"}`)
in := buildRequestInput(r, result, cd, body, false, int64(len(body)))
require.NotNil(t, in, "buildRequestInput must return an envelope")
assert.Equal(t, middleware.SlotOnRequest, in.Slot, "request input runs in the on-request slot")
assert.Equal(t, "svc-1", in.ServiceID, "service id must propagate")
assert.Equal(t, "acct-1", in.AccountID, "account id must propagate")
assert.Equal(t, "user-1", in.UserID, "user id must propagate")
assert.Equal(t, "user@example.com", in.UserEmail, "user email must propagate")
assert.Equal(t, []string{"grp-admins", "grp-users"}, in.UserGroups,
"CapturedData groups MUST reach the middleware Input — llm_router authorises against this")
assert.Equal(t, []string{"Admins", "Users"}, in.UserGroupNames, "group names must propagate")
assert.Equal(t, "oidc", in.AuthMethod, "auth method must propagate")
assert.Equal(t, "100.90.1.14", in.SourceIP, "client IP must propagate")
assert.True(t, in.AgentNetwork, "agent-network target flag must reach the Input")
assert.Equal(t, body, in.Body, "captured body must reach the Input")
}
+16
View File
@@ -8,6 +8,8 @@ import (
"strings"
"time"
"github.com/netbirdio/netbird/proxy/internal/middleware"
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
"github.com/netbirdio/netbird/proxy/internal/types"
)
@@ -32,6 +34,20 @@ type PathTarget struct {
// over the embedded NetBird WireGuard client when forwarding requests
// to this target. Default false → embedded client (existing behaviour).
DirectUpstream bool
// Middlewares is the validated per-target middleware chain. Nil or empty
// for non-agent-network targets, keeping them on the no-middleware fast path.
Middlewares []middleware.Spec
// CaptureConfig holds the per-target body-capture limits used by the
// middleware chain. Nil for targets without body-inspecting middlewares.
CaptureConfig *bodytap.Config
// AgentNetwork marks this target as a synthesised agent-network target so
// the proxy can tag access-log entries and gate agent-network behaviour.
AgentNetwork bool
// DisableAccessLog suppresses the per-request access-log emission for this
// target. Defaults false so non-agent-network targets continue to log
// unchanged. The agent-network synthesizer sets this true only when the
// account's EnableLogCollection toggle is off.
DisableAccessLog bool
}
// Mapping describes how a domain is routed by the HTTP reverse proxy.
+30
View File
@@ -0,0 +1,30 @@
package proxy
import (
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
)
func TestStripUpstreamPathPrefix(t *testing.T) {
cases := []struct {
name string
path string
prefix string
want string
}{
{"strips matching namespace prefix", "/bedrock/model/x/invoke", "/bedrock", "/model/x/invoke"},
{"no-op when prefix absent", "/model/x/invoke", "/bedrock", "/model/x/invoke"},
{"no-op on empty prefix", "/bedrock/model/x/invoke", "", "/bedrock/model/x/invoke"},
{"no-op on non-segment match", "/bedrockfoo/model/x", "/bedrock", "/bedrockfoo/model/x"},
{"bare prefix collapses to root", "/bedrock", "/bedrock", "/"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest("POST", tc.path, nil)
stripUpstreamPathPrefix(r, tc.prefix)
assert.Equal(t, tc.want, r.URL.Path, "stripped path for %q", tc.path)
})
}
}