mirror of
https://github.com/netbirdio/netbird.git
synced 2026-09-17 20:29:07 +02:00
[management,proxy,client] 0.74.0 version (#6563)
* [management,proxy] Agent network: per-account LLM gateway (policy, metering, multi-provider) (#6555) * [agent-network] Shared proto, OpenAPI schema, and generated types * [agent-network] Management: store, manager, synthesizer, policy engine, provider catalog, HTTP/gRPC API Adds the account-scoped agent-network module: provider/policy/budget CRUD and store, the reverse-proxy service synthesizer, policy selection + limit enforcement, the provider catalog (incl. Vertex AI and AWS Bedrock entries), and the management HTTP + proxy gRPC surfaces. * [management] Fix agent-network proxy-peer fan-out on affected-peer recompute The affected-peers resolver loaded only persisted reverse-proxy services, but agent-network services are synthesized on demand and never persisted. As a result the embedded proxy peer was never folded into the affected set when a client's group changed, so the proxy received no network-map update for a newly authorised client and rejected its handshake until a full resync (restart). loadProxyServices now merges the synthesized agent-network services (injected via a registration hook to avoid an import cycle), so proxy peers learn newly authorised clients immediately. * [proxy] Reverse-proxy middleware framework, chain, and request plumbing The per-target middleware chain (slots, dispatcher, mutation gate, metadata merger), body capture, access-log terminal sink, and the proxy wiring that builds + runs chains for synthesized agent-network services. * [proxy] LLM parsers, pricing, and builtin middlewares (OpenAI, Anthropic, Vertex AI, AWS Bedrock) Request/response parsers and SSE/event-stream metering, the embedded pricing table, and the builtin middleware set: request parser, router, policy limit-check/record, cost meter, guardrail, identity inject, response parser. Includes the path-routed providers — Google Vertex AI (keyfile:: service-account OAuth minting) and AWS Bedrock (bearer auth, invoke/converse/streaming, optional /bedrock prefix) — plus the Models allowlist and unmeterable-publisher deny. * [proxy] IPv6 in-place apply and TCP accept-loop hardening on netstack listeners * [agent-network] End-to-end test suite, module docs, and deployment preset * [agent-network] Fix codespell typos and exclude false positives - labelgen word pool: vermillion -> vermilion, racoon -> raccoon. - codespell ignore list: add flate (Go compress/flate package), recordin (a test-local identifier), and unparseable (a valid alternative spelling used consistently across identifiers + a metadata-value constant). * [management] Set LastSeen on injected proxy peer in realstack test (MySQL strict-mode) The injected embedded proxy peer had a PeerStatus with a zero LastSeen, which serializes to '0000-00-00' and is rejected by MySQL in strict mode (SQLite tolerates it). Set LastSeen to a valid time so SaveAccount succeeds on both engines. * [agent-network] Remove e2e shell-script suite from this branch The end-to-end shell scripts under scripts/e2e/ are maintained in a separate testing suite and are not part of this change set. * [agent-network] Polish module docs: remove internal review scaffolding, fix links, verify diagrams Strip PR-review framing, commit references, absolute paths, and stale internal references from the agent-network module docs; fix broken relative links; verify all diagrams against the current architecture. Remove the internal AI-reviewer prompt file. * [management] Refine session expiration handling to support 3-state encoding for SSO deadlines * [agent-network] Relocate agentnetwork package to internals/modules Move management/server/agentnetwork (and its catalog/, labelgen/, types/ subpackages) to management/internals/modules/agentnetwork, alongside the reverse-proxy module, and rewrite all importers. Pure relocation: package names, the synthesizer + affectedpeers registration hook, and store access (shared store.Store) are unchanged, so no import cycle is introduced (affectedpeers still depends only on the agentnetwork/types leaf). * [agent-network] Co-locate HTTP handlers in the module (RegisterEndpoints) Move the agent-network HTTP handlers from server/http/handlers/agentnetwork into the module at internals/modules/agentnetwork/handlers (package handlers) and rename the entrypoint AddEndpoints -> RegisterEndpoints, matching the reverse-proxy module convention. Wiring in http/handler.go updated accordingly. * Update getting started to point to rc when agent network enabled * Add a reference to a commercial license * Fix docs localhost link * Fix docs localhost link * Add private services domain note * [management] Add agent-network telemetry metrics (#6561) Surface agent-network adoption and usage in the self-hosted metrics worker: distinct accounts, providers, policies, budget rules, accounts with log collection enabled, and aggregated input/output tokens plus cost. Tokens and cost are summed from agent_network_request_usage (the always-written per-request ledger) so the figures are accurate regardless of the log-collection toggle and carry no double-counting. All values come from a handful of indexed aggregate queries run only on the worker's periodic tick. Adds store.AgentNetworkMetrics with GetAgentNetworkMetrics on the Store interface, the SqlStore implementation, and a zero-valued FileStore stub. * Update NetBird server and proxy image versions to 0.74.0-rc.2 * [management,proxy] Reduce agent-network cognitive complexity (#6566) Address the SonarCloud quality-gate findings in new agent-network code by extracting focused helpers. No behavior change. - synthesizer.go: split buildIdentityInjectConfigJSON into per-shape rule builders; extract mergeGuardrail from mergeGuardrails to cut nesting depth. - llm_identity_inject: extract injectionEmitsAnything validation predicate from New. - llm_response_parser/streaming.go: extract applyOpenAIStreamUsage and applyAnthropicStreamUsage (via a named anthropicStreamUsage type) and simplify the OpenAI scanner loop. - reverseproxy.go: decompose ServeHTTP into serveRouteError, buildTargetContext, serveDirect, serveWithChain, captureRequestForChain, serveDeny, newResponseWriter, observeResponse, and forwardUpstream, preserving the defer ordering so response observation still reads the captured writer before it is released. * [management] Move agent-network access-log ingest into the agentnetwork module (#6568) The agent-network access-log ingest path (metaKey wire contract, flatten, usage derivation, and the dual-write of the usage ledger + settings-gated full row) lived in the reverseproxy accesslogs manager, even though the agentnetwork module already owns the rest of that domain — types, read (ListAccessLogs / GetUsageOverview), the budget-counter writes, and retention cleanup. Move it next to the rest: a stateless agentnetwork.IngestAccessLog(ctx, store, entry) that the reverseproxy SaveAccessLog delegates to when the entry is agent-network. Removes the agentNetworkTypes import from the reverseproxy manager. No behavior change; the write/read table separation is unchanged. Adds real-store coverage for the disable->enable log-collection toggle (usage ledger always written, full row gated) plus the metadata parse and group-dedup helpers, which previously had no dedicated tests. * Add session view support in the access log * [management,proxy] Container-based agent-network e2e harness (#6577) * [e2e] Add container-based agent-network e2e harness (Pillar 1) Introduce a self-contained, OIDC-free e2e harness that stands up NetBird in containers, so suites no longer depend on the hand-maintained Tilt stack or a real IdP. - harness brings up the combined server (management + signal + relay + STUN + embedded IdP) in a single container built from combined/Dockerfile.multistage, and mints an admin PAT through the unauthenticated /api/setup bootstrap (NB_SETUP_PAT_ENABLED). API access goes through the existing shared/management/client/rest typed client. - the image is built via the docker CLI (BuildKit) so the Dockerfile's cache mounts are honored; testcontainers then runs the tagged image. - everything is behind the `e2e` build tag so normal builds and unit tests never pull in testcontainers. Adds BuildKit cache mounts to combined/Dockerfile.multistage so source changes recompile incrementally rather than from scratch. Pillar 1 proven by TestCombinedBootstrap: server builds, boots, mints a PAT, and the PAT authenticates a real management API call. * [e2e] Add management-side agent-network scenarios (Pillar 2) Port the API-driven agent-network scenarios from the bash suites to Go, sharing one combined server per package run (TestMain) with each test owning its resource cleanup. Drives the /api/agent-network/* endpoints through the shared REST client's NewRequest primitive with the generated api types. Scenarios: - provider lifecycle (create/get/list/delete + 404 after delete) - provider validation (missing api_key, unknown catalog id → 4xx) - settings collection-toggle round-trip with cluster/subdomain immutability - policy window floor (reject <60s enabled limit, accept at 60s) - consumption read endpoint returns an array All deterministic and dependency-free (dummy provider keys; no upstream calls), so they run headless in CI. * [e2e] Add live chat-through-proxy scenario (Pillar 3) Stand up the full agent-network data path in containers and drive a real chat-completion through the gateway: - harness: a shared docker network (combined server reachable by alias), a proxy container built from the published reverse-proxy image (NB_PROXY_PRIVATE, NB_PROXY_ALLOW_INSECURE, NB_RELAY_TRANSPORT=ws to match the combined server's WS-multiplexed relay) with a generated self-signed wildcard cert, and a netbird client container that joins via a setup key. - the combined image, proxy image, and client image default to the published rc.2 releases (overridable via NB_E2E_*_IMAGE; a bare local tag is built from source instead). Geolocation download is disabled so the server starts without external fetches. - one shared domain is used for the management exposed address, the proxy domain, and the agent-network cluster; the proxy token is minted via the server CLI (global) to match the manual install. TestChatCompletionThroughProxy provisions provider+policy+group+setup key, runs proxy+client, drives an OpenAI chat-completion through the tunnel, and asserts a 200 plus the ingested access-log row. Requires OPENAI_TOKEN (skips otherwise). The provider must be created with enabled=true explicitly — the create default is false despite the API doc. * [e2e] Run the live chat scenario across a provider matrix Replace the single-provider chat test with a data-driven matrix that runs the same scenario through every provider whose credentials are present in the environment (keys/URLs sourced from ~/.llm-keys locally, Actions secrets in CI): - OpenAI (chat), Anthropic (messages), Vercel, OpenRouter, Cloudflare (OpenAI-compatible gateways), and Bedrock (path-routed, bearer, via the messages shape) — covering both wire shapes and the gateway routing. - all providers are created enabled with a unique model string so the proxy's connect-time snapshot carries them all and model->provider routing is unambiguous (provider toggles after connect don't reconcile to a connected proxy). - the client supports both wire shapes (/v1/chat/completions and /v1/messages); Cloudflare gets the openai provider segment appended to its gateway URL. Each provider must return 200 through the tunnel and produce an ingested access-log row. Vertex is intentionally excluded from the uniform matrix: it needs a bespoke rawPredict request shape rather than the shared chat/messages path, so it warrants a dedicated scenario. * [ci] Add manual workflow for the agent-network e2e suite The e2e suite (build tag `e2e`) stands up the combined server + proxy + client in Docker and drives live chat-completions, so it is slow and needs provider credentials. Gate it out of normal CI (it already is, via the build tag) and run it on demand via workflow_dispatch. Provider scenarios skip when their secret is unset, so it degrades gracefully. * [e2e] Add Vertex to the provider matrix; run e2e on ubuntu-latest Vertex (Anthropic-on-Vertex) doesn't share the chat/messages wire shapes: the model travels in a rawPredict path and the proxy mints the service account's OAuth token. Add a Vertex client method that posts /v1/projects/<project>/locations/<region>/publishers/anthropic/models/<model>:rawPredict with the Vertex anthropic_version body, and wire it into the matrix as a path-routed provider (created without a models array). It is keyed off GOOGLE_VERTEX_SA_BASE64 + GOOGLE_VERTEX_PROJECT (region defaults to "global", model to a pinned claude snapshot, both overridable). Also bump the e2e workflow runner to ubuntu-latest and add the Vertex secrets. * Add docker/docker and docker/go-connections as direct dependencies in go.mod * [ci] Trigger agent-network e2e workflow on push to main and pull requests * [e2e] Fix proxy cert permission denied on Linux CI runners The proxy bind-mounts a temp dir of self-signed certs. MkdirTemp creates it 0700 and the key was 0600, which Docker Desktop on macOS ignores but a non-root proxy container on Linux runners cannot traverse/read, so the cert watcher failed with "open /certs/tls.crt: permission denied" and the container exited. Widen the cert dir to 0755 and write the throwaway key 0644 so the proxy uid can read the bind-mounted material. * [e2e] Build images from source by default instead of pulling rc.2 The agent-network code under test lives in this branch, so the e2e should exercise it rather than a frozen published release. Flip the harness default: combined/proxy/client are now built from their in-repo Dockerfiles (combined/Dockerfile.multistage, proxy/Dockerfile.multistage, e2e/harness/Dockerfile.client) under local tags. Pulling a published image stays available by setting NB_E2E_*_IMAGE to a registry reference. Builds now go through buildx --load so the Dockerfile cache mounts are honored and the result is loaded for testcontainers. The CI workflow adds a container-driver builder and a local layer cache (NB_E2E_BUILDX_CACHE) persisted via actions/cache, which caches the base/apt/dep-download layers across runs. The Go compile still re-runs each time, as BuildKit mount caches cannot be exported to the GitHub cache. * [e2e] Cover real providers in lifecycle + assert real consumption metering - TestProviderLifecycle now runs per available real provider (create → get → list → delete → 404) instead of a single dummy provider, exercising each catalog's create and field round-trip. Create is offline, so it stays fast and burns no provider quota; falls back to a synthetic OpenAI provider when no keys are set. - TestProvidersMatrix attaches a token limit (high caps, 60s window) to its policy, which switches on usage metering, and asserts consumption rows are recorded with positive token counts after the live traffic. Consumption is account-scoped (keyed by source group / user and window, not per provider), so the assertion is aggregate. - TestProviderValidation gains invalid-upstream and blank-name cases. Create validation is uniform across catalogs (no per-provider required-field rules), so per-provider rejection cases would be redundant. * [e2e] Assert session id propagates per provider Each matrix request now sends a unique session id as the universal x-session-id header and asserts it round-trips into that provider's access-log row. This guards the session-grouping contract end to end for every provider (header extraction runs in llm_request_parser ahead of the parser-specific body extraction, so it is provider-agnostic). * [e2e] Drop accidentally committed sync-phases dashboard netbird-sync-phases.json was swept into the Pillar 1 commit by a broad git add; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Remove it from the branch so the PR diff is scoped to the e2e changes. * [e2e] Revert accidentally committed sync-phase ingest spec The netbird_sync_phase measurement spec in metrics ingest was swept into the Pillar 1 commit; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Its emission side never landed here, so the spec was orphaned anyway. Restore ingest/main.go to its origin/main state. * Fix golint issues * Fix sonar * Add access log session test * Fix access log tests --------- Co-authored-by: braginini <bangvalo@gmail.com> Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
This commit is contained in:
co-authored by
braginini
Zoltan Papp
parent
3be90f06b2
commit
92a66cdd20
@@ -0,0 +1,321 @@
|
||||
package proxy_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
"google.golang.org/grpc/test/bufconn"
|
||||
|
||||
rpservice "github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service"
|
||||
mgmtgrpc "github.com/netbirdio/netbird/management/internals/shared/grpc"
|
||||
"github.com/netbirdio/netbird/management/internals/modules/agentnetwork"
|
||||
agentNetworkTypes "github.com/netbirdio/netbird/management/internals/modules/agentnetwork/types"
|
||||
"github.com/netbirdio/netbird/management/server/store"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
|
||||
mwbuiltin "github.com/netbirdio/netbird/proxy/internal/middleware/builtin"
|
||||
// Side-effect imports register every builtin middleware factory.
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/cost_meter"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_guardrail"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_identity_inject"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_limit_check"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_limit_record"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_request_parser"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_response_parser"
|
||||
_ "github.com/netbirdio/netbird/proxy/internal/middleware/builtin/llm_router"
|
||||
"github.com/netbirdio/netbird/proxy/internal/proxy"
|
||||
nbproxytypes "github.com/netbirdio/netbird/proxy/internal/types"
|
||||
"github.com/netbirdio/netbird/shared/management/proto"
|
||||
|
||||
log "github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// TestReverseProxy_AgentNetworkRequest_FullChain is the self-contained Go
|
||||
// replacement for the bash 50 + 51 legs. It drives a real agent-network
|
||||
// request through proxy.ReverseProxy.ServeHTTP with the actual middleware
|
||||
// chain the synthesizer produces, against an in-process management gRPC and a
|
||||
// httptest fake upstream — no tilt, no docker, no real LLM provider, no
|
||||
// WireGuard tunnel. The test guarantees:
|
||||
//
|
||||
// 1. The reverse proxy's response-leg input construction copies UserGroups
|
||||
// onto respInput so llm_limit_record sends a non-empty group_ids field
|
||||
// on RecordLLMUsage. This is the exact bug class that motivated the
|
||||
// reverseproxy.go fix — its regression would land the request OK but
|
||||
// leave consumption at zero, defeating any group-targeted budget rule.
|
||||
// 2. With settings.RedactPii=true the parsers ship redacted text on both
|
||||
// llm.request_prompt_raw and llm.response_completion — proving the
|
||||
// end-to-end wiring (synth → proto → spec → parser config) carries the
|
||||
// toggle through to runtime emission.
|
||||
// 3. The full chain (request + response + recorder) runs against a real
|
||||
// management stack and the consumption row for the bound group dim
|
||||
// increments.
|
||||
//
|
||||
// If any of those three guarantees regresses, this single test fails.
|
||||
func TestReverseProxy_AgentNetworkRequest_FullChain(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("sqlite store not supported on Windows")
|
||||
}
|
||||
|
||||
const (
|
||||
testAccountID = "acct-fullchain-1"
|
||||
testAdminUser = "user-admin-1"
|
||||
adminGroupID = "grp-admins"
|
||||
providerID = "prov-openai-test"
|
||||
cluster = "test.proxy.local"
|
||||
subdomain = "fullchain"
|
||||
)
|
||||
testLogger := log.New()
|
||||
testLogger.SetLevel(log.PanicLevel) // keep test output clean
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// ---- 1. Fake upstream that returns OpenAI-shaped JSON with PII in the
|
||||
// completion. The reverse proxy's chain will redact this when the synth
|
||||
// stamps redact_pii=true on the response parser config.
|
||||
completion := "Sample record: Alice Johnson alice.johnson@example.com SSN 123-45-6789 phone (202) 555-0147 also Bob 202/555/0108"
|
||||
upstreamBody := []byte(`{"id":"x","model":"gpt-5.4","choices":[{"message":{"role":"assistant","content":"` + completion + `"}}],"usage":{"prompt_tokens":12,"completion_tokens":40,"total_tokens":52}}`)
|
||||
var upstreamHits atomic.Int64
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
upstreamHits.Add(1)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(upstreamBody)
|
||||
}))
|
||||
t.Cleanup(upstream.Close)
|
||||
upstreamHost := strings.TrimPrefix(upstream.URL, "http://")
|
||||
|
||||
// ---- 2. In-process management gRPC server (bufconn) backed by a real
|
||||
// sqlite store + real agentnetwork.Manager. The proxy's middlewares talk
|
||||
// to this client.
|
||||
st, cleanup, err := store.NewTestStoreFromSQL(ctx, "", t.TempDir())
|
||||
require.NoError(t, err, "real sqlite test store must come up")
|
||||
t.Cleanup(cleanup)
|
||||
|
||||
anMgr := agentnetwork.NewManager(st, nil, nil, nil)
|
||||
server := &mgmtgrpc.ProxyServiceServer{}
|
||||
server.SetAgentNetworkLimitsService(anMgr)
|
||||
|
||||
lis := bufconn.Listen(1024 * 1024)
|
||||
srv := grpc.NewServer()
|
||||
proto.RegisterProxyServiceServer(srv, server)
|
||||
go func() { _ = srv.Serve(lis) }()
|
||||
t.Cleanup(srv.Stop)
|
||||
|
||||
conn, err := grpc.NewClient("passthrough:///bufnet",
|
||||
grpc.WithContextDialer(func(_ context.Context, _ string) (net.Conn, error) { return lis.Dial() }),
|
||||
grpc.WithTransportCredentials(insecure.NewCredentials()),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
mgmtClient := proto.NewProxyServiceClient(conn)
|
||||
|
||||
// ---- 3. Seed account state: settings (redact + capture on), provider
|
||||
// whose upstream URL points at our fake server, policy (catch-all-allow
|
||||
// over the Admins group → window=0 path), and a generous budget rule
|
||||
// targeting Admins so the curl succeeds and we can prove the counter
|
||||
// increments on the response leg.
|
||||
require.NoError(t, st.SaveAgentNetworkSettings(ctx, &agentNetworkTypes.Settings{
|
||||
AccountID: testAccountID,
|
||||
Cluster: cluster,
|
||||
Subdomain: subdomain,
|
||||
EnablePromptCollection: true,
|
||||
EnableLogCollection: true,
|
||||
RedactPii: true,
|
||||
}))
|
||||
require.NoError(t, st.SaveAgentNetworkProvider(ctx, &agentNetworkTypes.Provider{
|
||||
ID: providerID,
|
||||
AccountID: testAccountID,
|
||||
ProviderID: "openai_api",
|
||||
Name: "openai-fullchain-test",
|
||||
UpstreamURL: upstream.URL, // router rewrites to this
|
||||
APIKey: "sk-test",
|
||||
Enabled: true,
|
||||
Models: []agentNetworkTypes.ProviderModel{{ID: "gpt-5.4"}},
|
||||
SessionPrivateKey: "priv",
|
||||
SessionPublicKey: "pub",
|
||||
}))
|
||||
require.NoError(t, st.SaveAgentNetworkPolicy(ctx, &agentNetworkTypes.Policy{
|
||||
ID: "ainpol-fullchain",
|
||||
AccountID: testAccountID,
|
||||
Name: "admins-openai",
|
||||
Enabled: true,
|
||||
SourceGroups: []string{adminGroupID},
|
||||
DestinationProviderIDs: []string{providerID},
|
||||
// No token / budget caps → effectiveWindowSeconds=0 → exercises the
|
||||
// catch-all-allow path that the GC-2 record-on-window=0 fix targets.
|
||||
}))
|
||||
require.NoError(t, st.SaveAgentNetworkBudgetRule(ctx, &agentNetworkTypes.AccountBudgetRule{
|
||||
ID: "ainbud-admins-fullchain",
|
||||
AccountID: testAccountID,
|
||||
Name: "admins-monthly",
|
||||
Enabled: true,
|
||||
TargetGroups: []string{adminGroupID},
|
||||
Limits: agentNetworkTypes.PolicyLimits{
|
||||
TokenLimit: agentNetworkTypes.PolicyTokenLimit{Enabled: true, GroupCap: 1_000_000, UserCap: 1_000_000, WindowSeconds: 60},
|
||||
},
|
||||
}))
|
||||
|
||||
// ---- 4. Synth the service. This produces the exact middleware chain
|
||||
// configuration the production reconcile path ships to the proxy.
|
||||
services, err := agentnetwork.SynthesizeServices(ctx, st, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, services, 1, "exactly one synth service expected")
|
||||
synthSvc := services[0]
|
||||
require.NotEmpty(t, synthSvc.Targets, "synth target must exist")
|
||||
|
||||
// ---- 5. Wire the middleware framework — same registry the proxy uses
|
||||
// in production, configured with our bufconn-backed management client.
|
||||
mwbuiltin.Configure(ctx, t.TempDir(), nil, testLogger, mgmtClient)
|
||||
registry := mwbuiltin.DefaultRegistry()
|
||||
mwMetrics, err := middleware.NewMetrics(nil)
|
||||
require.NoError(t, err)
|
||||
mwMgr := middleware.NewManager(0, mwMetrics, testLogger)
|
||||
mwMgr.SetResolver(middleware.NewResolver(registry))
|
||||
|
||||
// Convert the synth's rpservice.MiddlewareConfig list into proxy
|
||||
// middleware.Spec values. Mirrors the proto→Spec translation server.go
|
||||
// does at runtime; kept inline here so the test isn't coupled to the
|
||||
// proxy server's private translateMiddlewareConfig helper.
|
||||
specs := make([]middleware.Spec, 0, len(synthSvc.Targets[0].Options.Middlewares))
|
||||
for _, mw := range synthSvc.Targets[0].Options.Middlewares {
|
||||
var slot middleware.Slot
|
||||
switch mw.Slot {
|
||||
case rpservice.MiddlewareSlotOnRequest:
|
||||
slot = middleware.SlotOnRequest
|
||||
case rpservice.MiddlewareSlotOnResponse:
|
||||
slot = middleware.SlotOnResponse
|
||||
case rpservice.MiddlewareSlotTerminal:
|
||||
slot = middleware.SlotTerminal
|
||||
default:
|
||||
t.Fatalf("unknown middleware slot %q on %s", mw.Slot, mw.ID)
|
||||
}
|
||||
specs = append(specs, middleware.Spec{
|
||||
ID: mw.ID,
|
||||
Slot: slot,
|
||||
Enabled: mw.Enabled,
|
||||
FailMode: middleware.FailOpen,
|
||||
Timeout: middleware.DefaultTimeout,
|
||||
RawConfig: append([]byte(nil), mw.ConfigJSON...),
|
||||
CanMutate: mw.CanMutate,
|
||||
})
|
||||
}
|
||||
|
||||
serviceIDStr := synthSvc.ID
|
||||
require.NoError(t, mwMgr.Rebuild(serviceIDStr, []middleware.PathTargetBinding{{
|
||||
ServiceID: serviceIDStr,
|
||||
PathID: "/",
|
||||
Specs: specs,
|
||||
}}))
|
||||
|
||||
// ---- 6. Build the reverse proxy, with a mapping whose target URL goes
|
||||
// straight to the fake upstream (the router middleware rewriting upstream
|
||||
// from the synth's noop placeholder isn't needed when we own the mapping
|
||||
// in-process — point the target at the fake URL directly so the body
|
||||
// arrives at the upstream the synth would have routed to).
|
||||
upstreamURL, err := url.Parse(upstream.URL)
|
||||
require.NoError(t, err)
|
||||
|
||||
rp := proxy.NewReverseProxy(http.DefaultTransport, "auto", nil, testLogger, proxy.WithMiddlewareManager(mwMgr))
|
||||
rp.AddMapping(proxy.Mapping{
|
||||
ID: nbproxytypes.ServiceID(serviceIDStr),
|
||||
AccountID: nbproxytypes.AccountID(testAccountID),
|
||||
Host: synthSvc.Domain,
|
||||
Paths: map[string]*proxy.PathTarget{
|
||||
"/": {
|
||||
URL: upstreamURL,
|
||||
DirectUpstream: true,
|
||||
AgentNetwork: true,
|
||||
Middlewares: specs,
|
||||
CaptureConfig: &bodytap.Config{
|
||||
MaxRequestBytes: 1 << 20,
|
||||
MaxResponseBytes: 1 << 20,
|
||||
ContentTypes: []string{"application/json", "text/event-stream"},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
// ---- 7. Send a request with the auth-stamped CapturedData (mimicking
|
||||
// what the tunnel-peer auth middleware does at the edge of the proxy).
|
||||
reqBody := `{"model":"gpt-5.4","client_metadata":{"session_id":"sess-fullchain-1"},"messages":[{"role":"user","content":"contact alice.johnson@example.com SSN 987-65-4321 phone (202)555-0156"}]}`
|
||||
req := httptest.NewRequest("POST", "https://"+synthSvc.Domain+"/v1/chat/completions", strings.NewReader(reqBody))
|
||||
req.Host = synthSvc.Domain
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
cd := proxy.NewCapturedData("test-request-1")
|
||||
cd.SetServiceID(nbproxytypes.ServiceID(serviceIDStr))
|
||||
cd.SetAccountID(nbproxytypes.AccountID(testAccountID))
|
||||
cd.SetUserID(testAdminUser)
|
||||
cd.SetUserGroups([]string{adminGroupID})
|
||||
cd.SetAuthMethod("tunnel_peer")
|
||||
req = req.WithContext(proxy.WithCapturedData(req.Context(), cd))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
rp.ServeHTTP(w, req)
|
||||
|
||||
require.Equal(t, http.StatusOK, w.Code, "upstream call must succeed end-to-end; body=%s", w.Body.String())
|
||||
assert.GreaterOrEqual(t, upstreamHits.Load(), int64(1), "fake upstream must have been hit")
|
||||
|
||||
// ---- 8. Assertions — the three guarantees this test exists for.
|
||||
|
||||
// 8a. The reverseproxy.go respInput construction carried UserGroups
|
||||
// into the response-leg middleware chain, so llm_limit_record sent a
|
||||
// non-empty group_ids on RecordLLMUsage. Verifying via the management
|
||||
// store directly bypasses the manager's permission gate (which is nil
|
||||
// in this test) — we want to confirm the row landed, not who saw it.
|
||||
require.Eventually(t, func() bool {
|
||||
rows, lerr := st.ListAgentNetworkConsumption(ctx, store.LockingStrengthNone, testAccountID)
|
||||
if lerr != nil {
|
||||
return false
|
||||
}
|
||||
for _, r := range rows {
|
||||
if r.DimensionKind == agentNetworkTypes.DimensionGroup &&
|
||||
r.DimensionID == adminGroupID &&
|
||||
r.WindowSeconds == 60 &&
|
||||
r.TokensInput+r.TokensOutput > 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}, 5*time.Second, 50*time.Millisecond,
|
||||
"Admins group consumption row must increment via the response leg — if this fails the proxy's respInput dropped UserGroups again or the parser/recorder wiring is broken")
|
||||
|
||||
// 8b. Both the captured prompt and the captured completion are
|
||||
// redacted — proves the synth threads redact_pii=true into BOTH parser
|
||||
// configs and the parsers honour it at emission time.
|
||||
md := cd.GetMetadata()
|
||||
promptRaw := md["llm.request_prompt_raw"]
|
||||
completionMeta := md["llm.response_completion"]
|
||||
|
||||
// 8a-bis. The session id from client_metadata.session_id flows through
|
||||
// the request parser into the captured metadata, so the access-log /
|
||||
// usage rows can group this request with the rest of its conversation.
|
||||
assert.Equal(t, "sess-fullchain-1", md["llm.session_id"],
|
||||
"session id must be extracted from client_metadata.session_id and carried through the chain")
|
||||
|
||||
assert.NotEmpty(t, promptRaw, "llm.request_prompt_raw must be present in captured metadata")
|
||||
assert.Contains(t, promptRaw, "[REDACTED:", "captured raw prompt must carry redaction markers")
|
||||
assert.NotContains(t, promptRaw, "alice.johnson@example.com", "raw email must NOT survive in prompt_raw")
|
||||
assert.NotContains(t, promptRaw, "987-65-4321", "raw SSN must NOT survive in prompt_raw")
|
||||
assert.NotContains(t, promptRaw, "(202)555-0156", "raw paren-no-space phone must NOT survive in prompt_raw")
|
||||
|
||||
assert.NotEmpty(t, completionMeta, "llm.response_completion must be present in captured metadata")
|
||||
assert.Contains(t, completionMeta, "[REDACTED:", "captured completion must carry redaction markers")
|
||||
assert.NotContains(t, completionMeta, "alice.johnson@example.com", "raw email must NOT survive in completion")
|
||||
assert.NotContains(t, completionMeta, "123-45-6789", "raw SSN must NOT survive in completion")
|
||||
assert.NotContains(t, completionMeta, "(202) 555-0147", "raw paren+space phone must NOT survive in completion")
|
||||
assert.NotContains(t, completionMeta, "202/555/0108", "raw slash phone must NOT survive in completion")
|
||||
|
||||
_ = upstreamHost // kept for future header-inspection assertions if needed
|
||||
}
|
||||
@@ -58,9 +58,11 @@ type CapturedData struct {
|
||||
// the JWT's group_names claim or from ValidateSession/Tunnel
|
||||
// responses. Slice may be shorter than userGroups for tokens minted
|
||||
// before names were resolvable.
|
||||
userGroupNames []string
|
||||
authMethod string
|
||||
metadata map[string]string
|
||||
userGroupNames []string
|
||||
authMethod string
|
||||
metadata map[string]string
|
||||
agentNetwork bool
|
||||
suppressAccessLog bool
|
||||
}
|
||||
|
||||
// NewCapturedData creates a CapturedData with the given request ID.
|
||||
@@ -178,6 +180,41 @@ func (c *CapturedData) SetUserGroups(groups []string) {
|
||||
c.userGroups = append(c.userGroups[:0], groups...)
|
||||
}
|
||||
|
||||
// SetAgentNetwork records whether the request hit a synthesised
|
||||
// agent-network target. The terminal access-log middleware stamps the
|
||||
// flag onto the proto so management can distinguish synthetic traffic.
|
||||
func (c *CapturedData) SetAgentNetwork(b bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.agentNetwork = b
|
||||
}
|
||||
|
||||
// GetAgentNetwork reports whether the request matched a synthesised
|
||||
// agent-network target.
|
||||
func (c *CapturedData) GetAgentNetwork() bool {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return c.agentNetwork
|
||||
}
|
||||
|
||||
// SetSuppressAccessLog records whether the per-request access-log emission
|
||||
// must be skipped for this request. Stamped from the matched target's
|
||||
// DisableAccessLog flag so the access-log middleware can short-circuit
|
||||
// log delivery for opted-out agent-network targets.
|
||||
func (c *CapturedData) SetSuppressAccessLog(b bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.suppressAccessLog = b
|
||||
}
|
||||
|
||||
// GetSuppressAccessLog reports whether access-log emission has been
|
||||
// suppressed for this request.
|
||||
func (c *CapturedData) GetSuppressAccessLog() bool {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return c.suppressAccessLog
|
||||
}
|
||||
|
||||
// GetUserGroups returns a copy of the authenticated user's group
|
||||
// memberships.
|
||||
func (c *CapturedData) GetUserGroups() []string {
|
||||
|
||||
@@ -2,6 +2,7 @@ package proxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
@@ -11,10 +12,13 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
log "github.com/sirupsen/logrus"
|
||||
|
||||
"github.com/netbirdio/netbird/proxy/auth"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
|
||||
"github.com/netbirdio/netbird/proxy/internal/roundtrip"
|
||||
"github.com/netbirdio/netbird/proxy/internal/types"
|
||||
"github.com/netbirdio/netbird/proxy/web"
|
||||
@@ -32,6 +36,25 @@ type ReverseProxy struct {
|
||||
mappingsMux sync.RWMutex
|
||||
mappings map[string]Mapping
|
||||
logger *log.Logger
|
||||
// middlewareManager, when non-nil, drives per-target middleware
|
||||
// dispatch. A nil manager (or an empty chain for the resolved
|
||||
// target) keeps the reverse-proxy hot path on the no-capture fast
|
||||
// path with no middleware overhead.
|
||||
middlewareManager *middleware.Manager
|
||||
}
|
||||
|
||||
// Option configures optional ReverseProxy behavior. Options exist so the core
|
||||
// constructor signature stays stable across additive features.
|
||||
type Option func(*ReverseProxy)
|
||||
|
||||
// WithMiddlewareManager attaches a middleware manager to the reverse
|
||||
// proxy. When the manager is nil or returns an empty chain for the
|
||||
// target, the request follows the fast path with no middleware
|
||||
// overhead.
|
||||
func WithMiddlewareManager(m *middleware.Manager) Option {
|
||||
return func(p *ReverseProxy) {
|
||||
p.middlewareManager = m
|
||||
}
|
||||
}
|
||||
|
||||
// NewReverseProxy configures a new NetBird ReverseProxy.
|
||||
@@ -40,29 +63,28 @@ type ReverseProxy struct {
|
||||
// between requested URLs and targets.
|
||||
// The internal mappings can be modified using the AddMapping
|
||||
// and RemoveMapping functions.
|
||||
func NewReverseProxy(transport http.RoundTripper, forwardedProto string, trustedProxies []netip.Prefix, logger *log.Logger) *ReverseProxy {
|
||||
func NewReverseProxy(transport http.RoundTripper, forwardedProto string, trustedProxies []netip.Prefix, logger *log.Logger, opts ...Option) *ReverseProxy {
|
||||
if logger == nil {
|
||||
logger = log.StandardLogger()
|
||||
}
|
||||
return &ReverseProxy{
|
||||
p := &ReverseProxy{
|
||||
transport: transport,
|
||||
forwardedProto: forwardedProto,
|
||||
trustedProxies: trustedProxies,
|
||||
mappings: make(map[string]Mapping),
|
||||
logger: logger,
|
||||
}
|
||||
for _, opt := range opts {
|
||||
opt(p)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
result, exists := p.findTargetForRequest(r)
|
||||
if !exists {
|
||||
if cd := CapturedDataFromContext(r.Context()); cd != nil {
|
||||
cd.SetOrigin(OriginNoRoute)
|
||||
}
|
||||
requestID := getRequestID(r)
|
||||
web.ServeErrorPage(w, r, http.StatusNotFound, "Service Not Found",
|
||||
"The requested service could not be found. Please check the URL, try refreshing, or check if the peer is running. If that doesn't work, see our documentation for help.",
|
||||
requestID, web.ErrorStatus{Proxy: true, Destination: false})
|
||||
p.serveRouteError(w, r, http.StatusNotFound, "Service Not Found",
|
||||
"The requested service could not be found. Please check the URL, try refreshing, or check if the peer is running. If that doesn't work, see our documentation for help.")
|
||||
return
|
||||
}
|
||||
|
||||
@@ -72,38 +94,23 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
// with 421 (Misdirected Request) so the caller sees an explicit
|
||||
// error instead of silently doubling tunnel traffic.
|
||||
if p.isSelfTargetLoop(r, result.target.URL) {
|
||||
if cd := CapturedDataFromContext(r.Context()); cd != nil {
|
||||
cd.SetOrigin(OriginNoRoute)
|
||||
}
|
||||
requestID := getRequestID(r)
|
||||
web.ServeErrorPage(w, r, http.StatusMisdirectedRequest, "Loop Detected",
|
||||
"This peer is the target of the requested service. Reach the backend directly instead of dialing the public service URL from the same machine.",
|
||||
requestID, web.ErrorStatus{Proxy: true, Destination: false})
|
||||
p.serveRouteError(w, r, http.StatusMisdirectedRequest, "Loop Detected",
|
||||
"This peer is the target of the requested service. Reach the backend directly instead of dialing the public service URL from the same machine.")
|
||||
return
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
// Set the account ID in the context for the roundtripper to use.
|
||||
ctx = roundtrip.WithAccountID(ctx, result.accountID)
|
||||
pt := result.target
|
||||
ctx := p.buildTargetContext(r.Context(), result)
|
||||
|
||||
// Populate captured data if it exists (allows middleware to read after handler completes).
|
||||
// This solves the problem of passing data UP the middleware chain: we put a mutable struct
|
||||
// pointer in the context, and mutate the struct here so outer middleware can read it.
|
||||
if capturedData := CapturedDataFromContext(ctx); capturedData != nil {
|
||||
capturedData := CapturedDataFromContext(ctx)
|
||||
if capturedData != nil {
|
||||
capturedData.SetServiceID(result.serviceID)
|
||||
capturedData.SetAccountID(result.accountID)
|
||||
}
|
||||
|
||||
pt := result.target
|
||||
|
||||
if pt.SkipTLSVerify {
|
||||
ctx = roundtrip.WithSkipTLSVerify(ctx)
|
||||
}
|
||||
if pt.RequestTimeout > 0 {
|
||||
ctx = types.WithDialTimeout(ctx, pt.RequestTimeout)
|
||||
}
|
||||
if pt.DirectUpstream {
|
||||
ctx = roundtrip.WithDirectUpstream(ctx)
|
||||
capturedData.SetAgentNetwork(result.target != nil && result.target.AgentNetwork)
|
||||
capturedData.SetSuppressAccessLog(result.target != nil && result.target.DisableAccessLog)
|
||||
}
|
||||
|
||||
rewriteMatchedPath := result.matchedPath
|
||||
@@ -111,6 +118,45 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
rewriteMatchedPath = ""
|
||||
}
|
||||
|
||||
chain := p.resolveChain(result)
|
||||
if chain == nil || chain.Empty() {
|
||||
p.serveDirect(w, r, ctx, result, rewriteMatchedPath)
|
||||
return
|
||||
}
|
||||
p.serveWithChain(w, r, ctx, result, chain, rewriteMatchedPath, capturedData)
|
||||
}
|
||||
|
||||
// serveRouteError marks the request as un-routed on any captured-data
|
||||
// context and renders the proxy error page.
|
||||
func (p *ReverseProxy) serveRouteError(w http.ResponseWriter, r *http.Request, status int, title, message string) {
|
||||
if cd := CapturedDataFromContext(r.Context()); cd != nil {
|
||||
cd.SetOrigin(OriginNoRoute)
|
||||
}
|
||||
web.ServeErrorPage(w, r, status, title, message, getRequestID(r),
|
||||
web.ErrorStatus{Proxy: true, Destination: false})
|
||||
}
|
||||
|
||||
// buildTargetContext layers the per-target roundtrip flags (account id,
|
||||
// TLS-verify skip, direct upstream, dial timeout) onto the request context.
|
||||
func (p *ReverseProxy) buildTargetContext(ctx context.Context, result targetResult) context.Context {
|
||||
pt := result.target
|
||||
ctx = roundtrip.WithAccountID(ctx, result.accountID)
|
||||
if pt.SkipTLSVerify {
|
||||
ctx = roundtrip.WithSkipTLSVerify(ctx)
|
||||
}
|
||||
if pt.DirectUpstream {
|
||||
ctx = roundtrip.WithDirectUpstream(ctx)
|
||||
}
|
||||
if pt.RequestTimeout > 0 {
|
||||
ctx = types.WithDialTimeout(ctx, pt.RequestTimeout)
|
||||
}
|
||||
return ctx
|
||||
}
|
||||
|
||||
// serveDirect forwards the request without a middleware chain — the common
|
||||
// path for plain reverse-proxy targets.
|
||||
func (p *ReverseProxy) serveDirect(w http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, rewriteMatchedPath string) {
|
||||
pt := result.target
|
||||
rp := &httputil.ReverseProxy{
|
||||
Rewrite: p.rewriteFunc(pt.URL, rewriteMatchedPath, result.passHostHeader, pt.PathRewrite, pt.CustomHeaders, result.stripAuthHeaders),
|
||||
Transport: p.transport,
|
||||
@@ -123,6 +169,344 @@ func (p *ReverseProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
rp.ServeHTTP(w, r.WithContext(ctx))
|
||||
}
|
||||
|
||||
// serveWithChain runs the per-target middleware chain around the upstream
|
||||
// request: request-leg capture and authorisation, then (on allow) the
|
||||
// upstream forward with response/terminal observation deferred so it reads
|
||||
// the captured response before the writer is released.
|
||||
func (p *ReverseProxy) serveWithChain(w http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, chain *middleware.Chain, rewriteMatchedPath string, capturedData *CapturedData) {
|
||||
middlewareIDs := chain.IDs()
|
||||
p.logger.Debugf("middleware chain matched: service=%s path=%s middlewares=%v", result.serviceID, result.matchedPath, middlewareIDs)
|
||||
|
||||
capturedBody, truncated, originalSize, releaseBudget := p.captureRequestForChain(ctx, r, result, capturedData)
|
||||
defer releaseBudget()
|
||||
|
||||
acc := middleware.NewAccumulator(middleware.MaxRequestMetadataBytes)
|
||||
reqInput := buildRequestInput(r, result, capturedData, capturedBody, truncated, originalSize)
|
||||
|
||||
denyOutput, requestMeta, upstreamRewrite, _ := chain.RunRequest(ctx, r, reqInput, acc)
|
||||
if capturedData != nil {
|
||||
for _, kv := range requestMeta {
|
||||
capturedData.SetMetadata(kv.Key, kv.Value)
|
||||
}
|
||||
}
|
||||
if denyOutput != nil {
|
||||
p.serveDeny(w, denyOutput, result, middlewareIDs)
|
||||
return
|
||||
}
|
||||
|
||||
respWriter, capturingWriter := p.newResponseWriter(ctx, w, result, capturedData)
|
||||
if capturingWriter != nil {
|
||||
defer capturingWriter.Release()
|
||||
defer p.observeResponse(ctx, chain, acc, reqInput, requestMeta, capturingWriter, w, capturedData, result, middlewareIDs)
|
||||
}
|
||||
|
||||
p.forwardUpstream(respWriter, r, ctx, result, rewriteMatchedPath, upstreamRewrite)
|
||||
}
|
||||
|
||||
// captureRequestForChain copies the request body for inspection by the
|
||||
// chain, records any capture bypass, and applies agent-network routing
|
||||
// recovery for oversized bodies. The returned release frees the capture
|
||||
// budget and must be deferred by the caller.
|
||||
func (p *ReverseProxy) captureRequestForChain(ctx context.Context, r *http.Request, result targetResult, capturedData *CapturedData) ([]byte, bool, int64, func()) {
|
||||
pt := result.target
|
||||
capturedBody, truncated, originalSize, bypass, releaseBudget, captureErr := bodytap.CaptureRequest(r, pt.CaptureConfig, p.middlewareManager.Budget())
|
||||
if captureErr != nil {
|
||||
p.logger.Debugf("middleware request body capture error: %v", captureErr)
|
||||
}
|
||||
if bypass != "" {
|
||||
if capturedData != nil {
|
||||
capturedData.SetMetadata("mw.capture.bypass_reason", bypass)
|
||||
}
|
||||
p.middlewareManager.Metrics().IncCaptureBypass(ctx, string(result.serviceID), bypass)
|
||||
}
|
||||
|
||||
// Routing recovery for oversized agent-network requests: when the body
|
||||
// exceeded the capture cap (bypassed or truncated), the captured copy
|
||||
// can't be parsed for the model, so llm_router would deny with
|
||||
// model_not_routable. Scan the full stream for just the routing fields
|
||||
// and hand the request parser a minimal stub so routing succeeds; the
|
||||
// prompt stays uncaptured and the upstream still gets the full body.
|
||||
if pt.AgentNetwork && (truncated || capturedBody == nil) {
|
||||
if model, stream, ok := bodytap.ScanRoutingFields(r, bodytap.MaxRoutingScanBytes); ok {
|
||||
capturedBody = buildRoutingStub(model, stream)
|
||||
truncated = false
|
||||
p.logger.Debugf("agent-network routing recovery: extracted model=%s stream=%t from oversized request body (service=%s)", model, stream, result.serviceID)
|
||||
}
|
||||
}
|
||||
return capturedBody, truncated, originalSize, releaseBudget
|
||||
}
|
||||
|
||||
// serveDeny renders the chain's deny response. Policy/budget/routing/guardrail
|
||||
// denials are expected runtime outcomes and can be high-volume under
|
||||
// misconfigured or hostile clients; per-request detail stays at Debug and
|
||||
// metrics/access logs carry the signal at scale.
|
||||
func (p *ReverseProxy) serveDeny(w http.ResponseWriter, denyOutput *middleware.Output, result targetResult, middlewareIDs []string) {
|
||||
middlewareID := "middleware"
|
||||
if denyOutput.DenyReason != nil && denyOutput.DenyReason.Code != "" {
|
||||
middlewareID = denyOutput.DenyReason.Code
|
||||
}
|
||||
p.logger.Debugf("middleware chain denied request: service=%s path=%s middlewares=%v reason=%s status=%d",
|
||||
result.serviceID, result.matchedPath, middlewareIDs, middlewareID, denyOutput.DenyStatus)
|
||||
middleware.RenderDenyResponse(w, middlewareID, denyOutput.DenyReason, denyOutput.DenyStatus)
|
||||
}
|
||||
|
||||
// newResponseWriter returns the writer the upstream forward should use. When
|
||||
// response capture is enabled and not bypassed it wraps w in a capturing
|
||||
// writer (also returned so the caller can release it and feed the response
|
||||
// leg); otherwise the capturing writer is nil and w is used directly.
|
||||
func (p *ReverseProxy) newResponseWriter(ctx context.Context, w http.ResponseWriter, result targetResult, capturedData *CapturedData) (http.ResponseWriter, *bodytap.CapturingResponseWriter) {
|
||||
pt := result.target
|
||||
if pt.CaptureConfig == nil || pt.CaptureConfig.MaxResponseBytes <= 0 {
|
||||
return w, nil
|
||||
}
|
||||
capturingWriter := bodytap.NewCapturingResponseWriter(w, pt.CaptureConfig.MaxResponseBytes, p.middlewareManager.Budget())
|
||||
if capturingWriter.Bypassed() {
|
||||
if capturedData != nil {
|
||||
capturedData.SetMetadata("mw.capture.bypass_reason", capturingWriter.BypassReason())
|
||||
}
|
||||
p.middlewareManager.Metrics().IncCaptureBypass(ctx, string(result.serviceID), capturingWriter.BypassReason())
|
||||
capturingWriter.Release()
|
||||
return w, nil
|
||||
}
|
||||
return capturingWriter, capturingWriter
|
||||
}
|
||||
|
||||
// observeResponse runs the response and terminal middleware slots after the
|
||||
// body has been forwarded. It is deferred by serveWithChain so it reads the
|
||||
// captured response before the writer is released.
|
||||
func (p *ReverseProxy) observeResponse(ctx context.Context, chain *middleware.Chain, acc *middleware.Accumulator, reqInput *middleware.Input, requestMeta []middleware.KV, capturingWriter *bodytap.CapturingResponseWriter, w http.ResponseWriter, capturedData *CapturedData, result targetResult, middlewareIDs []string) {
|
||||
respInput := &middleware.Input{
|
||||
Slot: middleware.SlotOnResponse,
|
||||
RequestID: reqInput.RequestID,
|
||||
TargetID: reqInput.TargetID,
|
||||
Method: reqInput.Method,
|
||||
URL: reqInput.URL,
|
||||
Headers: reqInput.Headers,
|
||||
Status: capturingWriter.Status(),
|
||||
RespHeaders: headerToKV(w.Header()),
|
||||
RespBody: capturingWriter.Body(),
|
||||
RespBodyTruncated: capturingWriter.Truncated(),
|
||||
OriginalRespSize: capturingWriter.BytesWritten(),
|
||||
ServiceID: reqInput.ServiceID,
|
||||
AccountID: reqInput.AccountID,
|
||||
UserID: reqInput.UserID,
|
||||
// UserEmail / UserGroups / UserGroupNames must flow into the
|
||||
// response leg too — llm_limit_record needs UserGroups to send
|
||||
// group_ids on RecordLLMUsage so management's account-budget
|
||||
// fan-out can match group-targeted rules; identity-stamping and
|
||||
// any future response-side authorisation also depend on these.
|
||||
UserEmail: reqInput.UserEmail,
|
||||
UserGroups: reqInput.UserGroups,
|
||||
UserGroupNames: reqInput.UserGroupNames,
|
||||
AuthMethod: reqInput.AuthMethod,
|
||||
SourceIP: reqInput.SourceIP,
|
||||
Metadata: requestMeta,
|
||||
AgentNetwork: reqInput.AgentNetwork,
|
||||
}
|
||||
// The response/terminal phase runs after the body is forwarded, so
|
||||
// a streaming client (e.g. Codex) has usually disconnected by now,
|
||||
// cancelling r.Context(). These middlewares only observe and record
|
||||
// (token/cost metering, usage recording) and must still complete —
|
||||
// otherwise the dispatcher short-circuits each to fail-mode and the
|
||||
// usage is silently lost. Detach from client cancellation, keep ctx
|
||||
// values, and bound the work.
|
||||
obsCtx, obsCancel := context.WithTimeout(context.WithoutCancel(ctx), observabilityPhaseTimeout)
|
||||
defer obsCancel()
|
||||
|
||||
respMeta := chain.RunResponse(obsCtx, respInput, acc)
|
||||
if capturedData != nil {
|
||||
for _, kv := range respMeta {
|
||||
capturedData.SetMetadata(kv.Key, kv.Value)
|
||||
}
|
||||
}
|
||||
|
||||
// Terminal slot sees the merged metadata bag from request and
|
||||
// response phases.
|
||||
mergedMeta := append(append([]middleware.KV(nil), requestMeta...), respMeta...)
|
||||
termInput := *respInput
|
||||
termInput.Slot = middleware.SlotTerminal
|
||||
termInput.Metadata = mergedMeta
|
||||
termMeta := chain.RunTerminal(obsCtx, &termInput, acc)
|
||||
if capturedData != nil {
|
||||
for _, kv := range termMeta {
|
||||
capturedData.SetMetadata(kv.Key, kv.Value)
|
||||
}
|
||||
}
|
||||
|
||||
p.logger.Debugf("middleware chain ran: service=%s path=%s middlewares=%v status=%d req_meta=%d resp_meta=%d term_meta=%d",
|
||||
result.serviceID, result.matchedPath, middlewareIDs, capturingWriter.Status(), len(requestMeta), len(respMeta), len(termMeta))
|
||||
}
|
||||
|
||||
// forwardUpstream applies any middleware-emitted upstream rewrite and proxies
|
||||
// the request to the effective upstream URL.
|
||||
func (p *ReverseProxy) forwardUpstream(respWriter http.ResponseWriter, r *http.Request, ctx context.Context, result targetResult, rewriteMatchedPath string, upstreamRewrite *middleware.UpstreamRewrite) {
|
||||
pt := result.target
|
||||
effectiveURL := applyUpstreamRewrite(pt.URL, upstreamRewrite)
|
||||
if upstreamRewrite != nil {
|
||||
r.Host = effectiveURL.Host
|
||||
applyUpstreamHeaders(r, upstreamRewrite)
|
||||
stripUpstreamPathPrefix(r, upstreamRewrite.StripPathPrefix)
|
||||
}
|
||||
|
||||
rp := &httputil.ReverseProxy{
|
||||
Rewrite: p.rewriteFunc(effectiveURL, rewriteMatchedPath, result.passHostHeader, pt.PathRewrite, pt.CustomHeaders, result.stripAuthHeaders),
|
||||
Transport: p.transport,
|
||||
FlushInterval: -1,
|
||||
ErrorHandler: p.proxyErrorHandler,
|
||||
}
|
||||
if result.rewriteRedirects {
|
||||
rp.ModifyResponse = p.rewriteLocationFunc(effectiveURL, rewriteMatchedPath, r) //nolint:bodyclose
|
||||
}
|
||||
rp.ServeHTTP(respWriter, r.WithContext(ctx))
|
||||
}
|
||||
|
||||
// buildRoutingStub returns a minimal JSON request body carrying only the
|
||||
// model and stream fields. It feeds the LLM request parser when the real
|
||||
// body was too large to capture: the parser emits llm.model / llm.stream
|
||||
// so llm_router can route, while ExtractPrompt on the stub yields nothing
|
||||
// — no prompt is captured for oversized requests.
|
||||
func buildRoutingStub(model string, stream bool) []byte {
|
||||
b, err := json.Marshal(map[string]any{"model": model, "stream": stream})
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// applyUpstreamRewrite returns the effective upstream URL after
|
||||
// applying a middleware-emitted rewrite. When rewrite is nil or
|
||||
// incomplete, the original target is returned unchanged. The original
|
||||
// URL is never mutated; a clone is returned when a rewrite applies.
|
||||
//
|
||||
// Rewrite Path semantics: when non-empty, replaces the cloned URL's
|
||||
// path entirely. httputil.ProxyRequest.SetURL then joins target.Path
|
||||
// with the agent's request path, so an operator-configured upstream
|
||||
// path like "/v1/{account}/{gateway}/compat" gets prepended to
|
||||
// "/chat/completions" yielding the full Cloudflare-shaped path.
|
||||
// Empty rewrite.Path preserves the original target's path (the
|
||||
// historical, non-agent-network behavior).
|
||||
func applyUpstreamRewrite(orig *url.URL, rewrite *middleware.UpstreamRewrite) *url.URL {
|
||||
if rewrite == nil || orig == nil {
|
||||
return orig
|
||||
}
|
||||
if rewrite.Scheme == "" || rewrite.Host == "" {
|
||||
return orig
|
||||
}
|
||||
cloned := *orig
|
||||
cloned.Scheme = rewrite.Scheme
|
||||
cloned.Host = rewrite.Host
|
||||
if rewrite.Path != "" {
|
||||
cloned.Path = rewrite.Path
|
||||
cloned.RawPath = ""
|
||||
}
|
||||
return &cloned
|
||||
}
|
||||
|
||||
// stripUpstreamPathPrefix removes a gateway-namespace prefix (e.g. "/bedrock")
|
||||
// from the request path before it is forwarded, so the upstream receives its
|
||||
// native path. The chain has already run by this point, so metering/logging
|
||||
// keep the original client path; only the outbound path is rewritten. RawPath
|
||||
// is cleared so the escaped form is recomputed from the trimmed Path.
|
||||
func stripUpstreamPathPrefix(r *http.Request, prefix string) {
|
||||
if r == nil || r.URL == nil || prefix == "" {
|
||||
return
|
||||
}
|
||||
if !strings.HasPrefix(r.URL.Path, prefix+"/") && r.URL.Path != prefix {
|
||||
return
|
||||
}
|
||||
r.URL.Path = strings.TrimPrefix(r.URL.Path, prefix)
|
||||
if r.URL.Path == "" {
|
||||
r.URL.Path = "/"
|
||||
}
|
||||
r.URL.RawPath = ""
|
||||
}
|
||||
|
||||
// applyUpstreamHeaders strips the headers the rewrite asks for and
|
||||
// injects the resolved auth header on the in-flight request. It is
|
||||
// the proxy-trusted counterpart to chain.applyMutations: regular
|
||||
// middleware HeadersAdd/HeadersRemove pass through the framework
|
||||
// denylist (which blocks Authorization, Cookie, etc.), but the
|
||||
// router middleware needs to replace Authorization on the upstream
|
||||
// request as a first-class operation. AuthHeader/StripHeaders ride
|
||||
// on UpstreamRewrite so only the proxy's upstream-build path
|
||||
// unpacks them — middlewares can't smuggle these in via the
|
||||
// regular mutation surface.
|
||||
func applyUpstreamHeaders(r *http.Request, rewrite *middleware.UpstreamRewrite) {
|
||||
if r == nil || rewrite == nil {
|
||||
return
|
||||
}
|
||||
for _, name := range rewrite.StripHeaders {
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
r.Header.Del(name)
|
||||
}
|
||||
if rewrite.AuthHeader != nil && rewrite.AuthHeader.Name != "" {
|
||||
r.Header.Set(rewrite.AuthHeader.Name, rewrite.AuthHeader.Value)
|
||||
}
|
||||
}
|
||||
|
||||
// resolveChain returns the middleware chain registered for the
|
||||
// resolved target, or nil when middleware is disabled for the proxy
|
||||
// or the target.
|
||||
func (p *ReverseProxy) resolveChain(result targetResult) *middleware.Chain {
|
||||
if p.middlewareManager == nil {
|
||||
return nil
|
||||
}
|
||||
return p.middlewareManager.ChainFor(string(result.serviceID), result.matchedPath)
|
||||
}
|
||||
|
||||
// buildRequestInput gathers the per-request fields the middleware
|
||||
// chain needs. Body and captured metadata are passed in; the rest are
|
||||
// copied from the request and CapturedData.
|
||||
func buildRequestInput(r *http.Request, result targetResult, cd *CapturedData, body []byte, truncated bool, originalSize int64) *middleware.Input {
|
||||
in := &middleware.Input{
|
||||
Slot: middleware.SlotOnRequest,
|
||||
TargetID: result.matchedPath,
|
||||
Method: r.Method,
|
||||
URL: r.URL.String(),
|
||||
Headers: headerToKV(r.Header),
|
||||
Body: body,
|
||||
BodyTruncated: truncated,
|
||||
OriginalBodySize: originalSize,
|
||||
ServiceID: string(result.serviceID),
|
||||
AccountID: string(result.accountID),
|
||||
AgentNetwork: result.target != nil && result.target.AgentNetwork,
|
||||
}
|
||||
if cd != nil {
|
||||
in.RequestID = cd.GetRequestID()
|
||||
in.UserID = cd.GetUserID()
|
||||
in.UserEmail = cd.GetUserEmail()
|
||||
in.UserGroups = cd.GetUserGroups()
|
||||
in.UserGroupNames = cd.GetUserGroupNames()
|
||||
in.AuthMethod = cd.GetAuthMethod()
|
||||
if ip := cd.GetClientIP(); ip.IsValid() {
|
||||
in.SourceIP = ip.String()
|
||||
}
|
||||
}
|
||||
return in
|
||||
}
|
||||
|
||||
// headerToKV flattens an http.Header into the KV slice shape expected
|
||||
// by the middleware envelope, preserving value order under the same
|
||||
// key.
|
||||
func headerToKV(h http.Header) []middleware.KV {
|
||||
if len(h) == 0 {
|
||||
return nil
|
||||
}
|
||||
total := 0
|
||||
for _, v := range h {
|
||||
total += len(v)
|
||||
}
|
||||
out := make([]middleware.KV, 0, total)
|
||||
for k, vs := range h {
|
||||
for _, v := range vs {
|
||||
out = append(out, middleware.KV{Key: k, Value: v})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// isSelfTargetLoop reports whether an overlay-origin request is about to
|
||||
// be forwarded back to the very peer that initiated it. The detection
|
||||
// is intentionally narrow: it only fires when the request arrived on
|
||||
@@ -486,6 +870,14 @@ const (
|
||||
// comma or any non-printable byte are dropped at stamp time so the
|
||||
// list is unambiguously splittable by consumers.
|
||||
headerNetBirdGroups = "X-NetBird-Groups"
|
||||
|
||||
// observabilityPhaseTimeout bounds the detached response/terminal
|
||||
// metering phase. It runs after the client connection (and its context)
|
||||
// may be gone, so it can't borrow the request deadline; this ceiling
|
||||
// keeps a slow management round-trip (RecordLLMUsage) from pinning the
|
||||
// handler goroutine indefinitely while still allowing each middleware
|
||||
// its own per-invoke timeout.
|
||||
observabilityPhaseTimeout = 30 * time.Second
|
||||
)
|
||||
|
||||
// isHeaderValueSafe reports whether v is a valid RFC 7230 field-value:
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/netbirdio/netbird/proxy/auth"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware"
|
||||
"github.com/netbirdio/netbird/proxy/internal/roundtrip"
|
||||
"github.com/netbirdio/netbird/proxy/internal/types"
|
||||
"github.com/netbirdio/netbird/proxy/web"
|
||||
@@ -1407,3 +1408,45 @@ func TestStampNetBirdIdentity_CapturedDataPresentButEmpty(t *testing.T) {
|
||||
assert.Empty(t, pr.Out.Header.Get(headerNetBirdGroups),
|
||||
"X-NetBird-Groups must be stripped when CapturedData has no groups")
|
||||
}
|
||||
|
||||
// TestBuildRequestInput_PropagatesIdentityAndGroups locks the final wiring link
|
||||
// between auth and the middleware chain: CapturedData identity (user, groups,
|
||||
// auth method, client IP) and the target's AgentNetwork flag must land on the
|
||||
// middleware Input the chain runs against. If UserGroups stops flowing here,
|
||||
// llm_router denies every request with no_authorised_provider.
|
||||
func TestBuildRequestInput_PropagatesIdentityAndGroups(t *testing.T) {
|
||||
cd := NewCapturedData("req-123")
|
||||
cd.SetUserID("user-1")
|
||||
cd.SetUserEmail("user@example.com")
|
||||
cd.SetUserGroups([]string{"grp-admins", "grp-users"})
|
||||
cd.SetUserGroupNames([]string{"Admins", "Users"})
|
||||
cd.SetAuthMethod("oidc")
|
||||
cd.SetClientIP(netip.MustParseAddr("100.90.1.14"))
|
||||
|
||||
r := httptest.NewRequest(http.MethodPost, "http://agent.example.com/v1/chat/completions", nil)
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
|
||||
result := targetResult{
|
||||
target: &PathTarget{AgentNetwork: true},
|
||||
matchedPath: "/",
|
||||
serviceID: types.ServiceID("svc-1"),
|
||||
accountID: types.AccountID("acct-1"),
|
||||
}
|
||||
|
||||
body := []byte(`{"model":"gpt-5.4"}`)
|
||||
in := buildRequestInput(r, result, cd, body, false, int64(len(body)))
|
||||
|
||||
require.NotNil(t, in, "buildRequestInput must return an envelope")
|
||||
assert.Equal(t, middleware.SlotOnRequest, in.Slot, "request input runs in the on-request slot")
|
||||
assert.Equal(t, "svc-1", in.ServiceID, "service id must propagate")
|
||||
assert.Equal(t, "acct-1", in.AccountID, "account id must propagate")
|
||||
assert.Equal(t, "user-1", in.UserID, "user id must propagate")
|
||||
assert.Equal(t, "user@example.com", in.UserEmail, "user email must propagate")
|
||||
assert.Equal(t, []string{"grp-admins", "grp-users"}, in.UserGroups,
|
||||
"CapturedData groups MUST reach the middleware Input — llm_router authorises against this")
|
||||
assert.Equal(t, []string{"Admins", "Users"}, in.UserGroupNames, "group names must propagate")
|
||||
assert.Equal(t, "oidc", in.AuthMethod, "auth method must propagate")
|
||||
assert.Equal(t, "100.90.1.14", in.SourceIP, "client IP must propagate")
|
||||
assert.True(t, in.AgentNetwork, "agent-network target flag must reach the Input")
|
||||
assert.Equal(t, body, in.Body, "captured body must reach the Input")
|
||||
}
|
||||
|
||||
@@ -8,6 +8,8 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware"
|
||||
"github.com/netbirdio/netbird/proxy/internal/middleware/bodytap"
|
||||
"github.com/netbirdio/netbird/proxy/internal/types"
|
||||
)
|
||||
|
||||
@@ -32,6 +34,20 @@ type PathTarget struct {
|
||||
// over the embedded NetBird WireGuard client when forwarding requests
|
||||
// to this target. Default false → embedded client (existing behaviour).
|
||||
DirectUpstream bool
|
||||
// Middlewares is the validated per-target middleware chain. Nil or empty
|
||||
// for non-agent-network targets, keeping them on the no-middleware fast path.
|
||||
Middlewares []middleware.Spec
|
||||
// CaptureConfig holds the per-target body-capture limits used by the
|
||||
// middleware chain. Nil for targets without body-inspecting middlewares.
|
||||
CaptureConfig *bodytap.Config
|
||||
// AgentNetwork marks this target as a synthesised agent-network target so
|
||||
// the proxy can tag access-log entries and gate agent-network behaviour.
|
||||
AgentNetwork bool
|
||||
// DisableAccessLog suppresses the per-request access-log emission for this
|
||||
// target. Defaults false so non-agent-network targets continue to log
|
||||
// unchanged. The agent-network synthesizer sets this true only when the
|
||||
// account's EnableLogCollection toggle is off.
|
||||
DisableAccessLog bool
|
||||
}
|
||||
|
||||
// Mapping describes how a domain is routed by the HTTP reverse proxy.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package proxy
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestStripUpstreamPathPrefix(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
path string
|
||||
prefix string
|
||||
want string
|
||||
}{
|
||||
{"strips matching namespace prefix", "/bedrock/model/x/invoke", "/bedrock", "/model/x/invoke"},
|
||||
{"no-op when prefix absent", "/model/x/invoke", "/bedrock", "/model/x/invoke"},
|
||||
{"no-op on empty prefix", "/bedrock/model/x/invoke", "", "/bedrock/model/x/invoke"},
|
||||
{"no-op on non-segment match", "/bedrockfoo/model/x", "/bedrock", "/bedrockfoo/model/x"},
|
||||
{"bare prefix collapses to root", "/bedrock", "/bedrock", "/"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := httptest.NewRequest("POST", tc.path, nil)
|
||||
stripUpstreamPathPrefix(r, tc.prefix)
|
||||
assert.Equal(t, tc.want, r.URL.Path, "stripped path for %q", tc.path)
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user