Each matrix request now sends a unique session id as the universal
x-session-id header and asserts it round-trips into that provider's
access-log row. This guards the session-grouping contract end to end for
every provider (header extraction runs in llm_request_parser ahead of the
parser-specific body extraction, so it is provider-agnostic).
- TestProviderLifecycle now runs per available real provider (create → get →
list → delete → 404) instead of a single dummy provider, exercising each
catalog's create and field round-trip. Create is offline, so it stays fast
and burns no provider quota; falls back to a synthetic OpenAI provider when
no keys are set.
- TestProvidersMatrix attaches a token limit (high caps, 60s window) to its
policy, which switches on usage metering, and asserts consumption rows are
recorded with positive token counts after the live traffic. Consumption is
account-scoped (keyed by source group / user and window, not per provider),
so the assertion is aggregate.
- TestProviderValidation gains invalid-upstream and blank-name cases. Create
validation is uniform across catalogs (no per-provider required-field rules),
so per-provider rejection cases would be redundant.
Vertex (Anthropic-on-Vertex) doesn't share the chat/messages wire shapes:
the model travels in a rawPredict path and the proxy mints the service
account's OAuth token. Add a Vertex client method that posts
/v1/projects/<project>/locations/<region>/publishers/anthropic/models/<model>:rawPredict
with the Vertex anthropic_version body, and wire it into the matrix as a
path-routed provider (created without a models array). It is keyed off
GOOGLE_VERTEX_SA_BASE64 + GOOGLE_VERTEX_PROJECT (region defaults to
"global", model to a pinned claude snapshot, both overridable).
Also bump the e2e workflow runner to ubuntu-latest and add the Vertex
secrets.
Replace the single-provider chat test with a data-driven matrix that runs
the same scenario through every provider whose credentials are present in
the environment (keys/URLs sourced from ~/.llm-keys locally, Actions
secrets in CI):
- OpenAI (chat), Anthropic (messages), Vercel, OpenRouter, Cloudflare
(OpenAI-compatible gateways), and Bedrock (path-routed, bearer, via the
messages shape) — covering both wire shapes and the gateway routing.
- all providers are created enabled with a unique model string so the
proxy's connect-time snapshot carries them all and model->provider
routing is unambiguous (provider toggles after connect don't reconcile
to a connected proxy).
- the client supports both wire shapes (/v1/chat/completions and
/v1/messages); Cloudflare gets the openai provider segment appended to
its gateway URL.
Each provider must return 200 through the tunnel and produce an ingested
access-log row. Vertex is intentionally excluded from the uniform matrix:
it needs a bespoke rawPredict request shape rather than the shared
chat/messages path, so it warrants a dedicated scenario.
Stand up the full agent-network data path in containers and drive a real
chat-completion through the gateway:
- harness: a shared docker network (combined server reachable by alias),
a proxy container built from the published reverse-proxy image
(NB_PROXY_PRIVATE, NB_PROXY_ALLOW_INSECURE, NB_RELAY_TRANSPORT=ws to match
the combined server's WS-multiplexed relay) with a generated self-signed
wildcard cert, and a netbird client container that joins via a setup key.
- the combined image, proxy image, and client image default to the
published rc.2 releases (overridable via NB_E2E_*_IMAGE; a bare local tag
is built from source instead). Geolocation download is disabled so the
server starts without external fetches.
- one shared domain is used for the management exposed address, the proxy
domain, and the agent-network cluster; the proxy token is minted via the
server CLI (global) to match the manual install.
TestChatCompletionThroughProxy provisions provider+policy+group+setup key,
runs proxy+client, drives an OpenAI chat-completion through the tunnel, and
asserts a 200 plus the ingested access-log row. Requires OPENAI_TOKEN
(skips otherwise). The provider must be created with enabled=true explicitly
— the create default is false despite the API doc.
Port the API-driven agent-network scenarios from the bash suites to Go,
sharing one combined server per package run (TestMain) with each test
owning its resource cleanup. Drives the /api/agent-network/* endpoints
through the shared REST client's NewRequest primitive with the generated
api types.
Scenarios:
- provider lifecycle (create/get/list/delete + 404 after delete)
- provider validation (missing api_key, unknown catalog id → 4xx)
- settings collection-toggle round-trip with cluster/subdomain immutability
- policy window floor (reject <60s enabled limit, accept at 60s)
- consumption read endpoint returns an array
All deterministic and dependency-free (dummy provider keys; no upstream
calls), so they run headless in CI.
Introduce a self-contained, OIDC-free e2e harness that stands up NetBird
in containers, so suites no longer depend on the hand-maintained Tilt
stack or a real IdP.
- harness brings up the combined server (management + signal + relay +
STUN + embedded IdP) in a single container built from
combined/Dockerfile.multistage, and mints an admin PAT through the
unauthenticated /api/setup bootstrap (NB_SETUP_PAT_ENABLED). API access
goes through the existing shared/management/client/rest typed client.
- the image is built via the docker CLI (BuildKit) so the Dockerfile's
cache mounts are honored; testcontainers then runs the tagged image.
- everything is behind the `e2e` build tag so normal builds and unit
tests never pull in testcontainers.
Adds BuildKit cache mounts to combined/Dockerfile.multistage so source
changes recompile incrementally rather than from scratch.
Pillar 1 proven by TestCombinedBootstrap: server builds, boots, mints a
PAT, and the PAT authenticates a real management API call.
