mirror of
https://github.com/netbirdio/netbird.git
synced 2026-07-16 19:49:56 +00:00
* [management,proxy] Agent network: per-account LLM gateway (policy, metering, multi-provider) (#6555) * [agent-network] Shared proto, OpenAPI schema, and generated types * [agent-network] Management: store, manager, synthesizer, policy engine, provider catalog, HTTP/gRPC API Adds the account-scoped agent-network module: provider/policy/budget CRUD and store, the reverse-proxy service synthesizer, policy selection + limit enforcement, the provider catalog (incl. Vertex AI and AWS Bedrock entries), and the management HTTP + proxy gRPC surfaces. * [management] Fix agent-network proxy-peer fan-out on affected-peer recompute The affected-peers resolver loaded only persisted reverse-proxy services, but agent-network services are synthesized on demand and never persisted. As a result the embedded proxy peer was never folded into the affected set when a client's group changed, so the proxy received no network-map update for a newly authorised client and rejected its handshake until a full resync (restart). loadProxyServices now merges the synthesized agent-network services (injected via a registration hook to avoid an import cycle), so proxy peers learn newly authorised clients immediately. * [proxy] Reverse-proxy middleware framework, chain, and request plumbing The per-target middleware chain (slots, dispatcher, mutation gate, metadata merger), body capture, access-log terminal sink, and the proxy wiring that builds + runs chains for synthesized agent-network services. * [proxy] LLM parsers, pricing, and builtin middlewares (OpenAI, Anthropic, Vertex AI, AWS Bedrock) Request/response parsers and SSE/event-stream metering, the embedded pricing table, and the builtin middleware set: request parser, router, policy limit-check/record, cost meter, guardrail, identity inject, response parser. Includes the path-routed providers — Google Vertex AI (keyfile:: service-account OAuth minting) and AWS Bedrock (bearer auth, invoke/converse/streaming, optional /bedrock prefix) — plus the Models allowlist and unmeterable-publisher deny. * [proxy] IPv6 in-place apply and TCP accept-loop hardening on netstack listeners * [agent-network] End-to-end test suite, module docs, and deployment preset * [agent-network] Fix codespell typos and exclude false positives - labelgen word pool: vermillion -> vermilion, racoon -> raccoon. - codespell ignore list: add flate (Go compress/flate package), recordin (a test-local identifier), and unparseable (a valid alternative spelling used consistently across identifiers + a metadata-value constant). * [management] Set LastSeen on injected proxy peer in realstack test (MySQL strict-mode) The injected embedded proxy peer had a PeerStatus with a zero LastSeen, which serializes to '0000-00-00' and is rejected by MySQL in strict mode (SQLite tolerates it). Set LastSeen to a valid time so SaveAccount succeeds on both engines. * [agent-network] Remove e2e shell-script suite from this branch The end-to-end shell scripts under scripts/e2e/ are maintained in a separate testing suite and are not part of this change set. * [agent-network] Polish module docs: remove internal review scaffolding, fix links, verify diagrams Strip PR-review framing, commit references, absolute paths, and stale internal references from the agent-network module docs; fix broken relative links; verify all diagrams against the current architecture. Remove the internal AI-reviewer prompt file. * [management] Refine session expiration handling to support 3-state encoding for SSO deadlines * [agent-network] Relocate agentnetwork package to internals/modules Move management/server/agentnetwork (and its catalog/, labelgen/, types/ subpackages) to management/internals/modules/agentnetwork, alongside the reverse-proxy module, and rewrite all importers. Pure relocation: package names, the synthesizer + affectedpeers registration hook, and store access (shared store.Store) are unchanged, so no import cycle is introduced (affectedpeers still depends only on the agentnetwork/types leaf). * [agent-network] Co-locate HTTP handlers in the module (RegisterEndpoints) Move the agent-network HTTP handlers from server/http/handlers/agentnetwork into the module at internals/modules/agentnetwork/handlers (package handlers) and rename the entrypoint AddEndpoints -> RegisterEndpoints, matching the reverse-proxy module convention. Wiring in http/handler.go updated accordingly. * Update getting started to point to rc when agent network enabled * Add a reference to a commercial license * Fix docs localhost link * Fix docs localhost link * Add private services domain note * [management] Add agent-network telemetry metrics (#6561) Surface agent-network adoption and usage in the self-hosted metrics worker: distinct accounts, providers, policies, budget rules, accounts with log collection enabled, and aggregated input/output tokens plus cost. Tokens and cost are summed from agent_network_request_usage (the always-written per-request ledger) so the figures are accurate regardless of the log-collection toggle and carry no double-counting. All values come from a handful of indexed aggregate queries run only on the worker's periodic tick. Adds store.AgentNetworkMetrics with GetAgentNetworkMetrics on the Store interface, the SqlStore implementation, and a zero-valued FileStore stub. * Update NetBird server and proxy image versions to 0.74.0-rc.2 * [management,proxy] Reduce agent-network cognitive complexity (#6566) Address the SonarCloud quality-gate findings in new agent-network code by extracting focused helpers. No behavior change. - synthesizer.go: split buildIdentityInjectConfigJSON into per-shape rule builders; extract mergeGuardrail from mergeGuardrails to cut nesting depth. - llm_identity_inject: extract injectionEmitsAnything validation predicate from New. - llm_response_parser/streaming.go: extract applyOpenAIStreamUsage and applyAnthropicStreamUsage (via a named anthropicStreamUsage type) and simplify the OpenAI scanner loop. - reverseproxy.go: decompose ServeHTTP into serveRouteError, buildTargetContext, serveDirect, serveWithChain, captureRequestForChain, serveDeny, newResponseWriter, observeResponse, and forwardUpstream, preserving the defer ordering so response observation still reads the captured writer before it is released. * [management] Move agent-network access-log ingest into the agentnetwork module (#6568) The agent-network access-log ingest path (metaKey wire contract, flatten, usage derivation, and the dual-write of the usage ledger + settings-gated full row) lived in the reverseproxy accesslogs manager, even though the agentnetwork module already owns the rest of that domain — types, read (ListAccessLogs / GetUsageOverview), the budget-counter writes, and retention cleanup. Move it next to the rest: a stateless agentnetwork.IngestAccessLog(ctx, store, entry) that the reverseproxy SaveAccessLog delegates to when the entry is agent-network. Removes the agentNetworkTypes import from the reverseproxy manager. No behavior change; the write/read table separation is unchanged. Adds real-store coverage for the disable->enable log-collection toggle (usage ledger always written, full row gated) plus the metadata parse and group-dedup helpers, which previously had no dedicated tests. * Add session view support in the access log * [management,proxy] Container-based agent-network e2e harness (#6577) * [e2e] Add container-based agent-network e2e harness (Pillar 1) Introduce a self-contained, OIDC-free e2e harness that stands up NetBird in containers, so suites no longer depend on the hand-maintained Tilt stack or a real IdP. - harness brings up the combined server (management + signal + relay + STUN + embedded IdP) in a single container built from combined/Dockerfile.multistage, and mints an admin PAT through the unauthenticated /api/setup bootstrap (NB_SETUP_PAT_ENABLED). API access goes through the existing shared/management/client/rest typed client. - the image is built via the docker CLI (BuildKit) so the Dockerfile's cache mounts are honored; testcontainers then runs the tagged image. - everything is behind the `e2e` build tag so normal builds and unit tests never pull in testcontainers. Adds BuildKit cache mounts to combined/Dockerfile.multistage so source changes recompile incrementally rather than from scratch. Pillar 1 proven by TestCombinedBootstrap: server builds, boots, mints a PAT, and the PAT authenticates a real management API call. * [e2e] Add management-side agent-network scenarios (Pillar 2) Port the API-driven agent-network scenarios from the bash suites to Go, sharing one combined server per package run (TestMain) with each test owning its resource cleanup. Drives the /api/agent-network/* endpoints through the shared REST client's NewRequest primitive with the generated api types. Scenarios: - provider lifecycle (create/get/list/delete + 404 after delete) - provider validation (missing api_key, unknown catalog id → 4xx) - settings collection-toggle round-trip with cluster/subdomain immutability - policy window floor (reject <60s enabled limit, accept at 60s) - consumption read endpoint returns an array All deterministic and dependency-free (dummy provider keys; no upstream calls), so they run headless in CI. * [e2e] Add live chat-through-proxy scenario (Pillar 3) Stand up the full agent-network data path in containers and drive a real chat-completion through the gateway: - harness: a shared docker network (combined server reachable by alias), a proxy container built from the published reverse-proxy image (NB_PROXY_PRIVATE, NB_PROXY_ALLOW_INSECURE, NB_RELAY_TRANSPORT=ws to match the combined server's WS-multiplexed relay) with a generated self-signed wildcard cert, and a netbird client container that joins via a setup key. - the combined image, proxy image, and client image default to the published rc.2 releases (overridable via NB_E2E_*_IMAGE; a bare local tag is built from source instead). Geolocation download is disabled so the server starts without external fetches. - one shared domain is used for the management exposed address, the proxy domain, and the agent-network cluster; the proxy token is minted via the server CLI (global) to match the manual install. TestChatCompletionThroughProxy provisions provider+policy+group+setup key, runs proxy+client, drives an OpenAI chat-completion through the tunnel, and asserts a 200 plus the ingested access-log row. Requires OPENAI_TOKEN (skips otherwise). The provider must be created with enabled=true explicitly — the create default is false despite the API doc. * [e2e] Run the live chat scenario across a provider matrix Replace the single-provider chat test with a data-driven matrix that runs the same scenario through every provider whose credentials are present in the environment (keys/URLs sourced from ~/.llm-keys locally, Actions secrets in CI): - OpenAI (chat), Anthropic (messages), Vercel, OpenRouter, Cloudflare (OpenAI-compatible gateways), and Bedrock (path-routed, bearer, via the messages shape) — covering both wire shapes and the gateway routing. - all providers are created enabled with a unique model string so the proxy's connect-time snapshot carries them all and model->provider routing is unambiguous (provider toggles after connect don't reconcile to a connected proxy). - the client supports both wire shapes (/v1/chat/completions and /v1/messages); Cloudflare gets the openai provider segment appended to its gateway URL. Each provider must return 200 through the tunnel and produce an ingested access-log row. Vertex is intentionally excluded from the uniform matrix: it needs a bespoke rawPredict request shape rather than the shared chat/messages path, so it warrants a dedicated scenario. * [ci] Add manual workflow for the agent-network e2e suite The e2e suite (build tag `e2e`) stands up the combined server + proxy + client in Docker and drives live chat-completions, so it is slow and needs provider credentials. Gate it out of normal CI (it already is, via the build tag) and run it on demand via workflow_dispatch. Provider scenarios skip when their secret is unset, so it degrades gracefully. * [e2e] Add Vertex to the provider matrix; run e2e on ubuntu-latest Vertex (Anthropic-on-Vertex) doesn't share the chat/messages wire shapes: the model travels in a rawPredict path and the proxy mints the service account's OAuth token. Add a Vertex client method that posts /v1/projects/<project>/locations/<region>/publishers/anthropic/models/<model>:rawPredict with the Vertex anthropic_version body, and wire it into the matrix as a path-routed provider (created without a models array). It is keyed off GOOGLE_VERTEX_SA_BASE64 + GOOGLE_VERTEX_PROJECT (region defaults to "global", model to a pinned claude snapshot, both overridable). Also bump the e2e workflow runner to ubuntu-latest and add the Vertex secrets. * Add docker/docker and docker/go-connections as direct dependencies in go.mod * [ci] Trigger agent-network e2e workflow on push to main and pull requests * [e2e] Fix proxy cert permission denied on Linux CI runners The proxy bind-mounts a temp dir of self-signed certs. MkdirTemp creates it 0700 and the key was 0600, which Docker Desktop on macOS ignores but a non-root proxy container on Linux runners cannot traverse/read, so the cert watcher failed with "open /certs/tls.crt: permission denied" and the container exited. Widen the cert dir to 0755 and write the throwaway key 0644 so the proxy uid can read the bind-mounted material. * [e2e] Build images from source by default instead of pulling rc.2 The agent-network code under test lives in this branch, so the e2e should exercise it rather than a frozen published release. Flip the harness default: combined/proxy/client are now built from their in-repo Dockerfiles (combined/Dockerfile.multistage, proxy/Dockerfile.multistage, e2e/harness/Dockerfile.client) under local tags. Pulling a published image stays available by setting NB_E2E_*_IMAGE to a registry reference. Builds now go through buildx --load so the Dockerfile cache mounts are honored and the result is loaded for testcontainers. The CI workflow adds a container-driver builder and a local layer cache (NB_E2E_BUILDX_CACHE) persisted via actions/cache, which caches the base/apt/dep-download layers across runs. The Go compile still re-runs each time, as BuildKit mount caches cannot be exported to the GitHub cache. * [e2e] Cover real providers in lifecycle + assert real consumption metering - TestProviderLifecycle now runs per available real provider (create → get → list → delete → 404) instead of a single dummy provider, exercising each catalog's create and field round-trip. Create is offline, so it stays fast and burns no provider quota; falls back to a synthetic OpenAI provider when no keys are set. - TestProvidersMatrix attaches a token limit (high caps, 60s window) to its policy, which switches on usage metering, and asserts consumption rows are recorded with positive token counts after the live traffic. Consumption is account-scoped (keyed by source group / user and window, not per provider), so the assertion is aggregate. - TestProviderValidation gains invalid-upstream and blank-name cases. Create validation is uniform across catalogs (no per-provider required-field rules), so per-provider rejection cases would be redundant. * [e2e] Assert session id propagates per provider Each matrix request now sends a unique session id as the universal x-session-id header and asserts it round-trips into that provider's access-log row. This guards the session-grouping contract end to end for every provider (header extraction runs in llm_request_parser ahead of the parser-specific body extraction, so it is provider-agnostic). * [e2e] Drop accidentally committed sync-phases dashboard netbird-sync-phases.json was swept into the Pillar 1 commit by a broad git add; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Remove it from the branch so the PR diff is scoped to the e2e changes. * [e2e] Revert accidentally committed sync-phase ingest spec The netbird_sync_phase measurement spec in metrics ingest was swept into the Pillar 1 commit; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Its emission side never landed here, so the spec was orphaned anyway. Restore ingest/main.go to its origin/main state. * Fix golint issues * Fix sonar * Add access log session test * Fix access log tests --------- Co-authored-by: braginini <bangvalo@gmail.com> Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
661 lines
24 KiB
Go
661 lines
24 KiB
Go
package agentnetwork
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"math"
|
|
"sort"
|
|
"time"
|
|
|
|
"github.com/netbirdio/netbird/management/internals/modules/agentnetwork/types"
|
|
"github.com/netbirdio/netbird/management/server/store"
|
|
"github.com/netbirdio/netbird/shared/management/status"
|
|
)
|
|
|
|
// validateUsageDeltas rejects negative or non-finite usage counters before they
|
|
// reach the consumption store, so a bad delta can't decrement or poison totals.
|
|
// The store batch method enforces the same invariant; this is the manager-level
|
|
// guard so direct callers fail fast with a clear error.
|
|
func validateUsageDeltas(tokensIn, tokensOut int64, costUSD float64) error {
|
|
if tokensIn < 0 || tokensOut < 0 || costUSD < 0 || math.IsNaN(costUSD) || math.IsInf(costUSD, 0) {
|
|
return status.Errorf(status.InvalidArgument, "usage deltas must be non-negative and finite")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Deny codes the proxy surfaces back to the caller when every
|
|
// applicable policy is exhausted. The proxy converts these into
|
|
// upstream-shaped error responses.
|
|
const (
|
|
//nolint:gosec // policy deny code label, not a credential
|
|
denyCodeTokenCapExceeded = "llm_policy.token_cap_exceeded"
|
|
//nolint:gosec // policy deny code label, not a credential
|
|
denyCodeBudgetCapExceeded = "llm_policy.budget_cap_exceeded"
|
|
//nolint:gosec // account deny code label, not a credential
|
|
denyCodeAccountTokenCapExceeded = "llm_account.token_cap_exceeded"
|
|
//nolint:gosec // account deny code label, not a credential
|
|
denyCodeAccountBudgetCapExceeded = "llm_account.budget_cap_exceeded"
|
|
)
|
|
|
|
// consumptionCache holds the consumption counters prefetched for one
|
|
// policy-selection request, keyed by ConsumptionKey. A miss returns a zero
|
|
// counter — the same contract the store's single-row getter uses for absent
|
|
// rows — so the eval logic is identical whether a counter exists yet or not.
|
|
type consumptionCache map[types.ConsumptionKey]*types.Consumption
|
|
|
|
func (c consumptionCache) get(accountID string, kind types.ConsumptionDimension, dimID string, windowSeconds int64, windowStart time.Time) *types.Consumption {
|
|
key := types.ConsumptionKey{Kind: kind, DimID: dimID, WindowSeconds: windowSeconds, WindowStartUTC: windowStart.UTC()}
|
|
if row, ok := c[key]; ok && row != nil {
|
|
return row
|
|
}
|
|
return &types.Consumption{
|
|
AccountID: accountID,
|
|
DimensionKind: kind,
|
|
DimensionID: dimID,
|
|
WindowSeconds: windowSeconds,
|
|
WindowStartUTC: windowStart.UTC(),
|
|
}
|
|
}
|
|
|
|
// addLimitKeys records the user/group consumption keys a single enabled (token
|
|
// or budget) limit window reads for the given attribution group, into a dedup
|
|
// set. attrGroup may be empty (no group dimension applies).
|
|
func addLimitKeys(set map[types.ConsumptionKey]struct{}, userID, attrGroup string, windowSeconds int64, now time.Time) {
|
|
if windowSeconds <= 0 {
|
|
return
|
|
}
|
|
ws := types.WindowStart(now, windowSeconds)
|
|
if userID != "" {
|
|
set[types.ConsumptionKey{Kind: types.DimensionUser, DimID: userID, WindowSeconds: windowSeconds, WindowStartUTC: ws}] = struct{}{}
|
|
}
|
|
if attrGroup != "" {
|
|
set[types.ConsumptionKey{Kind: types.DimensionGroup, DimID: attrGroup, WindowSeconds: windowSeconds, WindowStartUTC: ws}] = struct{}{}
|
|
}
|
|
}
|
|
|
|
// prefetchConsumption loads, in one store round-trip, every consumption counter
|
|
// that the account-budget ceiling and the candidate policies will read while
|
|
// scoring this request. This replaces the per-cap point reads the selector
|
|
// previously issued one at a time (the N+1 on the hot path).
|
|
func (m *managerImpl) prefetchConsumption(ctx context.Context, in PolicySelectionInput, rules []*types.AccountBudgetRule, candidates []*types.Policy, now time.Time) (consumptionCache, error) {
|
|
set := make(map[types.ConsumptionKey]struct{})
|
|
for _, p := range candidates {
|
|
attr := lowestIntersect(p.SourceGroups, in.GroupIDs)
|
|
if p.Limits.TokenLimit.Enabled {
|
|
addLimitKeys(set, in.UserID, attr, p.Limits.TokenLimit.WindowSeconds, now)
|
|
}
|
|
if p.Limits.BudgetLimit.Enabled {
|
|
addLimitKeys(set, in.UserID, attr, p.Limits.BudgetLimit.WindowSeconds, now)
|
|
}
|
|
}
|
|
for _, r := range rules {
|
|
if r == nil || !r.Enabled || !budgetRuleApplies(r, in) {
|
|
continue
|
|
}
|
|
attr := lowestIntersect(r.TargetGroups, in.GroupIDs)
|
|
if r.Limits.TokenLimit.Enabled {
|
|
addLimitKeys(set, in.UserID, attr, r.Limits.TokenLimit.WindowSeconds, now)
|
|
}
|
|
if r.Limits.BudgetLimit.Enabled {
|
|
addLimitKeys(set, in.UserID, attr, r.Limits.BudgetLimit.WindowSeconds, now)
|
|
}
|
|
}
|
|
if len(set) == 0 {
|
|
return consumptionCache{}, nil
|
|
}
|
|
keys := make([]types.ConsumptionKey, 0, len(set))
|
|
for k := range set {
|
|
keys = append(keys, k)
|
|
}
|
|
rows, err := m.store.GetAgentNetworkConsumptionBatch(ctx, store.LockingStrengthNone, in.AccountID, keys)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("batch read consumption: %w", err)
|
|
}
|
|
return consumptionCache(rows), nil
|
|
}
|
|
|
|
// SelectPolicyForRequest picks the policy that "pays" for the
|
|
// incoming request. The chosen policy is the one with the largest
|
|
// pool that still has headroom — drain the bigger bucket first,
|
|
// fall through to the next-biggest only when the current one's
|
|
// group cap or shared per-user cap is exhausted. This matches
|
|
// operator intuition for layered tiers ("privileged group has the
|
|
// 10k budget, regular group has 1k as the safety net") and avoids
|
|
// the load-balancer flapping that fraction-based scoring produces
|
|
// once any cap has been touched.
|
|
//
|
|
// Ordering across non-exhausted candidates:
|
|
// 1. Policies with NO enabled caps (catch-all-allow) win over any
|
|
// capped policy — operators who configure unlimited access
|
|
// expect requests to attribute there until they explicitly add
|
|
// caps.
|
|
// 2. Larger group token cap wins.
|
|
// 3. Larger group budget USD cap wins.
|
|
// 4. Larger user token cap wins.
|
|
// 5. Larger user budget USD cap wins.
|
|
// 6. Older created_at wins (deterministic final tiebreak so
|
|
// multi-node selection converges).
|
|
//
|
|
// Returns Allow=true with empty SelectedPolicyID when no policy in
|
|
// the account targets the (provider, caller-groups) combination —
|
|
// llm_router is the gate that owns "no policy authorises this
|
|
// request" semantics; this function trusts that authorisation has
|
|
// already happened upstream and only does the limit-aware
|
|
// attribution.
|
|
func (m *managerImpl) SelectPolicyForRequest(ctx context.Context, in PolicySelectionInput) (*PolicySelectionResult, error) {
|
|
if in.AccountID == "" {
|
|
return nil, status.Errorf(status.InvalidArgument, "account_id is required")
|
|
}
|
|
|
|
now := time.Now().UTC()
|
|
|
|
rules, err := m.store.GetAccountAgentNetworkBudgetRules(ctx, store.LockingStrengthNone, in.AccountID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("list account budget rules: %w", err)
|
|
}
|
|
policies, err := m.store.GetAccountAgentNetworkPolicies(ctx, store.LockingStrengthNone, in.AccountID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("list account policies: %w", err)
|
|
}
|
|
candidates := filterApplicablePolicies(policies, in)
|
|
|
|
// Prefetch every consumption counter the ceiling + candidate policies will
|
|
// read, in a single store round-trip, then score against the cache.
|
|
cache, err := m.prefetchConsumption(ctx, in, rules, candidates, now)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// Account-level budget rules are an always-on ceiling, evaluated
|
|
// independently of policy selection (they bind even for catch-all-allow
|
|
// policies or requests that match no policy). All applicable rules must
|
|
// pass — this is where min-wins lives.
|
|
if deny, code, reason := checkAccountBudget(in, rules, cache, now); deny {
|
|
return &PolicySelectionResult{Allow: false, DenyCode: code, DenyReason: reason}, nil
|
|
}
|
|
|
|
if len(candidates) == 0 {
|
|
return &PolicySelectionResult{Allow: true}, nil
|
|
}
|
|
scored, lastDenyCode, lastDenyReason := scoreCandidates(in, candidates, cache, now)
|
|
if len(scored) == 0 {
|
|
return &PolicySelectionResult{
|
|
Allow: false,
|
|
DenyCode: lastDenyCode,
|
|
DenyReason: lastDenyReason,
|
|
}, nil
|
|
}
|
|
|
|
sort.SliceStable(scored, func(i, j int) bool {
|
|
// Catch-all-allow (no caps configured) wins outright over
|
|
// any capped policy.
|
|
iNoCap := isUncapped(scored[i].policy)
|
|
jNoCap := isUncapped(scored[j].policy)
|
|
if iNoCap != jNoCap {
|
|
return iNoCap
|
|
}
|
|
// Bigger pool drains first. Group caps dominate (shared
|
|
// across the group) before individual caps.
|
|
if a, b := groupCapTokens(scored[i].policy), groupCapTokens(scored[j].policy); a != b {
|
|
return a > b
|
|
}
|
|
if a, b := groupCapBudgetUsd(scored[i].policy), groupCapBudgetUsd(scored[j].policy); a != b {
|
|
return a > b
|
|
}
|
|
if a, b := userCapTokens(scored[i].policy), userCapTokens(scored[j].policy); a != b {
|
|
return a > b
|
|
}
|
|
if a, b := userCapBudgetUsd(scored[i].policy), userCapBudgetUsd(scored[j].policy); a != b {
|
|
return a > b
|
|
}
|
|
return scored[i].policy.CreatedAt.Before(scored[j].policy.CreatedAt)
|
|
})
|
|
|
|
winner := scored[0]
|
|
return &PolicySelectionResult{
|
|
Allow: true,
|
|
SelectedPolicyID: winner.policy.ID,
|
|
AttributionGroupID: winner.attributionGroup,
|
|
WindowSeconds: winner.windowSeconds,
|
|
}, nil
|
|
}
|
|
|
|
// filterApplicablePolicies returns the enabled policies that target
|
|
// the requested provider and have at least one of the caller's groups
|
|
// in their source_groups. Caller's group set is matched
|
|
// case-sensitively against policy.SourceGroups.
|
|
func filterApplicablePolicies(policies []*types.Policy, in PolicySelectionInput) []*types.Policy {
|
|
if len(policies) == 0 {
|
|
return nil
|
|
}
|
|
groupSet := make(map[string]struct{}, len(in.GroupIDs))
|
|
for _, g := range in.GroupIDs {
|
|
if g != "" {
|
|
groupSet[g] = struct{}{}
|
|
}
|
|
}
|
|
out := make([]*types.Policy, 0, len(policies))
|
|
for _, p := range policies {
|
|
if p == nil || !p.Enabled {
|
|
continue
|
|
}
|
|
if !sliceContains(p.DestinationProviderIDs, in.ProviderID) {
|
|
continue
|
|
}
|
|
if !anyGroupMatches(p.SourceGroups, groupSet) {
|
|
continue
|
|
}
|
|
out = append(out, p)
|
|
}
|
|
return out
|
|
}
|
|
|
|
// candidate is the per-policy intermediate the selector ranks. A
|
|
// policy that's been exhausted on any enabled cap never makes it
|
|
// into this slice; the selector's deny envelope carries the latest
|
|
// exhaustion's reason out separately.
|
|
type candidate struct {
|
|
policy *types.Policy
|
|
attributionGroup string
|
|
windowSeconds int64
|
|
}
|
|
|
|
// scoreCandidates evaluates every applicable policy against the
|
|
// caller's current consumption. Exhausted policies are filtered out
|
|
// of the returned slice; the most recent exhaustion's deny code +
|
|
// human reason is returned alongside so the caller can surface it
|
|
// when no candidate survives.
|
|
func scoreCandidates(
|
|
in PolicySelectionInput,
|
|
candidates []*types.Policy,
|
|
cache consumptionCache,
|
|
now time.Time,
|
|
) ([]candidate, string, string) {
|
|
out := make([]candidate, 0, len(candidates))
|
|
var lastDenyCode, lastDenyReason string
|
|
|
|
for _, p := range candidates {
|
|
c, exhausted, denyCode, denyReason := scoreOne(in, p, cache, now)
|
|
if exhausted {
|
|
lastDenyCode = denyCode
|
|
lastDenyReason = denyReason
|
|
continue
|
|
}
|
|
out = append(out, c)
|
|
}
|
|
return out, lastDenyCode, lastDenyReason
|
|
}
|
|
|
|
// scoreOne checks a single policy for cap exhaustion. Returns the
|
|
// candidate envelope when the policy still has headroom on every
|
|
// enabled cap; reports exhausted=true with a deny code naming the
|
|
// offending cap kind otherwise.
|
|
func scoreOne(
|
|
in PolicySelectionInput,
|
|
p *types.Policy,
|
|
cache consumptionCache,
|
|
now time.Time,
|
|
) (candidate, bool, string, string) {
|
|
attrGroup := lowestIntersect(p.SourceGroups, in.GroupIDs)
|
|
c := candidate{
|
|
policy: p,
|
|
attributionGroup: attrGroup,
|
|
windowSeconds: effectiveWindowSeconds(p),
|
|
}
|
|
|
|
if p.Limits.TokenLimit.Enabled && p.Limits.TokenLimit.WindowSeconds > 0 {
|
|
if exhausted, reason := evalTokenCap(cache, in.AccountID, in.UserID, attrGroup, p.Limits.TokenLimit, now, "policy "+p.ID); exhausted {
|
|
return candidate{}, true, denyCodeTokenCapExceeded, reason
|
|
}
|
|
}
|
|
|
|
if p.Limits.BudgetLimit.Enabled && p.Limits.BudgetLimit.WindowSeconds > 0 {
|
|
if exhausted, reason := evalBudgetCap(cache, in.AccountID, in.UserID, attrGroup, p.Limits.BudgetLimit, now, "policy "+p.ID); exhausted {
|
|
return candidate{}, true, denyCodeBudgetCapExceeded, reason
|
|
}
|
|
}
|
|
|
|
return c, false, "", ""
|
|
}
|
|
|
|
// evalTokenCap reports whether the token limit is already exhausted for the
|
|
// caller in its own window. attrGroup may be empty (no group dimension applies).
|
|
// label identifies the cap source ("policy <id>" or "account rule <id>") for the
|
|
// deny reason. It is the shared primitive behind both policy and account-rule
|
|
// enforcement.
|
|
func evalTokenCap(
|
|
cache consumptionCache,
|
|
accountID, userID, attrGroup string,
|
|
tl types.PolicyTokenLimit,
|
|
now time.Time,
|
|
label string,
|
|
) (bool, string) {
|
|
windowStart := types.WindowStart(now, tl.WindowSeconds)
|
|
|
|
if tl.UserCap > 0 && userID != "" {
|
|
row := cache.get(accountID, types.DimensionUser, userID, tl.WindowSeconds, windowStart)
|
|
used := row.TokensInput + row.TokensOutput
|
|
if used >= tl.UserCap {
|
|
return true, fmt.Sprintf("user token cap exhausted on %s (used %d of %d)", label, used, tl.UserCap)
|
|
}
|
|
}
|
|
|
|
if tl.GroupCap > 0 && attrGroup != "" {
|
|
row := cache.get(accountID, types.DimensionGroup, attrGroup, tl.WindowSeconds, windowStart)
|
|
used := row.TokensInput + row.TokensOutput
|
|
if used >= tl.GroupCap {
|
|
return true, fmt.Sprintf("group token cap exhausted on %s (used %d of %d)", label, used, tl.GroupCap)
|
|
}
|
|
}
|
|
|
|
return false, ""
|
|
}
|
|
|
|
// evalBudgetCap is the budget (USD) counterpart of evalTokenCap.
|
|
func evalBudgetCap(
|
|
cache consumptionCache,
|
|
accountID, userID, attrGroup string,
|
|
bl types.PolicyBudgetLimit,
|
|
now time.Time,
|
|
label string,
|
|
) (bool, string) {
|
|
windowStart := types.WindowStart(now, bl.WindowSeconds)
|
|
|
|
if bl.UserCapUsd > 0 && userID != "" {
|
|
row := cache.get(accountID, types.DimensionUser, userID, bl.WindowSeconds, windowStart)
|
|
if row.CostUSD >= bl.UserCapUsd {
|
|
return true, fmt.Sprintf("user budget cap exhausted on %s (used $%.4f of $%.4f)", label, row.CostUSD, bl.UserCapUsd)
|
|
}
|
|
}
|
|
|
|
if bl.GroupCapUsd > 0 && attrGroup != "" {
|
|
row := cache.get(accountID, types.DimensionGroup, attrGroup, bl.WindowSeconds, windowStart)
|
|
if row.CostUSD >= bl.GroupCapUsd {
|
|
return true, fmt.Sprintf("group budget cap exhausted on %s (used $%.4f of $%.4f)", label, row.CostUSD, bl.GroupCapUsd)
|
|
}
|
|
}
|
|
|
|
return false, ""
|
|
}
|
|
|
|
// checkAccountBudget evaluates every applicable account-level budget rule as an
|
|
// all-must-pass ceiling. A rule applies when the caller is in its TargetUsers,
|
|
// one of its TargetGroups, or it has no targets at all (account-wide). Returns
|
|
// deny=true with an llm_account.* code on the first exhausted rule. Group caps
|
|
// attribute to the lowest intersecting group (the same model policies use), so
|
|
// multi-group behavior is unchanged.
|
|
func checkAccountBudget(in PolicySelectionInput, rules []*types.AccountBudgetRule, cache consumptionCache, now time.Time) (bool, string, string) {
|
|
for _, r := range rules {
|
|
if r == nil || !r.Enabled || !budgetRuleApplies(r, in) {
|
|
continue
|
|
}
|
|
attrGroup := lowestIntersect(r.TargetGroups, in.GroupIDs)
|
|
label := "account rule " + r.ID
|
|
|
|
if r.Limits.TokenLimit.Enabled && r.Limits.TokenLimit.WindowSeconds > 0 {
|
|
if exhausted, reason := evalTokenCap(cache, in.AccountID, in.UserID, attrGroup, r.Limits.TokenLimit, now, label); exhausted {
|
|
return true, denyCodeAccountTokenCapExceeded, reason
|
|
}
|
|
}
|
|
|
|
if r.Limits.BudgetLimit.Enabled && r.Limits.BudgetLimit.WindowSeconds > 0 {
|
|
if exhausted, reason := evalBudgetCap(cache, in.AccountID, in.UserID, attrGroup, r.Limits.BudgetLimit, now, label); exhausted {
|
|
return true, denyCodeAccountBudgetCapExceeded, reason
|
|
}
|
|
}
|
|
}
|
|
|
|
return false, "", ""
|
|
}
|
|
|
|
// budgetRuleApplies reports whether an account budget rule binds the caller:
|
|
// a direct user match, a group intersection, or an untargeted (account-wide)
|
|
// rule.
|
|
func budgetRuleApplies(r *types.AccountBudgetRule, in PolicySelectionInput) bool {
|
|
if len(r.TargetUsers) == 0 && len(r.TargetGroups) == 0 {
|
|
return true
|
|
}
|
|
if in.UserID != "" && sliceContains(r.TargetUsers, in.UserID) {
|
|
return true
|
|
}
|
|
groupSet := make(map[string]struct{}, len(in.GroupIDs))
|
|
for _, g := range in.GroupIDs {
|
|
if g != "" {
|
|
groupSet[g] = struct{}{}
|
|
}
|
|
}
|
|
return anyGroupMatches(r.TargetGroups, groupSet)
|
|
}
|
|
|
|
// RecordAccountBudgetUsage fans the served request's usage out to every
|
|
// applicable account budget rule's own (dimension, window) counter. The user
|
|
// dimension is always booked when a rule has a user-applicable cap; the group
|
|
// dimension books against the rule's lowest intersecting group. This runs
|
|
// alongside the policy-window record so account ceilings accumulate in their own
|
|
// windows (commonly monthly) independently of the per-policy window.
|
|
func (m *managerImpl) RecordAccountBudgetUsage(ctx context.Context, accountID, userID string, groupIDs []string, tokensIn, tokensOut int64, costUSD float64) error {
|
|
if accountID == "" {
|
|
return status.Errorf(status.InvalidArgument, "account_id is required")
|
|
}
|
|
if err := validateUsageDeltas(tokensIn, tokensOut, costUSD); err != nil {
|
|
return err
|
|
}
|
|
rules, err := m.store.GetAccountAgentNetworkBudgetRules(ctx, store.LockingStrengthNone, accountID)
|
|
if err != nil {
|
|
return fmt.Errorf("list account budget rules: %w", err)
|
|
}
|
|
set := make(map[types.ConsumptionKey]struct{})
|
|
addAccountBudgetKeys(set, PolicySelectionInput{AccountID: accountID, UserID: userID, GroupIDs: groupIDs}, rules, time.Now().UTC())
|
|
if len(set) == 0 {
|
|
return nil
|
|
}
|
|
return m.store.IncrementAgentNetworkConsumptionBatch(ctx, accountID, keysSlice(set), tokensIn, tokensOut, costUSD)
|
|
}
|
|
|
|
// RecordUsageInput carries everything RecordUsage books for one served request.
|
|
type RecordUsageInput struct {
|
|
AccountID string
|
|
UserID string
|
|
AttributionGroupID string // selected policy's attribution group (policy window)
|
|
GroupIDs []string
|
|
WindowSeconds int64 // selected policy's window; 0 means no policy cap
|
|
TokensIn int64
|
|
TokensOut int64
|
|
CostUSD float64
|
|
}
|
|
|
|
// RecordUsage books a served request's usage against every counter it touches —
|
|
// the selected policy's per-(user, group) window plus every applicable account
|
|
// budget rule's own window — deduplicated and written in a single transaction.
|
|
// Two counters that collapse to the same (dimension, window) tuple are booked
|
|
// once, so a single request can never double-count against one cap.
|
|
func (m *managerImpl) RecordUsage(ctx context.Context, in RecordUsageInput) error {
|
|
if in.AccountID == "" {
|
|
return status.Errorf(status.InvalidArgument, "account_id is required")
|
|
}
|
|
if err := validateUsageDeltas(in.TokensIn, in.TokensOut, in.CostUSD); err != nil {
|
|
return err
|
|
}
|
|
now := time.Now().UTC()
|
|
set := make(map[types.ConsumptionKey]struct{})
|
|
|
|
// Policy-window dimensions are booked only when a policy cap bound this
|
|
// request (window > 0). A zero window means catch-all-allow / no policy cap;
|
|
// the account fan-out below still books against the budget rules' windows.
|
|
if in.WindowSeconds > 0 {
|
|
addLimitKeys(set, in.UserID, in.AttributionGroupID, in.WindowSeconds, now)
|
|
}
|
|
|
|
rules, err := m.store.GetAccountAgentNetworkBudgetRules(ctx, store.LockingStrengthNone, in.AccountID)
|
|
if err != nil {
|
|
return fmt.Errorf("list account budget rules: %w", err)
|
|
}
|
|
addAccountBudgetKeys(set, PolicySelectionInput{AccountID: in.AccountID, UserID: in.UserID, GroupIDs: in.GroupIDs}, rules, now)
|
|
|
|
if len(set) == 0 {
|
|
return nil
|
|
}
|
|
return m.store.IncrementAgentNetworkConsumptionBatch(ctx, in.AccountID, keysSlice(set), in.TokensIn, in.TokensOut, in.CostUSD)
|
|
}
|
|
|
|
// addAccountBudgetKeys adds the (dimension, window) keys a served request books
|
|
// against every applicable account budget rule into the dedup set.
|
|
func addAccountBudgetKeys(set map[types.ConsumptionKey]struct{}, in PolicySelectionInput, rules []*types.AccountBudgetRule, now time.Time) {
|
|
for _, r := range rules {
|
|
if r == nil || !r.Enabled || !budgetRuleApplies(r, in) {
|
|
continue
|
|
}
|
|
attrGroup := lowestIntersect(r.TargetGroups, in.GroupIDs)
|
|
for _, window := range ruleWindows(r) {
|
|
addLimitKeys(set, in.UserID, attrGroup, window, now)
|
|
}
|
|
}
|
|
}
|
|
|
|
// keysSlice flattens a ConsumptionKey set into a slice.
|
|
func keysSlice(set map[types.ConsumptionKey]struct{}) []types.ConsumptionKey {
|
|
keys := make([]types.ConsumptionKey, 0, len(set))
|
|
for k := range set {
|
|
keys = append(keys, k)
|
|
}
|
|
return keys
|
|
}
|
|
|
|
// ruleWindows returns the distinct enabled window lengths a budget rule books
|
|
// against (token window and/or budget window, deduplicated).
|
|
func ruleWindows(r *types.AccountBudgetRule) []int64 {
|
|
var windows []int64
|
|
if r.Limits.TokenLimit.Enabled && r.Limits.TokenLimit.WindowSeconds > 0 {
|
|
windows = append(windows, r.Limits.TokenLimit.WindowSeconds)
|
|
}
|
|
if r.Limits.BudgetLimit.Enabled && r.Limits.BudgetLimit.WindowSeconds > 0 {
|
|
bw := r.Limits.BudgetLimit.WindowSeconds
|
|
if len(windows) == 0 || windows[0] != bw {
|
|
windows = append(windows, bw)
|
|
}
|
|
}
|
|
return windows
|
|
}
|
|
|
|
// effectiveWindowSeconds returns the window length the proxy should
|
|
// hand back to RecordLLMUsage. When both halves are enabled with
|
|
// different windows, token_limit wins (the more common config); when
|
|
// only one is enabled that one wins; when neither is enabled the
|
|
// returned value is 0 — RecordLLMUsage treats 0 as "no limit
|
|
// tracking" and skips the increment, which is the right pass-through
|
|
// for catch-all-allow policies with no caps configured.
|
|
func effectiveWindowSeconds(p *types.Policy) int64 {
|
|
if p.Limits.TokenLimit.Enabled && p.Limits.TokenLimit.WindowSeconds > 0 {
|
|
return p.Limits.TokenLimit.WindowSeconds
|
|
}
|
|
if p.Limits.BudgetLimit.Enabled && p.Limits.BudgetLimit.WindowSeconds > 0 {
|
|
return p.Limits.BudgetLimit.WindowSeconds
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// lowestIntersect returns the lowest-by-string-sort element of
|
|
// callerGroups ∩ sourceGroups. Empty when the intersection is empty.
|
|
// Lowest is deterministic so multi-node selection converges.
|
|
func lowestIntersect(sourceGroups, callerGroups []string) string {
|
|
if len(sourceGroups) == 0 || len(callerGroups) == 0 {
|
|
return ""
|
|
}
|
|
srcSet := make(map[string]struct{}, len(sourceGroups))
|
|
for _, g := range sourceGroups {
|
|
srcSet[g] = struct{}{}
|
|
}
|
|
var best string
|
|
for _, g := range callerGroups {
|
|
if _, ok := srcSet[g]; !ok {
|
|
continue
|
|
}
|
|
if best == "" || g < best {
|
|
best = g
|
|
}
|
|
}
|
|
return best
|
|
}
|
|
|
|
func anyGroupMatches(sourceGroups []string, callerSet map[string]struct{}) bool {
|
|
for _, g := range sourceGroups {
|
|
if _, ok := callerSet[g]; ok {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// isUncapped reports whether a policy has any enabled cap with a
|
|
// positive limit value. Mirrors the eval functions' guards: a policy
|
|
// with token_limit.enabled=true but every cap value at 0 still
|
|
// counts as uncapped because the eval would query nothing and bind
|
|
// nothing.
|
|
func isUncapped(p *types.Policy) bool {
|
|
tl := p.Limits.TokenLimit
|
|
if tl.Enabled && tl.WindowSeconds > 0 && (tl.GroupCap > 0 || tl.UserCap > 0) {
|
|
return false
|
|
}
|
|
bl := p.Limits.BudgetLimit
|
|
if bl.Enabled && bl.WindowSeconds > 0 && (bl.GroupCapUsd > 0 || bl.UserCapUsd > 0) {
|
|
return false
|
|
}
|
|
return true
|
|
}
|
|
|
|
// groupCapTokens returns the policy's group-token cap when the token
|
|
// limit is enabled, zero otherwise. Drives the primary "bigger pool
|
|
// first" sort.
|
|
func groupCapTokens(p *types.Policy) int64 {
|
|
if p.Limits.TokenLimit.Enabled {
|
|
return p.Limits.TokenLimit.GroupCap
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// groupCapBudgetUsd returns the policy's group-budget cap in USD
|
|
// when the budget limit is enabled, zero otherwise. Secondary sort
|
|
// key after token group cap so budget-only policies still order
|
|
// predictably.
|
|
func groupCapBudgetUsd(p *types.Policy) float64 {
|
|
if p.Limits.BudgetLimit.Enabled {
|
|
return p.Limits.BudgetLimit.GroupCapUsd
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// userCapTokens returns the policy's per-user token cap when the
|
|
// token limit is enabled, zero otherwise. Tertiary sort key, used
|
|
// when group caps tie or are absent.
|
|
func userCapTokens(p *types.Policy) int64 {
|
|
if p.Limits.TokenLimit.Enabled {
|
|
return p.Limits.TokenLimit.UserCap
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// userCapBudgetUsd returns the policy's per-user budget cap in USD
|
|
// when the budget limit is enabled, zero otherwise. Quaternary sort
|
|
// key for budget-only policies whose group caps tie or are absent.
|
|
func userCapBudgetUsd(p *types.Policy) float64 {
|
|
if p.Limits.BudgetLimit.Enabled {
|
|
return p.Limits.BudgetLimit.UserCapUsd
|
|
}
|
|
return 0
|
|
}
|
|
|
|
func sliceContains(haystack []string, needle string) bool {
|
|
for _, v := range haystack {
|
|
if v == needle {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// mockManager fallback so tests that don't care about selection still
|
|
// compile.
|
|
func (*mockManager) SelectPolicyForRequest(_ context.Context, _ PolicySelectionInput) (*PolicySelectionResult, error) {
|
|
return &PolicySelectionResult{Allow: true}, nil
|
|
}
|