Compare commits

..

2 Commits

Author SHA1 Message Date
Viktor Liu
af1f94bbe9 Merge branch 'main' into reverse-proxy-crowdsec-appsec
# Conflicts:
#	management/server/store/sql_store.go
#	management/server/store/sql_store_service_test.go
2026-07-31 22:00:33 +02:00
Viktor Liu
663468e199 Add CrowdSec AppSec request inspection to the reverse proxy 2026-07-28 10:51:54 +02:00
53 changed files with 3917 additions and 1637 deletions

514
AGENTS.md
View File

@@ -1,514 +0,0 @@
# NetBird Agent Guidelines
**NetBird** is an open source connectivity platform: a WireGuard®-based overlay
network with a control plane. The **agent** (`client/`) runs on user machines as
a privileged daemon and manages the WireGuard interface, routing, firewall, and
DNS. **Management** (`management/`) is the control plane and REST/gRPC API,
**Signal** (`signal/`) brokers peer handshakes, **Relay** (`relay/`) carries
traffic when a direct tunnel is impossible, and **Proxy** (`proxy/`) is the
identity-aware proxy behind Agent Network.
This file applies to the whole repository, and is the single source of truth for
agent guidance here. `CLAUDE.md` is a one-line pointer to it — keep the guidance
in this file, not duplicated there.
## Contents
- [NetBird Agent Guidelines](#netbird-agent-guidelines)
- [Contents](#contents)
- [STOP and ask the user before](#stop-and-ask-the-user-before)
- [Quick reference](#quick-reference)
- [Structure](#structure)
- [Where to look](#where-to-look)
- [Repo-wide principles](#repo-wide-principles)
- [Error handling](#error-handling)
- [Comments](#comments)
- [Testing](#testing)
- [Pitfalls](#pitfalls)
- [Commits, PRs, releases](#commits-prs-releases)
- [After you push: CI and review bots](#after-you-push-ci-and-review-bots)
- [Discussion and support](#discussion-and-support)
## STOP and ask the user before
- **Opening a pull request for anything beyond a trivial fix, without an agreed
ticket.** Ask the user directly: *"Is there a discussion or issue for this
change?"* NetBird is discussion-first — community reports start in
[Discussions](https://github.com/netbirdio/netbird/discussions), DevRel
validates them, and only validated discussions become issues. A PR that
changes behavior with no linked issue may be closed on arrival. If there is no
ticket, offer to draft the discussion post **instead of** the PR, and wait for
the user's call. Only typos, broken links, documentation corrections, and
one-line fixes that already have an issue can skip this.
- **Designing in any high-risk area** (see
[CONTRIBUTING.md](CONTRIBUTING.md#high-risk-areas)): public API and OpenAPI
schema, gRPC protos, behavior existing deployments would notice after an
upgrade, peer connectivity (ICE, NAT traversal, relay selection, WireGuard® or
Rosenpass key handling), client system integration (routing, firewall, DNS,
interface), authentication and authorization, CLI or service flags, config
file format, daemon IPC, store schema and migrations, or a new feature. The
design gets agreed in the ticket before code is written.
- **Writing a store migration or changing a persisted model.** Migrations are
one-way in the field and both the GORM and pgx paths may need the change.
- **Hand-editing generated code.** `*.pb.go`, `*.gen.go`, and mocks are outputs.
Edit the source (`.proto`, `openapi.yml`) and rerun the matching
`generate.sh`.
- **Adding, removing, or bumping a dependency**, and never vendor a fork.
- **Weakening a security control** — authentication, authorization, certificate
verification, privilege dropping, or peer identity checks — even when it is
the fastest way to make a test pass.
- **Force-pushing to `main`**, force-pushing any branch that is already under
review, amending pushed commits, or bypassing hooks with `--no-verify`.
## Quick reference
```bash
# Build
go build ./...
cd client && CGO_ENABLED=0 go build . # agent
cd management && go build . # management service
cd signal && go build . # signal service
# Verify (run before every push)
go fmt ./...
make lint # golangci-lint on files changed vs origin/main (also the pre-push hook)
make lint-all # full-repository lint, matches CI
make test-unit # host-safe unit tests, -tags devcert, no sudo
make test-privileged # privileged-tagged suite in a Docker container with NET_ADMIN
make setup-hooks # wire make lint into .githooks/pre-push
# Narrow runs
go test ./client/internal/dns/...
go test -race -run TestPeerConn ./client/internal/peer/...
PRIV_RUN=TestNftablesManager PRIV_PKGS=./client/firewall/nftables/... make test-privileged
# Code generation (never hand-edit the output)
./shared/management/http/api/generate.sh # REST types from openapi.yml
./shared/management/proto/generate.sh
./shared/signal/proto/generate.sh
./client/proto/generate.sh
./flow/proto/generate.sh
# Run locally (lab only, never on a machine you rely on)
sudo ./client/netbird up --log-level debug --log-file console
sudo ./client/netbird down # teardown: restores routing, firewall, DNS
./signal/signal run --log-level debug --log-file console
./management/management management --log-level debug --log-file console --config ./management.json
```
`netbird up` needs root and rewrites the host's routing table, firewall rules,
DNS configuration, and WireGuard® interface. Run it only in a disposable test
environment (a VM, container, or throwaway host) that you can rebuild, never on
a workstation or server whose connectivity matters. Run `sudo netbird down`
before you stop working, before rebuilding the binary, and on every failure
path, so the host's networking state is restored instead of left half-applied.
See [Pitfalls](#pitfalls) for why cleanup on every exit path matters.
## Structure
```text
netbird/
├── client/ NetBird agent
│ ├── cmd/ agent CLI
│ ├── internal/ agent business logic (engine, peer, dns, routemanager, ...)
│ ├── server/ daemon for background execution
│ ├── proto/ daemon gRPC protos
│ ├── iface/ WireGuard® interface management
│ ├── firewall/ nftables, iptables, pf, WFP, userspace backends
│ ├── ssh/ built-in SSH server and client
│ ├── ui/ desktop UI (Wails v3 + React)
│ ├── android/, ios/ mobile bindings
│ ├── wasm/ WebAssembly build
│ └── mdm/, system/ MDM policy, host information
├── management/ control plane
│ └── server/ account, peer, groups, networks, posture, permissions,
│ settings, store, http (REST), idp, integrations, migration
├── signal/ handshake broker (peer/, server/)
├── relay/ relay service (protocol/, server/, healthcheck/)
├── proxy/ identity-aware proxy (llm/, acme/, accesslog/, middleware/, tcp/, udp/)
├── agent-network/ Agent Network overview
├── shared/ imported by both agent and services
│ ├── management/ proto/, client/, http/api (OpenAPI + generated types)
│ ├── signal/ proto/, client/
│ └── relay/, auth/, sshauth/, metrics/
├── e2e/ end-to-end suites and harness
├── encryption/, dns/, route/, stun/, sharedsock/, util/, flow/
├── infrastructure_files/ docker compose and getting-started templates
└── release_files/ files packaged into releases
```
## Where to look
| Task | Location |
| --------------------------- | ------------------------------------------------------------ |
| REST API / OpenAPI | `shared/management/http/api/` + `management/server/http/` |
| Management gRPC protocol | `shared/management/proto/` |
| Signal protocol | `shared/signal/proto/` |
| Daemon IPC protocol | `client/proto/` |
| Peer connection and NAT | `client/internal/peer/` |
| Network map handling | `client/internal/engine.go`, `shared/management/networkmap/` |
| Routing | `client/internal/routemanager/`, `route/` |
| Firewall backends | `client/firewall/` |
| DNS | `client/internal/dns/`, `dns/` |
| WireGuard® interface | `client/iface/` |
| Persistence and migrations | `management/server/store/`, `management/server/migration/` |
| IdP integrations | `management/server/idp/` |
| Permissions model | `management/server/permissions/` |
| LLM routing / Agent Network | `proxy/internal/llm/`, `agent-network/` |
| End-to-end tests | `e2e/` |
## Repo-wide principles
1. **Run `go fmt` on every modified Go file.** Formatting is not optional.
2. **Zero unaddressed diagnostics.** Fix IDE and linter warnings on code you
touch, and delete imports, helpers, and parameters your refactor orphaned.
Exception: unused parameters in shared code may be consumed by builds outside
this repository — do not remove them, ask instead.
3. **Function comments are mandatory for exported functions**, written as full
sentences with a period, starting with the identifier name.
4. **Prefer private functions and constants.** Export only what a caller outside
the package genuinely needs.
5. **Early returns and guard clauses.** Handle errors and edge cases first
instead of nesting `if`/`else` chains.
6. **Split complex functions.** If a function trips a complexity warning, break
it into named helpers rather than silencing the warning.
7. **Avoid LLM-slop tells:** em dashes, hedging narration, restating the diff in
prose, trailing summaries. Defaults, not absolute bans. Applies to code,
comments, commit messages, and PR descriptions alike.
8. **Concurrency: do a two-pass race analysis after every change** that adds
shared state. Guard maps and slices with a mutex, keep critical sections
short, and run `go test -race` on the touched packages.
9. **Cross-platform builds must keep working.** The agent targets Linux, macOS,
Windows, FreeBSD, Android, and iOS. When you add a platform-specific file,
add the counterpart or a build-tagged fallback for the others.
10. **Never hand-edit generated files.** Change the source and regenerate.
11. **Never log secrets** — private keys, setup keys, tokens, PAT values — and
keep peer IPs and hostnames out of logs above debug level.
## Error handling
Use single-assignment form when the error is only needed inside the `if`:
```go
// Good
if err := someCall(); err != nil {
return fmt.Errorf("context: %w", err)
}
// Bad - unnecessary split
err := someCall()
if err != nil {
return fmt.Errorf("context: %w", err)
}
```
Use multiple assignment when the value is needed after the block:
```go
result, err := someCall()
if err != nil {
return fmt.Errorf("context: %w", err)
}
```
Add short, meaningful context, and **do not** start `fmt.Errorf` messages with
obvious words like "failed to" or "error":
```go
// Good
return fmt.Errorf("parse remote address: %w", err)
return fmt.Errorf("listen on %s: %w", addr, err)
// Bad
return fmt.Errorf("failed to parse remote address: %w", err)
return fmt.Errorf("error listening on %s: %w", addr, err)
// "failed" is fine in log messages
log.Debugf("failed to parse remote address: %v", err)
```
Skip the wrapping when a function only extracts or delegates and the wrap would
add nothing:
```go
func parseAddr(addr string) (string, int, error) {
host, portStr, err := net.SplitHostPort(addr)
if err != nil {
return "", 0, err
}
// ...
}
```
Log the errors you choose not to act on:
- `log.Debugf()` for errors that do not affect program flow but help debugging.
- `log.Tracef()` for very verbose errors that would otherwise spam logs.
- **Never ignore** errors from writes, network sends, or critical cleanup.
- Close errors may be ignored for read-only operations; log them at debug for
writes.
## Comments
Comment the **why**, never the **what**. Default to no comment, and add one only
when a hidden constraint or workaround would surprise a future reader. Never
reference the current task, PR, or your own changes in a comment.
```go
// Bad - trailing comments explaining the obvious
defer localConn.Close() // Close the connection
if err != nil { // Check if error occurred
// Good
defer localConn.Close()
// Good - explains a non-obvious constraint
// Use incremental checksum update per RFC 1624 for performance.
checksum = updateChecksum(checksum, oldPort, newPort)
```
### Length budget
- **90 characters per line.** Wrap the comment, do not run past it.
- **250 characters per comment**, roughly three wrapped lines. Doc comments on
exported identifiers may exceed it when the API genuinely needs the
explanation; inline comments inside a function body may not.
The budget is a smell detector, not a rule to game. Do not compress a needed
explanation into cryptic shorthand to fit — if a block of code needs more than
250 characters of prose, the code is doing too much. Fix the code:
- **Extract a named function.** A well-named function replaces the comment: the
name says *what*, the body shows *how*, and the comment you no longer write
was the *what* anyway. Clean Code calls this "explain yourself in code".
- **Extract a named constant or predicate.** `if isExpiredSetupKey(key)` needs
no comment; `if key.ExpiresAt.Before(now) && !key.Revoked && key.UsageLimit > 0`
does.
- **Keep the surviving comment for the why** — the RFC, the kernel quirk, the
ordering constraint. That part is usually one or two lines.
### Long switch and if/else chains
A `switch` whose cases carry multi-line explanations is the usual place this
budget is breached, and the comment is a symptom. In order of preference:
1. **Extract each case body into a named function.** The case becomes one line,
the name carries the meaning, and the switch reads as a table of contents.
2. **Replace the switch with a lookup table**`map[Kind]handlerFunc` — when the
branches are uniform. Adding a case stops meaning editing a growing function.
3. **Replace conditional with polymorphism** when branches vary by type and the
same switch shape starts appearing in more than one place. Clean Code's rule
of thumb: tolerate a switch statement if it appears **once**, is buried in a
factory that returns an interface, and no other switch dispatches on the same
type. A second switch over the same enum is the signal to introduce the
interface.
Do not restructure a switch purely to satisfy the budget when the cases are one
line each and self-evident — a flat, boring `switch` over an enum is fine and
needs no comments at all.
Explanatory comments in tests are welcome — they document the scenario being set
up, and the 250-character budget does not apply to them.
## Testing
- **Unit tests** live beside the code as `_test.go`. `make test-unit` runs the
host-safe set with `-tags devcert` and no sudo.
- **Privileged tests** carry the `privileged` build tag and mutate host
networking. They run through `make test-privileged`, inside a Docker container
with `NET_ADMIN`. Never bypass that harness by running them directly on the
host.
- **End-to-end suites** live in `e2e/` with a shared harness.
- **Test real behavior, not API existence.** Assert on the observable end state
a consumer would see — bytes that arrived, the packet after translation, the
row after the write — not merely that a method exists or returns an error.
- **Avoid mocks for code we own.** Exercise the real store, manager, or
controller and assert what the caller actually receives.
- **`require` for setup and preconditions, `assert` for the conditions under
test.** Use `require` whenever a later line would panic or be meaningless
otherwise.
- **Message guidance:** optional for `NoError`/`Error`; always give context for
comparison, boolean, and collection assertions.
```go
server, err := StartTestServer()
require.NoError(t, err, "Test server setup must succeed")
defer server.Close()
result, err := client.DoOperation()
assert.NoError(t, err)
assert.Equal(t, expectedResult, result, "Result should match expected")
```
## Pitfalls
- **The agent runs as root.** Anything touching routing, firewall, DNS, or the
interface can take a user's machine off the network. Prefer a reversible
change and make sure cleanup runs on every exit path.
- **Management has two account loaders** (GORM and pgx). Adding a relation to an
account often means updating both, or it silently comes back empty in
production.
- **`go test ./...` without `-tags devcert` skips tests** that need the
development certificate. Use `make test-unit`.
- **`make lint` only checks the diff against `origin/main`.** CI runs
`make lint-all`; run it too before pushing a large change.
- **Protos are consumed by released clients.** An old agent must keep working
against a new Management, so fields are added, never renumbered or removed.
- **Windows requires the wintun driver**, and the daemon serves a named pipe
(`npipe://netbird`) rather than loopback TCP. Loopback TCP carries no caller
identity, so privileged operations are refused over it.
## Commits, PRs, releases
- **PR titles must start with a bracketed tag.** Before you propose a title,
**read [`.github/workflows/pr-title-check.yml`](.github/workflows/pr-title-check.yml)
and take the allowed tags from the `allowedTags` array in that file.** It is
the only source of truth, it changes as components are added, and the check
runs on every title edit — a tag that is not in that array is a red build. Do
not rely on a list memorized from anywhere else, including this file.
```text
[client] Authorize daemon IPC callers by their local identity
[management,client] Add MDM policy support
```
Multiple tags are comma-separated inside one pair of brackets. Match the tag
to the component you actually changed, not to the one you read the most.
- **Use the repository's PR template.** Fill in
[`.github/pull_request_template.md`](.github/pull_request_template.md) rather
than replacing it with your own summary: describe the change, link the issue,
tick the checklist honestly (including "ran locally" and "single purpose"),
and complete the documentation section. Do not tick a box you have not
verified, and do not delete rows that do not apply.
- **Keep the PR description short.** Under 1000 words on top of the template's
own text, and usually far less — a few paragraphs. Reviewers read the diff;
the description exists to explain what the diff cannot say for itself. This is
well below what an agent will produce by default, so cut before you post.
- **Body: why before what.** Lead with the problem and the reason for this
approach, then the shape of the change. No bullet list of files changed, no
per-function walkthrough, no restating the diff in prose, no trailing summary
section, no self-congratulatory closing line.
- **No `Co-Authored-By` or tool-attribution trailers in the PR description**,
and none in commits either. Contributors own their contributions. Whatever
tooling produced the diff, the person opening the PR is its author: they have
read every line, they can explain why it works, they can answer review
questions without going back to a model, and they are accountable for the
consequences of merging it. Do not add a trailer, footer, or description line
that spreads that ownership onto a tool.
- **Commit subjects follow the same `[scope] Subject` convention.** Keep the
subject short, and use the body for why before what. No bullet lists of files
changed.
- **Push review fixes as separate commits.** The PR is squashed on merge, so
there is no reason to rewrite history mid-review; many small commits make the
re-review readable.
- **Do not force-push a branch that is under review.** A force-push detaches
existing review comments from the lines they were written against, destroys
the "changes since your last review" diff a reviewer relies on, and discards
the CI history that showed which commit broke what. Add commits instead —
including for fixups and reverts. Force-push only when there is no
alternative: a rebase to clear a genuine conflict, or removing a secret or a
large binary that was committed by mistake. When you must, ask the user first,
then say so in a PR comment so reviewers know their anchors moved. Never
force-push `main`, and never force-push a branch you do not own.
- **One PR, one purpose.** Split refactors out of fixes and fixes out of
features.
- **Keep the PR small.** Size is the single strongest predictor of how long a PR
waits. Aim for **under ~400 changed lines across under ~20 files**; past
roughly **1000 lines or 50 files** a community PR is likely to be sent back to
be split, or left unreviewed until it is. Large PRs from outside the core team
may be blocked outright when the size was never agreed in the ticket —
reviewing a sprawling change against a privileged networking daemon is a
security risk in itself, not just a time cost.
Judge the size by hand-written code: exclude generated output, `go.sum`,
vendored files, and test fixtures from the estimate, but do not use their
presence to argue a 3000-line PR is small.
When a change genuinely cannot be small — a protocol migration, a
cross-component rename — agree the split in the ticket **before** writing
code, and land it as a sequence of PRs that each build, test, and make sense
on their own. Propose that split to the user rather than opening one large PR
and hoping.
- **User-facing changes need a docs PR** in
[netbirdio/docs](https://github.com/netbirdio/docs), linked from the PR
description.
## After you push: CI and review bots
Opening the PR is not the end of the task. Watch the run, read what the bots
say, and drive the PR to green before you report the work as done.
```bash
gh pr checks <pr> --watch # all checks, live
gh run view <run-id> --log-failed # only the failing steps
gh pr view <pr> --comments # bot and human review comments
```
**Never report a change as finished while checks are pending or red**, and never
describe a red PR as passing. If you ran out of turn before CI finished, say
which checks were still running.
### The checks
- **Go tests** — `golang-test-{linux,darwin,windows,freebsd}.yml`, sharded per
component. A failure in a component you did not touch is usually a real
interaction, not noise; read the log before assuming flake.
- **golangci-lint** — `golangci-lint.yml` runs the full repository, while
`make lint` only checks your diff. A clean local lint does not guarantee green
CI on a large change.
- **PR Title Check** — `pr-title-check.yml`, see above.
- **Codecov** — uploaded from the Linux test workflow with per-component flags
(`unit,client`, `unit,management`, `unit,relay`, `unit,proxy`, `unit,signal`,
`integration,management`). Coverage on new code should not go backwards. Add
tests for the paths you introduced; do not adjust thresholds or exclude files
to clear the report.
- **CodeRabbit** — configured in [`.coderabbit.yaml`](.coderabbit.yaml): `chill`
profile, auto-review on every non-draft PR, TypeScript/JavaScript/SVG paths
filtered out. Chat auto-reply is on, so `@coderabbitai` in a comment reaches
it.
- **SonarCloud** — project `netbirdio_netbird`, quality gate on new code (bugs,
vulnerabilities, code smells, duplication, coverage).
- **Snyk** — dependency and code scanning.
Sonar and Snyk report as GitHub App checks rather than workflows in this
repository, so their detail lives on the PR check, not in the Actions logs.
### Handling bot findings
- **Read every comment and act on it.** Either fix it, or reply with the reason
it does not apply. Do not bulk-resolve threads to clear the count, and do not
silently ignore a finding because the check is advisory.
- **Bots are frequently wrong here.** NetBird has privileged, platform-specific,
and concurrency-heavy code that static analysis reads poorly. A confident
CodeRabbit or Sonar comment can still be nonsense. Verify the claim against
the code before you change anything — never edit correct code just to silence
a bot.
- **Security findings get the opposite default.** For a Snyk or Sonar
vulnerability, or a CodeRabbit comment about authentication, authorization,
certificate verification, or key handling, assume it is real until you have
disproved it. Surface it to the user rather than dismissing it yourself.
- **A new vulnerable dependency is a stop.** Bumping or replacing dependencies
needs the user's decision, as above.
- **Never change a workflow, threshold, lint exclusion, or bot config to make a
check pass.** If a check is genuinely wrong, say so and let the user decide.
- **Do not paper over flakes with blind re-runs.** Identify the failure first. If
it is a known flake, name it; if you cannot tell, report it as unresolved
rather than re-running until it goes green.
## Discussion and support
- Discussions: <https://github.com/netbirdio/netbird/discussions>
- Slack: <https://docs.netbird.io/slack-url>
- Docs: <https://docs.netbird.io>
- Security: <https://github.com/netbirdio/netbird/security/policy> — never in public
- Contribution process: [CONTRIBUTING.md](CONTRIBUTING.md)

View File

@@ -1 +0,0 @@
See [AGENTS.md](AGENTS.md) for the agent guidelines in this repository.

View File

@@ -66,41 +66,11 @@ Typical bug fixes, internal refactors, documentation updates, and tests do not
need a design discussion, but should still be tied to an issue so the work is
visible and nobody duplicates it.
### Using AI coding agents
We have no policy for or against using an AI agent to write NetBird code. That
choice is yours, and we are not going to interrogate anyone about their tools.
What we do have is a lot of incoming contributions that were plainly drafted with
one, and enough experience reviewing them to see the same avoidable problems
again and again: no ticket behind the change, a diff far too large to review, a
description longer than the code it describes, an approach that was never going
to be accepted, and an author who cannot answer questions about their own PR.
None of that is caused by the tooling — it is what happens when a tool is pointed
at a repository whose expectations it has never been told.
So rather than a rule, there is a guide. [AGENTS.md](AGENTS.md) restates the
expectations from this document in the form agents read automatically
(`CLAUDE.md` points to it), so pointing your tool at the repository is usually
enough. Among other things it tells the agent to ask you for the
discussion or issue before drafting a PR, to keep the change small and
single-purpose, to run the tests locally, to use this repository's PR template
and title tags, and to write a description a reviewer can get through.
The guardrails are the point, and they are the same ones we apply to everyone: an
agreed ticket, a change you have actually run, a diff small enough to review with
care, and an author who can explain it. Whatever wrote the diff, you are its
author — you own every line you submit and the consequences of opening a PR with it.
We may assess whether a contribution is maintainable and whether its merged code
aligns with our security standards and design expectations.
## Contents
- [Contributing to NetBird](#contributing-to-netbird)
- [Ticket first, PR second](#ticket-first-pr-second)
- [High-risk areas](#high-risk-areas)
- [Using AI coding agents](#using-ai-coding-agents)
- [Contents](#contents)
- [Code of conduct](#code-of-conduct)
- [Directory structure](#directory-structure)
@@ -478,7 +448,7 @@ go test -race ./client/internal/dns/...
## Checklist before submitting a PR
As a critical network service and open source project, we must enforce a few
As a critical network service and open-source project, we must enforce a few
things before submitting a pull request. The
[pull request template](/.github/pull_request_template.md) mirrors this list —
fill it in rather than deleting it.

View File

@@ -130,7 +130,7 @@ In November 2022, NetBird joined the [StartUpSecure program](https://www.forschu
![CISPA_Logo_BLACK_EN_RZ_RGB (1)](https://user-images.githubusercontent.com/700848/203091324-c6d311a0-22b5-4b05-a288-91cbc6cdcc46.png)
### Acknowledgements
We build on open source technologies like [WireGuard®](https://www.wireguard.com/), [Pion ICE](https://github.com/pion/ice), and [Rosenpass](https://rosenpass.eu). We greatly appreciate the work these projects are doing, and we'd love it if you could support them too (e.g., by starring or contributing).
We build on open-source technologies like [WireGuard®](https://www.wireguard.com/), [Pion ICE](https://github.com/pion/ice), and [Rosenpass](https://rosenpass.eu). We greatly appreciate the work these projects are doing, and we'd love it if you could support them too (e.g., by starring or contributing).
### Legal
This repository is licensed under the BSD-3-Clause license, which applies to all parts of the repository except for the directories management/, signal/ and relay/.

View File

@@ -14,7 +14,7 @@ Report security issues one of these two ways:
on this repository. This is the preferred route: it keeps the discussion, the draft advisory, and the credit in one place.
- **Email** — `security@netbird.io`.
If the finding affects NetBird Cloud or our hosted infrastructure rather than the open source code, email us rather than
If the finding affects NetBird Cloud or our hosted infrastructure rather than the open-source code, email us rather than
filing a repository report.
### What to include

View File

@@ -115,7 +115,7 @@ sequenceDiagram
Resp->>Resp: parse usage tokens, completion
Note over Resp: capture_completion gates raw<br/>completion capture
Resp->>Cost: tokens
Cost->>Cost: lookup rates from config-delivered<br/>pricing table + compute cost
Cost->>Cost: lookup pricing.yaml + compute cost
Cost->>Rec: tokens + cost
Rec->>MgmtGrpc: RecordLLMUsage(provider, model, prompt_t, completion_t, cost, groups, user)
Rec-->>Log: emit access-log entry<br/>(if EnableLogCollection)

View File

@@ -15,10 +15,6 @@ Inside the package: `manager.go` is the CRUD + permissions-gated facade; `synthe
| ---- | ---- |
| `agentnetwork/manager.go` | Manager interface + CRUD + permission gates + bootstrap-settings + reconcile trigger |
| `agentnetwork/synthesizer.go` | Settings/policy → wire-format synthesis; sole writer of the proxy middleware chain |
| `agentnetwork/synthesizer_pricing.go` | `buildCostMeterConfigJSON` — default table + per-provider prices → `cost_meter` config |
| `agentnetwork/pricing/defaults.go` | Default pricing table derived from the catalog + supplementals; `DefaultTable`, `LookupDefault`, wire `Entry` |
| `agentnetwork/pricing/override.go` | `LoadFile`/`StartReloader` for `AgentNetwork.PricingDefaultsFile` (mtime poll, merge over compiled-in base) |
| `agentnetwork/pricing/{exampleyaml,gen}.go` | Generates `defaults_llm_pricing.example.yaml` from the compiled-in table (golden-tested) |
| `agentnetwork/policyselect.go` | Per-request policy attribution + account-budget ceiling (min-wins) |
| `agentnetwork/reconcile.go` | Per-account synth diff vs in-memory cache → Create/Update/Delete |
| `agentnetwork/catalog/catalog.go` | Static provider catalogue (auth headers, identity-injection shapes) |
@@ -52,8 +48,6 @@ flowchart TD
I --> J[indexProviderGroups: providerID -> sorted source groups]
J --> K[buildRouterConfigJSON drops orphan providers]
J --> L[buildIdentityInjectConfigJSON per catalog entry]
J --> K2[buildCostMeterConfigJSON: default table + per-provider prices]
K2 --> P
H --> M[mergeGuardrails: union allowlist, OR redact]
M --> N[applyAccountCollectionControls account toggle = SOLE capture control]
N --> O[marshalGuardrailConfig]
@@ -66,84 +60,6 @@ flowchart TD
R --> T[accountManager.UpdateAccountPeers — fans synth ACLs into network map]
```
### LLM pricing (management is the sole authority)
**The proxy carries no price list.** Management synthesizes the entire pricing
table and ships it inside `cost_meter`'s `ConfigJSON`, so a price change reaches
the proxies as an ordinary mapping push — the chain rebuild installs a fresh
table and there is nothing to reload on the proxy side.
```mermaid
flowchart TD
A[catalog.All — PricingSurfaces x Models] --> B[buildDefaultTable + supplementalDefaults]
B --> C{AgentNetwork.PricingDefaultsFile}
C -- absent --> D[compiled-in table serves]
C -- loaded --> E[LoadFile: merge file entries WHOLE over compiled base]
E --> F[mergedTable atomic.Pointer]
D --> G[DefaultTable]
F --> G
G --> H[buildCostMeterConfigJSON — pricing.defaults]
I[types.Provider.Models operator prices] --> J[normalizePricingModelID<br/>bedrock ARN/region/version, vertex @version]
J --> K[materializeEntry: default entry as base,<br/>operator input/output verbatim,<br/>cache pointers only when non-nil]
K --> L[pricing.providers keyed by provider record ID]
H --> M[cost_meter ConfigJSON]
L --> M
G --> N[GET /catalog — applyDefaultPricing prefills dashboard rows]
O[StartReloader: mtime poll every ReloadInterval 1m] --> E
```
**Two tiers, resolved per request on the proxy** (`synthesizer_pricing.go:22-35`):
- `pricing.defaults` — surface (`openai`/`anthropic`/`bedrock`) → normalized model
id → rates. The **full** default table ships to every account: it is small
(~10 KB) and it is what keeps gateway-style providers (which enumerate no
models, so they claim every model) priced.
- `pricing.providers` — provider **record** id → normalized model id → rates,
matched against the `llm.resolved_provider_id` the router stamps. Entries are
**fully materialized here**, at synth time: `materializeEntry` starts from the
default entry for that model so cache rates the operator didn't state are
inherited, overlays operator `input`/`output` verbatim (**including an explicit
0**, which prices a self-hosted or internal endpoint as free rather than
silently reverting to list price), and overlays cache-rate **pointers only when
non-nil** — `nil` means "inherit the default", an explicit `0` means "no
discount, bill this bucket at the input rate". The proxy therefore does two map
lookups and no merging.
Same orphan rule as the router: a provider no enabled policy authorises is
unreachable, so its prices aren't shipped. Model ids are normalized with the
**same** functions the request parser uses (`NormalizeBedrockModel` /
`NormalizeVertexModel`), which is what makes the per-record lookup key compare
equal to the `llm.model` the proxy meters. Post-normalization duplicates resolve
first-occurrence-wins, matching the routing dedup order.
**`AgentNetwork.PricingDefaultsFile`** (`config.go:190-207`) lets an operator
replace default rates without a rebuild. Schema is `surface → model → rates`
(`input_per_1k`, `output_per_1k`, and optional `cached_input_per_1k` /
`cache_read_per_1k` / `cache_creation_per_1k`). Semantics:
- A **relative** path resolves against `<Datadir>`, so a bare filename lands
alongside the store. Empty config probes `<Datadir>/defaults_llm_pricing.yaml`.
- An **explicitly configured** path is *required to load*: a typo or malformed
file fails startup, because the operator believes those rates are live. The
conventional probe is optional — an absent file just serves compiled-in
defaults, and the path stays watched in case it appears later.
- File entries **replace** the compiled-in entry for the same (surface, model)
**whole** — they are not field-merged, so an entry must repeat the cache rates
it wants to keep. Everything the file doesn't mention keeps built-in rates.
- Unknown YAML fields are rejected (`KnownFields(true)`) and every rate must be
finite and non-negative — the same constraints the HTTP API enforces on
operator per-provider prices.
- Reload is an mtime poll (`ReloadInterval`, 1 min) and is **lenient at runtime**:
a parse error keeps the previous table, a deleted file reverts to compiled-in
defaults. A mid-edit save can never take pricing down.
The live table feeds **both** consumers, which is what keeps them consistent: the
synthesizer (what proxies actually bill with) and `GET /api/agent-network/catalog`
via `applyDefaultPricing` (what the dashboard's model-row prices prefill with).
`defaults_llm_pricing.example.yaml` is generated from the compiled-in table
(`go generate ./management/internals/modules/agentnetwork/pricing`) and
golden-tested, so operators start from a file matching the built-in rates exactly.
### Budget rule resolution (min-wins, group+user bound)
```mermaid
@@ -208,7 +124,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
| on_request | 3 | `llm_identity_inject` | `{"providers":[{provider_id, header_pair?, json_metadata?, extra_headers?}]}` | **true** |
| on_request | 4 | `llm_guardrail` | `{"provider_allowlists"?: {providerID: []model}, "prompt_capture":{enabled,redact_pii}}` | |
| on_response | 5 | `llm_limit_record` | `{}` (runs LAST at runtime) | |
| on_response | 6 | `cost_meter` | `{"pricing":{"defaults":{surface:{model:rates}},"providers"?:{providerRecordID:{model:rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}` | |
| on_response | 6 | `cost_meter` | `{}` | |
| on_response | 7 | `llm_response_parser` | `{"capture_completion": <bool>, "redact_pii"?: true}` | |
- **Synthesized service shape** (`synthesizer.go:739`): `Mode=HTTP`, `Private=true`, `Domain=<subdomain>.<cluster>`, `AccessGroups=unionSourceGroups(enabledPolicies)`, one `TargetTypeCluster` target with `Host=noop.invalid:443` (router rewrites per request), `Options.{DirectUpstream,AgentNetwork}=true`, `DisableAccessLog=!settings.EnableLogCollection`, `CaptureMax{Req,Resp}Bytes=1<<20`, `CaptureContentTypes=["application/json","text/event-stream"]`.
@@ -223,12 +139,6 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Orphan providers (no enabled policy authorises them) NEVER reach the router** (`synthesizer.go:351-357`); skipped from `identity_inject` for symmetry.
- **Provider creation refuses empty `api_key`** (`manager.go:175`); **deletion refuses while any policy still references it** (`manager.go:265-273`).
- **Session keypair stability across provider edits** (`manager.go:226-228`) — server-managed, copied through every `UpdateProvider`, never API-surfaced.
- **Management is the sole pricing authority.** The proxy has no embedded price list, so an account whose `cost_meter` config carries no `pricing` block bills **nothing** (`cost.skipped=unknown_model`, $0) rather than falling back to stale built-ins. The top-level `pricing` wrapper is also the feature-detection signal in both directions: an old proxy ignores it as an unknown field, and a new proxy reads its absence as "old management".
- **Per-provider prices are materialized at synth time, not merged on the proxy** (`synthesizer_pricing.go:114-131`). A per-record entry is always complete, so the proxy's lookup is per-record-then-defaults with no field-level fallback between tiers.
- **An explicit operator price of `0` prices the model as free** — it must not be treated as "unset" and reverted to list price (`synthesizer_pricing.go:49-54`). Only *cache*-rate fields distinguish unset from zero, via `*float64`.
- **Pricing model ids are normalized with the same functions the request parser uses** (`normalizePricingModelID`). If the two ever diverge, per-record prices silently stop matching and every request falls through to surface defaults.
- **The default table's coverage is structural, not curated.** It is derived from the catalog via each provider's `PricingSurfaces`; `TestDefaultTable_CoversEveryCatalogModel` fails on an unpriced catalog model and `TestDefaultTable_NoConflictingContributions` fails if two providers contribute the same (surface, model) at different rates.
- **A pricing-defaults file failure is fatal only at startup, and only for an explicitly configured path.** Runtime reload failures keep the previous table; a deleted file reverts to compiled-in defaults (`pricing/override.go:62-81, 113-148`).
## Things to scrutinize
@@ -266,12 +176,10 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Capture-pointer semantics (restated):** non-agent-network callers see no field → legacy nil-default emit, identical to pre-PR. Agent-network targets always carry an explicit `capture_*` value.
- **`TestSynthesizeServices_HappyPath` was updated:** request-parser config moved from `{}` to `{"capture_prompt":false}` (`synthesizer_test.go:174`). External snapshot tests against synth output need updating.
- **`MergedGuardrails` retains zeroed `TokenLimits`/`Budget`/`Retention`** even though `Policy.Limits` carries the real values now; `llm_limit_check` is the authoritative enforcement. Comment at `synthesizer.go:940-948` calls this out.
- **`cost_meter`'s `pricing` block is version-skew-safe in both directions.** A proxy predating config-delivered pricing ignores the field as unknown JSON (it previously priced from its own embedded table, so it keeps billing — at its own rates, which is the skew to be aware of during a rolling upgrade). A current proxy paired with old management sees no `pricing` block, logs one warning at chain-build time, and records `cost.skipped=unknown_model` — token counting and cap enforcement are unaffected, only the USD annotation goes to $0.
### Performance
- **`SynthesizeServices` runs on every controller tick / mutation reconcile.** Cost: 4 store reads + optional per-provider keypair backfill. Sort + index + merge are O(N log N) / O(P × G); dominant cost is JSON marshalling. No nested loops escape these dimensions.
- **The full default pricing table is marshalled into every account's `cost_meter` config on every synth** (~10 KB serialized). This is a deliberate trade: it keeps gateway-style providers priced for every catalog model, and it is the largest single contributor to the synth JSON. `DefaultTable()` itself is a pointer load (or a `sync.Once`-built map) — the cost is the marshal, not the build.
- **`reconcile.diffMappings` is O(N + M)** with N=M=1 per account today — effectively constant.
- **`SynthesizeServicesForCluster`** (`synthesizer.go:71`) walks every account on a cluster; per-account failures are **swallowed** (`synthesizer.go:91-93`) so a single misconfigured account doesn't drop the cluster. Runs per proxy reconnect.
@@ -280,7 +188,6 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
- **Activity codes:** `AgentNetwork{Provider,Policy,Guardrail,BudgetRule}{Created,Updated,Deleted}`; `AgentNetworkSettingsUpdated` with `log_collection/prompt_collection/redact_pii` payload (`manager.go:567-571`). **No activity code for `SelectPolicyForRequest` denies** — surfaced via proxy access log only (likely intentional given volume).
- **Deny codes** namespaced: `llm_policy.{token,budget}_cap_exceeded`, `llm_account.{token,budget}_cap_exceeded` (`policyselect.go:18-26`).
- **Reconcile failures are logged at warn and swallowed** (`reconcile.go:42-44`). Persistent synth failures (e.g. unknown catalog id) silently keep the proxy out of sync — consider a manager-level synth-health surface if this becomes a support burden.
- **Pricing-file lifecycle logs at info** (load, reload, revert-to-built-ins) and **at warn** for a runtime reload failure; the mtime check itself is `Debugf`. There is no metric on reload failures, so an operator who breaks the file mid-flight keeps billing at the previous table with only a log line to show it (`pricing/override.go:113-148`).
## Test coverage
@@ -291,9 +198,6 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
| `synthesizer_guardrail_realstore_test.go` | `PromptCaptureAccountIsSoleControl`; `PromptCaptureFlowsWhenAccountOptsIn`; `AccountRedactWithoutGuardrailRedact`; `NoGuardrail_CaptureOff`. |
| `synthesizer_log_collection_realstore_test.go` | `LogCollection{Off_SuppressesAccessLog,On_PermitsAccessLog}` — verifies `DisableAccessLog` propagation through `ToProtoMapping`. |
| `synthesizer_parser_redact_realstore_test.go` | **Capture-pointer regression suite:** `ParserConfigsCarryRedactPii`; `ParserConfigsSuppressCaptureWhenLogCollectionOnly` (log=on/prompt=off ⇒ both capture flags false); `ParserConfigsOmitRedactPiiWhenOff`. |
| `synthesizer_pricing_test.go` | `BuildCostMeterConfig_{BedrockModelNormalization,CacheRateNilVsZero,OrphanAndGatewayProviders}` — the per-record tier's three load-bearing rules: keys normalized like the parser's, `nil` cache pointer inherits vs explicit `0` bills at input rate, and orphan / gateway (empty `Models`) providers ship no per-record entry. |
| `pricing/defaults_test.go` | `DefaultTable_{CoversEveryCatalogModel,NoConflictingContributions,AllRatesFiniteNonNegative,PinnedRates}`; `LookupDefault_SurfaceOrder`. Catalog-derived coverage + rate sanity are structural, not curated. |
| `pricing/override_test.go` | `LoadFile_{MergesOverCompiledDefaults,MissingPath,RejectsInvalid}`; `Reload_LifeCycle` (mtime detect, parse error keeps previous, delete reverts to built-ins); `ExampleYAML_InSyncWithBuiltins` golden. |
| `policyselect_test.go` | Mock-store: `NoApplicablePolicies`; `AllowWithLowestGroupAttribution`; `LargerPoolWinsAcrossUsageLevels`; `StaysOnLargerPoolAfterPartialDrain`; `FallsThroughToSmallerPoolWhenLargerExhausted`; `TiebreakBy{LargerGroupPool,CreatedAt}`; `DeniesWhenAllExhausted`; `UncappedPolicyAlwaysWinsAgainstCapped`; `DisabledPolicyIgnored`; `StoreErrorPropagates`; `RejectsEmptyAccount`; `SharesGroupCounterAcrossPolicies`; `AntiFallThroughOnLowestGroup`; `BudgetOnlyExhaustionDenies`; `BudgetTighterThanTokenWins`. |
| `policyselect_realstore_test.go` | Real-sqlite regression guard: `NoApplicablePolicies`; `AllowAndLowestGroupAttribution`; `LargerPoolWins_FallsThroughWhenExhausted`; `BudgetCapDenies`; `GroupCounterSharedAcrossPolicies`; `DisabledPolicyIgnored`. |
| `policyselect_account_realstore_test.go` | Account budget rules: `AccountCeilingBindsEvenWithUncappedPolicy` (min-wins); `AccountGroupCeiling`; `AccountTargetUsersBindsOnlyThatUser`; `AccountRuleRecordsToOwnWindow`. |

View File

@@ -5,7 +5,7 @@ LLM request. The two highest-blast-radius areas are the **capture-pointer
semantics** and the **limit_check ⇒ limit_record** record-once invariant.
Sibling module: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — the SDK
adapters + pricing table and cost formula this chain delegates to.
adapters + pricing catalog this chain delegates to.
---
@@ -34,7 +34,7 @@ rewrites.
| `llm_identity_inject` | OnRequest | `llm.{resolved_provider_id,authorising_groups}`, `Input.{UserEmail,UserID,UserGroups,UserGroupNames}` | none | header strip/inject + optional body rewrite |
| `llm_guardrail` | OnRequest | `llm.{model,request_prompt_raw}` | `llm_policy.{decision,reason}`, `llm.request_prompt` | none (model allowlist deny) |
| `llm_response_parser` | OnResponse | `llm.provider`, `Input.{RespHeaders,RespBody,Status}` | `llm.{input,output,total,cached_input,cache_creation}_tokens`, `llm.response_completion` | none |
| `cost_meter` | OnResponse | `llm.{provider,model,resolved_provider_id}`, token buckets | `cost.usd_{input,cached_input,cache_creation,output,total,cache}` or `cost.skipped` | none (in-memory pricing lookup) |
| `cost_meter` | OnResponse | `llm.{provider,model}`, token buckets | `cost.usd_total` or `cost.skipped` | pricing lookup |
| `llm_limit_record` | OnResponse | `llm.{attribution_group_id,attribution_window_seconds,input_tokens,output_tokens}`, `cost.usd_total` | none | gRPC `RecordLLMUsage` |
[all_test.go:2640](../../../proxy/internal/middleware/builtin/all_test.go)
@@ -44,7 +44,7 @@ locks the ID set; adding or removing one is a conscious extension.
| File | LOC | Notes |
|---|---:|---|
| `builtin.go` | 90 | Registry + `FactoryContext` (ctx, meter, logger, mgmt client) |
| `builtin.go` | 86 | Registry + `FactoryContext` (ctx, data dir, meter, logger, mgmt client) |
| `all_test.go` | 41 | Locks the 8-ID registry surface |
| `agentnetwork_chain_integration_test.go` | 319 | Live sqlite + real gRPC bufconn; gate→recorder wire path |
| `llm_request_parser/*` | 162 / 66 / 356 | Provider detection, body parse, prompt extraction with capture-pointer gating |
@@ -53,7 +53,7 @@ locks the ID set; adding or removing one is a conscious extension.
| `llm_identity_inject/*` | 440 / 108 / 666 | HeaderPair (LiteLLM) + JSONMetadata (Portkey) + ExtraHeaders |
| `llm_guardrail/*` | 176 / 82 / 75 / 219 / 217 | Model allowlist + optional prompt capture with PII redaction |
| `llm_response_parser/*` | 258 / 222 / 43 / 433 / 169 / 111 | Buffered + SSE accumulation; AWS event-stream accumulator (`streaming_bedrock.go`) for Bedrock; capture-pointer gates completion emit |
| `cost_meter/*` | 236 / 98 / 586 | Token → USD via `proxy/internal/llm/pricing`; both pricing tiers arrive in the middleware config |
| `cost_meter/*` | 181 / 84 / 439 | Token → USD via `proxy/internal/llm/pricing` |
| `llm_limit_record/*` | 144 / 35 / 191 | Post-flight `RecordLLMUsage` (5s, debug-on-error) |
## Per-middleware
@@ -168,46 +168,12 @@ token schema.
### cost_meter
Reads `llm.provider` + `llm.model` + token buckets, looks up the per-1k rates,
and emits the full `cost.usd_*` breakdown (four per-bucket values plus the
`_total` and `_cache` aggregates) or a closed-set `cost.skipped` reason
(`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
`unknown_model`).
**Management owns pricing.** The proxy carries no embedded price list: the whole
table arrives in this middleware's `ConfigJSON` as
`{pricing: {defaults, providers}}`, synthesized by management from the catalog
plus the operator's stored per-provider prices
([factory.go:1334](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
Both tiers are validated by `pricing.NewTable` / `pricing.NewEntries` at
construction, so a non-finite or negative rate fails the chain build. A price
change is an ordinary mapping push — the chain rebuild yields a fresh instance
over a fresh immutable table, so there is no data dir, no pricing file, no
reload goroutine, and nothing to invalidate.
**Two-tier lookup**
([middleware.go:165183](../../../proxy/internal/middleware/builtin/cost_meter/middleware.go)):
1. **Per-provider-record** — the operator's stored price for the route that
actually served the request, keyed by the `llm.resolved_provider_id` that
`llm_router` stamped on the allow path, then by normalized model id. Entries
arrive fully materialized (management folds default cache rates in at synth
time), so there is no merging here. Absent metadata — no router in the chain
— skips this tier.
2. **Surface defaults** — the catalog-derived table keyed by `llm.provider`
(`openai`/`anthropic`/`bedrock`). This is also what prices gateway-style
providers, which enumerate no models and therefore get no per-record entry.
**Backward compatibility:** a config with no `pricing` block means management
predates config-delivered pricing. The factory logs one warning at build time
and the instance records `cost.skipped=unknown_model` ($0) for every request
rather than falling back to a stale built-in price list
([factory.go:5560](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
**Key invariant:** the provider-shape switch lives in `pricing.EntryCosts`
(sibling doc) and is selected by the **surface**, not by which tier the entry
came from — `cost_meter` stays provider-agnostic, and a per-record override on
an Anthropic route still bills its cache buckets additively.
Reads `llm.provider` + `llm.model` + token buckets, looks up per-1k rate via
`pricing.Loader`, emits `cost.usd_total` or a closed-set `cost.skipped`
reason (`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
`unknown_model`). Loader's hot-reload goroutine is bound to proxy-lifetime
context via `startReloader`. **Key invariant:** provider-shape switch lives
in `pricing.Table.Cost` (sibling doc) — `cost_meter` stays provider-agnostic.
### llm_limit_record
@@ -280,14 +246,12 @@ no mocks. Tests: `TestChain_AllowPath_StampsAttributionAndRecordsCounter`
| `llm_identity_inject` | `{providers: [{provider_id, header_pair?|json_metadata?, extra_headers?}]}` |
| `llm_guardrail` | `{provider_allowlists: {providerID: []string}, prompt_capture: {enabled, redact_pii}}` — allowlist keyed by resolved provider id; a provider absent from the map is unrestricted (fail-closed backstop; authoritative per-policy/group check is management's `CheckLLMPolicyLimits`) |
| `llm_response_parser` | `{redact_pii?, capture_completion?: *bool}` |
| `cost_meter` | `{pricing: {defaults: {surface: {model: rates}}, providers: {providerRecordID: {model: rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}`. A missing `pricing` key means "management predates config-delivered pricing": every request records `cost.skipped=unknown_model` |
| `cost_meter` | `{pricing_path?}` (basename inside data-dir; defaults `pricing.yaml`) |
| `llm_limit_record` | `{}` — same pattern as `llm_limit_check` |
All factories accept empty / null / `{}` / whitespace as zero-value config;
only structurally invalid JSON is rejected so misconfig surfaces at chain
build time. `cost_meter` adds a semantic check on top of that: a `pricing`
block carrying a negative or non-finite rate fails the build too, rather than
mispricing live traffic.
build time.
## Invariants
@@ -356,11 +320,10 @@ non-object `metadata` field
— header path still attributes, but body-level tag-budget enforcement
doesn't run for that request.
**Concurrency.** `cost_meter`'s two pricing tables are built once from the
middleware config and never mutated, so the lookup path needs no lock or atomic
swap — a price change replaces the whole instance. Every middleware is
otherwise a stateless value receiver. Integration test uses real bufconn gRPC —
race detector is the meaningful bar.
**Concurrency.** `cost_meter` shares a `pricing.Loader` via
`atomic.Pointer[Table]`; readers always see a consistent table. Every
middleware is a stateless value receiver. Integration test uses real bufconn
gRPC — race detector is the meaningful bar.
**Perf.** Hot path is `lookupKV` linear scan over <10 KVs; `cost_meter.Cost`
is O(1); SSE accumulation is single-pass. No map allocation per call.
@@ -386,13 +349,13 @@ counter accuracy.
| `llm_guardrail/redact_test.go` | 15 | Email, SSN, phone (E.164 + NA), bearer, IPv4; fixture-driven |
| `llm_response_parser/middleware_test.go` | 18 | Buffered OAI+Anthro, capture-pointer, redact, truncation |
| `llm_response_parser/streaming_test.go` | 7 | OAI usage frame, Anthro message_delta, truncated body best-effort |
| `cost_meter/middleware_test.go` | 22 | Each skip reason, provider-shape formulas, config-delivered defaults, per-record-beats-defaults + miss-falls-back, per-record uses surface formula, nil-pricing skips everything, invalid-rate rejection |
| `cost_meter/middleware_test.go` | 17 | Each skip reason, provider-shape, pricing loader integration |
| `llm_limit_record/middleware_test.go` | 7 | Skip-on-no-signal, skip-on-missing-attribution, RPC failure swallowed |
## Cross-references
- Sibling: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — SDK adapters
+ SSE framer + pricing table and cost formula.
+ SSE framer + pricing loader.
- Path-routed providers (Vertex AI + Bedrock), `keyfile::` credential, GCP
token minting, `/bedrock` prefix:
[50-path-routed-providers.md](./50-path-routed-providers.md).

View File

@@ -9,7 +9,7 @@ pricing table's per-provider cost formula is the highest-leverage place a
small bug would silently mis-bill operators.
Sibling module: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
— the 8 middlewares that consume this package's parsers + pricing table.
— the 8 middlewares that consume this package's parsers + pricing loader.
---
@@ -24,9 +24,8 @@ proxy-framework dependencies:
- `openai.go` / `anthropic.go` / `bedrock.go` — per-provider `Parser` impls.
- `sse.go` — SSE scanner (`Scanner`, `Event`, `NewScanner`).
- `errors.go` — sentinels callers branch on with `errors.Is`.
- `pricing/`immutable pricing table + the per-surface cost formula. The
rates themselves come from management inside `cost_meter`'s middleware
config; this package holds no price list and reads no files.
- `pricing/`embedded-default + hot-reload override table with
symlink-safe Unix loader (build-tagged stub elsewhere).
- `fixtures/` — captured request/response/stream bodies the tests replay.
The package carries zero proxy-framework dependencies so the same parsers can
@@ -48,9 +47,12 @@ be reused later by a WASM adapter
| `sse_test.go` | 175 | 12 tests; fixture replay + multiline + size limits |
| `parser_test.go` | 53 | `Parsers()`, `DetectParser`, provider enum values |
| `errors.go` | 31 | 6 sentinels: `Err{Unknown,Unsupported}Provider/Model`, `Err{NotLLM,Malformed}Response`, `ErrStreamingUnsupported`, `ErrMalformedRequest` |
| `pricing/pricing.go` | 234 | `Table`, `Entry`, `EntryJSON`, `Costs`; `NewTable`/`NewEntries` validation + `EntryCosts` formula. No I/O, no reload, no embedded rates |
| `pricing/pricing_test.go` | 177 | 10 tests — provider-shape formulas, cached clamp, rate fallback, nil-safety, rate validation |
| `fixtures/*` | 2159 | OAI chat/responses/stream + Anthro messages/stream |
| `pricing/pricing.go` | 421 | `Loader`, `Table`, `Entry`; embedded defaults + atomic swap + mtime reload |
| `pricing/pricing_unix.go` | 69 | `O_NOFOLLOW` + fstat-from-FD + 1 MiB cap |
| `pricing/pricing_other.go` | 21 | Stub returning "not supported on this platform" |
| `pricing/pricing_test.go` | 432 | 21 tests — symlink rejection, reload race, path traversal, oversize |
| `pricing/defaults_pricing.yaml` | 85 | go:embed source of truth |
| `fixtures/*` | 2159 | OAI chat/responses/stream + Anthro messages/stream + pricing starter |
## Request body → parser dispatch
@@ -186,11 +188,9 @@ response leg, covering both Bedrock body shapes:
`totalTokens`). `firstNonZero` folds the two naming conventions into one
`Usage`; when Converse omits `totalTokens` the parser sums the buckets.
`ProviderName()` returns `"bedrock"` — its own pricing surface in the table
management ships, keyed by the **normalised** model id (region prefix + version
suffix stripped by the request parser; management normalises its keys the same
way at synth time so the two compare equal). `ParseResponse` returns
`ErrStreamingUnsupported` for an
`ProviderName()` returns `"bedrock"` — its own `defaults_pricing.yaml` block,
keyed by the **normalised** model id (region prefix + version suffix stripped by
the request parser). `ParseResponse` returns `ErrStreamingUnsupported` for an
AWS binary event-stream content-type (`application/vnd.amazon.eventstream`,
`isAWSEventStream`) so the caller routes to the streaming accumulator instead.
@@ -205,34 +205,11 @@ response body. Streaming accumulators live in the middleware package
([llm_response_parser/streaming.go](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go))
but use `llm.NewScanner` so the framing contract stays here.
### Pricing table
### Pricing catalog
**Management is the sole pricing authority.** The proxy carries no embedded
price list and reads no pricing file: the whole table arrives inside
`cost_meter`'s `ConfigJSON` on the ordinary mapping push, and a price change
is just another push — the chain rebuild constructs a fresh `Table`, so there
is nothing to reload
([pricing.go:17](../../../proxy/internal/llm/pricing/pricing.go)). The
management side of the contract (catalog defaults, the operator's stored
per-provider prices, and `AgentNetwork.PricingDefaultsFile`) is covered in the
management-side module guide; `cost_meter`'s wire shape is in
[31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md).
`EntryJSON`
([pricing.go:3645](../../../proxy/internal/llm/pricing/pricing.go)) is the
management→proxy contract — five USD-per-1k rates under `input_per_1k`,
`output_per_1k`, `cached_input_per_1k`, `cache_read_per_1k`,
`cache_creation_per_1k`. Management's `pricing.Entry` marshals the identical
names, and `EntryJSON`/`Entry` are field-identical so `NewEntries` converts by
direct struct conversion rather than field-by-field copying (a new rate can't
be silently dropped in transit).
`EntryCosts`
([pricing.go:183234](../../../proxy/internal/llm/pricing/pricing.go))
is the cost formula — most security-relevant math in this module. The
**surface** (the `llm.provider` value the request parser stamped) selects the
formula, never the tier the entry came from: a per-provider-record override on
an Anthropic route still bills its cache buckets additively.
`Table.Cost`
([pricing.go:129174](../../../proxy/internal/llm/pricing/pricing.go))
is the cost formula — most security-relevant math in this module:
| Provider | Formula |
|---|---|
@@ -241,7 +218,7 @@ an Anthropic route still bills its cache buckets additively.
| default | `inTokens × InputPer1K + outTokens × OutputPer1K` |
`bedrock` shares the Anthropic additive-cache formula
([pricing.go:214229](../../../proxy/internal/llm/pricing/pricing.go)):
([pricing.go:172-174](../../../proxy/internal/llm/pricing/pricing.go)):
Anthropic-on-Bedrock reports the same additive cache buckets, while non-Anthropic
Bedrock models (Nova, Llama) simply report zero in those buckets so cost reduces
to `input + output`.
@@ -249,12 +226,15 @@ to `input + output`.
Each per-bucket rate falls back to `InputPer1K` when zero — operators opt in
to discounts by setting the field.
`Costs`
([pricing.go:143163](../../../proxy/internal/llm/pricing/pricing.go)) is the
per-request split. The four per-bucket fields are the base; `TotalUSD` and
`CacheUSD` are **derived** in `newCosts` so the aggregates can never drift from
the breakdown. `InputUSD` is always the non-cached input bucket on both
provider shapes, so input and cached-input never double-count.
`Loader`
([pricing.go:212268](../../../proxy/internal/llm/pricing/pricing.go))
overlays an optional `pricing.yaml` from data-dir on top of the go:embed
defaults. Atomic pointer swap means readers never observe a partial update.
The mtime-poll reloader (30s default cadence) keeps the previous table on
parse failure so cost annotation never goes blank during a botched edit.
`defaults_pricing.yaml` is the source of truth for built-in pricing.
Operator overrides only carry the entries they want to change.
## Public contracts
@@ -284,38 +264,29 @@ Order matters: `DetectFromURL` ties resolve by registration order.
`ProviderBedrock = 3`. Numeric values are persisted in nothing today but treat
them as wire-stable — new providers must take fresh numbers.
**`Pricing` construction + lookup**
([pricing.go:60130](../../../proxy/internal/llm/pricing/pricing.go)):
**`Pricing` lookup**
([pricing.go:129](../../../proxy/internal/llm/pricing/pricing.go)):
```go
func NewEntries(raw map[string]map[string]EntryJSON) (map[string]map[string]Entry, error)
func NewTable(raw map[string]map[string]EntryJSON) (*Table, error)
func (t *Table) Lookup(provider, model string) (Entry, bool)
func (t *Table) Cost(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (float64, bool)
func (t *Table) Costs(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (Costs, bool)
func EntryCosts(entry Entry, surface string, inTokens, outTokens, cachedInput, cacheCreation int64) Costs
```
`NewTable` is the surface-keyed defaults table; `NewEntries` returns the raw
two-level map `cost_meter` uses for the per-provider-record tier (it looks up an
`Entry` directly and calls `EntryCosts`, so it needs no `Table` wrapper). Both
reject any non-finite or negative rate, so a corrupt config fails the chain
build rather than mispricing silently. Nil input yields an empty,
never-matching table.
Nil-safe: `t.Cost`/`t.Lookup` on a nil receiver returns `ok=false`
([pricing.go:9699](../../../proxy/internal/llm/pricing/pricing.go)).
`ok=false` means the surface or model is absent from the table management sent;
the caller emits `cost.skipped=unknown_model`.
Nil-safe: `t.Cost` on a nil receiver returns `(0, false)`
([pricing.go:130132](../../../proxy/internal/llm/pricing/pricing.go)).
`ok=false` means provider or model is absent from the loaded table; the caller
emits `cost.skipped=unknown_model`.
## Invariants
1. **The pricing package is pure and platform-independent.** No file I/O, no
`//go:embed`, no goroutines, no build tags — the rates arrive as config, so
there is nothing platform-specific left to port. Anything reintroducing a
read-from-disk path here re-splits pricing authority between management and
the proxy, which is exactly what this design removed.
1. **Cross-platform pricing build.** `pricing_unix.go` carries the only
functional `loadPricing` (uses `syscall.O_NOFOLLOW` and `f.Stat()` on an
open descriptor — both Unix-only). `pricing_other.go` is a build-tag
fallback that returns `"not supported on this platform"`
([pricing_other.go:1416](../../../proxy/internal/llm/pricing/pricing_other.go)).
The proxy is Linux-only in production today; a Windows port needs an
equivalent path-as-handle implementation. Reviewers building on Windows
should expect this surface to return an error at startup if an override
file is configured.
2. **SSE scanner handles partial chunks.** A buffered prefix that doesn't end
in `\n\n` still yields its accumulated event before `io.EOF`
@@ -327,45 +298,38 @@ the caller emits `cost.skipped=unknown_model`.
usage rather than aborting
([streaming.go:6873, 144150](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go)).
3. **Management is the only source of rates.** `Table` has no constructor that
invents prices: the only way in is `NewTable`/`NewEntries` over the wire map
management sent. A missing or empty `pricing` block therefore means *no
prices at all* (`cost_meter` records `cost.skipped=unknown_model`, $0) —
never a stale built-in fallback that would silently bill list price.
3. **`defaults_pricing.yaml` is the source of truth.** Compiled into the
binary via `//go:embed`
([pricing.go:2930](../../../proxy/internal/llm/pricing/pricing.go)).
`DefaultTable()` parses once and panics on parse failure
([pricing.go:4249](../../../proxy/internal/llm/pricing/pricing.go))
— by design: a broken embedded YAML must not ship to production.
4. **Tables are immutable once built.** `Table.entries` is written only in
`NewEntries` and never mutated afterwards, and `cost_meter`'s `perRecord`
map is likewise build-time-only
([pricing.go:4752](../../../proxy/internal/llm/pricing/pricing.go)). This
is what makes the no-reload design safe: a price change arrives as a mapping
push that builds a new middleware instance over a new table, so concurrent
readers can't observe a half-updated price list and no atomic swap or lock
is needed on the hot path.
4. **Loader path validation.** `resolveMiddlewareDataPath`
([pricing.go:370394](../../../proxy/internal/llm/pricing/pricing.go))
rejects absolute paths, traversal segments, and basenames that fail
`basenameRegex = ^[a-zA-Z0-9._-]+$`. The resolved path must remain
inside `baseDir` even after `filepath.Clean`. Tests:
`TestNewLoader_PathValidation`, `TestNewLoader_PathValidation_Extended`,
`TestNewLoader_SymlinkOutsideBaseDirRejected`, `TestNewLoader_SymlinkRejected`.
5. **Rate validation happens at chain-build time, not per request.**
`NewEntries` rejects negative, NaN, and ±Inf rates field by field
([pricing.go:6083](../../../proxy/internal/llm/pricing/pricing.go)), naming
the offending surface/model/field in the error. Management enforces the same
constraints at its API boundary and in its YAML parser, so this is
defense-in-depth — but it means a corrupt push fails loudly at build instead
of producing negative costs on live traffic. Test:
`TestNewTable_ValidatesRates`.
5. **Unix loader symlink safety.** `O_NOFOLLOW` on open, `f.Stat()` on the
open descriptor (never re-stat by path), `info.Mode().IsRegular()` check,
`io.LimitReader(f, maxPricingBytes+1)` with a final size assertion
([pricing_unix.go:2557](../../../proxy/internal/llm/pricing/pricing_unix.go)).
A mid-read symlink swap is detected because the fstat is on the original
fd. Test: `TestNewLoader_RejectsOversizedFile_FixesM4`.
6. **New rates must be added to `Entry`, `EntryJSON`, *and* management's
`pricing.Entry` together.** `NewEntries` converts by direct struct
conversion `Entry(e)`
([pricing.go:7678](../../../proxy/internal/llm/pricing/pricing.go)), which
only compiles while the two structs stay field-identical — so the proxy half
is compiler-enforced. The management half is not: a rate added there but not
here unmarshals into nothing and prices that bucket at `InputPer1K`.
6. **`yaml.NewDecoder(...).KnownFields(true)`**
([pricing.go:397398](../../../proxy/internal/llm/pricing/pricing.go))
rejects YAML files that carry fields not in the schema. A typo in an
operator override file fails loud instead of silently zeroing rates.
## Things to scrutinise
**Correctness.** Verify the OpenAI cached-prompt clamp at
[pricing.go:203206](../../../proxy/internal/llm/pricing/pricing.go)
short-circuits before subtraction. Negative token counts are clamped to zero up
front ([pricing.go:186197](../../../proxy/internal/llm/pricing/pricing.go)) so
no formula can yield a negative cost. `Anthropic.TotalTokens` sums all four
**Correctness.** Verify OpenAI cached-prompt clamp at
[pricing.go:147149](../../../proxy/internal/llm/pricing/pricing.go)
short-circuits before subtraction. `Anthropic.TotalTokens` sums all four
buckets (in + out + cache_read + cache_creation) — downstream dashboards
need to know this differs from `input + output`.
`OpenAIParser.ExtractPrompt` falls through `messages → input → prompt`; a
@@ -374,27 +338,22 @@ noting).
**Security.** `Scanner.maxLine = 1 MiB`; a 2 MiB single-line `data:` event
errors from `Scanner.Next` and both accumulators stop with partial usage.
Pricing is no longer file-backed, so the loader's path-traversal / symlink /
oversize surface is gone entirely — the config channel (an authenticated
mapping push from management) is now the only way rates enter the proxy, and
`NewEntries` is the validation boundary on it. A new rate added to management's
`pricing.Entry` but not to `EntryJSON` here is the remaining silent-mispricing
path (see invariant 6).
Pricing file 1 MiB cap is orders of magnitude larger than realistic. Confirm
new schema additions are mirrored in both `pricingFile` and `Entry`;
`KnownFields(true)` will reject silently-typo'd operator overrides
otherwise.
**Concurrency.** Nothing in this package is shared mutable state: tables are
built once and never written again, so `cost_meter`'s hot path is lock-free by
construction rather than by atomic swap. Per-call `Scanner` instances mean no
shared state across concurrent response-parser calls.
**Concurrency.** `Loader.table` is `atomic.Pointer[Table]`; readers never
block or see a torn table. `Loader.Reload` is one goroutine, cancelled via
context (`TestLoader_ReloadBackgroundLoopCancellation`). `DefaultTable()`
uses `sync.Once`. Per-call `Scanner` instances mean no shared state across
concurrent response-parser calls.
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1); the
per-provider-record tier adds at most one more lookup. `Scanner.Next` is one
`ReadString('\n')` per line. No background goroutines and no per-request
allocation of pricing state.
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1).
`Scanner.Next` is one `ReadString('\n')` per line. Pricing reload poll 30s.
**Observability.** A config carrying no `pricing` block logs one warning at
chain-build time (`cost_meter` factory) and then records
`cost.skipped=unknown_model` per request, so an old-management deployment is
visible in both logs and the access log rather than quietly reporting $0.
**Observability.** Reload failures count via `metric.Int64Counter` keyed
`plugin`; warning log rate-limited at 5 min so a broken file doesn't flood.
Parser errors return sentinels — middleware uses `errors.Is` to map to the
right `cost.skipped` reason.
@@ -406,7 +365,7 @@ right `cost.skipped` reason.
| `openai_test.go` | 11 | Chat Completions + Responses API + legacy `prompt`; cached-tokens subset for both naming conventions; fixture replays |
| `anthropic_test.go` | 7 | Messages + legacy `/v1/complete`; streaming REJECTED on `ParseResponse` (must use scanner); fixture replays |
| `sse_test.go` | 12 | Fixture replay both providers; multiline `data:`; CRLF; comment skip; trailing-event-without-blank-line; oversize rejection |
| `pricing/pricing_test.go` | 10 | Provider-shape switch (surface selects the formula); cached-rate + cache-read/creation fallback to `InputPer1K`; cached-clamp; negative-token clamp; nil-receiver safety; rate validation (negative / NaN / Inf rejected); nil + empty table |
| `pricing/pricing_test.go` | 21 | Provider-shape switch; cached-rate fallback; cached-clamp; symlink rejection (target outside basedir + symlink to file); path validation matrix; oversize rejection; reload-keeps-previous-on-parse-error; mtime change detection; goroutine cancellation |
**Fixtures** ([proxy/internal/llm/fixtures/](../../../proxy/internal/llm/fixtures/)):
`openai_chat_completion.json` (chat.completions with usage),
@@ -414,15 +373,14 @@ right `cost.skipped` reason.
`openai_stream.txt` (3 deltas + usage + `[DONE]`),
`anthropic_messages.json` (Messages API non-streaming),
`anthropic_stream.txt` (full 7-event sequence: message_start →
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop).
No pricing fixture: the table is config-delivered, so pricing tests construct
it in-process from a wire-shape map.
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop),
`pricing.yaml` (realistic-pricing starter for operator overrides).
## Cross-references
- Sibling: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
— the chain that calls `llm.Parsers()`, `llm.ParserByName`,
`llm.NewScanner`, `pricing.NewTable` / `pricing.NewEntries`.
`llm.NewScanner`, `pricing.NewLoader`.
- Path-routed providers (Vertex AI + Bedrock), credential syntax, and the
Bedrock AWS event-stream accumulator:
[50-path-routed-providers.md](./50-path-routed-providers.md).

View File

@@ -1,7 +1,7 @@
# proxy/runtime — translate + serve + log
> **Risk level:** High — every config push from management is translated here, and the chain runs on every HTTP request to a synth target.
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path. Middleware config is entirely wire-delivered — no proxy-side data dir is involved, including for LLM pricing, which management ships inside `cost_meter`'s config.
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareDataDir`, `MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path.
## Module boundary
@@ -114,7 +114,8 @@ At **request time** the access-log middleware stamps `CapturedData`; the auth ch
## Public contracts touched
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:249-253). There is no `MiddlewareDataDir`: no built-in middleware reads config from disk, so `builtin.FactoryContext` carries only the proxy-lifetime context, meter, logger, and management client.
- `proxy.Server.MiddlewareDataDir` (string) — base dir for file-backed middleware config (server.go:238-241).
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:248-250).
- `proxy/internal/proxy.WithMiddlewareManager(*middleware.Manager) Option` — new option on `NewReverseProxy`; nil keeps the fast path (reverseproxy.go:48-56).
- `proxy/internal/proxy.PathTarget` adds `Middlewares`, `CaptureConfig`, `AgentNetwork`, `DisableAccessLog` (servicemapping.go:27-51), all zero-default.
- `proxy/internal/proxy.CapturedData` adds `agentNetwork`, `suppressAccessLog`, `userGroupNames` behind `sync.RWMutex`; slices deep-copied (context.go:47-66, 183-258).

View File

@@ -87,9 +87,9 @@ strips the `@version` suffix from the model, and maps the publisher to a parser
surface via `vertexPublisherVendor`:
- `anthropic``llm.provider="anthropic"` → metered through the Anthropic
parser, priced under the **`anthropic`** surface of the pricing table
management ships (the parser emits the standard Anthropic provider label, so
Vertex Claude reuses first-party Anthropic prices).
parser, priced under the **`anthropic`** block in `defaults_pricing.yaml`
(the parser emits the standard Anthropic provider label, so Vertex Claude
reuses first-party Anthropic prices).
- `openai``llm.provider="openai"` (reserved; not in the catalog lineup
today).
- anything else (notably `google` / Gemini) → empty vendor → **no parser**.
@@ -104,9 +104,8 @@ is omitted from the catalog.
> Caveat: cross-region inference profiles in `eu` / `apac` carry a ~10% price
> premium that the base per-token rates do **not** model — cost annotations for
> those regions read low. Operators who need exact regional billing set the
> affected models' prices on the provider record, or replace the default entries
> via management's `AgentNetwork.PricingDefaultsFile`.
> those regions read low. Operators who need exact regional billing override
> the affected entries in `pricing.yaml`.
## AWS Bedrock (`bedrock_api`)
@@ -212,19 +211,15 @@ so a model-listing call can't be rewritten onto an upstream that would 404 it.
## Catalog ↔ pricing cross-check
Catalog prices and context windows are cross-checked against LiteLLM's
`model_prices_and_context_window.json`. The **catalog is the source of default
prices**: management's `pricing.DefaultTable` folds every catalog provider's
models into the surfaces that provider declares (`PricingSurfaces`), so coverage
is structural rather than maintained in a parallel file
([pricing/defaults.go](../../../management/internals/modules/agentnetwork/pricing/defaults.go)).
`TestDefaultTable_CoversEveryCatalogModel` fails if a catalog model ends up
unpriced, and `TestDefaultTable_NoConflictingContributions` fails if two
providers contribute the same (surface, model) at different rates. Bedrock
entries are keyed by the **normalised** id the request parser emits (region
prefix + version suffix stripped) — management applies the same normalisation to
per-provider prices at synth time, so the two keys compare equal. Vertex Claude
carries no Bedrock-style prefix, so it prices straight off the `anthropic`
surface.
`model_prices_and_context_window.json`. The proxy's embedded
`defaults_pricing.yaml` covers **every metered first-party model** the catalog
enumerates — guarded by
`TestDefaultTable_FirstPartyModelCoverage`
([pricing/defaults_coverage_test.go](../../../proxy/internal/llm/pricing/defaults_coverage_test.go)),
which fails if a catalog model has no embedded price. Bedrock entries are keyed
by the **normalised** id the request parser emits (region prefix + version
suffix stripped). Vertex Claude carries no Bedrock-style prefix, so it prices
straight off the `anthropic` block.
## Things to scrutinise
@@ -237,17 +232,16 @@ operator-misconfigured Vertex provider and unmetered Gemini traffic; verify
publishers).
**Correctness.** `normalizeBedrockModel` is the join between the wire id and the
pricing key — a model that normalises to something absent from the shipped
pricing table meters at `cost.skipped=unknown_model` rather than failing the
request. The
pricing key — a model that normalises to something not in `defaults_pricing.yaml`
meters at `cost.skipped=unknown_model` rather than failing the request. The
`/bedrock` prefix strip must run on both the parser side (so the model is
extracted) and the router side (so the upstream path is native); a regression in
either silently breaks the other.
**Metering caveats.** eu/apac cross-region Bedrock + Vertex profiles carry a
~10% premium not modelled by base pricing — flagged in the catalog comment.
Operators needing exact regional billing set per-provider prices on the model
rows (or replace the default entries via `AgentNetwork.PricingDefaultsFile`).
~10% premium not modelled by base pricing — flagged in both the catalog comment
and `defaults_pricing.yaml`. Operators needing exact regional billing override
the relevant entries.
## Cross-references

View File

@@ -6,7 +6,7 @@
"name": "NetBird GmbH",
"email": "hello@netbird.io",
"phone": "",
"description": "NetBird GmbH is a Berlin-based software company specializing in the development of open source network security solutions. Network security is utterly complex and expensive, accessible only to companies with multi-million dollar IT budgets. In contrast, there are millions of companies left behind. Our mission is to create an advanced network and cybersecurity platform that is both easy-to-use and affordable for teams of all sizes and budgets. By leveraging the open source strategy and technological advancements, NetBird aims to set the industry standard for connecting and securing IT infrastructure.",
"description": "NetBird GmbH is a Berlin-based software company specializing in the development of open-source network security solutions. Network security is utterly complex and expensive, accessible only to companies with multi-million dollar IT budgets. In contrast, there are millions of companies left behind. Our mission is to create an advanced network and cybersecurity platform that is both easy-to-use and affordable for teams of all sizes and budgets. By leveraging the open-source strategy and technological advancements, NetBird aims to set the industry standard for connecting and securing IT infrastructure.",
"webpageUrl": {
"url": "https://github.com/netbirdio"
}
@@ -15,7 +15,7 @@
{
"guid": "netbird",
"name": "NetBird",
"description": "NetBird is a configuration-free peer-to-peer private network and a centralized access control system combined in a single open source platform. It makes it easy to create secure WireGuard-based private networks for your organization or home.",
"description": "NetBird is a configuration-free peer-to-peer private network and a centralized access control system combined in a single open-source platform. It makes it easy to create secure WireGuard-based private networks for your organization or home.",
"webpageUrl": {
"url": "https://github.com/netbirdio/netbird"
},
@@ -59,7 +59,7 @@
"guid": "support-yearly",
"status": "active",
"name": "Support Open Source Development and Maintenance - Yearly",
"description": "This will help us partially cover the yearly cost of maintaining the open source NetBird project.",
"description": "This will help us partially cover the yearly cost of maintaining the open-source NetBird project.",
"amount": 100000,
"currency": "USD",
"frequency": "yearly",
@@ -72,7 +72,7 @@
"guid": "support-one-time-year",
"status": "active",
"name": "Support Open Source Development and Maintenance - One Year",
"description": "This will help us partially cover the yearly cost of maintaining the open source NetBird project.",
"description": "This will help us partially cover the yearly cost of maintaining the open-source NetBird project.",
"amount": 100000,
"currency": "USD",
"frequency": "one-time",
@@ -85,7 +85,7 @@
"guid": "support-one-time-monthly",
"status": "active",
"name": "Support Open Source Development and Maintenance - Monthly",
"description": "This will help us partially cover the monthly cost of maintaining the open source NetBird project.",
"description": "This will help us partially cover the monthly cost of maintaining the open-source NetBird project.",
"amount": 10000,
"currency": "USD",
"frequency": "monthly",
@@ -98,7 +98,7 @@
"guid": "support-monthly",
"status": "active",
"name": "Support Open Source Development and Maintenance - One Month",
"description": "This will help us partially cover the monthly cost of maintaining the open source NetBird project.",
"description": "This will help us partially cover the monthly cost of maintaining the open-source NetBird project.",
"amount": 10000,
"currency": "USD",
"frequency": "monthly",

View File

@@ -228,6 +228,8 @@ read_enable_crowdsec() {
echo "CrowdSec checks client IPs against a community threat intelligence database" > /dev/stderr
echo "and blocks known malicious sources before they reach your services." > /dev/stderr
echo "A local CrowdSec LAPI container will be added to your deployment." > /dev/stderr
echo "It also enables the AppSec (WAF) endpoint, so services can inspect HTTP" > /dev/stderr
echo "requests for exploits. Both stay off per service until you enable them." > /dev/stderr
echo -n "Enable CrowdSec? [y/N]: " > /dev/stderr
read -r CHOICE < /dev/tty
@@ -497,7 +499,8 @@ generate_configuration_files() {
# TCP ServersTransport for PROXY protocol v2 to the proxy backend
render_traefik_dynamic > traefik-dynamic.yaml
if [[ "$ENABLE_CROWDSEC" == "true" ]]; then
mkdir -p crowdsec
mkdir -p crowdsec/acquis.d
render_crowdsec_appsec_acquis > crowdsec/acquis.d/appsec.yaml
fi
fi
;;
@@ -531,6 +534,23 @@ generate_configuration_files() {
return 0
}
# The AppSec (WAF) listener only exists if an appsec acquisition datasource is
# configured. One datasource is one listener carrying one merged rule set: the
# protocol has no rule-set selector, so per-service rule variation would need
# either a second datasource on another port or pre_eval hooks filtering on
# req.Host.
render_crowdsec_appsec_acquis() {
cat <<EOF
source: appsec
listen_addr: 0.0.0.0:7422
appsec_configs:
- crowdsecurity/appsec-default
labels:
type: appsec
EOF
return 0
}
start_services_and_show_instructions() {
# For built-in Traefik, start containers immediately
# For NPM, start containers first (NPM needs services running to create proxy)
@@ -742,7 +762,11 @@ render_docker_compose_traefik_builtin() {
restart: unless-stopped
networks: [netbird]
environment:
COLLECTIONS: crowdsecurity/linux
# appsec-generic-rules is required alongside appsec-virtual-patching:
# the appsec-default config references crowdsecurity/generic-* and
# crowdsecurity/experimental-*, which only that collection provides, and
# the engine exits at startup if they are missing.
COLLECTIONS: crowdsecurity/linux crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
volumes:
- ./crowdsec:/etc/crowdsec
- crowdsec_db:/var/lib/crowdsec/data
@@ -1007,6 +1031,11 @@ EOF
cat <<EOF
NB_PROXY_CROWDSEC_API_URL=http://crowdsec:8080
NB_PROXY_CROWDSEC_API_KEY=$CROWDSEC_BOUNCER_KEY
# AppSec (WAF) request inspection. Separate endpoint from the LAPI above and
# validated with the same bouncer key. Setting it makes the proxy advertise the
# AppSec capability, which is what lets a service select appsec_mode; nothing is
# inspected until a service opts in.
NB_PROXY_CROWDSEC_APPSEC_URL=http://crowdsec:7422/
EOF
fi

View File

@@ -23,6 +23,9 @@ type Domain struct {
// SupportsCrowdSec is populated at query time from proxy cluster capabilities.
// Not persisted.
SupportsCrowdSec *bool `gorm:"-"`
// SupportsAppSec is populated at query time from proxy cluster capabilities.
// Not persisted.
SupportsAppSec *bool `gorm:"-"`
// SupportsPrivate is populated at query time from proxy cluster capabilities. Not persisted.
SupportsPrivate *bool `gorm:"-"`
}

View File

@@ -49,6 +49,7 @@ func domainToApi(d *domain.Domain) api.ReverseProxyDomain {
SupportsCustomPorts: d.SupportsCustomPorts,
RequireSubdomain: d.RequireSubdomain,
SupportsCrowdsec: d.SupportsCrowdSec,
SupportsAppsec: d.SupportsAppSec,
SupportsPrivate: d.SupportsPrivate,
}
if d.TargetCluster != "" {

View File

@@ -35,6 +35,7 @@ type proxyManager interface {
ClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool
ClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool
ClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool
}
@@ -94,6 +95,7 @@ func (m Manager) GetDomains(ctx context.Context, accountID, userID string) ([]*d
d.SupportsCustomPorts = m.proxyManager.ClusterSupportsCustomPorts(ctx, cluster)
d.RequireSubdomain = m.proxyManager.ClusterRequireSubdomain(ctx, cluster)
d.SupportsCrowdSec = m.proxyManager.ClusterSupportsCrowdSec(ctx, cluster)
d.SupportsAppSec = m.proxyManager.ClusterSupportsAppSec(ctx, cluster)
d.SupportsPrivate = m.proxyManager.ClusterSupportsPrivate(ctx, cluster)
ret = append(ret, d)
}
@@ -111,6 +113,7 @@ func (m Manager) GetDomains(ctx context.Context, accountID, userID string) ([]*d
if d.TargetCluster != "" {
cd.SupportsCustomPorts = m.proxyManager.ClusterSupportsCustomPorts(ctx, d.TargetCluster)
cd.SupportsCrowdSec = m.proxyManager.ClusterSupportsCrowdSec(ctx, d.TargetCluster)
cd.SupportsAppSec = m.proxyManager.ClusterSupportsAppSec(ctx, d.TargetCluster)
cd.SupportsPrivate = m.proxyManager.ClusterSupportsPrivate(ctx, d.TargetCluster)
}
// Custom domains never require a subdomain by default since

View File

@@ -40,6 +40,10 @@ func (m *mockProxyManager) ClusterSupportsCrowdSec(_ context.Context, _ string)
return nil
}
func (m *mockProxyManager) ClusterSupportsAppSec(_ context.Context, _ string) *bool {
return nil
}
func (m *mockProxyManager) ClusterSupportsPrivate(_ context.Context, _ string) *bool {
return nil
}

View File

@@ -19,6 +19,7 @@ type Manager interface {
ClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool
ClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool
ClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool
CleanupStale(ctx context.Context, inactivityDuration time.Duration) error
GetAccountProxy(ctx context.Context, accountID string) (*Proxy, error)

View File

@@ -21,6 +21,7 @@ type store interface {
GetClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool
GetClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool
CleanupStaleProxies(ctx context.Context, inactivityDuration time.Duration) error
GetProxyByAccountID(ctx context.Context, accountID string) (*proxy.Proxy, error)
@@ -138,6 +139,13 @@ func (m Manager) ClusterSupportsCrowdSec(ctx context.Context, clusterAddr string
return m.store.GetClusterSupportsCrowdSec(ctx, clusterAddr)
}
// ClusterSupportsAppSec returns whether all active proxies in the cluster have
// a CrowdSec AppSec endpoint configured (unanimous). Returns nil when no proxy
// has reported capabilities.
func (m Manager) ClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool {
return m.store.GetClusterSupportsAppSec(ctx, clusterAddr)
}
// ClusterSupportsPrivate reports whether any active proxy claims the private capability (nil = unreported).
func (m Manager) ClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool {
return m.store.GetClusterSupportsPrivate(ctx, clusterAddr)

View File

@@ -99,6 +99,9 @@ func (m *mockStore) GetClusterRequireSubdomain(_ context.Context, _ string) *boo
func (m *mockStore) GetClusterSupportsCrowdSec(_ context.Context, _ string) *bool {
return nil
}
func (m *mockStore) GetClusterSupportsAppSec(_ context.Context, _ string) *bool {
return nil
}
func (m *mockStore) GetClusterSupportsPrivate(_ context.Context, _ string) *bool {
return nil
}

View File

@@ -50,20 +50,6 @@ func (mr *MockManagerMockRecorder) CleanupStale(ctx, inactivityDuration interfac
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanupStale", reflect.TypeOf((*MockManager)(nil).CleanupStale), ctx, inactivityDuration)
}
// ClusterSupportsCustomPorts mocks base method.
func (m *MockManager) ClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ClusterSupportsCustomPorts", ctx, clusterAddr)
ret0, _ := ret[0].(*bool)
return ret0
}
// ClusterSupportsCustomPorts indicates an expected call of ClusterSupportsCustomPorts.
func (mr *MockManagerMockRecorder) ClusterSupportsCustomPorts(ctx, clusterAddr interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ClusterSupportsCustomPorts", reflect.TypeOf((*MockManager)(nil).ClusterSupportsCustomPorts), ctx, clusterAddr)
}
// ClusterRequireSubdomain mocks base method.
func (m *MockManager) ClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
@@ -78,6 +64,20 @@ func (mr *MockManagerMockRecorder) ClusterRequireSubdomain(ctx, clusterAddr inte
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ClusterRequireSubdomain", reflect.TypeOf((*MockManager)(nil).ClusterRequireSubdomain), ctx, clusterAddr)
}
// ClusterSupportsAppSec mocks base method.
func (m *MockManager) ClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ClusterSupportsAppSec", ctx, clusterAddr)
ret0, _ := ret[0].(*bool)
return ret0
}
// ClusterSupportsAppSec indicates an expected call of ClusterSupportsAppSec.
func (mr *MockManagerMockRecorder) ClusterSupportsAppSec(ctx, clusterAddr interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ClusterSupportsAppSec", reflect.TypeOf((*MockManager)(nil).ClusterSupportsAppSec), ctx, clusterAddr)
}
// ClusterSupportsCrowdSec mocks base method.
func (m *MockManager) ClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
@@ -92,6 +92,20 @@ func (mr *MockManagerMockRecorder) ClusterSupportsCrowdSec(ctx, clusterAddr inte
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ClusterSupportsCrowdSec", reflect.TypeOf((*MockManager)(nil).ClusterSupportsCrowdSec), ctx, clusterAddr)
}
// ClusterSupportsCustomPorts mocks base method.
func (m *MockManager) ClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ClusterSupportsCustomPorts", ctx, clusterAddr)
ret0, _ := ret[0].(*bool)
return ret0
}
// ClusterSupportsCustomPorts indicates an expected call of ClusterSupportsCustomPorts.
func (mr *MockManagerMockRecorder) ClusterSupportsCustomPorts(ctx, clusterAddr interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ClusterSupportsCustomPorts", reflect.TypeOf((*MockManager)(nil).ClusterSupportsCustomPorts), ctx, clusterAddr)
}
// ClusterSupportsPrivate mocks base method.
func (m *MockManager) ClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
@@ -121,6 +135,35 @@ func (mr *MockManagerMockRecorder) Connect(ctx, proxyID, sessionID, clusterAddre
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Connect", reflect.TypeOf((*MockManager)(nil).Connect), ctx, proxyID, sessionID, clusterAddress, ipAddress, accountID, capabilities)
}
// CountAccountProxies mocks base method.
func (m *MockManager) CountAccountProxies(ctx context.Context, accountID string) (int64, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "CountAccountProxies", ctx, accountID)
ret0, _ := ret[0].(int64)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// CountAccountProxies indicates an expected call of CountAccountProxies.
func (mr *MockManagerMockRecorder) CountAccountProxies(ctx, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CountAccountProxies", reflect.TypeOf((*MockManager)(nil).CountAccountProxies), ctx, accountID)
}
// DeleteAccountCluster mocks base method.
func (m *MockManager) DeleteAccountCluster(ctx context.Context, clusterAddress, accountID string) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "DeleteAccountCluster", ctx, clusterAddress, accountID)
ret0, _ := ret[0].(error)
return ret0
}
// DeleteAccountCluster indicates an expected call of DeleteAccountCluster.
func (mr *MockManagerMockRecorder) DeleteAccountCluster(ctx, clusterAddress, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteAccountCluster", reflect.TypeOf((*MockManager)(nil).DeleteAccountCluster), ctx, clusterAddress, accountID)
}
// Disconnect mocks base method.
func (m *MockManager) Disconnect(ctx context.Context, proxyID, sessionID string) error {
m.ctrl.T.Helper()
@@ -135,6 +178,21 @@ func (mr *MockManagerMockRecorder) Disconnect(ctx, proxyID, sessionID interface{
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Disconnect", reflect.TypeOf((*MockManager)(nil).Disconnect), ctx, proxyID, sessionID)
}
// GetAccountProxy mocks base method.
func (m *MockManager) GetAccountProxy(ctx context.Context, accountID string) (*Proxy, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetAccountProxy", ctx, accountID)
ret0, _ := ret[0].(*Proxy)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// GetAccountProxy indicates an expected call of GetAccountProxy.
func (mr *MockManagerMockRecorder) GetAccountProxy(ctx, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetAccountProxy", reflect.TypeOf((*MockManager)(nil).GetAccountProxy), ctx, accountID)
}
// GetActiveClusterAddresses mocks base method.
func (m *MockManager) GetActiveClusterAddresses(ctx context.Context) ([]string, error) {
m.ctrl.T.Helper()
@@ -150,6 +208,7 @@ func (mr *MockManagerMockRecorder) GetActiveClusterAddresses(ctx interface{}) *g
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetActiveClusterAddresses", reflect.TypeOf((*MockManager)(nil).GetActiveClusterAddresses), ctx)
}
// GetActiveClusterAddressesForAccount mocks base method.
func (m *MockManager) GetActiveClusterAddressesForAccount(ctx context.Context, accountID string) ([]string, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetActiveClusterAddressesForAccount", ctx, accountID)
@@ -158,6 +217,7 @@ func (m *MockManager) GetActiveClusterAddressesForAccount(ctx context.Context, a
return ret0, ret1
}
// GetActiveClusterAddressesForAccount indicates an expected call of GetActiveClusterAddressesForAccount.
func (mr *MockManagerMockRecorder) GetActiveClusterAddressesForAccount(ctx, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetActiveClusterAddressesForAccount", reflect.TypeOf((*MockManager)(nil).GetActiveClusterAddressesForAccount), ctx, accountID)
@@ -177,36 +237,6 @@ func (mr *MockManagerMockRecorder) Heartbeat(ctx, p interface{}) *gomock.Call {
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Heartbeat", reflect.TypeOf((*MockManager)(nil).Heartbeat), ctx, p)
}
// GetAccountProxy mocks base method.
func (m *MockManager) GetAccountProxy(ctx context.Context, accountID string) (*Proxy, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetAccountProxy", ctx, accountID)
ret0, _ := ret[0].(*Proxy)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// GetAccountProxy indicates an expected call of GetAccountProxy.
func (mr *MockManagerMockRecorder) GetAccountProxy(ctx, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetAccountProxy", reflect.TypeOf((*MockManager)(nil).GetAccountProxy), ctx, accountID)
}
// CountAccountProxies mocks base method.
func (m *MockManager) CountAccountProxies(ctx context.Context, accountID string) (int64, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "CountAccountProxies", ctx, accountID)
ret0, _ := ret[0].(int64)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// CountAccountProxies indicates an expected call of CountAccountProxies.
func (mr *MockManagerMockRecorder) CountAccountProxies(ctx, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CountAccountProxies", reflect.TypeOf((*MockManager)(nil).CountAccountProxies), ctx, accountID)
}
// IsClusterAddressAvailable mocks base method.
func (m *MockManager) IsClusterAddressAvailable(ctx context.Context, clusterAddress, accountID string) (bool, error) {
m.ctrl.T.Helper()
@@ -222,20 +252,6 @@ func (mr *MockManagerMockRecorder) IsClusterAddressAvailable(ctx, clusterAddress
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsClusterAddressAvailable", reflect.TypeOf((*MockManager)(nil).IsClusterAddressAvailable), ctx, clusterAddress, accountID)
}
// DeleteAccountCluster mocks base method.
func (m *MockManager) DeleteAccountCluster(ctx context.Context, clusterAddress, accountID string) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "DeleteAccountCluster", ctx, clusterAddress, accountID)
ret0, _ := ret[0].(error)
return ret0
}
// DeleteAccountCluster indicates an expected call of DeleteAccountCluster.
func (mr *MockManagerMockRecorder) DeleteAccountCluster(ctx, clusterAddress, accountID interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteAccountCluster", reflect.TypeOf((*MockManager)(nil).DeleteAccountCluster), ctx, clusterAddress, accountID)
}
// MockController is a mock of Controller interface.
type MockController struct {
ctrl *gomock.Controller

View File

@@ -20,6 +20,9 @@ type Capabilities struct {
RequireSubdomain *bool
// SupportsCrowdsec indicates whether this proxy has CrowdSec configured.
SupportsCrowdsec *bool
// SupportsAppsec indicates whether this proxy has a CrowdSec AppSec (WAF)
// endpoint configured.
SupportsAppsec *bool
// Private indicates whether this proxy supports inbound access via Wireguard
// tunnel and netbird-only authentication policies
Private *bool
@@ -74,5 +77,6 @@ type Cluster struct {
SupportsCustomPorts *bool
RequireSubdomain *bool
SupportsCrowdSec *bool
SupportsAppSec *bool
Private *bool
}

View File

@@ -204,6 +204,7 @@ func (h *handler) getClusters(w http.ResponseWriter, r *http.Request) {
SupportsCustomPorts: c.SupportsCustomPorts,
RequireSubdomain: c.RequireSubdomain,
SupportsCrowdsec: c.SupportsCrowdSec,
SupportsAppsec: c.SupportsAppSec,
Private: c.Private,
})
}

View File

@@ -82,6 +82,7 @@ type CapabilityProvider interface {
ClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool
ClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool
ClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool
ClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool
}
@@ -137,6 +138,7 @@ func (m *Manager) GetClusters(ctx context.Context, accountID, userID string) ([]
clusters[i].SupportsCustomPorts = m.capabilities.ClusterSupportsCustomPorts(ctx, clusters[i].Address)
clusters[i].RequireSubdomain = m.capabilities.ClusterRequireSubdomain(ctx, clusters[i].Address)
clusters[i].SupportsCrowdSec = m.capabilities.ClusterSupportsCrowdSec(ctx, clusters[i].Address)
clusters[i].SupportsAppSec = m.capabilities.ClusterSupportsAppSec(ctx, clusters[i].Address)
clusters[i].Private = m.capabilities.ClusterSupportsPrivate(ctx, clusters[i].Address)
}

View File

@@ -165,6 +165,18 @@ type AccessRestrictions struct {
AllowedCountries []string `json:"allowed_countries,omitempty" gorm:"serializer:json"`
BlockedCountries []string `json:"blocked_countries,omitempty" gorm:"serializer:json"`
CrowdSecMode string `json:"crowdsec_mode,omitempty" gorm:"serializer:json"`
// AppSecMode is the CrowdSec AppSec (WAF) request inspection mode: "",
// "off", "enforce", or "observe". HTTP services only.
AppSecMode string `json:"appsec_mode,omitempty" gorm:"serializer:json"`
}
// isEmpty reports whether no restriction is configured. Both conversions drop
// the object entirely in that case, so a field missing from this check is
// silently discarded on the way to the API and the proxy.
func (r AccessRestrictions) isEmpty() bool {
return len(r.AllowedCIDRs) == 0 && len(r.BlockedCIDRs) == 0 &&
len(r.AllowedCountries) == 0 && len(r.BlockedCountries) == 0 &&
r.CrowdSecMode == "" && r.AppSecMode == ""
}
// Copy returns a deep copy of the AccessRestrictions.
@@ -175,6 +187,7 @@ func (r AccessRestrictions) Copy() AccessRestrictions {
AllowedCountries: slices.Clone(r.AllowedCountries),
BlockedCountries: slices.Clone(r.BlockedCountries),
CrowdSecMode: r.CrowdSecMode,
AppSecMode: r.AppSecMode,
}
}
@@ -808,13 +821,17 @@ func restrictionsFromAPI(r *api.AccessRestrictions) (AccessRestrictions, error)
}
res.CrowdSecMode = string(*r.CrowdsecMode)
}
if r.AppsecMode != nil {
if !r.AppsecMode.Valid() {
return AccessRestrictions{}, fmt.Errorf("invalid appsec_mode %q", *r.AppsecMode)
}
res.AppSecMode = string(*r.AppsecMode)
}
return res, nil
}
func restrictionsToAPI(r AccessRestrictions) *api.AccessRestrictions {
if len(r.AllowedCIDRs) == 0 && len(r.BlockedCIDRs) == 0 &&
len(r.AllowedCountries) == 0 && len(r.BlockedCountries) == 0 &&
r.CrowdSecMode == "" {
if r.isEmpty() {
return nil
}
res := &api.AccessRestrictions{}
@@ -834,13 +851,15 @@ func restrictionsToAPI(r AccessRestrictions) *api.AccessRestrictions {
mode := api.AccessRestrictionsCrowdsecMode(r.CrowdSecMode)
res.CrowdsecMode = &mode
}
if r.AppSecMode != "" {
mode := api.AccessRestrictionsAppsecMode(r.AppSecMode)
res.AppsecMode = &mode
}
return res
}
func restrictionsToProto(r AccessRestrictions) *proto.AccessRestrictions {
if len(r.AllowedCIDRs) == 0 && len(r.BlockedCIDRs) == 0 &&
len(r.AllowedCountries) == 0 && len(r.BlockedCountries) == 0 &&
r.CrowdSecMode == "" {
if r.isEmpty() {
return nil
}
return &proto.AccessRestrictions{
@@ -849,6 +868,7 @@ func restrictionsToProto(r AccessRestrictions) *proto.AccessRestrictions {
AllowedCountries: r.AllowedCountries,
BlockedCountries: r.BlockedCountries,
CrowdsecMode: r.CrowdSecMode,
AppsecMode: r.AppSecMode,
}
}
@@ -874,6 +894,11 @@ func (s *Service) Validate() error {
if err := validateAccessRestrictions(&s.Restrictions); err != nil {
return err
}
// AppSec inspects HTTP requests, so it cannot apply to the L4 modes, which
// forward opaque byte streams.
if appSecEnabled(s.Restrictions.AppSecMode) && s.Mode != ModeHTTP {
return fmt.Errorf("appsec_mode is only supported for HTTP services, got mode %q", s.Mode)
}
if err := s.validatePrivateRequirements(); err != nil {
return err
}
@@ -1242,10 +1267,27 @@ func validateCrowdSecMode(mode string) error {
}
}
func validateAppSecMode(mode string) error {
switch mode {
case "", "off", "enforce", "observe":
return nil
default:
return fmt.Errorf("appsec_mode %q is invalid", mode)
}
}
// appSecEnabled reports whether the mode asks for request inspection.
func appSecEnabled(mode string) bool {
return mode == "enforce" || mode == "observe"
}
func validateAccessRestrictions(r *AccessRestrictions) error {
if err := validateCrowdSecMode(r.CrowdSecMode); err != nil {
return err
}
if err := validateAppSecMode(r.AppSecMode); err != nil {
return err
}
if len(r.AllowedCIDRs) > maxCIDREntries {
return fmt.Errorf("allowed_cidrs: exceeds maximum of %d entries", maxCIDREntries)

View File

@@ -26,6 +26,17 @@ func validProxy() *Service {
}
}
// validL4Proxy returns a service that passes validation in one of the L4 modes.
func validL4Proxy(mode string) *Service {
rp := validProxy()
rp.Mode = mode
rp.ListenPort = 9000
rp.Targets = []*Target{
{TargetId: "peer-1", TargetType: TargetTypePeer, Host: "10.0.0.1", Port: 5432, Protocol: mode, Enabled: true},
}
return rp
}
func TestValidate_Valid(t *testing.T) {
require.NoError(t, validProxy().Validate())
}
@@ -1315,3 +1326,68 @@ func TestValidate_Private_RejectsNonHTTPMode(t *testing.T) {
}}
assert.ErrorContains(t, rp.Validate(), "HTTP")
}
func TestRestrictions_AppSecMode_RoundTrip(t *testing.T) {
mode := api.AccessRestrictionsAppsecModeEnforce
apiIn := &api.AccessRestrictions{AppsecMode: &mode}
model, err := restrictionsFromAPI(apiIn)
require.NoError(t, err)
assert.Equal(t, "enforce", model.AppSecMode)
// appsec_mode alone must keep the restrictions object alive on both the API
// and proto legs: it is meaningful without any CIDR or country entry.
apiOut := restrictionsToAPI(model)
require.NotNil(t, apiOut, "appsec_mode alone must not collapse the restrictions to nil")
require.NotNil(t, apiOut.AppsecMode)
assert.Equal(t, api.AccessRestrictionsAppsecModeEnforce, *apiOut.AppsecMode)
protoOut := restrictionsToProto(model)
require.NotNil(t, protoOut, "appsec_mode alone must reach the proxy")
assert.Equal(t, "enforce", protoOut.AppsecMode)
}
func TestRestrictions_AppSecMode_EmptyIsOmitted(t *testing.T) {
model, err := restrictionsFromAPI(&api.AccessRestrictions{
AllowedCidrs: &[]string{"203.0.113.0/24"},
})
require.NoError(t, err)
assert.Empty(t, model.AppSecMode)
apiOut := restrictionsToAPI(model)
require.NotNil(t, apiOut)
assert.Nil(t, apiOut.AppsecMode, "empty appsec_mode is omitted from the API response")
}
func TestRestrictions_AppSecMode_CopyIsDeep(t *testing.T) {
original := AccessRestrictions{AppSecMode: "observe", CrowdSecMode: "enforce"}
assert.Equal(t, original, original.Copy(), "Copy must carry every mode field")
}
func TestValidate_RejectsInvalidAppSecMode(t *testing.T) {
rp := validProxy()
rp.Restrictions = AccessRestrictions{AppSecMode: "sometimes"}
assert.ErrorContains(t, rp.Validate(), "appsec_mode")
}
func TestValidate_RejectsAppSecOnL4Modes(t *testing.T) {
// AppSec inspects HTTP requests, so the L4 modes cannot honor it. Accepting
// the field there would report protection that never runs.
for _, mode := range []string{ModeTCP, ModeUDP, ModeTLS} {
t.Run(mode, func(t *testing.T) {
rp := validL4Proxy(mode)
rp.Restrictions = AccessRestrictions{AppSecMode: "enforce"}
assert.ErrorContains(t, rp.Validate(), "appsec_mode is only supported for HTTP services")
})
}
}
func TestValidate_AllowsAppSecOffOnL4Modes(t *testing.T) {
for _, mode := range []string{ModeTCP, ModeUDP, ModeTLS} {
t.Run(mode, func(t *testing.T) {
rp := validL4Proxy(mode)
rp.Restrictions = AccessRestrictions{AppSecMode: "off"}
require.NoError(t, rp.Validate(), "an explicit off must not be rejected on L4 services")
})
}
}

View File

@@ -29,6 +29,7 @@ import (
"github.com/netbirdio/netbird/shared/management/domain"
"github.com/netbirdio/netbird/management/internals/modules/agentnetwork"
"github.com/netbirdio/netbird/management/internals/modules/peers"
"github.com/netbirdio/netbird/management/internals/modules/reverseproxy/accesslogs"
"github.com/netbirdio/netbird/management/internals/modules/reverseproxy/proxy"
@@ -36,7 +37,6 @@ import (
"github.com/netbirdio/netbird/management/internals/modules/reverseproxy/sessionkey"
"github.com/netbirdio/netbird/management/server/idp"
"github.com/netbirdio/netbird/management/server/peer"
"github.com/netbirdio/netbird/management/internals/modules/agentnetwork"
"github.com/netbirdio/netbird/management/server/types"
"github.com/netbirdio/netbird/management/server/users"
proxyauth "github.com/netbirdio/netbird/proxy/auth"
@@ -506,6 +506,7 @@ func (s *ProxyServiceServer) registerProxyConnection(ctx context.Context, params
SupportsCustomPorts: c.SupportsCustomPorts,
RequireSubdomain: c.RequireSubdomain,
SupportsCrowdsec: c.SupportsCrowdsec,
SupportsAppsec: c.SupportsAppsec,
Private: c.Private,
}
}

View File

@@ -6439,6 +6439,7 @@ var validCapabilityColumns = map[string]struct{}{
"supports_custom_ports": {},
"require_subdomain": {},
"supports_crowdsec": {},
"supports_appsec": {},
"private": {},
}
@@ -6469,6 +6470,14 @@ func (s *SqlStore) GetClusterSupportsCrowdSec(ctx context.Context, clusterAddr s
return s.getClusterUnanimousCapability(ctx, clusterAddr, "supports_crowdsec")
}
// GetClusterSupportsAppSec returns whether all active proxies in the cluster
// have a CrowdSec AppSec endpoint configured. Returns nil when no proxy
// reported the capability. Unanimous for the same reason as CrowdSec: a single
// proxy without AppSec would let requests through uninspected.
func (s *SqlStore) GetClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool {
return s.getClusterUnanimousCapability(ctx, clusterAddr, "supports_appsec")
}
// getClusterUnanimousCapability returns an aggregated boolean capability
// requiring all active proxies in the cluster to report true.
func (s *SqlStore) getClusterUnanimousCapability(ctx context.Context, clusterAddr, column string) *bool {

View File

@@ -0,0 +1,119 @@
package store
import (
"context"
"fmt"
"os"
"runtime"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/netbirdio/netbird/management/internals/modules/reverseproxy/proxy"
)
// Capabilities travel proxy → gRPC → embedded gorm columns → aggregation → API.
// A field dropped at any of those hops reads as "capability absent", which is
// indistinguishable from a proxy that never reported it: the dashboard simply
// hides the feature and nothing fails. These assertions cover the persistence
// and aggregation hops.
func TestSqlStore_ClusterCapabilityAggregation(t *testing.T) {
if os.Getenv("CI") == "true" && (runtime.GOOS == "darwin" || runtime.GOOS == "windows") {
t.Skip("skip CI tests on darwin and windows")
}
yes, no := true, false
tests := []struct {
name string
reported []*bool // one entry per connected proxy in the cluster
wantAppSec *bool
wantAssertion string
}{
{
name: "unreported stays unknown",
reported: []*bool{nil},
wantAppSec: nil,
wantAssertion: "an unreported capability must not read as false",
},
{
name: "single proxy reporting true",
reported: []*bool{&yes},
wantAppSec: &yes,
wantAssertion: "a reported capability must survive persistence",
},
{
name: "one proxy without it disables the cluster",
reported: []*bool{&yes, &no},
wantAppSec: &no,
wantAssertion: "capability must be unanimous, so a rolling upgrade cannot leave traffic uninspected",
},
{
name: "one proxy yet to report disables the cluster",
reported: []*bool{&yes, nil},
wantAppSec: &no,
wantAssertion: "a proxy that has not reported must not count as capable",
},
}
runTestForAllEngines(t, "", func(t *testing.T, store Store) {
ctx := context.Background()
for i, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cluster := fmt.Sprintf("cluster-%d.proxy.example", i)
for j, reported := range tt.reported {
require.NoError(t, store.SaveProxy(ctx, &proxy.Proxy{
ID: fmt.Sprintf("proxy-%d-%d", i, j),
ClusterAddress: cluster,
Status: proxy.StatusConnected,
LastSeen: time.Now(),
Capabilities: proxy.Capabilities{SupportsAppsec: reported},
}))
}
got := store.GetClusterSupportsAppSec(ctx, cluster)
if tt.wantAppSec == nil {
assert.Nil(t, got, tt.wantAssertion)
return
}
require.NotNil(t, got, tt.wantAssertion)
assert.Equal(t, *tt.wantAppSec, *got, tt.wantAssertion)
})
}
})
}
// AppSec and IP reputation are separate endpoints, so a cluster can have either
// without the other. Gating one on the other would silently disable a feature
// the operator configured.
func TestSqlStore_ClusterCapabilitiesAreIndependent(t *testing.T) {
if os.Getenv("CI") == "true" && (runtime.GOOS == "darwin" || runtime.GOOS == "windows") {
t.Skip("skip CI tests on darwin and windows")
}
runTestForAllEngines(t, "", func(t *testing.T, store Store) {
ctx := context.Background()
const cluster = "independent.proxy.example"
yes, no := true, false
require.NoError(t, store.SaveProxy(ctx, &proxy.Proxy{
ID: "proxy-independent",
ClusterAddress: cluster,
Status: proxy.StatusConnected,
LastSeen: time.Now(),
Capabilities: proxy.Capabilities{
SupportsAppsec: &yes,
SupportsCrowdsec: &no,
},
}))
appsec := store.GetClusterSupportsAppSec(ctx, cluster)
crowdsec := store.GetClusterSupportsCrowdSec(ctx, cluster)
require.NotNil(t, appsec)
require.NotNil(t, crowdsec)
assert.True(t, *appsec, "AppSec must not be gated on CrowdSec")
assert.False(t, *crowdsec, "CrowdSec must not be implied by AppSec")
})
}

View File

@@ -133,3 +133,42 @@ func TestSqlStore_GetAccount_ServiceTargetOptionsRoundtrip(t *testing.T) {
assert.True(t, tg.Options.DisableAccessLog, "options disable access log")
})
}
// Restrictions are stored as a JSON blob, and the Postgres read path lists
// columns by hand: a mode that is not read there is silently off on Postgres
// while working in SQLite dev.
func TestSqlStore_GetAccount_ServiceRestrictionsRoundtrip(t *testing.T) {
if os.Getenv("CI") == "true" && (runtime.GOOS == "darwin" || runtime.GOOS == "windows") {
t.Skip("skip CI tests on darwin and windows")
}
runTestForAllEngines(t, "", func(t *testing.T, store Store) {
ctx := context.Background()
account := newAccountWithId(ctx, "account_svc_restrictions", "testuser", "")
require.NoError(t, store.SaveAccount(ctx, account))
svc := &rpservice.Service{
ID: "svc-restrictions",
AccountID: account.Id,
Name: "restricted-svc",
Domain: "restricted.example",
Enabled: true,
Mode: rpservice.ModeHTTP,
Restrictions: rpservice.AccessRestrictions{
AllowedCIDRs: []string{"203.0.113.0/24"},
CrowdSecMode: "observe",
AppSecMode: "enforce",
},
}
require.NoError(t, store.CreateService(ctx, svc))
loaded, err := store.GetAccount(ctx, account.Id)
require.NoError(t, err)
require.Len(t, loaded.Services, 1)
got := loaded.Services[0].Restrictions
assert.Equal(t, []string{"203.0.113.0/24"}, got.AllowedCIDRs, "restrictions allowed CIDRs")
assert.Equal(t, "observe", got.CrowdSecMode, "restrictions crowdsec mode")
assert.Equal(t, "enforce", got.AppSecMode, "restrictions appsec mode")
})
}

View File

@@ -321,6 +321,7 @@ type Store interface {
GetClusterSupportsCustomPorts(ctx context.Context, clusterAddr string) *bool
GetClusterRequireSubdomain(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool
GetClusterSupportsPrivate(ctx context.Context, clusterAddr string) *bool
CleanupStaleProxies(ctx context.Context, inactivityDuration time.Duration) error
GetAllProxies(ctx context.Context) ([]*proxy.Proxy, error)

View File

@@ -1835,6 +1835,20 @@ func (mr *MockStoreMockRecorder) GetClusterRequireSubdomain(ctx, clusterAddr int
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetClusterRequireSubdomain", reflect.TypeOf((*MockStore)(nil).GetClusterRequireSubdomain), ctx, clusterAddr)
}
// GetClusterSupportsAppSec mocks base method.
func (m *MockStore) GetClusterSupportsAppSec(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetClusterSupportsAppSec", ctx, clusterAddr)
ret0, _ := ret[0].(*bool)
return ret0
}
// GetClusterSupportsAppSec indicates an expected call of GetClusterSupportsAppSec.
func (mr *MockStoreMockRecorder) GetClusterSupportsAppSec(ctx, clusterAddr interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetClusterSupportsAppSec", reflect.TypeOf((*MockStore)(nil).GetClusterSupportsAppSec), ctx, clusterAddr)
}
// GetClusterSupportsCrowdSec mocks base method.
func (m *MockStore) GetClusterSupportsCrowdSec(ctx context.Context, clusterAddr string) *bool {
m.ctrl.T.Helper()

View File

@@ -79,6 +79,11 @@ var (
geoDataDir string
crowdsecAPIURL string
crowdsecAPIKey string
appsecURL string
appsecTimeout time.Duration
appsecMaxBodyBytes int64
captureBudgetBytes int64
appsecMaxConcurrent int
)
var rootCmd = &cobra.Command{
@@ -125,6 +130,11 @@ func init() {
rootCmd.Flags().StringVar(&geoDataDir, "geo-data-dir", envStringOrDefault("NB_PROXY_GEO_DATA_DIR", "/var/lib/netbird/geolocation"), "Directory for the GeoLite2 MMDB file (auto-downloaded if missing)")
rootCmd.Flags().StringVar(&crowdsecAPIURL, "crowdsec-api-url", envStringOrDefault("NB_PROXY_CROWDSEC_API_URL", ""), "CrowdSec LAPI URL for IP reputation checks")
rootCmd.Flags().StringVar(&crowdsecAPIKey, "crowdsec-api-key", envStringOrDefault("NB_PROXY_CROWDSEC_API_KEY", ""), "CrowdSec bouncer API key")
rootCmd.Flags().StringVar(&appsecURL, "crowdsec-appsec-url", envStringOrDefault("NB_PROXY_CROWDSEC_APPSEC_URL", ""), "CrowdSec AppSec (WAF) endpoint for HTTP request inspection, e.g. http://127.0.0.1:7422/ (reuses the bouncer API key)")
rootCmd.Flags().DurationVar(&appsecTimeout, "crowdsec-appsec-timeout", envDurationOrDefault("NB_PROXY_CROWDSEC_APPSEC_TIMEOUT", 0), "Timeout for a single AppSec inspection call (0 = 200ms)")
rootCmd.Flags().IntVar(&appsecMaxConcurrent, "crowdsec-appsec-max-concurrent", int(envInt64OrDefault("NB_PROXY_CROWDSEC_APPSEC_MAX_CONCURRENT", 0)), "Cap on AppSec inspections in flight; further requests are denied in enforce mode rather than queued (0 = 256, negative = no cap)")
rootCmd.Flags().Int64Var(&captureBudgetBytes, "capture-budget-bytes", envInt64OrDefault("NB_PROXY_CAPTURE_BUDGET_BYTES", 0), "Total in-flight request-body buffering across the proxy, shared by AppSec inspection and agent-network capture (0 = 256MiB)")
rootCmd.Flags().Int64Var(&appsecMaxBodyBytes, "crowdsec-appsec-max-body-bytes", envInt64OrDefault("NB_PROXY_CROWDSEC_APPSEC_MAX_BODY_BYTES", 0), "Cap on the request body mirrored to AppSec (0 = 64KiB, negative = headers and URI only)")
}
// Execute runs the root command.
@@ -218,47 +228,59 @@ func runServer(cmd *cobra.Command, args []string) error {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()
srv := proxy.New(ctx, proxy.Config{
ListenAddr: addr,
Logger: logger,
Version: Version,
ManagementAddress: mgmtAddr,
ProxyURL: proxyDomain,
ProxyToken: proxyToken,
CertificateDirectory: certDir,
CertificateFile: certFile,
CertificateKeyFile: certKeyFile,
GenerateACMECertificates: acmeCerts,
ACMEChallengeAddress: acmeAddr,
ACMEDirectory: acmeDir,
ACMEEABKID: acmeEABKID,
ACMEEABHMACKey: acmeEABHMACKey,
ACMEChallengeType: acmeChallengeType,
DebugEndpointEnabled: debugEndpoint,
DebugEndpointAddress: debugEndpointAddr,
HealthAddr: healthAddr,
ForwardedProto: forwardedProto,
TrustedProxies: parsedTrustedProxies,
CertLockMethod: nbacme.CertLockMethod(certLockMethod),
WildcardCertDir: wildcardCertDir,
WireguardPort: wgPort,
Performance: perf,
ProxyProtocol: proxyProtocol,
PreSharedKey: preSharedKey,
SupportsCustomPorts: supportsCustomPorts,
RequireSubdomain: requireSubdomain,
Private: private,
MaxDialTimeout: maxDialTimeout,
MaxSessionIdleTimeout: maxSessionIdleTimeout,
MappingBatchWatchdog: envDurationOrDefault("NB_PROXY_MAPPING_BATCH_WATCHDOG", 0),
GeoDataDir: geoDataDir,
CrowdSecAPIURL: crowdsecAPIURL,
CrowdSecAPIKey: crowdsecAPIKey,
})
srv := proxy.New(ctx, serverConfig(logger, proxyToken, parsedTrustedProxies, perf))
return srv.ListenAndServe(ctx, addr)
}
// serverConfig maps the parsed flags and environment onto the proxy config.
// Kept separate from runServer so registering a new flag does not grow the
// startup path.
func serverConfig(logger *log.Logger, proxyToken string, trustedProxyList *trustedproxy.List, perf embed.Performance) proxy.Config {
return proxy.Config{
ListenAddr: addr,
Logger: logger,
Version: Version,
ManagementAddress: mgmtAddr,
ProxyURL: proxyDomain,
ProxyToken: proxyToken,
CertificateDirectory: certDir,
CertificateFile: certFile,
CertificateKeyFile: certKeyFile,
GenerateACMECertificates: acmeCerts,
ACMEChallengeAddress: acmeAddr,
ACMEDirectory: acmeDir,
ACMEEABKID: acmeEABKID,
ACMEEABHMACKey: acmeEABHMACKey,
ACMEChallengeType: acmeChallengeType,
DebugEndpointEnabled: debugEndpoint,
DebugEndpointAddress: debugEndpointAddr,
HealthAddr: healthAddr,
ForwardedProto: forwardedProto,
TrustedProxies: trustedProxyList,
CertLockMethod: nbacme.CertLockMethod(certLockMethod),
WildcardCertDir: wildcardCertDir,
WireguardPort: wgPort,
Performance: perf,
ProxyProtocol: proxyProtocol,
PreSharedKey: preSharedKey,
SupportsCustomPorts: supportsCustomPorts,
RequireSubdomain: requireSubdomain,
Private: private,
MaxDialTimeout: maxDialTimeout,
MaxSessionIdleTimeout: maxSessionIdleTimeout,
MappingBatchWatchdog: envDurationOrDefault("NB_PROXY_MAPPING_BATCH_WATCHDOG", 0),
GeoDataDir: geoDataDir,
CrowdSecAPIURL: crowdsecAPIURL,
CrowdSecAPIKey: crowdsecAPIKey,
CrowdSecAppSecURL: appsecURL,
CrowdSecAppSecTimeout: appsecTimeout,
CrowdSecAppSecMaxBodyBytes: appsecMaxBodyBytes,
CrowdSecAppSecMaxConcurrent: appsecMaxConcurrent,
MiddlewareCaptureBudgetBytes: captureBudgetBytes,
}
}
func envBoolOrDefault(key string, def bool) bool {
v, exists := os.LookupEnv(key)
if !exists {
@@ -293,6 +315,19 @@ func envUint16OrDefault(key string, def uint16) uint16 {
return uint16(parsed)
}
func envInt64OrDefault(key string, def int64) int64 {
v, exists := os.LookupEnv(key)
if !exists {
return def
}
parsed, err := strconv.ParseInt(v, 10, 64)
if err != nil {
log.Warnf("parse %s=%q: %v, using default %d", key, v, err, def)
return def
}
return parsed
}
func envDurationOrDefault(key string, def time.Duration) time.Duration {
v, exists := os.LookupEnv(key)
if !exists {

View File

@@ -0,0 +1,163 @@
package appsec
import (
"bytes"
"errors"
"io"
"mime"
"net/http"
"net/url"
"slices"
"strings"
)
// bufferBody reads up to limit+1 bytes from r.Body and always restores r.Body so
// the request stays forwardable. oversize reports that the body exceeded limit, in
// which case the returned prefix must not be used for inspection: the bytes are
// only read so they can be replayed to the backend.
func bufferBody(r *http.Request, limit int64) (body []byte, oversize bool, err error) {
original := r.Body
buf, readErr := io.ReadAll(io.LimitReader(original, limit+1))
if readErr != nil && !errors.Is(readErr, io.EOF) {
// Restore what was read so a downstream retry sees a consistent stream,
// then surface the failure.
r.Body = replay(buf, original)
return nil, false, readErr
}
if int64(len(buf)) > limit {
r.Body = replay(buf, original)
return nil, true, nil
}
// The whole body is buffered, so the original is drained and can be closed.
// A close error on a drained read-only body does not invalidate the bytes.
_ = original.Close()
r.Body = io.NopCloser(bytes.NewReader(buf))
// Framing is deliberately left as the client sent it. Rewriting a chunked
// request to a fixed Content-Length here would be invisible to the client
// but not to the rest of the chain: a later body capture with a smaller cap
// sees a known length over its cap and skips capture entirely, where an
// unknown length would have given it a truncated prefix. Inspecting a
// request must not change what any other layer gets to inspect.
return buf, false, nil
}
// replay returns a ReadCloser that yields the already-read prefix followed by
// the remainder of the original stream, and closes the original.
func replay(prefix []byte, rest io.ReadCloser) io.ReadCloser {
return struct {
io.Reader
io.Closer
}{
Reader: io.MultiReader(bytes.NewReader(prefix), rest),
Closer: rest,
}
}
// redactedPlaceholder replaces a credential value in the mirrored body. It is
// inert for rule matching, and its fixed length leaks nothing about the secret.
const redactedPlaceholder = "redacted"
// redactFormFields returns the body to mirror for a URL-encoded form, with the
// values of the named fields replaced. The proxy's own password / PIN login
// form posts to the service path itself, so without this the plaintext
// credential would reach the Security Engine.
//
// Only the credential values are removed, never the whole body: dropping the
// body outright would let a caller exempt any payload from inspection just by
// appending a field named "password". Everything else in the form stays
// inspectable, which is the point.
//
// Returns body unchanged when it is not a URL-encoded form or carries none of
// the fields.
//
// Substitution happens on the raw bytes rather than by re-encoding parsed
// values. Re-encoding would drop pairs that url.ParseQuery rejects, so a
// payload hidden in a malformed pair alongside a credential-named field would
// never be inspected while a tolerant backend parser still acted on it. Working
// byte-wise also avoids reordering keys and normalizing escapes, so the engine
// sees the same bytes the backend will.
//
// Field names match case-sensitively, on purpose: the caller passes the exact
// names the login handler reads via r.FormValue, and that lookup is itself
// case-sensitive. A "Password" field is therefore never a credential as far as
// the proxy is concerned, and redacting it would only blind the WAF to a value
// the proxy does not own.
func redactFormFields(contentType string, body []byte, fields []string) []byte {
if len(fields) == 0 || len(body) == 0 {
return body
}
media, _, err := mime.ParseMediaType(contentType)
if err != nil || media != "application/x-www-form-urlencoded" {
return body
}
return redactURLEncoded(body, fields)
}
// redactURLEncoded replaces the values of the named keys in a URL-encoded
// key/value sequence, the shared syntax of a query string and a form body.
func redactURLEncoded(raw []byte, fields []string) []byte {
// Split on "&" only, matching how Go's form parser delimits pairs.
segments := bytes.Split(raw, []byte("&"))
redacted := false
for i, segment := range segments {
rawKey, _, hasValue := bytes.Cut(segment, []byte("="))
if !hasValue {
continue
}
// Compare the decoded name, so an escaped spelling of the field
// ("pass%77ord") is redacted too: the reader decodes before looking it
// up. A key that fails to decode never reaches that reader either,
// since the parser drops the pair.
name, err := url.QueryUnescape(string(rawKey))
if err != nil || !slices.Contains(fields, name) {
continue
}
// Keep the key bytes as sent and replace only the value. Assigning a
// fresh slice leaves raw untouched, which matters: the caller restored
// the request body from the same buffer.
segments[i] = []byte(string(rawKey) + "=" + redactedPlaceholder)
redacted = true
}
if !redacted {
return raw
}
return bytes.Join(segments, []byte("&"))
}
// redactQuery replaces the values of the named query parameters in a raw query
// string, leaving every other byte as sent.
func redactQuery(rawQuery string, params []string) string {
if len(params) == 0 || rawQuery == "" {
return rawQuery
}
return string(redactURLEncoded([]byte(rawQuery), params))
}
// redactCookieHeader replaces the values of the named cookies in a Cookie
// header, keeping the others intact: cookies are a zone WAF rules match on, so
// dropping the whole header would cost real coverage.
func redactCookieHeader(value string, names []string) string {
if len(names) == 0 || value == "" {
return value
}
parts := strings.Split(value, ";")
redacted := false
for i, part := range parts {
name, _, hasValue := strings.Cut(part, "=")
if !hasValue {
continue
}
// Cookie names are case-sensitive and are not percent-decoded.
if !slices.Contains(names, strings.TrimSpace(name)) {
continue
}
parts[i] = name + "=" + redactedPlaceholder
redacted = true
}
if !redacted {
return value
}
return strings.Join(parts, ";")
}

View File

@@ -0,0 +1,571 @@
// Package appsec implements the CrowdSec AppSec (WAF) side of the remediation
// component protocol: each inspected HTTP request is mirrored to the Security
// Engine's AppSec endpoint, which replies with an allow / ban / captcha verdict
// for that request.
//
// This is a separate endpoint from the LAPI decision stream used by the
// crowdsec package: LAPI answers "is this IP known bad", AppSec answers "is
// this request an attack". The two are configured and enabled independently.
package appsec
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strings"
"sync"
"time"
log "github.com/sirupsen/logrus"
"github.com/netbirdio/netbird/proxy/internal/netutil"
"github.com/netbirdio/netbird/proxy/internal/restrict"
)
// Header names the AppSec component reads off the mirrored request. IP, URI and
// Verb are mandatory: the engine answers 500 when any of them is missing.
const (
headerAPIKey = "X-Crowdsec-Appsec-Api-Key" //nolint:gosec // G101: a header name, not a credential
headerIP = "X-Crowdsec-Appsec-Ip"
headerURI = "X-Crowdsec-Appsec-Uri"
headerVerb = "X-Crowdsec-Appsec-Verb"
headerHost = "X-Crowdsec-Appsec-Host"
headerUserAgent = "X-Crowdsec-Appsec-User-Agent"
headerHTTPVersion = "X-Crowdsec-Appsec-Http-Version"
headerTransactionID = "X-Crowdsec-Appsec-Transaction-Id"
)
// headerPrefix covers every protocol header. Any client-supplied header in this
// namespace is dropped before forwarding so a caller cannot influence the
// engine's view of its own address, or replay an API key.
const headerPrefix = "X-Crowdsec-Appsec-"
// Remediation actions the engine can return.
const (
actionAllow = "allow"
actionBan = "ban"
actionCaptcha = "captcha"
)
const (
// DefaultTimeout matches the 200ms budget CrowdSec's remediation component
// spec sets for the blocking AppSec call.
DefaultTimeout = 200 * time.Millisecond
// MinTimeout and MaxTimeout bound the configured inspection timeout.
// Inspection is synchronous, so the upper bound is what keeps a
// mis-set value from parking every request to an inspected service on a
// slow engine; the lower bound keeps the call from timing out before the
// engine can realistically answer. Mirrors the per-middleware bounds the
// proxy already applies to in-path calls.
MinTimeout = 10 * time.Millisecond
MaxTimeout = 5 * time.Second
// DefaultMaxBodyBytes caps the request body mirrored to the engine.
// Requests with a larger body are inspected on headers and URI only.
DefaultMaxBodyBytes int64 = 64 << 10
// DefaultMaxConcurrent bounds inspections in flight toward the engine. The
// point is to fail fast instead of parking a goroutine per request for the
// whole timeout once the engine is saturated: a slow engine otherwise turns
// a traffic burst into a pile of waiters that all time out anyway. Sized so
// a healthy engine (single-digit milliseconds per call) never reaches it.
DefaultMaxConcurrent = 256
// MaxConcurrentLimit is the ceiling for that bound.
MaxConcurrentLimit = 4096
// MaxBodyBytesLimit is the ceiling for that cap. A single request can hold
// this much in memory; the shared Budget is what bounds the total across
// concurrent requests. Matches the proxy-wide body-capture ceiling.
MaxBodyBytesLimit int64 = 8 << 20
// maxResponseBytes bounds how much of a verdict response is read. The
// engine answers with a two-field JSON object, so anything beyond this is
// not a response we can act on.
maxResponseBytes int64 = 4 << 10
)
// Reasons the request body was not mirrored. Reported so an access-log reader
// can distinguish "inspected and clean" from "never inspected", and so an
// oversize opt-out is visible rather than silent.
const (
BypassOversize = "oversize"
BypassUpgrade = "upgrade"
BypassDisabled = "disabled"
BypassBudget = "budget_exhausted"
)
// ErrUnavailable reports that the engine could not produce a verdict: the call
// failed, timed out, or the engine rejected it (401 bad key, 500 malformed).
// Distinguished from a block verdict so the caller can apply the per-service
// mode: enforce fails closed, observe allows.
var ErrUnavailable = errors.New("appsec engine unavailable")
// Config configures a Client.
type Config struct {
// URL is the AppSec endpoint, e.g. http://127.0.0.1:7422/.
URL string
// APIKey is the CrowdSec bouncer API key. The AppSec component validates it
// against LAPI, so the same key used for the decision stream works here.
APIKey string
// Timeout bounds a single inspection call. Zero means DefaultTimeout.
Timeout time.Duration
// MaxBodyBytes caps the mirrored request body. Zero means
// DefaultMaxBodyBytes; negative disables body forwarding entirely.
MaxBodyBytes int64
// MaxConcurrent bounds inspections in flight toward the engine. Zero means
// DefaultMaxConcurrent; negative disables the bound.
MaxConcurrent int
// Budget bounds the total body buffering in flight across all inspected
// requests. Nil disables that ceiling, which leaves the worst case at
// MaxBodyBytes times the concurrent request count; callers serving
// untrusted traffic should share the proxy-wide capture budget here.
Budget Budget
Logger *log.Entry
}
// Budget is the shared allowance for in-flight body buffering. Acquire reports
// whether n bytes could be reserved; every successful Acquire is matched by a
// Release of the same n. Satisfied by the proxy's capture budget, so AppSec and
// the middleware body tap draw down one pool rather than two independent ones.
type Budget interface {
Acquire(n int64) bool
Release(n int64)
}
// Client mirrors HTTP requests to a CrowdSec AppSec endpoint. It holds no
// per-service state and is safe for concurrent use.
type Client struct {
url string
apiKey string
maxBodyBytes int64
// sem bounds in-flight inspections. Nil when the bound is disabled.
sem chan struct{}
budget Budget
http *http.Client
logger *log.Entry
}
// New validates the config and returns a Client. The endpoint is not contacted
// here: the engine may come up after the proxy.
func New(cfg Config) (*Client, error) {
if cfg.URL == "" {
return nil, errors.New("appsec url is empty")
}
if cfg.APIKey == "" {
return nil, errors.New("appsec api key is empty")
}
parsed, err := url.Parse(cfg.URL)
if err != nil {
return nil, fmt.Errorf("parse appsec url: %w", err)
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return nil, fmt.Errorf("appsec url scheme %q is not http(s)", parsed.Scheme)
}
if parsed.Host == "" {
return nil, errors.New("appsec url has no host")
}
logger := cfg.Logger
if logger == nil {
logger = log.NewEntry(log.StandardLogger())
}
timeout := cfg.Timeout
switch {
case timeout <= 0:
timeout = DefaultTimeout
case timeout < MinTimeout:
logger.Warnf("appsec timeout %s is below the minimum, using %s", timeout, MinTimeout)
timeout = MinTimeout
case timeout > MaxTimeout:
logger.Warnf("appsec timeout %s exceeds the maximum, using %s", timeout, MaxTimeout)
timeout = MaxTimeout
}
// A negative cap is meaningful: forward no body at all.
maxBody := cfg.MaxBodyBytes
switch {
case maxBody == 0:
maxBody = DefaultMaxBodyBytes
case maxBody > MaxBodyBytesLimit:
logger.Warnf("appsec max body %d exceeds the maximum, using %d", maxBody, MaxBodyBytesLimit)
maxBody = MaxBodyBytesLimit
}
maxConcurrent := cfg.MaxConcurrent
switch {
case maxConcurrent == 0:
maxConcurrent = DefaultMaxConcurrent
case maxConcurrent > MaxConcurrentLimit:
logger.Warnf("appsec max concurrent %d exceeds the maximum, using %d", maxConcurrent, MaxConcurrentLimit)
maxConcurrent = MaxConcurrentLimit
}
var sem chan struct{}
if maxConcurrent > 0 {
sem = make(chan struct{}, maxConcurrent)
}
return &Client{
url: cfg.URL,
apiKey: cfg.APIKey,
maxBodyBytes: maxBody,
sem: sem,
budget: cfg.Budget,
logger: logger,
http: &http.Client{
Timeout: timeout,
Transport: &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 32,
IdleConnTimeout: 90 * time.Second,
},
},
}, nil
}
// Request is one inspection request.
type Request struct {
// HTTP is the in-flight client request. Inspect buffers and restores its
// body, so the request stays forwardable afterwards.
HTTP *http.Request
// ClientIP is the resolved client address (after trusted-proxy handling).
ClientIP netip.Addr
// TransactionID correlates the engine's alert with the proxy's access log
// entry. Empty lets the engine generate its own UUID.
TransactionID string
// RedactBodyFields lists form fields whose values are replaced before the
// body is mirrored. Used to keep credentials submitted to the proxy's own
// login form out of the engine while still inspecting the rest.
RedactBodyFields []string
// RedactHeaders, RedactCookies and RedactQueryParams name the credentials
// the proxy already withholds from backends: the header-auth values, its
// session cookie, and the OIDC session token. The engine logs and alerts on
// what it inspects, so mirroring them there would reintroduce the leak the
// upstream strippers exist to prevent. Only the values are replaced, so the
// surrounding headers, cookies and query stay inspectable.
RedactHeaders []string
RedactCookies []string
RedactQueryParams []string
}
// Result is the outcome of an inspection.
type Result struct {
Verdict restrict.Verdict
// BodyBypass names why the request body was not mirrored, empty when it
// was (or when the request had none). The engine still saw the headers and
// URI, so this is a coverage note, not a failure.
BodyBypass string
// Release returns the buffered body's budget reservation. Never nil, so it
// is always safe to defer. It must run only once the request has been
// served, not when Inspect returns: the buffer stays alive as r.Body for
// the backend to read, so releasing earlier would let the budget admit
// buffering that is still resident.
Release func()
}
// noopRelease is the Release for inspections that reserved no budget.
func noopRelease() {}
// Inspect mirrors r to the AppSec engine and returns its verdict. A nil error
// with restrict.Allow means the request passed. On failure it returns
// DenyAppSecUnavailable wrapped with ErrUnavailable; the caller decides whether
// that blocks, based on the per-service mode.
func (c *Client) Inspect(ctx context.Context, req Request) (Result, error) {
if c == nil {
return Result{Verdict: restrict.DenyAppSecUnavailable, Release: noopRelease}, ErrUnavailable
}
if req.HTTP == nil {
return Result{Verdict: restrict.DenyAppSecUnavailable, Release: noopRelease}, fmt.Errorf("%w: nil request", ErrUnavailable)
}
// release is carried out to the caller rather than deferred here: the
// buffered body outlives this call as r.Body.
if !c.acquireSlot() {
// Deny rather than wave through: a flood must not be a way to switch
// inspection off. Enforce blocks, observe logs and allows, exactly as
// for an unreachable engine.
return Result{Verdict: restrict.DenyAppSecUnavailable, Release: noopRelease},
fmt.Errorf("%w: %d inspections already in flight", ErrUnavailable, cap(c.sem))
}
defer c.releaseSlot()
body, bypass, release, err := c.readBody(req)
if err != nil {
return Result{Verdict: restrict.DenyAppSecUnavailable, Release: release}, fmt.Errorf("%w: read body: %w", ErrUnavailable, err)
}
outbound, err := c.buildRequest(ctx, req, body)
if err != nil {
return Result{Verdict: restrict.DenyAppSecUnavailable, BodyBypass: bypass, Release: release}, fmt.Errorf("%w: %w", ErrUnavailable, err)
}
resp, err := c.http.Do(outbound)
if err != nil {
return Result{Verdict: restrict.DenyAppSecUnavailable, BodyBypass: bypass, Release: release}, fmt.Errorf("%w: %w", ErrUnavailable, err)
}
defer func() {
// Drain before closing. net/http only returns a connection to the idle
// pool once its body is read to EOF; closing with bytes outstanding
// discards it. Every verdict carries a JSON body, so skipping this
// would cost a fresh handshake per inspected request, inside the
// timeout budget.
if _, err := io.Copy(io.Discard, io.LimitReader(resp.Body, maxResponseBytes)); err != nil {
c.logger.Tracef("drain appsec response body: %v", err)
}
if err := resp.Body.Close(); err != nil {
c.logger.Tracef("close appsec response body: %v", err)
}
}()
verdict, err := c.verdict(resp)
return Result{Verdict: verdict, BodyBypass: bypass, Release: release}, err
}
// acquireSlot takes an in-flight slot without blocking, reporting false when
// the engine is already at capacity.
func (c *Client) acquireSlot() bool {
if c.sem == nil {
return true
}
select {
case c.sem <- struct{}{}:
return true
default:
return false
}
}
// releaseSlot returns the slot. Scoped to the engine call, not the request: the
// buffered body outlives the call but the engine's attention does not.
func (c *Client) releaseSlot() {
if c.sem == nil {
return
}
select {
case <-c.sem:
default:
}
}
// readBody buffers the body so it can be mirrored, always restoring it on the
// original request. Returns nil when there is no body to forward: no body at
// all, an upgrade request, or a body over the cap. A login form is forwarded
// with its credential values redacted rather than suppressed.
// release is never nil; the caller invokes it once the request has been served.
func (c *Client) readBody(req Request) (body []byte, bypass string, release func(), err error) {
r := req.HTTP
if r.Body == nil || r.Body == http.NoBody {
return nil, "", noopRelease, nil
}
if c.maxBodyBytes < 0 {
return nil, BypassDisabled, noopRelease, nil
}
// A genuine upgrade request carries no body to inspect (net/http hands us
// http.NoBody, caught above); the hijacked stream is reached through
// Hijacker, never r.Body. The test has to be the forwarder's own, because a
// looser one would skip inspection for requests the forwarder still
// delivers to the backend with their body intact.
if netutil.IsUpgradeRequest(r.Header) {
return nil, BypassUpgrade, noopRelease, nil
}
// A Content-Length over the cap is known to be too large before reading.
if r.ContentLength > c.maxBodyBytes {
return nil, BypassOversize, noopRelease, nil
}
// Reserve the whole cap rather than the eventual length: the reservation
// has to be made before the body is read, and until then the only bound
// known is the cap. Skipping inspection when the pool is drained keeps a
// burst of large bodies from being an out-of-memory lever; the bypass is
// recorded so the gap in coverage is visible.
release = noopRelease
if c.budget != nil {
if !c.budget.Acquire(c.maxBodyBytes) {
c.logger.Debugf("appsec buffer budget exhausted, inspecting headers and URI only")
return nil, BypassBudget, noopRelease, nil
}
var once sync.Once
release = func() { once.Do(func() { c.budget.Release(c.maxBodyBytes) }) }
}
buffered, oversize, err := bufferBody(r, c.maxBodyBytes)
if err != nil {
// bufferBody restored r.Body from the bytes it did read, so the
// reservation stays held until the caller releases it.
return nil, "", release, err
}
// An oversize body was only partially read: a truncated prefix changes the
// engine's verdict in both directions, so inspect headers and URI only.
if oversize {
return nil, BypassOversize, release, nil
}
return redactFormFields(r.Header.Get("Content-Type"), buffered, req.RedactBodyFields), "", release, nil
}
// buildRequest assembles the mirrored request. Per the protocol it is a GET
// when there is no body and a POST otherwise; bytes.Reader gives the outbound
// request an accurate Content-Length, which the engine relies on to read the
// body at all.
func (c *Client) buildRequest(ctx context.Context, req Request, body []byte) (*http.Request, error) {
method := http.MethodGet
var payload io.Reader
if len(body) > 0 {
method = http.MethodPost
payload = bytes.NewReader(body)
}
outbound, err := http.NewRequestWithContext(ctx, method, c.url, payload)
if err != nil {
return nil, fmt.Errorf("build appsec request: %w", err)
}
r := req.HTTP
copyInspectableHeaders(outbound.Header, r.Header)
redactSecrets(outbound.Header, req)
outbound.Header.Set(headerAPIKey, c.apiKey)
outbound.Header.Set(headerIP, req.ClientIP.Unmap().String())
outbound.Header.Set(headerURI, mirroredURI(r.URL, req.RedactQueryParams))
outbound.Header.Set(headerVerb, r.Method)
outbound.Header.Set(headerHost, r.Host)
if ua := r.UserAgent(); ua != "" {
outbound.Header.Set(headerUserAgent, ua)
}
outbound.Header.Set(headerHTTPVersion, httpVersion(r))
if req.TransactionID != "" {
outbound.Header.Set(headerTransactionID, req.TransactionID)
}
return outbound, nil
}
// verdict maps the engine's response to a restrict.Verdict. 200 is a pass and
// 401/500 are engine-side failures; every other status carries a remediation in
// the body. The blocked status code is operator-configurable
// (blocked_http_code), so the action field decides, not the status.
func (c *Client) verdict(resp *http.Response) (restrict.Verdict, error) {
switch resp.StatusCode {
case http.StatusUnauthorized:
return restrict.DenyAppSecUnavailable, fmt.Errorf("%w: rejected api key", ErrUnavailable)
case http.StatusInternalServerError:
return restrict.DenyAppSecUnavailable, fmt.Errorf("%w: engine error", ErrUnavailable)
}
// Every status, 200 included, has to carry a decodable remediation. Taking a
// bare 200 as a pass would mean a URL pointing at anything that answers 200
// (a health endpoint, a load balancer's default page) silently allows every
// request while the service reports itself as enforcing.
var decoded struct {
Action string `json:"action"`
}
if err := json.NewDecoder(io.LimitReader(resp.Body, maxResponseBytes)).Decode(&decoded); err != nil {
// Every remediation carries a decodable action, so a response without
// one is not a verdict: most often the URL points at something that is
// not the AppSec endpoint, which answers 404 with HTML. Reported as
// unavailable rather than a ban so the access log names the real fault
// instead of sending an operator hunting for a rule that never fired.
// Enforce still blocks either way; only the recorded reason differs.
return restrict.DenyAppSecUnavailable, fmt.Errorf("%w: undecodable response (status %d): %w", ErrUnavailable, resp.StatusCode, err)
}
switch decoded.Action {
case actionAllow:
return restrict.Allow, nil
case actionCaptcha:
return restrict.DenyAppSecCaptcha, nil
case actionBan:
return restrict.DenyAppSecBan, nil
case "":
// Decodable JSON without a remediation is not a verdict either: the
// endpoint answered, but not as the engine. Same reasoning as an
// undecodable body, and the same reason to point at configuration.
return restrict.DenyAppSecUnavailable,
fmt.Errorf("%w: response carried no remediation (status %d)", ErrUnavailable, resp.StatusCode)
default:
// A remediation we do not implement still means the engine flagged the
// request, so deny.
c.logger.Debugf("unknown appsec action %q (status %d), treating as ban", decoded.Action, resp.StatusCode)
return restrict.DenyAppSecBan, nil
}
}
// copyInspectableHeaders copies the client's headers, which are what the WAF
// rules actually match on, dropping hop-by-hop headers that describe the
// proxy-to-engine connection rather than the client request, and any header in
// the AppSec protocol namespace.
func copyInspectableHeaders(dst, src http.Header) {
for name, values := range src {
if hopByHopHeaders[http.CanonicalHeaderKey(name)] {
continue
}
if strings.HasPrefix(http.CanonicalHeaderKey(name), headerPrefix) {
continue
}
dst[http.CanonicalHeaderKey(name)] = append([]string(nil), values...)
}
// Content-Length describes the mirrored payload, not the client's: net/http
// sets it from the body we actually attach. Content-Type is kept either way
// so rules matching on it still fire when the body was not forwarded.
dst.Del("Content-Length")
}
// redactSecrets replaces the credential values the proxy withholds from
// backends, so the mirrored copy does not carry them either.
func redactSecrets(dst http.Header, req Request) {
for _, name := range req.RedactHeaders {
// Presence, not Get: a header whose first value is empty still carries
// its later values to the engine, while the upstream strip deletes the
// name outright. Set collapses every value into the placeholder.
if len(dst.Values(name)) > 0 {
dst.Set(name, redactedPlaceholder)
}
}
// Every Cookie line, not just the first: a client may send several, and Get
// would leave the session cookie in any later one mirrored in the clear.
if cookies := dst.Values("Cookie"); len(cookies) > 0 {
redacted := make([]string, len(cookies))
for i, cookie := range cookies {
redacted[i] = redactCookieHeader(cookie, req.RedactCookies)
}
dst["Cookie"] = redacted
}
}
// mirroredURI renders the request target for the URI header, with the named
// query parameter values replaced.
func mirroredURI(u *url.URL, redactParams []string) string {
uri := u.RequestURI()
if u.RawQuery == "" || len(redactParams) == 0 {
return uri
}
redacted := redactQuery(u.RawQuery, redactParams)
if redacted == u.RawQuery {
return uri
}
// RequestURI is path + "?" + RawQuery; swap only the query part so the
// path keeps its original encoding.
return strings.TrimSuffix(uri, u.RawQuery) + redacted
}
var hopByHopHeaders = map[string]bool{
"Connection": true,
"Keep-Alive": true,
"Proxy-Authenticate": true,
"Proxy-Authorization": true,
"Proxy-Connection": true,
"Te": true,
"Trailer": true,
"Transfer-Encoding": true,
"Upgrade": true,
}
// httpVersion renders the two-digit form the engine parses ("11", "20").
func httpVersion(r *http.Request) string {
major, minor := r.ProtoMajor, r.ProtoMinor
if major < 0 || major > 9 || minor < 0 || minor > 9 {
return ""
}
return fmt.Sprintf("%d%d", major, minor)
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,306 @@
package auth
import (
"crypto/ed25519"
"encoding/base64"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/netbirdio/netbird/proxy/internal/appsec"
"github.com/netbirdio/netbird/proxy/internal/proxy"
"github.com/netbirdio/netbird/proxy/internal/restrict"
"github.com/netbirdio/netbird/proxy/internal/types"
)
// appsecEngine is a stub AppSec component returning a fixed remediation.
func appsecEngine(t *testing.T, status int, body string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(status)
if body != "" {
_, _ = w.Write([]byte(body))
}
}))
t.Cleanup(srv.Close)
return srv
}
// serveWithAppSec runs a request through the middleware for a domain configured
// with the given AppSec mode, returning the response and the captured metadata.
func serveWithAppSec(t *testing.T, mode restrict.AppSecMode, client *appsec.Client, r *http.Request) (*httptest.ResponseRecorder, map[string]string, bool) {
t.Helper()
mw := NewMiddleware(log.StandardLogger(), nil, nil)
mw.SetAppSec(client)
require.NoError(t, mw.AddDomain("svc.example.com", DomainSettings{
AccountID: types.AccountID("acct-1"),
ServiceID: types.ServiceID("svc-1"),
AppSecMode: mode,
}))
reached := false
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
reached = true
w.WriteHeader(http.StatusOK)
}))
cd := proxy.NewCapturedData("req-1")
r = r.WithContext(proxy.WithCapturedData(r.Context(), cd))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, r)
return rec, cd.GetMetadata(), reached
}
func appsecRequest() *http.Request {
r := httptest.NewRequest(http.MethodGet, "http://svc.example.com/?x=/etc/passwd", nil)
r.Host = "svc.example.com"
r.RemoteAddr = "203.0.113.7:44444"
return r
}
func TestCheckAppSec_EnforceBlocksBannedRequest(t *testing.T) {
srv := appsecEngine(t, http.StatusForbidden, `{"action":"ban","http_status":403}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, appsecRequest())
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.False(t, reached, "a banned request must not reach the backend")
assert.Equal(t, "appsec_ban", meta["appsec_verdict"])
assert.NotContains(t, meta, "appsec_mode", "enforce is the default, only observe is annotated")
}
func TestCheckAppSec_ObserveAllowsAndRecordsVerdict(t *testing.T) {
srv := appsecEngine(t, http.StatusForbidden, `{"action":"ban","http_status":403}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
rec, meta, reached := serveWithAppSec(t, restrict.AppSecObserve, client, appsecRequest())
assert.Equal(t, http.StatusOK, rec.Code)
assert.True(t, reached, "observe mode must not block")
assert.Equal(t, "appsec_ban", meta["appsec_verdict"])
assert.Equal(t, "observe", meta["appsec_mode"])
}
func TestCheckAppSec_AllowedRequestPasses(t *testing.T) {
srv := appsecEngine(t, http.StatusOK, `{"action":"allow","http_status":200}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, appsecRequest())
assert.Equal(t, http.StatusOK, rec.Code)
assert.True(t, reached)
assert.NotContains(t, meta, "appsec_verdict", "a clean request records no verdict")
}
func TestCheckAppSec_OffSkipsInspection(t *testing.T) {
// An engine that would ban everything; the mode must keep us away from it.
srv := appsecEngine(t, http.StatusForbidden, `{"action":"ban"}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
rec, meta, reached := serveWithAppSec(t, restrict.AppSecOff, client, appsecRequest())
assert.Equal(t, http.StatusOK, rec.Code)
assert.True(t, reached)
assert.Empty(t, meta)
}
func TestCheckAppSec_EnforceFailsClosedWithoutClient(t *testing.T) {
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, nil, appsecRequest())
assert.Equal(t, http.StatusForbidden, rec.Code,
"enforce with no configured endpoint must deny rather than pass traffic uninspected")
assert.False(t, reached)
assert.Equal(t, "appsec_unavailable", meta["appsec_verdict"])
}
func TestCheckAppSec_ObserveAllowsWithoutClient(t *testing.T) {
rec, meta, reached := serveWithAppSec(t, restrict.AppSecObserve, nil, appsecRequest())
assert.Equal(t, http.StatusOK, rec.Code)
assert.True(t, reached)
assert.Equal(t, "appsec_unavailable", meta["appsec_verdict"])
assert.Equal(t, "observe", meta["appsec_mode"])
}
func TestCheckAppSec_EnforceFailsClosedWhenEngineUnreachable(t *testing.T) {
client, err := appsec.New(appsec.Config{URL: "http://127.0.0.1:1/", APIKey: "k"})
require.NoError(t, err)
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, appsecRequest())
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.False(t, reached)
assert.Equal(t, "appsec_unavailable", meta["appsec_verdict"])
}
func TestCheckAppSec_InspectsOverlayTraffic(t *testing.T) {
srv := appsecEngine(t, http.StatusForbidden, `{"action":"ban"}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
// Requests arriving over the WireGuard overlay skip the geo and IP-reputation
// checks, but request content is just as inspectable.
r := appsecRequest()
r = r.WithContext(types.WithOverlayOrigin(r.Context()))
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, r)
assert.Equal(t, http.StatusForbidden, rec.Code, "overlay traffic must still be inspected")
assert.False(t, reached)
assert.Equal(t, "appsec_ban", meta["appsec_verdict"])
}
func TestCheckAppSec_UnresolvableClientIPFailsClosed(t *testing.T) {
srv := appsecEngine(t, http.StatusOK, `{"action":"allow"}`)
client, err := appsec.New(appsec.Config{URL: srv.URL, APIKey: "k"})
require.NoError(t, err)
r := appsecRequest()
r.RemoteAddr = "not-an-address"
rec, meta, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, r)
assert.Equal(t, http.StatusForbidden, rec.Code,
"the engine requires a client address; a request we cannot attribute must not pass")
assert.False(t, reached)
assert.Equal(t, "appsec_unavailable", meta["appsec_verdict"])
}
// The redaction sets are resolved from the domain's schemes at registration, so
// what AppSec withholds cannot drift from what those schemes actually accept.
func TestAddDomain_ResolvesRedactionSetsFromSchemes(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
require.NoError(t, mw.AddDomain("svc.example.com", DomainSettings{
Schemes: []Scheme{
NewPassword(nil, "svc-1", "acct-1"),
NewHeader(nil, "svc-1", "acct-1", "X-Api-Key"),
},
SessionPublicKey: base64.StdEncoding.EncodeToString(make([]byte, ed25519.PublicKeySize)),
SessionExpiration: time.Hour,
AppSecMode: restrict.AppSecEnforce,
}))
mw.domainsMux.RLock()
config := mw.domains["svc.example.com"]
mw.domainsMux.RUnlock()
assert.Equal(t, []string{"password"}, config.redactBodyFields)
assert.Equal(t, []string{"X-Api-Key"}, config.redactHeaders)
// r.FormValue merges the query into the form, so a credential passed there
// authenticates and must be redacted alongside the OIDC session token.
assert.Equal(t, []string{"session_token", "password"}, config.redactQueryParams)
}
// countingBudget records reservations so a test can observe when the
// middleware hands them back.
type countingBudget struct {
mu sync.Mutex
total int64
used int64
maxAtOnce int64
}
func (b *countingBudget) Acquire(n int64) bool {
b.mu.Lock()
defer b.mu.Unlock()
if b.used+n > b.total {
return false
}
b.used += n
if b.used > b.maxAtOnce {
b.maxAtOnce = b.used
}
return true
}
func (b *countingBudget) Release(n int64) {
b.mu.Lock()
defer b.mu.Unlock()
b.used -= n
}
func (b *countingBudget) inUse() int64 {
b.mu.Lock()
defer b.mu.Unlock()
return b.used
}
// The buffered body stays alive as r.Body until the backend has read it, so
// Protect must hold the reservation for the whole request and return it only
// once the handler chain has unwound. Releasing inside Inspect would let the
// budget admit buffering that is still resident.
func TestProtect_AppSecBudgetHeldForRequestAndReleasedAfter(t *testing.T) {
srv := appsecEngine(t, http.StatusOK, `{"action":"allow"}`)
budget := &countingBudget{total: 1 << 20}
client, err := appsec.New(appsec.Config{
URL: srv.URL,
APIKey: "k",
MaxBodyBytes: 4096,
Budget: budget,
})
require.NoError(t, err)
mw := NewMiddleware(log.StandardLogger(), nil, nil)
mw.SetAppSec(client)
require.NoError(t, mw.AddDomain("svc.example.com", DomainSettings{
AccountID: types.AccountID("acct-1"),
ServiceID: types.ServiceID("svc-1"),
AppSecMode: restrict.AppSecEnforce,
}))
var inHandler int64
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// The backend reads the buffered body here, so the reservation must
// still be held at this point.
inHandler = budget.inUse()
_, _ = io.ReadAll(r.Body)
w.WriteHeader(http.StatusOK)
}))
r := httptest.NewRequest(http.MethodPost, "http://svc.example.com/", strings.NewReader("payload"))
r.Host = "svc.example.com"
cd := proxy.NewCapturedData("req-1")
r = r.WithContext(proxy.WithCapturedData(r.Context(), cd))
handler.ServeHTTP(httptest.NewRecorder(), r)
assert.Equal(t, int64(4096), inHandler, "the reservation must be held while the backend reads the body")
assert.Equal(t, int64(0), budget.inUse(), "Protect must release the reservation once the request is served")
}
// A denied request never reaches the backend, but Protect still has to hand the
// reservation back or the pool leaks one cap per blocked request.
func TestProtect_AppSecBudgetReleasedOnDeny(t *testing.T) {
srv := appsecEngine(t, http.StatusForbidden, `{"action":"ban"}`)
budget := &countingBudget{total: 1 << 20}
client, err := appsec.New(appsec.Config{
URL: srv.URL,
APIKey: "k",
MaxBodyBytes: 4096,
Budget: budget,
})
require.NoError(t, err)
r := httptest.NewRequest(http.MethodPost, "http://svc.example.com/", strings.NewReader("payload"))
r.Host = "svc.example.com"
rec, _, reached := serveWithAppSec(t, restrict.AppSecEnforce, client, r)
assert.False(t, reached, "a banned request must not reach the backend")
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Equal(t, int64(0), budget.inUse(), "a blocked request must not leak its reservation")
}

View File

@@ -39,6 +39,11 @@ func (Header) Type() auth.Method {
return auth.MethodHeader
}
// HeaderName returns the request header this scheme reads its credential from.
func (h Header) HeaderName() string {
return h.headerName
}
// Authenticate checks for the configured header in the request. If absent,
// returns empty (unauthenticated). If present, validates via gRPC.
func (h Header) Authenticate(r *http.Request) (string, string, error) {

View File

@@ -18,6 +18,7 @@ import (
"google.golang.org/grpc"
"github.com/netbirdio/netbird/proxy/auth"
"github.com/netbirdio/netbird/proxy/internal/appsec"
"github.com/netbirdio/netbird/proxy/internal/proxy"
"github.com/netbirdio/netbird/proxy/internal/restrict"
"github.com/netbirdio/netbird/proxy/internal/types"
@@ -25,6 +26,11 @@ import (
"github.com/netbirdio/netbird/shared/management/proto"
)
// sessionCookieNames is the cookie set AppSec redacts before mirroring: the
// proxy's session cookie is a bearer credential for the service, and the
// reverse proxy already strips it before forwarding upstream.
var sessionCookieNames = []string{auth.SessionCookieName}
// errValidationUnavailable indicates that session validation failed due to
// an infrastructure error (e.g. gRPC unavailable), not an invalid token.
var errValidationUnavailable = errors.New("session validation unavailable")
@@ -59,6 +65,14 @@ type DomainConfig struct {
IPRestrictions *restrict.Filter
// Private routes the domain through ValidateTunnelPeer; failure → 403.
Private bool
// AppSecMode enables CrowdSec AppSec request inspection for this domain.
AppSecMode restrict.AppSecMode
// redact* name the credentials this domain's schemes accept, resolved once
// at registration. AppSec replaces their values before mirroring a request,
// matching what the reverse proxy strips before forwarding upstream.
redactBodyFields []string
redactHeaders []string
redactQueryParams []string
}
type validationResult struct {
@@ -82,6 +96,9 @@ type Middleware struct {
sessionValidator SessionValidator
geo restrict.GeoResolver
tunnelCache *tunnelValidationCache
// appsec is the shared CrowdSec AppSec client, nil when the proxy has no
// AppSec endpoint configured. Set once during startup, before serving.
appsec *appsec.Client
}
// NewMiddleware creates a new authentication middleware. The sessionValidator is
@@ -99,6 +116,12 @@ func NewMiddleware(logger *log.Logger, sessionValidator SessionValidator, geo re
}
}
// SetAppSec installs the shared CrowdSec AppSec client. Must be called during
// startup, before the middleware serves any request.
func (mw *Middleware) SetAppSec(client *appsec.Client) {
mw.appsec = client
}
// Protect wraps next with per-domain authentication and IP restriction checks.
// Requests whose Host is not registered pass through unchanged.
func (mw *Middleware) Protect(next http.Handler) http.Handler {
@@ -123,6 +146,14 @@ func (mw *Middleware) Protect(next http.Handler) http.Handler {
return
}
// Deferred, not released here: the inspected body stays alive as r.Body
// until the backend has read it, which happens inside next.ServeHTTP.
appSecAllowed, releaseAppSec := mw.checkAppSec(w, r, config)
defer releaseAppSec()
if !appSecAllowed {
return
}
// Private services bypass operator schemes and gate on tunnel peer.
if config.Private {
if mw.forwardWithTunnelPeer(w, r, host, config, next) {
@@ -262,6 +293,134 @@ func (mw *Middleware) checkIPRestrictions(w http.ResponseWriter, r *http.Request
return false
}
// checkAppSec mirrors the request to the CrowdSec AppSec engine when the domain
// enables inspection. Returns false when the request was blocked and a response
// has been written.
//
// The returned release frees the body-buffering budget the inspection reserved
// and is never nil. It must run only after the request has been served, since
// the buffered body stays alive as r.Body for the backend to read.
//
// Every non-allow remediation blocks with 403, captcha included: the proxy has
// no challenge flow to serve. The distinct verdict is still recorded so the
// access log shows which remediation the engine actually chose.
//
// Unlike the geo and IP-reputation checks, this runs for overlay traffic too:
// AppSec inspects request content, which is just as meaningful when the caller
// reached the proxy through the WireGuard tunnel.
func (mw *Middleware) checkAppSec(w http.ResponseWriter, r *http.Request, config DomainConfig) (bool, func()) {
if !config.AppSecMode.Enabled() {
return true, func() {}
}
verdict, release := mw.inspectAppSec(r, config)
if verdict == restrict.Allow {
return true, release
}
observe := config.AppSecMode == restrict.AppSecObserve
if cd := proxy.CapturedDataFromContext(r.Context()); cd != nil {
cd.SetMetadata("appsec_verdict", verdict.String())
if observe {
cd.SetMetadata("appsec_mode", "observe")
}
}
if observe {
mw.logger.Debugf("AppSec observe: would block %s for %s (%s)", r.RemoteAddr, r.Host, verdict)
return true, release
}
mw.markDenied(r, verdict.String())
mw.logger.Debugf("AppSec: %s for %s %s", verdict, r.Host, r.RemoteAddr)
http.Error(w, "Forbidden", http.StatusForbidden)
return false, release
}
// inspectAppSec runs the AppSec call and returns its verdict. Failures come
// back as DenyAppSecUnavailable regardless of mode so observe mode still
// records that inspection did not happen; the caller decides what blocks. The
// returned release is never nil.
func (mw *Middleware) inspectAppSec(r *http.Request, config DomainConfig) (restrict.Verdict, func()) {
// Mode requested but the proxy has no AppSec endpoint configured. Management
// gates this on the cluster capability; a stale mapping can still arrive.
if mw.appsec == nil {
mw.logger.Debugf("AppSec mode %q requested for %s but no AppSec endpoint is configured", config.AppSecMode, r.Host)
return restrict.DenyAppSecUnavailable, func() {}
}
clientIP := mw.resolveClientIP(r)
if !clientIP.IsValid() {
// The engine requires a client address, and a request whose source we
// cannot establish is exactly the kind we must not wave through.
mw.logger.Debugf("AppSec: cannot resolve client address for %q", r.RemoteAddr)
return restrict.DenyAppSecUnavailable, func() {}
}
req := appsec.Request{
HTTP: r,
ClientIP: clientIP,
RedactBodyFields: config.redactBodyFields,
RedactHeaders: config.redactHeaders,
RedactCookies: sessionCookieNames,
RedactQueryParams: config.redactQueryParams,
}
if cd := proxy.CapturedDataFromContext(r.Context()); cd != nil {
req.TransactionID = cd.GetRequestID()
}
result, err := mw.appsec.Inspect(r.Context(), req)
if err != nil {
mw.logger.Debugf("AppSec inspection failed for %s: %v", r.Host, err)
}
// Record when the body went uninspected: headers and URI were still
// checked, but an operator reading the log should not read a clean verdict
// as "the payload was examined". Oversize is reachable by padding, so its
// absence from the log would hide a deliberate opt-out.
if result.BodyBypass != "" {
if cd := proxy.CapturedDataFromContext(r.Context()); cd != nil {
cd.SetMetadata("appsec_body_bypass", result.BodyBypass)
}
}
return result.Verdict, result.Release
}
// credentialFormFields lists the login form fields whose values are redacted
// from the mirrored body, so a password or PIN submitted to the proxy's own
// login form never reaches the Security Engine.
func credentialFormFields(schemes []Scheme) []string {
var fields []string
for _, s := range schemes {
switch s.Type() {
case auth.MethodPassword:
fields = append(fields, passwordFormId)
case auth.MethodPIN:
fields = append(fields, pinFormId)
}
}
return fields
}
// credentialHeaders lists the request headers whose values are redacted from
// the mirrored request. A header-auth scheme carries a session token the proxy
// validates and never forwards upstream, so the engine must not see it either.
func credentialHeaders(schemes []Scheme) []string {
var names []string
for _, s := range schemes {
// Structural, not a concrete Header assertion: if the scheme is ever
// registered as a pointer, a type assertion would quietly stop matching
// and the header would start reaching the engine again.
named, ok := s.(interface{ HeaderName() string })
if !ok {
continue
}
if name := named.HeaderName(); name != "" {
names = append(names, name)
}
}
return names
}
// resolveClientIP extracts the real client IP from CapturedData, falling back to r.RemoteAddr.
func (mw *Middleware) resolveClientIP(r *http.Request) netip.Addr {
if cd := proxy.CapturedDataFromContext(r.Context()); cd != nil {
@@ -281,12 +440,18 @@ func (mw *Middleware) resolveClientIP(r *http.Request) netip.Addr {
return addr.Unmap()
}
// blockIPRestriction sets captured data fields for an IP-restriction block event.
func (mw *Middleware) blockIPRestriction(r *http.Request, reason string) {
// markDenied records the deny reason on the captured data so the access log
// attributes the response to the proxy rather than the backend.
func (mw *Middleware) markDenied(r *http.Request, reason string) {
if cd := proxy.CapturedDataFromContext(r.Context()); cd != nil {
cd.SetOrigin(proxy.OriginAuth)
cd.SetAuthMethod(reason)
}
}
// blockIPRestriction sets captured data fields for an IP-restriction block event.
func (mw *Middleware) blockIPRestriction(r *http.Request, reason string) {
mw.markDenied(r, reason)
mw.logger.Debugf("IP restriction: %s for %s", reason, r.RemoteAddr)
}
@@ -637,45 +802,61 @@ func wasCredentialSubmitted(r *http.Request, method auth.Method) bool {
case auth.MethodPassword:
return r.FormValue("password") != ""
case auth.MethodOIDC:
return r.URL.Query().Get("session_token") != ""
return r.URL.Query().Get(sessionTokenParam) != ""
}
return false
}
// AddDomain registers authentication schemes for the given domain. With schemes a valid session public key is required.
// private=true forces ValidateTunnelPeer enforcement (403 on failure) regardless of the schemes list.
func (mw *Middleware) AddDomain(domain string, schemes []Scheme, publicKeyB64 string, expiration time.Duration, accountID types.AccountID, serviceID types.ServiceID, ipRestrictions *restrict.Filter, private bool) error {
if len(schemes) == 0 {
mw.domainsMux.Lock()
defer mw.domainsMux.Unlock()
mw.domains[domain] = DomainConfig{
AccountID: accountID,
ServiceID: serviceID,
IPRestrictions: ipRestrictions,
Private: private,
}
return nil
// DomainSettings is the per-domain configuration AddDomain applies.
type DomainSettings struct {
Schemes []Scheme
// SessionPublicKey is the base64-encoded ed25519 key used to verify session
// cookies. Required when Schemes is non-empty.
SessionPublicKey string
SessionExpiration time.Duration
AccountID types.AccountID
ServiceID types.ServiceID
IPRestrictions *restrict.Filter
// Private forces ValidateTunnelPeer enforcement (403 on failure) regardless
// of the schemes list.
Private bool
AppSecMode restrict.AppSecMode
}
// AddDomain registers authentication schemes for the given domain. With schemes
// a valid session public key is required.
func (mw *Middleware) AddDomain(domain string, settings DomainSettings) error {
credentialFields := credentialFormFields(settings.Schemes)
config := DomainConfig{
AccountID: settings.AccountID,
ServiceID: settings.ServiceID,
IPRestrictions: settings.IPRestrictions,
Private: settings.Private,
AppSecMode: settings.AppSecMode,
redactBodyFields: credentialFields,
redactHeaders: credentialHeaders(settings.Schemes),
// A credential can arrive in the query too: r.FormValue merges the URL
// query into the form, so "?password=..." authenticates just as a form
// post does and must not be mirrored in the clear either.
redactQueryParams: append([]string{sessionTokenParam}, credentialFields...),
}
pubKeyBytes, err := base64.StdEncoding.DecodeString(publicKeyB64)
if err != nil {
return fmt.Errorf("decode session public key for domain %s: %w", domain, err)
}
if len(pubKeyBytes) != ed25519.PublicKeySize {
return fmt.Errorf("invalid session public key size for domain %s: got %d, want %d", domain, len(pubKeyBytes), ed25519.PublicKeySize)
if len(settings.Schemes) > 0 {
pubKeyBytes, err := base64.StdEncoding.DecodeString(settings.SessionPublicKey)
if err != nil {
return fmt.Errorf("decode session public key for domain %s: %w", domain, err)
}
if len(pubKeyBytes) != ed25519.PublicKeySize {
return fmt.Errorf("invalid session public key size for domain %s: got %d, want %d", domain, len(pubKeyBytes), ed25519.PublicKeySize)
}
config.Schemes = settings.Schemes
config.SessionPublicKey = pubKeyBytes
config.SessionExpiration = settings.SessionExpiration
}
mw.domainsMux.Lock()
defer mw.domainsMux.Unlock()
mw.domains[domain] = DomainConfig{
Schemes: schemes,
SessionPublicKey: pubKeyBytes,
SessionExpiration: expiration,
AccountID: accountID,
ServiceID: serviceID,
IPRestrictions: ipRestrictions,
Private: private,
}
mw.domains[domain] = config
return nil
}
@@ -730,10 +911,10 @@ func (mw *Middleware) validateSessionToken(ctx context.Context, host, token stri
// parameter removed so it doesn't linger in the browser's address bar or history.
func stripSessionTokenParam(u *url.URL) string {
q := u.Query()
if !q.Has("session_token") {
if !q.Has(sessionTokenParam) {
return u.RequestURI()
}
q.Del("session_token")
q.Del(sessionTokenParam)
clean := *u
clean.RawQuery = q.Encode()
return clean.RequestURI()

View File

@@ -64,7 +64,7 @@ func TestAddDomain_ValidKey(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
err := mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour})
require.NoError(t, err)
mw.domainsMux.RLock()
@@ -81,7 +81,7 @@ func TestAddDomain_EmptyKey(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
err := mw.AddDomain("example.com", []Scheme{scheme}, "", time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionExpiration: time.Hour})
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid session public key size")
@@ -95,7 +95,7 @@ func TestAddDomain_InvalidBase64(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
err := mw.AddDomain("example.com", []Scheme{scheme}, "not-valid-base64!!!", time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: "not-valid-base64!!!", SessionExpiration: time.Hour})
require.Error(t, err)
assert.Contains(t, err.Error(), "decode session public key")
@@ -110,7 +110,7 @@ func TestAddDomain_WrongKeySize(t *testing.T) {
shortKey := base64.StdEncoding.EncodeToString([]byte("tooshort"))
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
err := mw.AddDomain("example.com", []Scheme{scheme}, shortKey, time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: shortKey, SessionExpiration: time.Hour})
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid session public key size")
@@ -123,7 +123,7 @@ func TestAddDomain_WrongKeySize(t *testing.T) {
func TestAddDomain_NoSchemes_NoKeyRequired(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{SessionExpiration: time.Hour})
require.NoError(t, err, "domains with no auth schemes should not require a key")
mw.domainsMux.RLock()
@@ -139,8 +139,8 @@ func TestAddDomain_OverwritesPreviousConfig(t *testing.T) {
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp1.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp2.PublicKey, 2*time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp1.PublicKey, SessionExpiration: time.Hour}))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp2.PublicKey, SessionExpiration: 2 * time.Hour}))
mw.domainsMux.RLock()
config := mw.domains["example.com"]
@@ -156,7 +156,7 @@ func TestRemoveDomain(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
mw.RemoveDomain("example.com")
@@ -180,7 +180,7 @@ func TestProtect_UnknownDomainPassesThrough(t *testing.T) {
func TestProtect_DomainWithNoSchemesPassesThrough(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
require.NoError(t, mw.AddDomain("example.com", nil, "", time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -197,7 +197,7 @@ func TestProtect_UnauthenticatedRequestIsBlocked(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
var backendCalled bool
backend := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -218,7 +218,7 @@ func TestProtect_HostWithPortIsMatched(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
var backendCalled bool
backend := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -239,7 +239,7 @@ func TestProtect_ValidSessionCookiePassesThrough(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
token, err := sessionkey.SignToken(kp.PrivateKey, "test-user", "", "example.com", auth.MethodPIN, nil, nil, time.Hour)
require.NoError(t, err)
@@ -272,7 +272,7 @@ func TestProtect_SessionCookieGroupsPropagate(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
groups := []string{"engineering", "sre"}
token, err := sessionkey.SignToken(kp.PrivateKey, "test-user", "", "example.com", auth.MethodPIN, groups, nil, time.Hour)
@@ -337,7 +337,7 @@ func TestProtect_PrivateService_TunnelPeerGroupsPropagate(t *testing.T) {
kp := generateTestKeyPair(t)
// Private service: no operator schemes — auth gates solely on the tunnel peer.
require.NoError(t, mw.AddDomain("agent.example.com", nil, kp.PublicKey, time.Hour, "acct-1", "svc-1", nil, true))
require.NoError(t, mw.AddDomain("agent.example.com", DomainSettings{SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acct-1", ServiceID: "svc-1", Private: true}))
cd := proxy.NewCapturedData("")
cd.SetClientIP(netip.MustParseAddr("100.90.1.14")) // CGNAT tunnel source
@@ -377,7 +377,7 @@ func TestProtect_PrivateService_TunnelPeerDenied(t *testing.T) {
}}
mw := NewMiddleware(log.StandardLogger(), validator, nil)
kp := generateTestKeyPair(t)
require.NoError(t, mw.AddDomain("agent.example.com", nil, kp.PublicKey, time.Hour, "acct-1", "svc-1", nil, true))
require.NoError(t, mw.AddDomain("agent.example.com", DomainSettings{SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acct-1", ServiceID: "svc-1", Private: true}))
cd := proxy.NewCapturedData("")
cd.SetClientIP(netip.MustParseAddr("100.90.1.14"))
@@ -405,7 +405,7 @@ func TestProtect_ExpiredSessionCookieIsRejected(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
// Sign a token that expired 1 second ago.
token, err := sessionkey.SignToken(kp.PrivateKey, "test-user", "", "example.com", auth.MethodPIN, nil, nil, -time.Second)
@@ -431,7 +431,7 @@ func TestProtect_WrongDomainCookieIsRejected(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
// Token signed for a different domain audience.
token, err := sessionkey.SignToken(kp.PrivateKey, "test-user", "", "other.com", auth.MethodPIN, nil, nil, time.Hour)
@@ -458,7 +458,7 @@ func TestProtect_WrongKeyCookieIsRejected(t *testing.T) {
kp2 := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp1.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp1.PublicKey, SessionExpiration: time.Hour}))
// Token signed with a different private key.
token, err := sessionkey.SignToken(kp2.PrivateKey, "test-user", "", "example.com", auth.MethodPIN, nil, nil, time.Hour)
@@ -495,7 +495,7 @@ func TestProtect_SchemeAuthRedirectsWithCookie(t *testing.T) {
return "", "pin", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
var backendCalled bool
backend := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -548,7 +548,7 @@ func TestProtect_FailedAuthDoesNotSetCookie(t *testing.T) {
return "", "pin", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -584,7 +584,7 @@ func TestProtect_MultipleSchemes(t *testing.T) {
return "", "password", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{pinScheme, passwordScheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{pinScheme, passwordScheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
var backendCalled bool
backend := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -614,7 +614,7 @@ func TestProtect_InvalidTokenFromSchemeReturns400(t *testing.T) {
return "invalid-jwt-token", "", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -638,7 +638,7 @@ func TestAddDomain_RandomBytes32NotEd25519(t *testing.T) {
key := base64.StdEncoding.EncodeToString(randomBytes)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
err = mw.AddDomain("example.com", []Scheme{scheme}, key, time.Hour, "", "", nil, false)
err = mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: key, SessionExpiration: time.Hour})
require.NoError(t, err, "any 32-byte key should be accepted at registration time")
}
@@ -647,10 +647,10 @@ func TestAddDomain_InvalidKeyDoesNotCorruptExistingConfig(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
// Attempt to overwrite with an invalid key.
err := mw.AddDomain("example.com", []Scheme{scheme}, "bad", time.Hour, "", "", nil, false)
err := mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: "bad", SessionExpiration: time.Hour})
require.Error(t, err)
// The original valid config should still be intact.
@@ -674,7 +674,7 @@ func TestProtect_FailedPinAuthCapturesAuthMethod(t *testing.T) {
return "", "pin", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
capturedData := proxy.NewCapturedData("")
handler := mw.Protect(newPassthroughHandler())
@@ -701,7 +701,7 @@ func TestProtect_FailedPasswordAuthCapturesAuthMethod(t *testing.T) {
return "", "password", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
capturedData := proxy.NewCapturedData("")
handler := mw.Protect(newPassthroughHandler())
@@ -728,7 +728,7 @@ func TestProtect_NoCredentialsDoesNotCaptureAuthMethod(t *testing.T) {
return "", "pin", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
capturedData := proxy.NewCapturedData("")
handler := mw.Protect(newPassthroughHandler())
@@ -815,8 +815,7 @@ func TestWasCredentialSubmitted(t *testing.T) {
func TestCheckIPRestrictions_UnparseableAddress(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", 0, "acc1", "svc1",
restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}}), false)
err := mw.AddDomain("example.com", DomainSettings{AccountID: "acc1", ServiceID: "svc1", IPRestrictions: restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}})})
require.NoError(t, err)
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -851,8 +850,7 @@ func TestCheckIPRestrictions_UsesCapturedDataClientIP(t *testing.T) {
// trusted proxies), checkIPRestrictions should use that IP, not RemoteAddr.
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", 0, "acc1", "svc1",
restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"203.0.113.0/24"}}), false)
err := mw.AddDomain("example.com", DomainSettings{AccountID: "acc1", ServiceID: "svc1", IPRestrictions: restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"203.0.113.0/24"}})})
require.NoError(t, err)
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -892,8 +890,7 @@ func TestCheckIPRestrictions_NilGeoWithCountryRules(t *testing.T) {
// Geo is nil, country restrictions are configured: must deny (fail-close).
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", 0, "acc1", "svc1",
restrict.ParseFilter(restrict.FilterConfig{AllowedCountries: []string{"US"}}), false)
err := mw.AddDomain("example.com", DomainSettings{AccountID: "acc1", ServiceID: "svc1", IPRestrictions: restrict.ParseFilter(restrict.FilterConfig{AllowedCountries: []string{"US"}})})
require.NoError(t, err)
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -916,11 +913,10 @@ func TestCheckIPRestrictions_NilGeoWithCountryRules(t *testing.T) {
func TestCheckIPRestrictions_OverlayOriginSkipsCountryRules(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", 0, "acc1", "svc1",
restrict.ParseFilter(restrict.FilterConfig{
AllowedCIDRs: []string{"100.64.0.0/10"},
AllowedCountries: []string{"US"},
}), false)
err := mw.AddDomain("example.com", DomainSettings{AccountID: "acc1", ServiceID: "svc1", IPRestrictions: restrict.ParseFilter(restrict.FilterConfig{
AllowedCIDRs: []string{"100.64.0.0/10"},
AllowedCountries: []string{"US"},
})})
require.NoError(t, err)
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -953,8 +949,7 @@ func TestCheckIPRestrictions_OverlayOriginSkipsCountryRules(t *testing.T) {
func TestCheckIPRestrictions_OverlayOriginRespectsCIDR(t *testing.T) {
mw := NewMiddleware(log.StandardLogger(), nil, nil)
err := mw.AddDomain("example.com", nil, "", 0, "acc1", "svc1",
restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"100.64.0.0/16"}}), false)
err := mw.AddDomain("example.com", DomainSettings{AccountID: "acc1", ServiceID: "svc1", IPRestrictions: restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"100.64.0.0/16"}})})
require.NoError(t, err)
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -982,7 +977,7 @@ func TestProtect_OIDCOnlyRedirectsDirectly(t *testing.T) {
return "", oidcURL, nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1011,7 +1006,7 @@ func TestProtect_OIDCWithOtherMethodShowsLoginPage(t *testing.T) {
return "", "pin", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{oidcScheme, pinScheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{oidcScheme, pinScheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1055,7 +1050,7 @@ func TestProtect_HeaderAuth_ForwardsOnSuccess(t *testing.T) {
kp := generateTestKeyPair(t)
hdr := newHeaderSchemeWithToken(t, kp, "X-API-Key", "secret-key")
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
var backendCalled bool
capturedData := proxy.NewCapturedData("")
@@ -1098,7 +1093,7 @@ func TestProtect_HeaderAuth_MissingHeaderFallsThrough(t *testing.T) {
hdr := newHeaderSchemeWithToken(t, kp, "X-API-Key", "secret-key")
// Also add a PIN scheme so we can verify fallthrough behavior.
pinScheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr, pinScheme}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr, pinScheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
handler := mw.Protect(newPassthroughHandler())
@@ -1118,7 +1113,7 @@ func TestProtect_HeaderAuth_WrongValueReturns401(t *testing.T) {
return &proto.AuthenticateResponse{Success: false}, nil
}}
hdr := NewHeader(mock, "svc1", "acc1", "X-API-Key")
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
capturedData := proxy.NewCapturedData("")
handler := mw.Protect(newPassthroughHandler())
@@ -1141,7 +1136,7 @@ func TestProtect_HeaderAuth_InfraErrorReturns502(t *testing.T) {
return nil, errors.New("gRPC unavailable")
}}
hdr := NewHeader(mock, "svc1", "acc1", "X-API-Key")
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
handler := mw.Protect(newPassthroughHandler())
@@ -1158,7 +1153,7 @@ func TestProtect_HeaderAuth_SubsequentRequestUsesSessionCookie(t *testing.T) {
kp := generateTestKeyPair(t)
hdr := newHeaderSchemeWithToken(t, kp, "X-API-Key", "secret-key")
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
@@ -1218,7 +1213,7 @@ func TestProtect_HeaderAuth_MultipleValuesSameHeader(t *testing.T) {
// Single Header scheme (as if one entry existed), but the mock checks both values.
hdr := NewHeader(mock, "svc1", "acc1", "Authorization")
require.NoError(t, mw.AddDomain("example.com", []Scheme{hdr}, kp.PublicKey, time.Hour, "acc1", "svc1", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{hdr}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour, AccountID: "acc1", ServiceID: "svc1"}))
var backendCalled bool
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -1276,7 +1271,7 @@ func TestProtect_OIDCOnPlainHTTP_BlockedWith400(t *testing.T) {
return "", "https://idp.example.com/authorize", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1300,7 +1295,7 @@ func TestProtect_OIDCOverTLS_NotBlocked(t *testing.T) {
return "", "https://idp.example.com/authorize", nil
},
}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1320,7 +1315,7 @@ func TestProtect_NonOIDCSchemes_PlainHTTP_NotBlocked(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1350,7 +1345,7 @@ func TestProtect_TunnelPeerFastPath_RequiresInboundMarker(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())
@@ -1385,7 +1380,7 @@ func TestProtect_TunnelPeerFastPath_TakesPathWithInboundMarker(t *testing.T) {
kp := generateTestKeyPair(t)
scheme := &stubScheme{method: auth.MethodPIN, promptID: "pin"}
require.NoError(t, mw.AddDomain("example.com", []Scheme{scheme}, kp.PublicKey, time.Hour, "", "", nil, false))
require.NoError(t, mw.AddDomain("example.com", DomainSettings{Schemes: []Scheme{scheme}, SessionPublicKey: kp.PublicKey, SessionExpiration: time.Hour}))
handler := mw.Protect(newPassthroughHandler())

View File

@@ -13,6 +13,10 @@ import (
"github.com/netbirdio/netbird/shared/management/proto"
)
// sessionTokenParam is the query parameter the management server uses to hand
// the minted session token back to the proxy after an OIDC login.
const sessionTokenParam = "session_token"
type urlGenerator interface {
GetOIDCURL(context.Context, *proto.GetOIDCURLRequest, ...grpc.CallOption) (*proto.GetOIDCURLResponse, error)
}
@@ -43,7 +47,7 @@ func (o OIDC) Authenticate(r *http.Request) (string, string, error) {
// Check for the session_token query param (from OIDC redirects).
// The management server passes the token in the URL because it cannot set
// cookies for the proxy's domain (cookies are domain-scoped per RFC 6265).
if token := r.URL.Query().Get("session_token"); token != "" {
if token := r.URL.Query().Get(sessionTokenParam); token != "" {
return token, "", nil
}

View File

@@ -44,7 +44,7 @@ func (s *stubSessionValidator) ValidateTunnelPeer(_ context.Context, in *proto.V
func newTunnelMiddleware(t *testing.T, validator SessionValidator) *Middleware {
t.Helper()
mw := NewMiddleware(log.New(), validator, nil)
require.NoError(t, mw.AddDomain("svc.example", nil, "", 0, "acct-1", "svc-1", nil, false))
require.NoError(t, mw.AddDomain("svc.example", DomainSettings{AccountID: "acct-1", ServiceID: "svc-1"}))
return mw
}
@@ -235,8 +235,8 @@ func TestForwardWithTunnelPeer_RoutesAccountIDIntoCacheKey(t *testing.T) {
}
mw := NewMiddleware(log.New(), validator, nil)
require.NoError(t, mw.AddDomain("svc-a.example", nil, "", 0, "acct-a", "svc-a", nil, false))
require.NoError(t, mw.AddDomain("svc-b.example", nil, "", 0, "acct-b", "svc-b", nil, false))
require.NoError(t, mw.AddDomain("svc-a.example", DomainSettings{AccountID: "acct-a", ServiceID: "svc-a"}))
require.NoError(t, mw.AddDomain("svc-b.example", DomainSettings{AccountID: "acct-b", ServiceID: "svc-b"}))
// The fast-path requires the inbound-listener marker on the context.
// The peerstore lookup itself is account-agnostic at this level
@@ -299,7 +299,7 @@ func TestForwardWithTunnelPeer_LocalLookupShortCircuitDoesNotPopulateCache(t *te
func TestPrivateService_FailsClosedOnTunnelPeerFailure(t *testing.T) {
mw := NewMiddleware(log.New(), nil, nil)
require.NoError(t, mw.AddDomain("private.svc", nil, "", 0, "acct-1", "svc-1", nil, true))
require.NoError(t, mw.AddDomain("private.svc", DomainSettings{AccountID: "acct-1", ServiceID: "svc-1", Private: true}))
called := false
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -328,7 +328,7 @@ func TestPrivateService_ForwardsOnTunnelPeerSuccess(t *testing.T) {
},
}
mw := NewMiddleware(log.New(), validator, nil)
require.NoError(t, mw.AddDomain("private.svc", nil, "", 0, "acct-1", "svc-1", nil, true))
require.NoError(t, mw.AddDomain("private.svc", DomainSettings{AccountID: "acct-1", ServiceID: "svc-1", Private: true}))
called := false
handler := mw.Protect(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {

View File

@@ -21,6 +21,8 @@ import (
"strconv"
"strings"
"sync"
"github.com/netbirdio/netbird/proxy/internal/netutil"
)
// MaxRoutingScanBytes bounds how far ScanRoutingFields will read into a
@@ -34,7 +36,6 @@ const MaxRoutingScanBytes int64 = 32 << 20
// metadata key by the chain when a request body is not surfaced.
const (
BypassUpgradeHeader = "upgrade_header"
BypassConnectionUpgrd = "connection_upgrade"
BypassContentType = "content_type_not_allowed"
BypassBudget = "capture_budget_exhausted"
BypassNoConfig = "no_capture_config"
@@ -125,12 +126,13 @@ func CaptureRequest(r *http.Request, cfg *Config, b Budget) (body []byte, trunca
if cfg.MaxRequestBytes <= 0 {
return nil, false, 0, BypassCapZero, release, nil
}
if r.Header.Get("Upgrade") != "" {
// The predicate has to be the forwarder's own: a looser one (either header
// on its own) skips capture for requests the forwarder still delivers to
// the upstream with their body intact, which hides them from every
// deny-capable middleware in the chain.
if netutil.IsUpgradeRequest(r.Header) {
return nil, false, 0, BypassUpgradeHeader, release, nil
}
if strings.EqualFold(r.Header.Get("Connection"), "upgrade") {
return nil, false, 0, BypassConnectionUpgrd, release, nil
}
if !contentTypeAllowed(r.Header.Get("Content-Type"), cfg.ContentTypes) {
return nil, false, 0, BypassContentType, release, nil
}

View File

@@ -0,0 +1,23 @@
package netutil
import (
"net/http"
"golang.org/x/net/http/httpguts"
)
// IsUpgradeRequest reports whether r is a protocol-upgrade request, using the
// same predicate httputil.ReverseProxy applies when it decides to hand the
// connection over instead of proxying normally.
//
// Matching the forwarder exactly matters for anything that inspects a request
// before it is proxied: a looser test (an Upgrade header on its own, say) marks
// a request as an upgrade and skips inspection, while the forwarder still
// delivers it to the backend as an ordinary request with its body intact. That
// gap is a body-inspection bypass reachable by adding one header.
func IsUpgradeRequest(h http.Header) bool {
if !httpguts.HeaderValuesContainsToken(h["Connection"], "Upgrade") {
return false
}
return h.Get("Upgrade") != ""
}

View File

@@ -50,6 +50,37 @@ const (
CrowdSecObserve CrowdSecMode = "observe"
)
// AppSecMode is the per-service CrowdSec AppSec (WAF) enforcement mode.
type AppSecMode string
const (
// AppSecOff disables request inspection.
AppSecOff AppSecMode = ""
// AppSecEnforce blocks requests the engine flags, and fails closed when the
// engine is unreachable.
AppSecEnforce AppSecMode = "enforce"
// AppSecObserve records the verdict without blocking.
AppSecObserve AppSecMode = "observe"
)
// ParseAppSecMode maps a wire value to an AppSecMode. Unrecognized values map
// to AppSecOff so a typo never turns inspection into an unintended block.
func ParseAppSecMode(s string) AppSecMode {
switch AppSecMode(s) {
case AppSecEnforce:
return AppSecEnforce
case AppSecObserve:
return AppSecObserve
default:
return AppSecOff
}
}
// Enabled reports whether the mode asks for request inspection.
func (m AppSecMode) Enabled() bool {
return m == AppSecEnforce || m == AppSecObserve
}
// Filter evaluates IP restrictions. CIDR checks are performed first
// (cheap), followed by country lookups (more expensive) only when needed.
type Filter struct {
@@ -146,6 +177,13 @@ const (
// DenyCrowdSecUnavailable indicates enforce mode but the bouncer has not
// completed its initial sync.
DenyCrowdSecUnavailable
// DenyAppSecBan indicates a CrowdSec AppSec "ban" remediation.
DenyAppSecBan
// DenyAppSecCaptcha indicates a CrowdSec AppSec "captcha" remediation.
DenyAppSecCaptcha
// DenyAppSecUnavailable indicates enforce mode but the AppSec engine could
// not produce a verdict (unreachable, timed out, or it rejected the call).
DenyAppSecUnavailable
)
// String returns the deny reason string matching the HTTP auth mechanism names.
@@ -167,6 +205,12 @@ func (v Verdict) String() string {
return "crowdsec_throttle"
case DenyCrowdSecUnavailable:
return "crowdsec_unavailable"
case DenyAppSecBan:
return "appsec_ban"
case DenyAppSecCaptcha:
return "appsec_captcha"
case DenyAppSecUnavailable:
return "appsec_unavailable"
default:
return "unknown"
}
@@ -182,6 +226,16 @@ func (v Verdict) IsCrowdSec() bool {
}
}
// IsAppSec returns true when the verdict originates from an AppSec inspection.
func (v Verdict) IsAppSec() bool {
switch v {
case DenyAppSecBan, DenyAppSecCaptcha, DenyAppSecUnavailable:
return true
default:
return false
}
}
// IsObserveOnly returns true when v is a CrowdSec verdict and the filter is in
// observe mode. Callers should log the verdict but not block the request.
func (f *Filter) IsObserveOnly(v Verdict) bool {

View File

@@ -126,6 +126,23 @@ type Config struct {
// CrowdSecAPIKey is the CrowdSec bouncer API key. Empty disables
// CrowdSec.
CrowdSecAPIKey string
// CrowdSecAppSecURL is the CrowdSec AppSec (WAF) endpoint. Empty disables
// HTTP request inspection.
CrowdSecAppSecURL string
// CrowdSecAppSecTimeout bounds a single AppSec inspection call. Zero falls
// back to the internal default.
CrowdSecAppSecTimeout time.Duration
// CrowdSecAppSecMaxBodyBytes caps the request body mirrored to AppSec.
// Zero falls back to the internal default; negative forwards no body.
CrowdSecAppSecMaxBodyBytes int64
// CrowdSecAppSecMaxConcurrent bounds AppSec inspections in flight toward
// the engine. Zero falls back to the internal default; negative removes
// the bound.
CrowdSecAppSecMaxConcurrent int
// MiddlewareCaptureBudgetBytes bounds the total request-body buffering in
// flight across the proxy, shared by AppSec inspection and the
// agent-network capture. Zero falls back to the internal default.
MiddlewareCaptureBudgetBytes int64
}
// New builds a Server from cfg without performing any I/O. No goroutines
@@ -135,42 +152,47 @@ type Config struct {
// directly) byte-for-byte equivalent.
func New(ctx context.Context, cfg Config) *Server {
return &Server{
ctx: ctx,
ListenAddr: cfg.ListenAddr,
ID: cfg.ID,
Logger: cfg.Logger,
Version: cfg.Version,
ProxyURL: cfg.ProxyURL,
ManagementAddress: cfg.ManagementAddress,
ProxyToken: cfg.ProxyToken,
CertificateDirectory: cfg.CertificateDirectory,
CertificateFile: cfg.CertificateFile,
CertificateKeyFile: cfg.CertificateKeyFile,
GenerateACMECertificates: cfg.GenerateACMECertificates,
ACMEChallengeAddress: cfg.ACMEChallengeAddress,
ACMEDirectory: cfg.ACMEDirectory,
ACMEEABKID: cfg.ACMEEABKID,
ACMEEABHMACKey: cfg.ACMEEABHMACKey,
ACMEChallengeType: cfg.ACMEChallengeType,
CertLockMethod: cfg.CertLockMethod,
WildcardCertDir: cfg.WildcardCertDir,
DebugEndpointEnabled: cfg.DebugEndpointEnabled,
DebugEndpointAddress: cfg.DebugEndpointAddress,
HealthAddress: cfg.HealthAddr,
ForwardedProto: cfg.ForwardedProto,
TrustedProxies: cfg.TrustedProxies,
WireguardPort: cfg.WireguardPort,
ProxyProtocol: cfg.ProxyProtocol,
PreSharedKey: cfg.PreSharedKey,
Performance: cfg.Performance,
SupportsCustomPorts: cfg.SupportsCustomPorts,
RequireSubdomain: cfg.RequireSubdomain,
Private: cfg.Private,
MaxDialTimeout: cfg.MaxDialTimeout,
MaxSessionIdleTimeout: cfg.MaxSessionIdleTimeout,
MappingBatchWatchdog: cfg.MappingBatchWatchdog,
GeoDataDir: cfg.GeoDataDir,
CrowdSecAPIURL: cfg.CrowdSecAPIURL,
CrowdSecAPIKey: cfg.CrowdSecAPIKey,
ctx: ctx,
ListenAddr: cfg.ListenAddr,
ID: cfg.ID,
Logger: cfg.Logger,
Version: cfg.Version,
ProxyURL: cfg.ProxyURL,
ManagementAddress: cfg.ManagementAddress,
ProxyToken: cfg.ProxyToken,
CertificateDirectory: cfg.CertificateDirectory,
CertificateFile: cfg.CertificateFile,
CertificateKeyFile: cfg.CertificateKeyFile,
GenerateACMECertificates: cfg.GenerateACMECertificates,
ACMEChallengeAddress: cfg.ACMEChallengeAddress,
ACMEDirectory: cfg.ACMEDirectory,
ACMEEABKID: cfg.ACMEEABKID,
ACMEEABHMACKey: cfg.ACMEEABHMACKey,
ACMEChallengeType: cfg.ACMEChallengeType,
CertLockMethod: cfg.CertLockMethod,
WildcardCertDir: cfg.WildcardCertDir,
DebugEndpointEnabled: cfg.DebugEndpointEnabled,
DebugEndpointAddress: cfg.DebugEndpointAddress,
HealthAddress: cfg.HealthAddr,
ForwardedProto: cfg.ForwardedProto,
TrustedProxies: cfg.TrustedProxies,
WireguardPort: cfg.WireguardPort,
ProxyProtocol: cfg.ProxyProtocol,
PreSharedKey: cfg.PreSharedKey,
Performance: cfg.Performance,
SupportsCustomPorts: cfg.SupportsCustomPorts,
RequireSubdomain: cfg.RequireSubdomain,
Private: cfg.Private,
MaxDialTimeout: cfg.MaxDialTimeout,
MaxSessionIdleTimeout: cfg.MaxSessionIdleTimeout,
MappingBatchWatchdog: cfg.MappingBatchWatchdog,
GeoDataDir: cfg.GeoDataDir,
CrowdSecAPIURL: cfg.CrowdSecAPIURL,
CrowdSecAPIKey: cfg.CrowdSecAPIKey,
CrowdSecAppSecURL: cfg.CrowdSecAppSecURL,
CrowdSecAppSecTimeout: cfg.CrowdSecAppSecTimeout,
CrowdSecAppSecMaxBodyBytes: cfg.CrowdSecAppSecMaxBodyBytes,
CrowdSecAppSecMaxConcurrent: cfg.CrowdSecAppSecMaxConcurrent,
MiddlewareCaptureBudgetBytes: cfg.MiddlewareCaptureBudgetBytes,
}
}

45
proxy/lifecycle_test.go Normal file
View File

@@ -0,0 +1,45 @@
package proxy
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// New maps Config onto Server field by field, and a field left out of that
// literal still compiles: the knob is simply parsed and then dropped, so an
// operator setting it sees the default with no error anywhere. These assertions
// are the only thing standing between a new setting and that silent no-op.
func TestNew_ForwardsOperatorTuning(t *testing.T) {
cfg := Config{
ManagementAddress: "http://localhost:8080",
ProxyToken: "token",
CrowdSecAPIURL: "http://crowdsec:8080/",
CrowdSecAPIKey: "key",
CrowdSecAppSecURL: "http://crowdsec:7422/",
CrowdSecAppSecTimeout: 321 * time.Millisecond,
CrowdSecAppSecMaxBodyBytes: 4321,
CrowdSecAppSecMaxConcurrent: 17,
MiddlewareCaptureBudgetBytes: 5 << 20,
MaxDialTimeout: 7 * time.Second,
MaxSessionIdleTimeout: 11 * time.Second,
GeoDataDir: "/var/lib/geo",
}
srv := New(context.Background(), cfg)
require.NotNil(t, srv)
assert.Equal(t, cfg.CrowdSecAPIURL, srv.CrowdSecAPIURL)
assert.Equal(t, cfg.CrowdSecAPIKey, srv.CrowdSecAPIKey)
assert.Equal(t, cfg.CrowdSecAppSecURL, srv.CrowdSecAppSecURL)
assert.Equal(t, cfg.CrowdSecAppSecTimeout, srv.CrowdSecAppSecTimeout)
assert.Equal(t, cfg.CrowdSecAppSecMaxBodyBytes, srv.CrowdSecAppSecMaxBodyBytes)
assert.Equal(t, cfg.CrowdSecAppSecMaxConcurrent, srv.CrowdSecAppSecMaxConcurrent)
assert.Equal(t, cfg.MiddlewareCaptureBudgetBytes, srv.MiddlewareCaptureBudgetBytes)
assert.Equal(t, cfg.MaxDialTimeout, srv.MaxDialTimeout)
assert.Equal(t, cfg.MaxSessionIdleTimeout, srv.MaxSessionIdleTimeout)
assert.Equal(t, cfg.GeoDataDir, srv.GeoDataDir)
}

View File

@@ -240,6 +240,10 @@ func (m *testProxyManager) ClusterSupportsCrowdSec(_ context.Context, _ string)
return nil
}
func (m *testProxyManager) ClusterSupportsAppSec(_ context.Context, _ string) *bool {
return nil
}
func (m *testProxyManager) ClusterSupportsPrivate(_ context.Context, _ string) *bool {
return nil
}
@@ -562,16 +566,11 @@ func TestIntegration_ProxyConnection_ReconnectDoesNotDuplicateState(t *testing.T
addMappingCalls.Add(1)
// Apply to real auth middleware (idempotent)
err := authMw.AddDomain(
mapping.GetDomain(),
nil,
"",
0,
proxytypes.AccountID(mapping.GetAccountId()),
proxytypes.ServiceID(mapping.GetId()),
nil,
mapping.GetPrivate(),
)
err := authMw.AddDomain(mapping.GetDomain(), auth.DomainSettings{
AccountID: proxytypes.AccountID(mapping.GetAccountId()),
ServiceID: proxytypes.ServiceID(mapping.GetId()),
Private: mapping.GetPrivate(),
})
require.NoError(t, err)
// Apply to real proxy (idempotent)

View File

@@ -45,6 +45,7 @@ import (
"github.com/netbirdio/netbird/client/embed"
"github.com/netbirdio/netbird/proxy/internal/accesslog"
"github.com/netbirdio/netbird/proxy/internal/acme"
"github.com/netbirdio/netbird/proxy/internal/appsec"
"github.com/netbirdio/netbird/proxy/internal/auth"
"github.com/netbirdio/netbird/proxy/internal/certwatch"
"github.com/netbirdio/netbird/proxy/internal/conntrack"
@@ -126,6 +127,10 @@ type Server struct {
crowdsecMu sync.Mutex
crowdsecServices map[types.ServiceID]bool
// appsecClient is the shared CrowdSec AppSec client, nil when no AppSec
// endpoint is configured. Stateless, so it needs no per-service lifecycle.
appsecClient *appsec.Client
// routerReady is closed once mainRouter is fully initialized.
// The mapping worker waits on this before processing updates.
routerReady chan struct{}
@@ -238,6 +243,20 @@ type Server struct {
CrowdSecAPIURL string
// CrowdSecAPIKey is the CrowdSec bouncer API key. Empty disables CrowdSec.
CrowdSecAPIKey string
// CrowdSecAppSecURL is the CrowdSec AppSec (WAF) endpoint, e.g.
// http://127.0.0.1:7422/. Empty disables request inspection. Requires
// CrowdSecAPIKey, which the AppSec component validates against LAPI.
CrowdSecAppSecURL string
// CrowdSecAppSecTimeout bounds a single AppSec inspection call.
// Zero means appsec.DefaultTimeout.
CrowdSecAppSecTimeout time.Duration
// CrowdSecAppSecMaxBodyBytes caps the request body mirrored to the AppSec
// engine. Zero means appsec.DefaultMaxBodyBytes; negative disables body
// forwarding, leaving header and URI inspection.
CrowdSecAppSecMaxBodyBytes int64
// CrowdSecAppSecMaxConcurrent bounds AppSec inspections in flight. Zero
// means appsec.DefaultMaxConcurrent; negative removes the bound.
CrowdSecAppSecMaxConcurrent int
// MaxSessionIdleTimeout caps the per-service session idle timeout.
// Zero means no cap (the proxy honors whatever management sends).
// Set via NB_PROXY_MAX_SESSION_IDLE_TIMEOUT for shared deployments.
@@ -380,6 +399,13 @@ func (s *Server) Start(ctx context.Context) error {
s.crowdsecRegistry = crowdsec.NewRegistry(s.CrowdSecAPIURL, s.CrowdSecAPIKey, log.NewEntry(s.Logger))
s.crowdsecServices = make(map[types.ServiceID]bool)
// Must precede the mapping worker: the worker opens the management stream
// and reports proxyCapabilities, which reads appsecClient. Building it
// afterwards would both race the read and, when the worker won, advertise
// the proxy as AppSec-incapable for the lifetime of that stream.
if err := s.initAppSec(); err != nil {
return err
}
go s.newManagementMappingWorker(runCtx, s.mgmtClient)
@@ -407,6 +433,7 @@ func (s *Server) Start(ctx context.Context) error {
}()
s.auth = auth.NewMiddleware(s.Logger, s.mgmtClient, s.geo)
s.auth.SetAppSec(s.appsecClient)
s.accessLog = accesslog.NewLogger(s.mgmtClient, s.Logger, s.TrustedProxies)
s.startDebugEndpoint()
@@ -1270,6 +1297,7 @@ func (s *Server) newManagementMappingWorker(ctx context.Context, client proto.Pr
func (s *Server) proxyCapabilities() *proto.ProxyCapabilities {
supportsCrowdSec := s.crowdsecRegistry.Available()
supportsAppSec := s.appsecClient != nil
privateCapability := s.Private
// Always true: this build enforces ProxyMapping.private via the auth middleware.
supportsPrivateService := true
@@ -1277,6 +1305,7 @@ func (s *Server) proxyCapabilities() *proto.ProxyCapabilities {
SupportsCustomPorts: &s.SupportsCustomPorts,
RequireSubdomain: &s.RequireSubdomain,
SupportsCrowdsec: &supportsCrowdSec,
SupportsAppsec: &supportsAppSec,
Private: &privateCapability,
SupportsPrivateService: &supportsPrivateService,
}
@@ -1904,6 +1933,67 @@ func (s *Server) parseRestrictions(mapping *proto.ProxyMapping) *restrict.Filter
})
}
// initAppSec builds the shared AppSec client when an endpoint is configured.
// A configured-but-invalid endpoint is a startup error rather than a silent
// downgrade: services asking for enforce would otherwise fail closed on every
// request with no indication why.
//
// Runs before the management stream opens so the reported capability is stable;
// the auth middleware picks the client up separately once it exists.
func (s *Server) initAppSec() error {
if s.CrowdSecAppSecURL == "" {
return nil
}
if s.CrowdSecAPIKey == "" {
return errors.New("crowdsec appsec url is set but the crowdsec api key is empty")
}
// Share the middleware capture budget rather than opening a second pool:
// AppSec buffers before authentication, so its ceiling has to count against
// the same proxy-wide allowance the body tap draws from.
var budget appsec.Budget
if s.middlewareManager != nil {
budget = s.middlewareManager.Budget()
}
client, err := appsec.New(appsec.Config{
URL: s.CrowdSecAppSecURL,
APIKey: s.CrowdSecAPIKey,
Timeout: s.CrowdSecAppSecTimeout,
MaxBodyBytes: s.CrowdSecAppSecMaxBodyBytes,
MaxConcurrent: s.CrowdSecAppSecMaxConcurrent,
Budget: budget,
Logger: log.NewEntry(s.Logger),
})
if err != nil {
return fmt.Errorf("init crowdsec appsec: %w", err)
}
s.appsecClient = client
s.Logger.Infof("CrowdSec AppSec inspection available at %s", s.CrowdSecAppSecURL)
return nil
}
// appSecMode resolves the per-service AppSec mode. A service asking for
// inspection on a proxy with no AppSec endpoint keeps its mode so the auth
// middleware fails closed for enforce, mirroring the CrowdSec behavior.
func (s *Server) appSecMode(mapping *proto.ProxyMapping) restrict.AppSecMode {
raw := mapping.GetAccessRestrictions().GetAppsecMode()
mode := restrict.ParseAppSecMode(raw)
// An unrecognized value disables inspection, which is the safe default but a
// silent one: with a newer management and an older proxy, a mode this build
// does not know would look identical to "off" on a service the operator set
// to enforce. Say so rather than leaving it to be discovered.
if mode == restrict.AppSecOff && raw != "" && raw != "off" {
s.Logger.Warnf("service %s requests unrecognized AppSec mode %q; this build supports %q and %q, so inspection is disabled",
mapping.GetId(), raw, restrict.AppSecEnforce, restrict.AppSecObserve)
}
if mode.Enabled() && s.appsecClient == nil {
s.Logger.Warnf("service %s requests AppSec mode %q but proxy has no AppSec endpoint configured", mapping.GetId(), mode)
}
return mode
}
// releaseCrowdSec releases the CrowdSec bouncer reference for the given
// service if it had one.
func (s *Server) releaseCrowdSec(svcID types.ServiceID) {
@@ -2070,7 +2160,17 @@ func (s *Server) updateMapping(ctx context.Context, mapping *proto.ProxyMapping)
s.warnIfGeoUnavailable(mapping.GetDomain(), mapping.GetAccessRestrictions())
maxSessionAge := time.Duration(mapping.GetAuth().GetMaxSessionAgeSeconds()) * time.Second
if err := s.auth.AddDomain(mapping.GetDomain(), schemes, mapping.GetAuth().GetSessionKey(), maxSessionAge, accountID, svcID, ipRestrictions, mapping.GetPrivate()); err != nil {
settings := auth.DomainSettings{
Schemes: schemes,
SessionPublicKey: mapping.GetAuth().GetSessionKey(),
SessionExpiration: maxSessionAge,
AccountID: accountID,
ServiceID: svcID,
IPRestrictions: ipRestrictions,
Private: mapping.GetPrivate(),
AppSecMode: s.appSecMode(mapping),
}
if err := s.auth.AddDomain(mapping.GetDomain(), settings); err != nil {
return fmt.Errorf("auth setup for domain %s: %w", mapping.GetDomain(), err)
}
m := s.protoToMapping(ctx, mapping)

View File

@@ -3379,6 +3379,18 @@ components:
- "observe"
default: "off"
description: CrowdSec IP reputation mode. Only available when the proxy cluster supports CrowdSec.
appsec_mode:
type: string
enum:
- "off"
- "enforce"
- "observe"
default: "off"
description: >-
CrowdSec AppSec (WAF) request inspection mode. Only available when
the proxy cluster supports AppSec, and only applied to HTTP
services. "enforce" blocks requests the WAF flags; "observe" records
the verdict in the access log without blocking.
PasswordAuthConfig:
type: object
properties:
@@ -3515,6 +3527,10 @@ components:
type: boolean
description: Whether all active proxies in the cluster have CrowdSec configured
example: false
supports_appsec:
type: boolean
description: Whether all active proxies in the cluster have a CrowdSec AppSec (WAF) endpoint configured
example: false
private:
type: boolean
description: True when at least one connected proxy in this cluster is running embedded in a netbird client (`netbird proxy`) and serving over a WireGuard tunnel. Lets the dashboard distinguish per-peer / private clusters from centralised ones.
@@ -3574,6 +3590,10 @@ components:
type: boolean
description: Whether the proxy cluster has CrowdSec configured
example: false
supports_appsec:
type: boolean
description: Whether the proxy cluster has a CrowdSec AppSec (WAF) endpoint configured
example: false
supports_private:
type: boolean
description: Whether the proxy cluster supports private (NetBird-only) services. True when at least one connected proxy in the cluster runs embedded in a netbird client.
@@ -4607,7 +4627,7 @@ components:
FleetDMMatchAttributes:
type: object
description: Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
description: Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
additionalProperties: false
properties:
disk_encryption_enabled:

View File

@@ -17,6 +17,27 @@ const (
TokenAuthScopes tokenAuthContextKey = "TokenAuth.Scopes"
)
// Defines values for AccessRestrictionsAppsecMode.
const (
AccessRestrictionsAppsecModeEnforce AccessRestrictionsAppsecMode = "enforce"
AccessRestrictionsAppsecModeObserve AccessRestrictionsAppsecMode = "observe"
AccessRestrictionsAppsecModeOff AccessRestrictionsAppsecMode = "off"
)
// Valid indicates whether the value is a known member of the AccessRestrictionsAppsecMode enum.
func (e AccessRestrictionsAppsecMode) Valid() bool {
switch e {
case AccessRestrictionsAppsecModeEnforce:
return true
case AccessRestrictionsAppsecModeObserve:
return true
case AccessRestrictionsAppsecModeOff:
return true
default:
return false
}
}
// Defines values for AccessRestrictionsCrowdsecMode.
const (
AccessRestrictionsCrowdsecModeEnforce AccessRestrictionsCrowdsecMode = "enforce"
@@ -1540,6 +1561,9 @@ type AccessRestrictions struct {
// AllowedCountries ISO 3166-1 alpha-2 country codes to allow. If non-empty, only these countries are permitted.
AllowedCountries *[]string `json:"allowed_countries,omitempty"`
// AppsecMode CrowdSec AppSec (WAF) request inspection mode. Only available when the proxy cluster supports AppSec, and only applied to HTTP services. "enforce" blocks requests the WAF flags; "observe" records the verdict in the access log without blocking.
AppsecMode *AccessRestrictionsAppsecMode `json:"appsec_mode,omitempty"`
// BlockedCidrs CIDR blocklist. Connections from these CIDRs are rejected. Evaluated after allowed_cidrs.
BlockedCidrs *[]string `json:"blocked_cidrs,omitempty"`
@@ -1550,6 +1574,9 @@ type AccessRestrictions struct {
CrowdsecMode *AccessRestrictionsCrowdsecMode `json:"crowdsec_mode,omitempty"`
}
// AccessRestrictionsAppsecMode CrowdSec AppSec (WAF) request inspection mode. Only available when the proxy cluster supports AppSec, and only applied to HTTP services. "enforce" blocks requests the WAF flags; "observe" records the verdict in the access log without blocking.
type AccessRestrictionsAppsecMode string
// AccessRestrictionsCrowdsecMode CrowdSec IP reputation mode. Only available when the proxy cluster supports CrowdSec.
type AccessRestrictionsCrowdsecMode string
@@ -2852,7 +2879,7 @@ type EDRFleetDMRequest struct {
// LastSyncedInterval The devices last sync requirement interval in hours. Minimum value is 24 hours
LastSyncedInterval int `json:"last_synced_interval"`
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
MatchAttributes FleetDMMatchAttributes `json:"match_attributes"`
}
@@ -2885,7 +2912,7 @@ type EDRFleetDMResponse struct {
// LastSyncedInterval The devices last sync requirement interval in hours.
LastSyncedInterval int `json:"last_synced_interval"`
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
// MatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
MatchAttributes FleetDMMatchAttributes `json:"match_attributes"`
// UpdatedAt Timestamp of when the integration was last updated.
@@ -3105,7 +3132,7 @@ type Event struct {
// EventActivityCode The string code of the activity that occurred during the event
type EventActivityCode string
// FleetDMMatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open source version. Premium-only attributes are marked accordingly
// FleetDMMatchAttributes Attribute conditions to match when approving FleetDM hosts. Most attributes work with FleetDM's free/open-source version. Premium-only attributes are marked accordingly
type FleetDMMatchAttributes struct {
// DiskEncryptionEnabled Whether disk encryption (FileVault/BitLocker) must be enabled on the host
DiskEncryptionEnabled *bool `json:"disk_encryption_enabled,omitempty"`
@@ -4749,6 +4776,9 @@ type ProxyCluster struct {
// RequireSubdomain Whether services on this cluster must include a subdomain label
RequireSubdomain *bool `json:"require_subdomain,omitempty"`
// SupportsAppsec Whether all active proxies in the cluster have a CrowdSec AppSec (WAF) endpoint configured
SupportsAppsec *bool `json:"supports_appsec,omitempty"`
// SupportsCrowdsec Whether all active proxies in the cluster have CrowdSec configured
SupportsCrowdsec *bool `json:"supports_crowdsec,omitempty"`
@@ -4817,6 +4847,9 @@ type ReverseProxyDomain struct {
// RequireSubdomain Whether a subdomain label is required in front of this domain. When true, the domain cannot be used bare.
RequireSubdomain *bool `json:"require_subdomain,omitempty"`
// SupportsAppsec Whether the proxy cluster has a CrowdSec AppSec (WAF) endpoint configured
SupportsAppsec *bool `json:"supports_appsec,omitempty"`
// SupportsCrowdsec Whether the proxy cluster has CrowdSec configured
SupportsCrowdsec *bool `json:"supports_crowdsec,omitempty"`

File diff suppressed because it is too large Load Diff

View File

@@ -73,6 +73,10 @@ message ProxyCapabilities {
optional bool private = 4;
// Whether the proxy enforces ProxyMapping.private (fails closed on ValidateTunnelPeer failure). Management MUST NOT stream private mappings to proxies that don't claim this.
optional bool supports_private_service = 5;
// Whether the proxy has a CrowdSec AppSec (WAF) endpoint configured and can
// inspect HTTP requests. Independent of supports_crowdsec: AppSec is a
// separate endpoint on the Security Engine and applies to HTTP services only.
optional bool supports_appsec = 6;
}
// GetMappingUpdateRequest is sent to initialise a mapping stream.
@@ -203,6 +207,10 @@ message AccessRestrictions {
repeated string blocked_countries = 4;
// CrowdSec IP reputation mode: "", "off", "enforce", or "observe".
string crowdsec_mode = 5;
// CrowdSec AppSec (WAF) request inspection mode: "", "off", "enforce", or
// "observe". HTTP services only: "enforce" and "observe" are rejected at
// validation for TCP/UDP/TLS services, which carry no requests to inspect.
string appsec_mode = 7;
}
message ProxyMapping {