People who only ever reach private services through the reverse proxy were
invisible to activity accounting. Active users are counted from user.LastLogin
or from the LastSeen of a peer they own, and neither column was written on the
proxy paths — so a person signing in via SSO to a proxied service, or a peer
serving one over the mesh, never showed up in the 24 hour numbers.
Both writes now happen where the proxy already authenticates:
- GenerateSessionToken stamps LastLogin after the session token is signed,
the same column and the same way the dashboard and device login paths do.
- ValidateTunnelPeer stamps the calling peer's LastSeen, the column its owner
activates through.
The policy lives in a new reverseproxy/activity manager rather than in the gRPC
service, matching the module layout the other reverse proxy domains use. It
skips what can never count — service users, embedded proxy peers and WASM
clients — and throttles peer writes to once an hour, well inside the window
accounting asks about and far above the proxy's five minute tunnel cache.
The peer write is a single indexed UPDATE that touches only
peer_status_last_seen. Connected and SessionStartedAt are left alone so the
session-ownership fencing MarkPeerConnectedIfNewerSession relies on is never
disturbed, and the timestamp comes from the database clock rather than the
caller, for the same reason the other status writers take it from there. The
caller's cutoff travels into the statement's WHERE, so concurrent requests for
one peer collapse into a single write instead of each acting on its own stale
read, and a peer that was never seen — NULL last seen, since Status is an
embedded pointer — still records its first activity.
Nothing outside the reverse proxy changes behaviour: the only addition
elsewhere is the RefreshPeerLastSeen store method the manager calls.
Store the per-account gateway endpoint as {domain, proxy_address} with a
global unique index on the full hostname; dedicated = (domain ==
proxy_address). Bootstrap becomes an explicit POST carrying exactly one
of proxy_address (server allocates an adjective-noun label beneath it)
or endpoint (claimed verbatim, address-first); provider create loses its
bootstrap side effect. PUT is a full replace with every field required —
the immutable identity fields must be echoed unchanged and a mismatch is
rejected with 422. A guarded DELETE releases the endpoint: refused with
412 while providers exist or a proxy is actively serving the endpoint
hostname (matched case-insensitively); re-creating bootstraps fresh. A
self-addressed pin excludes its address from the account's cluster allow
list, and the live mapping update path now addresses the serving proxy
from the synthesized service. Existing rows are migrated on all three
store engines.
## Summary
Adds a unified `admin` CLI for self-hosted instance administrators in
both the management and combined binaries.
## User Management
### `admin user change-password`
- Changes a local embedded IdP user's password.
- Selects the user with `--email` or `--user-id`.
- Reads the new password from `--password` or `--password-file`.
- Clears the user's local authentication session so the new password is
required on the next login.
- **Alias:** `admin user set-password`.
### `admin user reset-mfa`
- Resets a local embedded IdP user's MFA enrollment.
- Selects the user with `--email` or `--user-id`.
- Clears TOTP/WebAuthn enrollment data and removes the local
authentication session.
- The user will re-enroll MFA on the next login.
## MFA Management
### `admin mfa status`
- Shows whether local MFA is enabled in the account settings.
- Checks the embedded IdP client configuration and reports whether MFA
is enabled there.
### `admin mfa enable`
- Enables local MFA for embedded IdP users.
- Updates embedded IdP clients and saves the account MFA setting.
- Records an audit event on a best-effort basis.
### `admin mfa disable`
- Disables local MFA for embedded IdP users.
- Updates embedded IdP clients and saves the account MFA setting.
- Records an audit event on a best-effort basis.
## Reverse Proxy Tokens
### `admin token create --name <name> [--expires-in <duration>]`
- Creates a reverse proxy access token.
- Prints the plaintext token once, along with the token ID.
- `--expires-in` supports values such as `24h`, `30d`, or `365d`. If
omitted, the token never expires.
### `admin token list`
- Lists reverse proxy access tokens.
- Shows the token ID, name, creation date, expiration, last-used time,
and revocation status.
- **Alias:** `admin token ls`.
### `admin token revoke <token-id>`
- Revokes a reverse proxy access token.
- Revoked tokens can no longer authenticate reverse proxy instances.
## Reverse Proxy Management
### `admin proxy disconnect-all`
- Lists registered reverse proxy instances and force-marks all connected
instances as disconnected.
- Useful for repairing stale proxy state after an unclean management
server shutdown.
- Prompts for confirmation by default.
- `--dry-run` previews the changes without applying them.
- `--force` skips the confirmation prompt.
- Live proxies may appear again after their next heartbeat, reconnect,
or re-registration.
## Compatibility Commands
### `token ...`
- Deprecated top-level compatibility path.
- Behaves the same as `admin token ...`.
- Retained so existing scripts using `token create`, `token list`, or
`token revoke` continue to work.
## Changes
- Adds reusable `management/cmd/admin` command package.
- Wires `admin` into `netbird-mgmt` and `combined`.
- Adds local user password reset with existing password strength
validation.
- Adds local MFA enrollment reset by clearing Dex TOTP/WebAuthn
credentials and local auth sessions.
- Adds local MFA enable/disable/status helpers for embedded IdP
deployments.
- Moves proxy access token commands under `admin token` for a single
admin-focused CLI entry point.
- Exports `server.ValidatePassword` for reuse by CLI helpers.
## Tests
```bash
go test ./management/cmd/...
go test ./management/cmd/admin ./management/cmd ./combined/cmd
go test ./management/server -run TestValidatePassword
```
Pre-push lint also passed.
### Checklist
- [ ] Is it a bug fix
- [ ] Is a typo/documentation fix
- [x] Is a feature enhancement
- [ ] It is a refactor
- [ ] Created tests that fail without the change (if possible)
- [ ] This change does **not** modify the public API, gRPC protocols,
functionality behavior, CLI / service flags, or introduce a new feature
— **OR** I have discussed it with the NetBird team beforehand (link the
issue / Slack thread in the description). See
[CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first).
> By submitting this pull request, you confirm that you have read and
agree to the terms of the [Contributor License
Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md).
## Documentation
Select exactly one:
- [x] I added/updated documentation for this change
- [ ] Documentation is **not needed** for this change (explain why)
### Docs PR URL (required if "docs added" is checked)
Paste the PR link from https://github.com/netbirdio/docs here:
https://github.com/netbirdio/docs/pull/832
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added self-hosted admin CLI commands for changing passwords, resetting
MFA (including WebAuthn), and managing embedded IdP client MFA
(enable/disable/status).
* Introduced a unified admin command entry point and improved
data-directory handling for embedded IdP storage.
* **Refactor**
* Centralized password strength validation into a shared exported
validator.
* **Tests**
* Added a comprehensive admin command test suite covering password
input, selectors, MFA reset, and client MFA state handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds a new "private" service mode for the reverse proxy: services reachable exclusively over the embedded WireGuard tunnel, gated by per-peer group membership instead of operator auth schemes.
Wire contract
- ProxyMapping.private (field 13): the proxy MUST call ValidateTunnelPeer and fail closed; operator schemes are bypassed.
- ProxyCapabilities.private (4) + supports_private_service (5): capability gate. Management never streams private mappings to proxies that don't claim the capability; the broadcast path applies the same filter via filterMappingsForProxy.
- ValidateTunnelPeer RPC: resolves an inbound tunnel IP to a peer, checks the peer's groups against service.AccessGroups, and mints a session JWT on success. checkPeerGroupAccess fails closed when a private service has empty AccessGroups.
- ValidateSession/ValidateTunnelPeer responses now carry peer_group_ids + peer_group_names so the proxy can authorise policy-aware middlewares without an extra management round-trip.
- ProxyInboundListener + SendStatusUpdate.inbound_listener: per-account inbound listener state surfaced to dashboards.
- PathTargetOptions.direct_upstream (11): bypass the embedded NetBird client and dial the target via the proxy host's network stack for upstreams reachable without WireGuard.
Data model
- Service.Private (bool) + Service.AccessGroups ([]string, JSON- serialised). Validate() rejects bearer auth on private services. Copy() deep-copies AccessGroups. pgx getServices loads the columns.
- DomainConfig.Private threaded into the proxy auth middleware. Request handler routes private services through forwardWithTunnelPeer and returns 403 on validation failure.
- Account-level SynthesizePrivateServiceZones (synthetic DNS) and injectPrivateServicePolicies (synthetic ACL) gate on len(svc.AccessGroups) > 0.
Proxy
- /netbird proxy --private (embedded mode) flag; Config.Private in proxy/lifecycle.go.
- Per-account inbound listener (proxy/inbound.go) binding HTTP/HTTPS on the embedded NetBird client's WireGuard tunnel netstack.
- proxy/internal/auth/tunnel_cache: ValidateTunnelPeer response cache with single-flight de-duplication and per-account eviction.
- Local peerstore short-circuit: when the inbound IP isn't in the account roster, deny fast without an RPC.
- proxy/server.go reports SupportsPrivateService=true and redacts the full ProxyMapping JSON from info logs (auth_token + header-auth hashed values now only at debug level).
Identity forwarding
- ValidateSessionJWT returns user_id, email, method, groups, group_names. sessionkey.Claims carries Email + Groups + GroupNames so the proxy can stamp identity onto upstream requests without an extra management round-trip on every cookie-bearing request.
- CapturedData carries userEmail / userGroups / userGroupNames; the proxy stamps X-NetBird-User and X-NetBird-Groups on r.Out from the authenticated identity (strips client-supplied values first to prevent spoofing).
- AccessLog.UserGroups: access-log enrichment captures the user's group memberships at write time so the dashboard can render group context without reverse-resolving stale memberships.
OpenAPI/dashboard surface
- ReverseProxyService gains private + access_groups; ReverseProxyCluster gains private + supports_private. ReverseProxyTarget target_type enum gains "cluster". ServiceTargetOptions gains direct_upstream. ProxyAccessLog gains user_groups.
The cluster listing now answers three questions in one round-trip
instead of forcing the dashboard to cross-reference the domains API:
which clusters can this account see, are they currently up, and what
do they support. The ProxyCluster wire type drops the boolean
self_hosted in favour of a `type` enum (`account` / `shared`) plus
explicit `online`, `supports_custom_ports`, `require_subdomain`, and
`supports_crowdsec` fields.
Store query reworked so offline clusters still appear (no last_seen
WHERE), with online and connected_proxies both derived from the
existing 2-min active window via portable CASE expressions; the
1-hour heartbeat reaper still removes long-stale rows. Service
manager enriches each cluster with the capability flags via the
existing per-cluster lookups (CapabilityProvider now also exposes
ClusterSupportsCrowdSec).
GetActiveClusterAddresses* keep their tight 2-min filter so service
routing and domain enumeration aren't pulled into the wider window.
The hard cut removes self_hosted from the response — the dashboard is
the only consumer and is updated in the matching PR; no transitional
field is shipped.
Adds a cross-engine regression test asserting offline clusters
surface, connected_proxies counts only fresh proxies, and
account-scoped BYOP clusters never leak across accounts.
* [management] Fence peer status updates with a session token
The connect/disconnect path used a best-effort LastSeen-after-streamStart
comparison to decide whether a status update should land. Under contention
— a re-sync arriving while the previous stream's disconnect was still in
flight, or two management replicas seeing the same peer at once — the
check was a read-then-decide-then-write window: any UPDATE in between
caused the wrong row to be written. The Go-side time.Now() that fed the
comparison also drifted under lock contention, since it was captured
seconds before the write actually committed.
Replace it with an integer-nanosecond fencing token stored alongside the
status. Every gRPC sync stream uses its open time (UnixNano) as its token.
Connects only land when the incoming token is strictly greater than the
stored one; disconnects only land when the incoming token equals the
stored one (i.e. we're the stream that owns the current session). Both
are single optimistic-locked UPDATEs — no read-then-write, no transaction
wrapper.
LastSeen is now written by the database itself (CURRENT_TIMESTAMP). The
caller never supplies it, so the value always reflects the real moment
of the UPDATE rather than the moment the caller queued the work — which
was already off by minutes under heavy lock contention.
Side effects (geo lookup, peer-login-expiration scheduling, network-map
fan-out) are explicitly documented as running after the fence UPDATE
commits, never inside it. Geo also skips the update when realIP equals
the stored ConnectionIP, dropping a redundant SavePeerLocation call on
same-IP reconnects.
Tests cover the three semantic cases (matched disconnect lands, stale
disconnect dropped, stale connect dropped) plus a 16-goroutine race test
that asserts the highest token always wins.
* [management] Add SessionStartedAt to peer status updates
Stored `SessionStartedAt` for fencing token propagation across goroutines and updated database queries/functions to handle the new field. Removed outdated geolocation handling logic and adjusted tests for concurrency safety.
* Rename `peer_status_required_approval` to `peer_status_requires_approval` in SQL store fields
The expose tracker used sync.Map for in-memory TTL tracking of active expose sessions, which broke and lost all sessions on restart.
Replace with SQL-backed operations that reuse the existing meta_last_renewed_at column:
- Add store methods: RenewEphemeralService, GetExpiredEphemeralServices, CountEphemeralServicesByPeer, EphemeralServiceExists
- Move duplicate/limit checks inside a transaction with row-level locking (SELECT ... FOR UPDATE) to prevent concurrent bypass
- Reaper re-checks expiry under row lock to avoid deleting a just-renewed service and prevent duplicate event emission
- Add composite index on (source, source_peer) for efficient queries
- Batch-limit and column-select the reaper query to avoid DB/GC spikes
- Filter out malformed rows with empty source_peer
dmitri-netbird