mirror of
https://github.com/netbirdio/docs.git
synced 2026-10-25 22:19:04 +01:00
Document which connections survive a Management and Signal outage (#1059)
* Document which connections survive a Management and Signal outage Established connections, direct and relayed, keep working while Management and Signal are down, as long as their Relay is up; Relay checks credentials only when a peer connects. Opening a connection needs Signal, so a connection that drops, a new connection, and a connection that lazy connections close for inactivity wait until Signal is back. - Lazy Connections: new section on behaviour while Management or Signal is unavailable - Self-hosted Enterprise FAQ: scope the relayed-sessions sentence to open connections - High availability, Step 8: add the stop-every-Management-and-Signal scenario * Clarify the outage behaviour after a readability review - Lazy Connections: title the section by Signal, name the default single-container deployment where relayed connections drop, list what cannot open, keep the idle-close rule as the one lazy-specific exception, and say where, on which peers and when to set NB_LAZY_CONN=off - Enterprise FAQ: say relayed sessions in the combined container drop and reconnect once the server is back; scope the lazy case to lazy connections on - HA Step 8: separate direct from relayed, explain credentialsTTL, and say policy and configuration changes cannot be made until Management is back * Tighten the outage sections and align their wording across pages Cut lead-ins and repeated statements, use one phrasing for the connections that cannot open while Signal is down, and say that Signal runs in the single server on the Enterprise FAQ. * Describe the Signal outage behaviour without recommending NB_LAZY_CONN=off * State which connections still close while Signal is down - A client or device restart closes every connection on that device, and none reopens until Signal is back. - A lazy idle close needs a userspace WireGuard peer (always the case on Windows and macOS); two kernel WireGuard peers never close an idle connection. - The Enterprise FAQ and HA row qualify 'keep working' with both cases.
This commit is contained in:
@@ -33,6 +33,21 @@ Warm-up only applies to names in [custom DNS zones](/manage/dns/custom-zones) an
|
||||
|
||||
Set `NB_DNS_LAZY_WARMUP_TIMEOUT` on the daemon to change this per-query wait. The value must be a positive Go duration, for example `5s`. Invalid, zero, or negative values fall back to the `2s` default.
|
||||
|
||||
### When the Signal server is down
|
||||
|
||||
A Signal outage usually comes with a Management outage, for example while a single self-hosted server restarts.
|
||||
|
||||
Opening a connection needs Signal, because the two peers exchange connection offers through it. Keeping a connection open does not: its traffic goes directly between the peers or through a Relay server. So while Signal is down:
|
||||
|
||||
- Open direct connections stay up.
|
||||
- Open relayed connections stay up if Relay runs separately from Signal, as on NetBird Cloud and in self-hosted deployments with [external relays](/selfhosted/maintenance/scaling/set-up-external-relays). In the default self-hosted deployment, Relay runs in the same `netbird-server` container as Signal, so relayed connections drop.
|
||||
- Restarting the NetBird client, or the device it runs on, closes every connection on that device, and none reopens until Signal is back.
|
||||
- No connection can open. That includes one that dropped (for example after a network change), one to a peer the device has never connected to, and one that lazy connections closed for inactivity.
|
||||
|
||||
With lazy connections on, an idle connection, direct or relayed, still closes after the inactivity threshold (15 minutes by default) if either peer runs userspace WireGuard, as the Windows and macOS clients always do. It cannot reopen until Signal is back. If both peers run kernel WireGuard (possible only on Linux), the connection does not close when idle. A connection in use stays up.
|
||||
|
||||
Once Signal is back, every connection that could not open opens on the next traffic to its peer. You don't need to restart anything.
|
||||
|
||||
## Enable Lazy Connections in Management
|
||||
|
||||
The account setting in the NetBird Dashboard is the normal source of truth: go to `Settings` > `Clients` and turn on the `Enable Lazy Connections` toggle. When it is enabled, compatible clients activate their lazy connection manager. When it is disabled, clients stop lazy mode and immediately attempt to connect to all applicable peers.
|
||||
|
||||
@@ -47,7 +47,7 @@ Yes. A script migrates your existing open source deployment to the commercial li
|
||||
|
||||
Yes, with the commercial license. It helps to be precise about what "downtime" means here, because control-plane downtime is not the same as network downtime.
|
||||
|
||||
**Single-server deployment (no HA).** Upgrading restarts the server for the duration of the update. **Direct peer-to-peer connections keep working throughout**, because they do not pass through your servers. **Relayed sessions depend on your topology.** In the default combined deployment, the `netbird-server` container bundles Management, Signal, and Relay together, so recreating it restarts Relay and active relayed sessions reconnect. If you run Relay as a separate external service, it stays up during a Management restart and relayed sessions continue as well. While the server is down you cannot change configuration, apply new policies, or onboard new peers. For many teams a short maintenance window is acceptable.
|
||||
**Single-server deployment (no HA).** Upgrading restarts the server for the duration of the update. **Open direct peer-to-peer connections keep working**, because they do not pass through your servers. **Relayed connections depend on your topology.** In the default combined deployment, the `netbird-server` container bundles Management, Signal, and Relay together, so restarting it drops open relayed connections. If you run Relay as a separate external service, it stays up during the restart and open relayed connections continue. Two things can still close a connection during the restart: restarting a client, or the device it runs on, closes all of its connections, and with [lazy connections](/manage/peers/lazy-connection) enabled, a connection idle for longer than the inactivity threshold (15 minutes by default) closes if either peer runs userspace WireGuard, as the Windows and macOS clients always do. Opening a connection needs Signal, which runs in this server, so until the server is back no connection can open, including one to a peer the device has never connected to, one that dropped, and one closed by a client restart or for inactivity. Once the server is back, each of those connections opens on the next traffic to its peer. While the server is down you cannot change policies or other configuration, or enroll peers. For many teams a short maintenance window is acceptable.
|
||||
|
||||
**High-availability deployment (commercial license).** Management, Signal, and Relay each run as a load-balanced pool of two or more instances, backed by shared PostgreSQL, Redis, and NATS. You upgrade by draining one instance, upgrading it, returning it to the load balancer, and repeating. This is a rolling upgrade with **no control-plane downtime** at any point. Schema migrations run once, automatically, on the first upgraded Management instance.
|
||||
|
||||
|
||||
@@ -825,6 +825,7 @@ Run each failure scenario and confirm the expected behavior. Do this in a stagin
|
||||
| Stop one Management replica | The Management LB marks it unhealthy within seconds; API traffic continues on remaining replicas. Established peer connections (signal, relay) are unaffected because they flow through the Signal LB and directly to the Relay instances, not through the Management LB. |
|
||||
| Stop one Signal instance | The Signal LB marks it unhealthy; peers connected to that instance reconnect via the LB and land on a surviving Signal instance. Cross-peer signaling continues via NATS without interruption to peers that were already connected to a different instance. |
|
||||
| Stop one Relay instance | Peers homed on that instance move to another by themselves: with option 1 to the next instance that answers, with option 2 to the instance DNS returns once its health check drops the stopped one. Relayed connections through the stopped instance drop and re-establish. Peers homed on other instances are unaffected. |
|
||||
| Stop every Management and Signal instance | Open direct connections keep working. Open relayed connections keep working while their Relay instance is up, even past `credentialsTTL` (the lifetime of the relay credentials Management issues). Relay checks credentials only when a peer connects to it. Two things can still close a connection during the outage: restarting a client, or the device it runs on, closes all of its connections, and with [lazy connections](/manage/peers/lazy-connection) enabled, a connection idle for longer than the inactivity threshold (15 minutes by default) closes if either peer runs userspace WireGuard, as the Windows and macOS clients always do. Opening a connection needs Signal, so until Signal is back no connection can open, including one to a peer the device has never connected to, one that dropped, and one closed by a client restart or for inactivity. Once Signal is back, each of those connections opens on the next traffic to its peer. You cannot change policies or other configuration, or enroll peers, until Management is back. |
|
||||
| Stop one NATS node | Cluster retains quorum (2 of 3 healthy). Writes to the `traffic-events` stream continue. Cross-instance Signal routing continues. |
|
||||
| Disconnect Redis | New OAuth flows fail until Redis returns; established peer connections continue working. Dynamic log/rate-limit changes pause. |
|
||||
| Trigger PostgreSQL failover (managed service) | Brief outage during the failover; Management replicas reconnect to the new primary and resume. Existing peer connections survive the gap because they don't touch PostgreSQL on every message. |
|
||||
|
||||
Reference in New Issue
Block a user