From c69c80509cbe1dbeb749a193f3f7ede8a2c74671 Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Fri, 16 Oct 2026 10:25:18 +0200 Subject: [PATCH] docs: correct and complete HA routing peer selection details Follow-up to #1043. Express the latency noise term in the uncertainty the client logs as +/- next to each latency (it is not the raw jitter), scope the margin examples to settled paths, correct the latency measurement and dwell rules, add the idle switch and the delay expiry trigger, state how much the relayed switch delay adds to a failover when a routing peer fails without shutting down, note the 10-event limit, and update the selection summaries on the Networks, Sizing Routing Peers and Network Routes pages. --- src/pages/client/environment-variables.mdx | 2 +- src/pages/manage/network-routes/index.mdx | 4 +-- .../networks/how-routing-peers-work.mdx | 26 +++++++++++-------- src/pages/manage/networks/index.mdx | 2 +- .../manage/networks/sizing-routing-peers.mdx | 2 +- 5 files changed, 20 insertions(+), 16 deletions(-) diff --git a/src/pages/client/environment-variables.mdx b/src/pages/client/environment-variables.mdx index 9b46aa1b..7a9ad078 100644 --- a/src/pages/client/environment-variables.mdx +++ b/src/pages/client/environment-variables.mdx @@ -35,7 +35,7 @@ To clear all saved service parameters (including env vars), run `sudo netbird se | `NB_DISABLE_CUSTOM_ROUTING` | All | Revert to the routing behavior from before exit node support was added. No exclusion routes or fwmark-based socket routing will be used; all dialers and listeners fall back to plain `net.Dial`/`net.Listen`. Routes with a prefix of /7 or larger (e.g. default routes, 0.0.0.0/0) will be rejected. | | `NB_ROUTE_PROTO_FLAG` | macOS, BSD | Set a custom route flag on routes the client adds to the routing table. Accepted values: `2` (RTF_PROTO2) or `3` (RTF_PROTO3); defaults to RTF_PROTO1. Not used on Linux (Linux uses netlink with its own protocol field). | | `NB_DISABLE_ROUTE_CACHE` | Windows | Disable the 2-second cache on Windows routing table lookups. The client reads the OS routing table (via Win32 API) when adding or removing routes; without the cache, each operation triggers a fresh read. Only needed for debugging route conflicts. | -| `NB_HA_RELAYED_SWITCH_DELAY` | All | How long a relayed routing peer keeps a highly available network before the route moves to an equivalent routing peer with a direct connection. Takes a duration (default `30s`), `0` to switch at once, or `never` to keep a routing peer whose direct connection fell back to relayed for as long as it stays connected. Available from v0.81.1. See [Relayed switch delay](/manage/networks/how-routing-peers-work#relayed-switch-delay). | +| `NB_HA_RELAYED_SWITCH_DELAY` | All | How long a relayed routing peer keeps a highly available network before the route moves to an equivalent routing peer with a direct connection. Takes a duration (default `30s`), `0` to switch at once, or `never` to keep a routing peer whose direct connection fell back to relayed for as long as it stays connected. Set it on the peer that uses the network, not on the routing peers. Available from v0.81.1. See [Relayed switch delay](/manage/networks/how-routing-peers-work#relayed-switch-delay). | | `NB_FWMARK_BASE` | Linux | Move the netfilter mark range the client uses (default base `0x1BD00`, covering `0x1BD00`-`0x1BD0FF`). Set this when other software on the host claims bits the NetBird marks use, so its rules act on NetBird traffic. The value is the base of the range, decimal or `0x`-prefixed, and the low byte identifies the individual mark, so it has to be left free: `NB_FWMARK_BASE=0x11000` puts the marks at `0x11000`-`0x110FF`. A value that is zero, larger than 32 bits, or has a bit set in its low byte is rejected with a warning in the log and the default range is used. | Container network plugins, CNIs and other VPNs claim bits of the mark space for themselves. When one of them has a rule matching a bit that a NetBird mark also sets, that rule acts on NetBird traffic, which shows up as traffic being dropped, masqueraded, or routed elsewhere. `NB_FWMARK_BASE` moves the whole NetBird range out of the way. diff --git a/src/pages/manage/network-routes/index.mdx b/src/pages/manage/network-routes/index.mdx index 318716f5..6f1d618e 100644 --- a/src/pages/manage/network-routes/index.mdx +++ b/src/pages/manage/network-routes/index.mdx @@ -73,7 +73,7 @@ Metric determines routing peer priority in high availability configurations. Low A highly available route combines multiple routes with the same network identifier and ranges but different routing peers or peer groups. This provides redundant paths between your peers and external networks. -Connected peers select a routing peer based on connection type and metric priority. +Each connected peer selects one routing peer by metric (lower wins), then connection type (direct beats relayed), then latency. See [High availability](/manage/networks/how-routing-peers-work#high-availability) for the selection rules and what makes a peer switch. ### DNS Routes @@ -177,7 +177,7 @@ Peers connected to either `aws-nb-europe-router-az-a` or `aws-nb-europe-router-a

-You can add unlimited routing peers to a highly available route. Each connected peer selects one routing peer based on metric priority (lower = higher priority) and connection type (direct or relayed). +You can add unlimited routing peers to a highly available route. Each connected peer selects one routing peer based on metric priority (lower = higher priority), then connection type (direct or relayed), then latency. ### Applying Different Routes to Peers with Group Attribution diff --git a/src/pages/manage/networks/how-routing-peers-work.mdx b/src/pages/manage/networks/how-routing-peers-work.mdx index 4d31015d..51ee312c 100644 --- a/src/pages/manage/networks/how-routing-peers-work.mdx +++ b/src/pages/manage/networks/how-routing-peers-work.mdx @@ -103,14 +103,14 @@ Multiple routing peers can serve the same network or route. Each peer that uses The choice is made by the peer that uses the network, not by the routing peers, so what follows depends on the NetBird version of that peer. -This section describes NetBird v0.81.1 and later. Earlier clients move to another routing peer as soon as their connection falls back to relayed, and for latency differences of 10 ms, which resets connections more often with masquerade on. +This section describes NetBird v0.81.1 and later. Earlier clients measure latency only once, when the connection to a routing peer is set up. They also move to another routing peer as soon as the connection to their routing peer falls back to relayed, and for latency differences of 10 ms or more, so with masquerade on they reset connections more often. ### How a peer picks a routing peer A peer compares the routing peers it can use, in this order: -1. **Connection status.** A connected routing peer beats one that is idle with [lazy connections](/manage/peers/lazy-connection). +1. **Connection status.** A connected routing peer beats one that is idle with [lazy connections](/manage/peers/lazy-connection). A routing peer whose connection is down and being re-established (`Connecting` in `netbird status`) is not considered. 2. **Metric.** The lowest metric wins, whatever the connection type or latency. 3. **Connection type.** Among equal metrics, a direct (peer-to-peer) connection beats a relayed one. 4. **Latency.** Among equal metrics and the same connection type, the lowest latency wins. @@ -119,21 +119,25 @@ Latency is the round-trip time of the direct connection, sampled every few secon ### What makes a peer switch -A peer repeats this comparison whenever a routing peer's connection state or latency changes, when the network's configuration is updated, and every 30 seconds. It switches: +A peer repeats this comparison whenever a routing peer's connection state or latency changes, when the network's configuration is updated, when one of the delays below runs out, and every 30 seconds. It switches: -- **At once, when its routing peer goes offline.** The peer moves to the next routing peer in the order above. +- **At once, when its routing peer is detected as offline.** The peer moves to the next routing peer in the order above. A routing peer that fails without shutting down can take longer to be detected: see the note under [Relayed switch delay](#relayed-switch-delay). +- **At once, when its routing peer's connection is idle and another routing peer's connection is open.** - **At once, when a routing peer with a lower metric comes back online**, even while that routing peer's connection is still relayed. - **After the relayed switch delay, when its routing peer is relayed and an equivalent routing peer is direct.** See [Relayed switch delay](#relayed-switch-delay). -- **When another routing peer is clearly closer.** The latency gain has to be at least 20 ms, at least 20% of the current latency, and more than four times the combined measurement noise (jitter) of both paths. A peer also stays at least 30 seconds on a routing peer it moved to for latency, and gives a routing peer whose direct connection just came up up to 30 seconds to get its first latency measurement before comparing. +- **When another routing peer is clearly closer**, once all three of these hold: + - **Switch margin.** The latency gain has to be at least 20 ms, at least 20% of the current latency, and at least four times the combined uncertainty of the two latency values, which the client logs as the `+/-` figure next to each latency (for example `latency: 40ms +/- 2ms`). The uncertainty grows with the jitter of a path and is small on a stable one. On stable paths that have been connected for a while, this means a peer on a routing peer at 40 ms moves to one at 15 ms but not to one at 25 ms, and a peer on a routing peer at 150 ms moves only to one at 120 ms or less. A newly connected path starts with a high uncertainty that settles as measurements accumulate, so right after connecting, the gain has to be larger. + - **Dwell time.** After a switch made for latency, the next switch for latency waits at least 30 seconds; this never delays a failover. + - **Latency measurement.** Both routing peers need a latency measurement. A routing peer whose direct connection just came up cannot take over until it has one, and a current routing peer without one is kept for up to 30 seconds while it gets one. ### Relayed switch delay A direct connection often falls back to relayed for a few seconds and recovers. Since every switch resets connections with masquerade on, a relayed routing peer is kept for a while before the route moves to an equivalent routing peer with a direct connection: - **Its direct connection fell back to relayed.** The delay counts from the fallback. If the direct connection comes back within the delay, the route does not move. -- **It came up relayed.** When the routing peers of a network connect at the same time, for example after waking up from idle or when the client reconnects, the relayed connections come up first and the direct ones follow, one routing peer after the other. The delay counts from the moment another routing peer turns direct, and starts over if that routing peer is no longer direct. +- **It came up relayed.** When the routing peers of a network connect at the same time, for example after waking up from idle or when the client reconnects, the relayed connections come up first and the direct ones follow, one routing peer after the other. The delay counts from the moment another routing peer turns direct, and starts over if no equivalent routing peer is direct any more. -If the routing peer is still relayed when the delay runs out, the route moves to the direct one, so traffic does not stay relayed for longer than the delay. +If the routing peer is still relayed when the delay runs out, the route moves to the direct one, so traffic does not stay relayed for longer than the delay, unless the delay is set to `never`. The delay is set with the `NB_HA_RELAYED_SWITCH_DELAY` [environment variable](/client/environment-variables) on the peer that uses the network: @@ -151,14 +155,14 @@ sudo netbird service reconfigure --service-env NB_HA_RELAYED_SWITCH_DELAY=60s ``` -When a routing peer fails completely, its direct connection is usually detected as down before its relayed one. For that short window the routing peer looks relayed, so the failover can wait for the relayed switch delay, and traffic to the network is dropped meanwhile. +When a routing peer fails without shutting down, for example on power loss or a network outage, its direct connection is detected as down within seconds, but its relayed connection only once the relay service notices that the routing peer is gone, which can take a minute. In between, the routing peer looks relayed rather than offline, so the relayed switch delay applies, and traffic to the network is dropped until the failover. With the default delay, such a failover takes about 30 seconds longer than one where the routing peer shuts down cleanly. With `never`, the failover waits until the relay service notices. Where fast failover matters more than avoiding connection resets, set a shorter delay, such as `10s`. ### What a switch does to open connections -With masquerade on (the default), every switch resets established TCP connections through the network, whatever caused it: a failover, a failback, a fallback to relayed, or a closer routing peer. The new routing peer translates the traffic to a different source address, so applications such as SSH must reconnect. Expect the connection to stall for several seconds before the reset, while the peer notices the change. With masquerade off, connections are not translated and behave differently: see [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). +With masquerade on (the default), every switch resets established TCP connections through the network, whatever caused it: a failover, a failback, a fallback to relayed, or a closer routing peer. The new routing peer translates the traffic to a different source address, so applications such as SSH must reconnect. After a failover, expect the connection to stall before the reset while the peer notices that its routing peer is gone: several seconds, or longer when the routing peer failed without shutting down (see [Relayed switch delay](#relayed-switch-delay)). With masquerade off, connections are not translated and behave differently: see [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). -If users report dropped connections while no routing peer went offline, look up the reason of each switch in the client log (see [Finding switches in the client log](#finding-switches-in-the-client-log)), and check whether connections stay relayed for longer than the relayed switch delay. [Troubleshooting relayed connections](/help/troubleshooting-relayed-connections) covers why a connection is relayed. +If users report dropped connections while no routing peer went offline, look up the reason for each switch in the client log (see [Finding switches in the client log](#finding-switches-in-the-client-log)), and check whether connections stay relayed for longer than the relayed switch delay. [Troubleshooting relayed connections](/help/troubleshooting-relayed-connections) covers why a connection is relayed. ### Primary / failover (different metrics) @@ -179,7 +183,7 @@ The client logs routing peer decisions at the default log level: - `keeping relayed routing peer …` or `keeping routing peer …` when a better-ranked routing peer is held back, with the rule that holds it: the relayed switch delay, the latency margin, the minimum time on a routing peer, or a missing latency measurement. Repeats of the same decision are logged at the debug level. - `network [] has not been assigned a routing peer …` when none of its routing peers is available. -Keys are WireGuard public keys. Switches and the loss of every routing peer are also recorded as `Routing peer changed` and `No routing peer available` events, which show in `netbird status -d` and in debug bundles whatever the log level. For an exit node (a default route), the loss of every routing peer is recorded as `Default route disconnected due to peer unreachability` instead. +Keys are WireGuard public keys. Switches and the loss of every routing peer are also recorded as `Routing peer changed` and `No routing peer available` events, which show in `netbird status -d` and in debug bundles whatever the log level. For an exit node (a default route), the loss of every routing peer is recorded as `Default route disconnected due to peer unreachability` instead. The client keeps only its 10 most recent events of any kind, so check soon after a report or read the client log. ### Failure domains diff --git a/src/pages/manage/networks/index.mdx b/src/pages/manage/networks/index.mdx index 67bd6894..5ac161cd 100644 --- a/src/pages/manage/networks/index.mdx +++ b/src/pages/manage/networks/index.mdx @@ -165,7 +165,7 @@ This single policy grants the `Development` group access to **every** resource i Before you depend on a Network in production, work through these: -- **High availability.** Add more than one routing peer to the same Network for redundancy. Added individually, each routing peer gets its own metric: the lowest is the primary and higher ones are failovers. With equal metrics, each peer picks one when it connects, by connection type and then latency; this is not load balancing. Added as a group, the routing peers share one metric, so they can't act as primary and failover. Keep highly available peers in different failure domains. See [High availability](/manage/networks/how-routing-peers-work#high-availability). +- **High availability.** Add more than one routing peer to the same Network for redundancy. Added individually, each routing peer gets its own metric: the lowest is the primary and higher ones are failovers. With equal metrics, each peer uses the best one by connection type and then latency, and also moves later if another becomes clearly closer; this is not load balancing. Added as a group, the routing peers share one metric, so they can't act as primary and failover. Keep highly available peers in different failure domains. See [High availability](/manage/networks/how-routing-peers-work#high-availability). - **Monitoring.** Enable the **Routing Peer Disconnected** event in [Notifications](/manage/settings/notifications) to get alerted by email, webhook, or Slack when a routing peer goes offline. - **Masquerade.** On by default and the simplest option. Turn it off only when you need source-IP visibility, and only on Linux routing peers, as that's the only platform where it can be disabled. Disabling it requires a return route in the destination network, and makes high availability something you have to arrange rather than something you get. See [Masquerade](/manage/networks/masquerade). - **Internal DNS.** Domain resources resolve on the routing peer, so it must be able to resolve the name. If it already can, nothing more is needed; if it can't, distribute a nameserver to the routing peer's group. See [Internal DNS Servers](/manage/dns/internal-dns-servers). diff --git a/src/pages/manage/networks/sizing-routing-peers.mdx b/src/pages/manage/networks/sizing-routing-peers.mdx index 053ff114..42c73ae4 100644 --- a/src/pages/manage/networks/sizing-routing-peers.mdx +++ b/src/pages/manage/networks/sizing-routing-peers.mdx @@ -143,7 +143,7 @@ Jumbo helps the [userspace datapath](#userspace-mode) too (about 2.5x), but user ## Scaling out: high availability and splitting load -A single peer has a ceiling. To go past it, run more than one, but understand what NetBird high availability does and does not do. It gives you two things: **failover** (different metrics: one peer carries all, the rest are reserve) and **nearest-peer selection** (equal metrics: each client uses its lowest-latency peer, switching only past a 20 ms difference). See [How Routing Peers Work](/manage/networks/how-routing-peers-work#high-availability) for the mechanism behind both. +A single peer has a ceiling. To go past it, run more than one, but understand what NetBird high availability does and does not do. It gives you two things: **failover** (different metrics: one peer carries all, the rest are reserve) and **nearest-peer selection** (equal metrics: each client uses its lowest-latency peer, switching only when another is clearly closer: at least 20 ms and at least 20% faster). See [How Routing Peers Work](/manage/networks/how-routing-peers-work#high-availability) for the mechanism behind both. Neither balances a Network's load. Nearest-peer *does* spread traffic when peers sit in different regions and users are spread among them — EU users on the EU peer, US users on the US — but it splits by geography, not evenly, and not at all when peers sit together. To split load deterministically, build more identical Networks as in [Step 4](#step-4-give-each-active-peer-its-own-network).