From 4b769c65c0ec85eb4373484dc42dd05d1d3a9ee2 Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Fri, 16 Oct 2026 09:53:03 +0200 Subject: [PATCH] docs: tighten HA routing peer selection wording Describe the latency uncertainty the switch margin uses instead of raw jitter, state the dwell time as time since the last latency switch, scope the failover stall to failovers, name the exit node events a switch records, and clarify how the relayed switch delay is set. --- src/pages/manage/network-routes/index.mdx | 2 +- .../networks/how-routing-peers-work.mdx | 32 +++++++++---------- src/pages/manage/networks/index.mdx | 2 +- .../manage/networks/sizing-routing-peers.mdx | 2 +- 4 files changed, 19 insertions(+), 19 deletions(-) diff --git a/src/pages/manage/network-routes/index.mdx b/src/pages/manage/network-routes/index.mdx index d6501495..6f1d618e 100644 --- a/src/pages/manage/network-routes/index.mdx +++ b/src/pages/manage/network-routes/index.mdx @@ -73,7 +73,7 @@ Metric determines routing peer priority in high availability configurations. Low A highly available route combines multiple routes with the same network identifier and ranges but different routing peers or peer groups. This provides redundant paths between your peers and external networks. -Each peer selects one routing peer by metric, then connection type (direct or relayed), then latency. See [High availability](/manage/networks/how-routing-peers-work#high-availability) for the selection rules and what makes a peer switch. +Each connected peer selects one routing peer by metric (lower wins), then connection type (direct beats relayed), then latency. See [High availability](/manage/networks/how-routing-peers-work#high-availability) for the selection rules and what makes a peer switch. ### DNS Routes diff --git a/src/pages/manage/networks/how-routing-peers-work.mdx b/src/pages/manage/networks/how-routing-peers-work.mdx index 324e4088..8ad9570f 100644 --- a/src/pages/manage/networks/how-routing-peers-work.mdx +++ b/src/pages/manage/networks/how-routing-peers-work.mdx @@ -110,16 +110,16 @@ This section describes NetBird clients v0.81.1 and later. Older clients measure A peer ranks the routing peers it can use, in this order: -1. **Connection state.** A routing peer the peer has an open connection to beats one whose connection is idle, as happens with [lazy connections](/manage/peers/lazy-connection). A routing peer that is still connecting is not considered. +1. **Connection state.** A routing peer with an open connection beats one whose connection is idle, as happens with [lazy connections](/manage/peers/lazy-connection). A routing peer whose connection is down and being re-established (`Connecting` in `netbird status`) is not considered at all. 2. **Metric.** The lowest metric wins, whatever the connection type or latency. 3. **Connection type.** Among equal metrics, a direct (peer-to-peer) connection beats a relayed one. -4. **Latency.** Among equal metrics and the same connection type, the lowest latency wins, subject to the switch margin described below. +4. **Latency.** Among equal metrics and the same connection type, the lowest latency wins. Leaving the current routing peer for a lower latency is subject to the switch margin and dwell time below. When two routing peers tie on all four, the peer keeps the one it already uses. ### How latency is measured -Latency is the round-trip time of the direct connection to the routing peer. The client reads it every 4 seconds from the connectivity checks the connection already exchanges, so measuring it adds no traffic. It smooths the readings into a moving average, so a brief spike barely moves the value while a lasting change shows up within tens of seconds. It also tracks how much the readings scatter around that average, which is how jittery the path is. +Latency is the round-trip time of the direct connection to the routing peer. The client reads it every 4 seconds from the connectivity checks the connection already exchanges, so measuring it adds no traffic. It smooths the readings into a moving average, so a brief spike barely moves the value while a lasting change shows up within tens of seconds. It also tracks how much the readings scatter around that average, that is, how jittery the path is, and derives from that how uncertain the smoothed value is. On a stable path, the uncertainty is close to zero. Only a direct connection has a latency. A relayed connection has none, and when a direct connection falls back to relayed its latency is cleared. Two relayed routing peers with the same metric are therefore never compared on latency: the peer stays on the one it uses. @@ -127,11 +127,11 @@ Only a direct connection has a latency. A relayed connection has none, and when A routing peer that is better only on latency takes over only when all three of these hold: -- **The gain clears the switch margin.** Its latency must be lower than the current routing peer's by at least the largest of: 20 ms, 20% of the current routing peer's latency, and four times the combined jitter of the two paths. The fixed 20 ms keeps fast paths from churning over a few milliseconds, the 20% scales the margin up for slow paths, and the jitter term keeps a noisy path from moving the route on readings that only look better for the moment. -- **The current routing peer has been in place for 30 seconds** since the last switch made for latency (the dwell time). Switches made for any other reason do not start this timer, so a failover never delays the next one. -- **Both routing peers have a latency reading.** A routing peer whose direct connection has just come up has no reading yet and cannot take over on latency. If the current routing peer is the one without a reading, the peer keeps it for up to 30 seconds while a reading arrives, then ranks it as unmeasured. +- **The gain clears the switch margin.** Its latency must be lower than the current routing peer's by at least the largest of: 20 ms, 20% of the current routing peer's latency, and four times the combined uncertainty of the two latency values. The fixed 20 ms keeps fast paths from churning over a few milliseconds, the 20% scales the margin up for slow paths, and the uncertainty term keeps a jittery path from moving the route on readings that only look better for the moment. +- **At least 30 seconds have passed since the last switch made for latency** (the dwell time). Only a switch made for latency restarts this timer, and the timer never delays a switch made for any other reason, such as a failover. +- **Both routing peers have a latency reading.** A routing peer whose direct connection has just come up has no reading yet and cannot take over on latency. If the current routing peer is the one without a reading, the peer keeps it for up to 30 seconds while it waits for a reading, then treats it as slower than any measured routing peer. -On stable paths, where the jitter term is near zero, this works out as follows: +On stable paths that have been connected for a while, so that the uncertainty term is near zero, and with the dwell time passed, this works out as follows: | Current routing peer | Other routing peer | Gain | Margin | Result | |---|---|---|---|---| @@ -140,11 +140,11 @@ On stable paths, where the jitter term is near zero, this works out as follows: | 150 ms | 125 ms | 25 ms | 30 ms (20%) | Stays | | 150 ms | 110 ms | 40 ms | 30 ms (20%) | Switches | -On a jittery path, the margin is larger than the table shows. A freshly connected path starts out with a high jitter estimate, which shrinks as readings accumulate, so a newly connected routing peer needs a clearer lead to win on latency than one that has been connected for a while. +On a jittery path, the margin is larger than the table shows. A freshly connected path starts with a high uncertainty that shrinks as readings accumulate, so a newly connected routing peer needs a clearer lead to win on latency than one that has been connected for a while. ### What makes a peer switch -A peer repeats the ranking whenever a routing peer's connection state changes, when a routing peer's latency or jitter moves by 5 ms or more, when the network's configuration is updated, and every 30 seconds in any case. It switches immediately when: +A peer repeats the ranking whenever a routing peer's connection state changes, a routing peer's latency or its uncertainty moves by 5 ms or more, the network's configuration is updated, or one of the delays described below runs out, and every 30 seconds in any case. It switches immediately when: - **Its routing peer goes offline.** The peer moves to the next routing peer in the order above. - **A routing peer with a lower metric comes back online.** The peer moves back as soon as that routing peer reconnects, even while its connection is still relayed. @@ -152,14 +152,14 @@ A peer repeats the ranking whenever a routing peer's connection state changes, w It switches after a delay when: -- **Its routing peer's connection is relayed, and another routing peer with the same metric is direct.** The peer keeps the relayed routing peer for 30 seconds by default and moves only if the connection is still relayed after that. The relay keeps carrying the traffic in the meantime, and a direct connection that dropped often recovers within seconds, so waiting avoids a reset that would not have been needed. If the routing peer dropped from direct to relayed, the 30 seconds count from the drop. If it was relayed from the start, as when all routing peers come up over relay and turn direct one after another, they count from when a direct alternative first appears. With different metrics this switch never happens, because metric outranks connection type. +- **Its routing peer's connection is relayed, and another routing peer with the same metric is direct.** The peer keeps the relayed routing peer for 30 seconds by default and moves only if the connection is still relayed after that. The relay keeps carrying the traffic in the meantime, and a direct connection that dropped often recovers within seconds, so waiting avoids a reset that would not have been needed. If the routing peer dropped from direct to relayed, the 30 seconds count from the drop. If it was relayed from the start, as when all routing peers come up over relay and turn direct one after another, they count from when a direct alternative first appears, and start over if no direct alternative remains. With different metrics this switch never happens, because metric outranks connection type. - To change the delay, set `NB_HA_RELAYED_SWITCH_DELAY` in the client service environment of the peers that use the network, not on the routing peers: the choice is made by each peer that uses the network. It takes a duration with a unit, such as `10s` or `2m`; `0` to switch at once; or `never` to keep a routing peer whose direct connection dropped until it goes offline. `never` does not apply to a routing peer that was relayed from the start, which still moves after 30 seconds, so it cannot leave a peer on a relay for good. An invalid value logs a warning and falls back to 30 seconds. The client reads the variable when its service starts and logs the value in use as `routing peer relayed switch delay: …`. See [Environment variables](/client/environment-variables) for how to set one on the client service. + To change the delay, set `NB_HA_RELAYED_SWITCH_DELAY` on the peers that use the network, not on the routing peers, because each of those peers makes its own choice. It takes a duration with a unit, such as `10s` or `2m`; `0` to switch at once; or `never` to keep a routing peer whose direct connection dropped for as long as it stays connected over the relay. `never` does not apply to a routing peer that was relayed from the start, which still moves after 30 seconds, so the setting cannot leave a peer on a relay for good. An invalid value logs a warning and falls back to 30 seconds. Set the variable in the client service environment, as described in [Environment variables](/client/environment-variables), and restart the service after changing it: the client reads it once and logs the value in use as `routing peer relayed switch delay: …`. - **Another routing peer with the same metric and connection type is clearly closer**, under the switch margin and dwell time above. ### What a switch does to open connections -With masquerade on (the default), every switch resets established TCP connections through the network, whatever caused it: a failover, a failback, a fallback to relayed, or a lower latency elsewhere. The new routing peer translates the traffic to a different source address, so applications such as SSH must reconnect. Expect the connection to stall for several seconds before the reset, while the peer notices the change. With masquerade off, connections are not translated and behave differently: see [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). +With masquerade on (the default), every switch resets established TCP connections through the network, whatever caused it: a failover, a failback, a fallback to relayed, or a lower latency elsewhere. The new routing peer translates the traffic to a different source address, so applications such as SSH must reconnect. After a failover, expect the connection to stall for several seconds before the reset, while the peer notices that its routing peer is gone. With masquerade off, connections are not translated and behave differently: see [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). If users report dropped connections while no routing peer went offline, check the switch reason the client recorded (see [Finding switches](#finding-switches)). A reason of `direct connection preferred over relayed` points at connections falling back to relayed: [Troubleshooting relayed connections](/help/troubleshooting-relayed-connections) covers why a connection is relayed. @@ -171,13 +171,13 @@ Use different metrics when you want every peer on one known routing peer, with t ### Nearest routing peer (equal metrics) -Use equal metrics when routing peers are spread across regions and you want each peer to use the closest one: for example, an EU-based and a US-based routing peer serving the same network, with users on both continents. Each peer uses the lowest-latency routing peer, though a direct connection still beats a relayed one whatever the latency. Because latency is measured continuously, a peer also moves later when another routing peer becomes clearly closer, for example when a laptop travels from Europe to the US while staying connected. Routing peers whose latencies are within the switch margin of each other are effectively tied, and the peer stays where it is. +Use equal metrics when routing peers are spread across regions and you want each peer to use the closest one: for example, an EU-based and a US-based routing peer serving the same network, with users on both continents. Each peer settles on the lowest-latency routing peer, though a direct connection still beats a relayed one whatever the latency. Because latency is measured continuously, a peer also moves later when another routing peer becomes clearly and lastingly closer, for example when the path to its current routing peer slows down and stays slow. Routing peers whose latencies are within the switch margin of each other are effectively tied, and the peer stays where it is. The trade-off: every move resets connections with masquerade on, whether a fallback to relayed or a lasting latency change caused it. ### Finding switches -The client records every switch as a warning event, whatever the log level. Run `netbird status -d` and look under **Events**, or open `status.txt` in a [debug bundle](/help/troubleshooting-client#debug-bundle): +The client records every move from one routing peer to another as a warning event, whatever the log level. Run `netbird status -d` and look under **Events**, or open `status.txt` in a [debug bundle](/help/troubleshooting-client#debug-bundle): ```text Events: @@ -198,9 +198,9 @@ Events: | `lower latency` | Another routing peer cleared the switch margin and the dwell time | | `route assigned to a different routing peer` | A configuration change moved the route to another routing peer | -When the last routing peer of a network goes offline, the client records a `No routing peer available` event instead. The client keeps only its 10 most recent events of any kind, so check soon after a report or read the client log. +When the last routing peer of a network goes offline, the client records a `No routing peer available` event instead, or, for an exit node, `Default route disconnected due to peer unreachability`. A switch between exit node routing peers also records the exit node's own disconnect and connect events. The client keeps only its 10 most recent events of any kind, so check soon after a report or read the client log. -The client log has the same switch as a warning line, `switching routing peer for network [] from to : `, followed by the new routing peer's connection state, metric, connection type and latency. The first choice for a network is logged at the info level as `new chosen route is …`. When a rule holds a better-ranked routing peer back, the client logs that at the info level once, naming the rule (the switch margin, the minimum dwell time, a missing latency reading, or the relayed switch delay), and repeats it only at the debug level for the next 10 minutes. +The client log has the same switch as a warning line, `switching routing peer for network [] from to : `, followed by the new routing peer's connection state, metric, connection type and latency. The first choice for a network is logged at the info level as `new chosen route is …`. When a rule holds a better-ranked routing peer back, the client logs that at the info level, naming the rule: the switch margin, the minimum dwell time, a missing latency reading, or the relayed switch delay. While the same rule keeps holding back the same routing peer, it repeats the message only at the debug level, and at the info level again after 10 minutes. ### Before v0.81.1 diff --git a/src/pages/manage/networks/index.mdx b/src/pages/manage/networks/index.mdx index 2c6c60da..5ac161cd 100644 --- a/src/pages/manage/networks/index.mdx +++ b/src/pages/manage/networks/index.mdx @@ -165,7 +165,7 @@ This single policy grants the `Development` group access to **every** resource i Before you depend on a Network in production, work through these: -- **High availability.** Add more than one routing peer to the same Network for redundancy. Added individually, each routing peer gets its own metric: the lowest is the primary and higher ones are failovers. With equal metrics, each peer uses the best one by connection type and then latency, and moves later only when another becomes clearly closer; this is not load balancing. Added as a group, the routing peers share one metric, so they can't act as primary and failover. Keep highly available peers in different failure domains. See [High availability](/manage/networks/how-routing-peers-work#high-availability). +- **High availability.** Add more than one routing peer to the same Network for redundancy. Added individually, each routing peer gets its own metric: the lowest is the primary and higher ones are failovers. With equal metrics, each peer uses the best one by connection type and then latency, and also moves later if another becomes clearly closer; this is not load balancing. Added as a group, the routing peers share one metric, so they can't act as primary and failover. Keep highly available peers in different failure domains. See [High availability](/manage/networks/how-routing-peers-work#high-availability). - **Monitoring.** Enable the **Routing Peer Disconnected** event in [Notifications](/manage/settings/notifications) to get alerted by email, webhook, or Slack when a routing peer goes offline. - **Masquerade.** On by default and the simplest option. Turn it off only when you need source-IP visibility, and only on Linux routing peers, as that's the only platform where it can be disabled. Disabling it requires a return route in the destination network, and makes high availability something you have to arrange rather than something you get. See [Masquerade](/manage/networks/masquerade). - **Internal DNS.** Domain resources resolve on the routing peer, so it must be able to resolve the name. If it already can, nothing more is needed; if it can't, distribute a nameserver to the routing peer's group. See [Internal DNS Servers](/manage/dns/internal-dns-servers). diff --git a/src/pages/manage/networks/sizing-routing-peers.mdx b/src/pages/manage/networks/sizing-routing-peers.mdx index 88c05611..42c73ae4 100644 --- a/src/pages/manage/networks/sizing-routing-peers.mdx +++ b/src/pages/manage/networks/sizing-routing-peers.mdx @@ -143,7 +143,7 @@ Jumbo helps the [userspace datapath](#userspace-mode) too (about 2.5x), but user ## Scaling out: high availability and splitting load -A single peer has a ceiling. To go past it, run more than one, but understand what NetBird high availability does and does not do. It gives you two things: **failover** (different metrics: one peer carries all, the rest are reserve) and **nearest-peer selection** (equal metrics: each client uses its lowest-latency peer, switching only when another is clearly closer, by at least 20 ms and 20%). See [How Routing Peers Work](/manage/networks/how-routing-peers-work#high-availability) for the mechanism behind both. +A single peer has a ceiling. To go past it, run more than one, but understand what NetBird high availability does and does not do. It gives you two things: **failover** (different metrics: one peer carries all, the rest are reserve) and **nearest-peer selection** (equal metrics: each client uses its lowest-latency peer, switching only when another is clearly closer: at least 20 ms and at least 20% faster). See [How Routing Peers Work](/manage/networks/how-routing-peers-work#high-availability) for the mechanism behind both. Neither balances a Network's load. Nearest-peer *does* spread traffic when peers sit in different regions and users are spread among them — EU users on the EU peer, US users on the US — but it splits by geography, not evenly, and not at all when peers sit together. To split load deterministically, build more identical Networks as in [Step 4](#step-4-give-each-active-peer-its-own-network).