diff --git a/src/components/NavigationDocs.jsx b/src/components/NavigationDocs.jsx index a8efff7d..4ca3fb76 100644 --- a/src/components/NavigationDocs.jsx +++ b/src/components/NavigationDocs.jsx @@ -598,6 +598,10 @@ export const docsNavigation = [ title: 'High Availability', href: '/selfhosted/maintenance/scaling/high-availability', }, + { + title: 'External Relays (Licensed)', + href: '/selfhosted/enterprise/external-relays', + }, { title: 'Grafana Dashboard', href: '/selfhosted/enterprise/grafana-dashboard', diff --git a/src/pages/selfhosted/enterprise/external-relays.mdx b/src/pages/selfhosted/enterprise/external-relays.mdx new file mode 100644 index 00000000..f5a7da96 --- /dev/null +++ b/src/pages/selfhosted/enterprise/external-relays.mdx @@ -0,0 +1,541 @@ +export const description = 'Move relay and STUN off your NetBird Enterprise server onto dedicated hosts, step by step, without breaking traffic event logging.' + +# External Relays on a Commercial License + +import {Note, Warning} from "@/components/mdx"; + +Your NetBird Enterprise server runs management, signal, relay and STUN in one `netbird-server` container. That works until relayed traffic starts competing with the control plane for the same host, or until users in another region relay through a server on the wrong continent. This guide moves the relay and STUN services onto dedicated hosts and points your server at them. + +One thing makes Enterprise different, and it is the reason this page exists rather than the [community guide](/selfhosted/maintenance/scaling/set-up-external-relays). When traffic flow is enabled, your deployment has a single shared secret doing **two** jobs: authenticating peers to the relay, and authenticating peers to the traffic-flow receiver. That is true however the deployment was built, whether installed fresh or migrated up from the community edition. Generate a new secret while following a guide that only knows about the first job, and your peers keep working perfectly while traffic event logging silently stops. Step 2 is where that is handled. + +## Before you start + +For each relay server you need: + +- A Linux VM with at least **1 CPU** and **1 GB RAM** +- A public IP address +- A domain name pointing at it, for example `relay-us.example.com` +- Docker and the Compose plugin installed +- These inbound ports open: **443/tcp** and **443/udp** (relay), **3478/udp** (STUN) + + +**No inbound port 80 is needed.** A relay proves its domain to Let's Encrypt over 443 itself, using the TLS-ALPN-01 challenge. Your main NetBird server is different and does still need 80, and so may a proxy in front of the relay. + +**Both 443 rules matter.** The relay serves two transports on that port: WebSocket over TCP and QUIC over UDP. Clients race both and keep whichever answers first. With 443/udp closed the QUIC attempt runs until it times out, WebSocket carries the connection, and nothing reports an error, so the missing rule is easy to overlook. + + +On the main server you need shell access to your deployment directory, which this guide calls `~/netbird-enterprise`, and the ability to restart containers there. + +This guide uses two relay servers, `relay-us.example.com` and `relay-eu.example.com`, and a main server at `netbird.example.com`. Substitute your own names throughout. One relay works fine; two is the smallest setup that shows the failover behavior. + +### Which names your deployment uses + +A deployment installed fresh with `getting-started-enterprise.sh` and one migrated up from the community edition with `migrate-to-enterprise.sh` use different names for the same things, and traffic flow is optional in both. On the main server, run: + +```bash +cd ~/netbird-enterprise +ls config.yaml.enterprise +docker compose config --services | grep -E '^(receiver|flow-receiver)$' +``` + +| | Fresh install | Migrated from the community edition | +|---|---|---| +| Server configuration | `config.yaml` | `config.yaml.enterprise` if `ls` lists it, otherwise `config.yaml` | +| Traffic-flow receiver service | `receiver` | `flow-receiver` | +| Receiver container | `netbird-receiver` | `netbird-flow-receiver` | +| Relay secret in `.env` | `NETBIRD_RELAY_AUTH_SECRET` | `NB_FLOW_AUTH_SECRET`, only if traffic flow was enabled | + +The commands on this page use the fresh-install names. On a migrated deployment, use the names from the second column instead. The configuration file matters most. `migrate-to-enterprise.sh` writes `config.yaml.enterprise` whenever it moves storage to PostgreSQL or enables traffic flow, and starts the server with that file instead of `config.yaml`. On such a deployment an edit to `config.yaml` has no effect. + +If the `grep` prints nothing, your deployment has no traffic-flow receiver. Skip the traffic event check in Step 1, all of Step 8, and the receiver commands in [Rotating to a new secret](#rotating-to-a-new-secret). Everything else on this page applies unchanged. + +## Step 1: Before you change anything + +**This is a cutover, not a gradual migration.** The moment you restart the server with the new configuration, the embedded relay stops and every peer that was using it reconnects to the external relays. There is no window where both are serving. So build and test the relay hosts completely, through the end of Step 5, before you touch the main server's configuration in Step 6. + +**Record whether traffic events are working now.** Step 8 checks that they survived the change, and that check can only tell you something if you know the answer beforehand. Generate a little traffic between two peers, for example `ping` one peer's NetBird IP from another, wait about 90 seconds, then on the main server run: + +```bash +cd ~/netbird-enterprise +docker compose logs --since 120s receiver | grep -c 'sending event' +``` + +Use `flow-receiver` on a migrated deployment, and skip this check if your deployment has no receiver ([Which names your deployment uses](#which-names-your-deployment-uses)). + +If it is `0` and the receiver still logs at its default level (`NB_FLOW_LOG_LEVEL` unset or `debug`), traffic events are not flowing **yet**, and that has nothing to do with relays. Traffic event logging is disabled by default at the account level even when the installer enabled traffic flow on the server: in the dashboard, go to `Settings > Networks` and turn on `Enable Traffic Events` under the Experimental section. See [Traffic Events Logging](/manage/activity/traffic-events-logging). Either enable it and re-run the check, or accept that Step 8 will have nothing to compare against. + +## Step 2: Read your existing shared secret + +**Do not generate a new secret.** Your deployment already has one, and the traffic-flow receiver, if you have one, uses the same value. Read it from the server configuration on the main server. The first line below picks `config.yaml.enterprise` when a migration created it, because that is the file the server reads, and the two files stop matching as soon as the secret is rotated: + +```bash +cd ~/netbird-enterprise +CONFIG=config.yaml; [ -f config.yaml.enterprise ] && CONFIG=config.yaml.enterprise +grep 'authSecret:' "$CONFIG" +``` + +Read it from the server configuration and not from `.env`. The server configuration carries this value on every deployment. `.env` does not: a deployment created by `getting-started-enterprise.sh` has it there as `NETBIRD_RELAY_AUTH_SECRET`, one migrated up from the community edition with traffic flow has `NB_FLOW_AUTH_SECRET` instead, and one migrated without traffic flow has neither, because the community installer never wrote the relay secret to `.env` in the first place. + +That value goes in two places later: `NB_AUTH_SECRET` on every relay host in Step 3, and `relays.secret` on the main server in Step 6. + + +`config.yaml` and `.env` are mode `600` and hold every secret in your deployment. Read the single line you need rather than opening the whole file, and keep the value out of shared terminal history. + + +If your policy requires a fresh secret, that works too, but it has more places to change than you would expect. Use [Rotating to a new secret](#rotating-to-a-new-secret) at the end of this page rather than improvising here. + +## Step 3: Create the relay configuration + +Do this on each relay server. + +```bash +mkdir -p ~/netbird-relay +cd ~/netbird-relay +``` + +Create `relay.env`. The relay can obtain and renew its own TLS certificate through Let's Encrypt: + +```bash +NB_LOG_LEVEL=info +NB_LISTEN_ADDRESS=:443 +NB_EXPOSED_ADDRESS=rels://relay-us.example.com:443 +NB_AUTH_SECRET=the-value-from-step-2 + +# TLS via Let's Encrypt +NB_LETSENCRYPT_DOMAINS=relay-us.example.com +NB_LETSENCRYPT_EMAIL=admin@example.com +NB_LETSENCRYPT_DATA_DIR=/data/letsencrypt + +# Embedded STUN (comma-separated for multiple ports, e.g. 3478,3479) +NB_ENABLE_STUN=true +NB_STUN_PORTS=3478 +``` + +Replace `relay-us.example.com` with this host's own domain and `the-value-from-step-2` with the secret you read. The file holds that secret, so make it readable by root only: + +```bash +chmod 600 relay.env +``` + +The relay reads `relay.env` when its container is created. If you change the file later on a running relay, apply it with `docker compose up -d`. `docker compose restart` keeps the old values. + +Create `docker-compose.yml`: + +```yaml +services: + relay: + image: netbirdio/relay:latest + container_name: netbird-relay + restart: unless-stopped + ports: + # Both transports. Docker reads a bare '443:443' as TCP only, + # so the UDP line is required for QUIC. + - '443:443/tcp' + - '443:443/udp' + # Expose every port listed in NB_STUN_PORTS + - '3478:3478/udp' + env_file: + - relay.env + volumes: + - relay_data:/data + logging: + driver: "json-file" + options: + max-size: "500m" + max-file: "2" + +volumes: + relay_data: +``` + +The relay image is the open source `netbirdio/relay`. A relay host takes no license key. + +### If you already have TLS certificates + +To use your own certificate instead of Let's Encrypt, delete the three `NB_LETSENCRYPT_*` lines from `relay.env` and add: + +```bash +NB_TLS_CERT_FILE=/certs/fullchain.pem +NB_TLS_KEY_FILE=/certs/privkey.pem +``` + +Then replace the relay service's `volumes:` list in `docker-compose.yml` with this one. Do not add it as a second `volumes:` key, which makes the file invalid: + +```yaml + volumes: + - /path/to/certs:/certs:ro + - relay_data:/data +``` + +`fullchain.pem` is your certificate followed by any intermediates, and the certificate must cover the relay's domain. Four things catch people out: + +- **Set both variables, and remove every `NB_LETSENCRYPT_*` line.** If a Let's Encrypt line remains, the relay uses Let's Encrypt and ignores your files. If only one `NB_TLS_*` variable is set, the relay starts with no TLS at all. Either way, the sign is a missing QUIC line in Step 4. +- **Nothing in the startup log names your certificate.** Step 5's `curl -v` is how you confirm it is the one being served. +- **The files are read once, at startup.** Restart the relay after every renewal: `docker compose restart relay`. +- **A private CA must be trusted by every peer.** NetBird checks the relay's certificate against each device's operating-system trust store, and has no setting for a separate CA. A device that does not trust yours reports the relay as `Unavailable` with `x509: certificate signed by unknown authority`, and cannot relay to other peers. Distribute the CA to every device before you switch. + +### If a load balancer or proxy sits in front of the relay + +In this shape your proxy holds the certificate and the relay serves plain HTTP behind it. + +Use this shape only because a load balancer or proxy is already in front of your relays. Do not add one just to terminate TLS on the relay host: the relay already does that itself, and a proxy in front of it costs you QUIC. + +**On the relay host**, `relay.env` has no `NB_LETSENCRYPT_*` lines, listens on an internal port, and names the proxy's address: + +```bash +NB_LOG_LEVEL=info +NB_LISTEN_ADDRESS=:8080 +NB_EXPOSED_ADDRESS=rels://relay-us.example.com:443 +NB_AUTH_SECRET=the-value-from-step-2 +NB_TRUSTED_PROXIES=10.20.0.10 +NB_ENABLE_STUN=true +NB_STUN_PORTS=3478 +``` + +Keep `rels://` written out in `NB_EXPOSED_ADDRESS`. With no TLS of its own, the relay would otherwise advertise `rel://`. + +In `docker-compose.yml`, remove the two `443` port lines and keep `3478:3478/udp` for STUN. Make port 8080 reachable from the proxy and nothing else. If the proxy is on another machine, publish the port with `'8080:8080/tcp'` and allow it at your firewall from the proxy's address only. + +**Your proxy must:** + +- hold a certificate for the relay's domain, and accept connections on 443/tcp; +- forward WebSocket upgrades over HTTP/1.1 to the relay's port 8080; +- set both `X-Real-Ip` and `X-Real-Port` to the address and source port of the connection it received, overwriting anything the client sent. + +The relay uses those two headers only when the connection comes from an address in `NB_TRUSTED_PROXIES`, and only when both are present. A proxy that sets only `X-Real-Ip` gets its own address logged for every peer. A proxy that passes the client's headers through lets any client write the address of its choice into your relay logs. + +`NB_TRUSTED_PROXIES` is the address the relay sees the proxy's connections come from, which is not always the proxy's public address. When the proxy and relay share a Docker network, it is the proxy container's address on that network. When the proxy is another machine on the same private network, it is that machine's private address: a published Docker port keeps the original source address. + + +Trust only the smallest proxy ranges you control. Never `0.0.0.0/0`, `::/0`, or a shared network where an untrusted system could reach the relay directly and forge these headers. Entries are IP addresses or CIDRs separated by commas. Hostnames are rejected, and a prefix such as `10.20.0.10/24` trusts the whole `/24`. + + +These headers decide only which client address the relay records, including in the `invalid signature` lines used for troubleshooting. Authentication does not depend on them. + +Apply a change to `NB_TRUSTED_PROXIES` with `docker compose up -d relay`. An invalid entry stops the relay: it exits with `failed to parse trusted proxies: ...` and restarts in a loop until you fix it. To confirm the setting works, check that the relay logs your peers' addresses rather than the proxy's: + +```bash +docker compose logs relay | grep 'WS client connected from' +``` + +The CLI equivalent of `NB_TRUSTED_PROXIES` is `--trusted-proxies`, available since **v0.75.0**. + +At startup the relay reports that it cannot serve QUIC: + +``` +WARN relay/server/server.go:78: Not starting QUIC listener: valid TLS config is required for QUIC listener +``` + +That is expected here. Peers connect over WebSocket, so neither the proxy nor the relay host needs 443/udp. The proxy needs 443/tcp, plus whatever it uses to obtain its own certificate, often 80/tcp. The relay host needs 3478/udp for STUN. + +## Step 4: Start each relay + +```bash +docker compose up -d +docker compose logs -f +``` + +You should see the relay announce its address, both listeners, and the STUN server: + +``` +INFO relay/cmd/root.go:242: server will be available on: rels://relay-us.example.com:443 +INFO relay/server/listener/ws/listener.go:51: WS server listening address: :443 +INFO relay/server/listener/quic/listener.go:39: QUIC server listening on address: :443 +INFO [component: stun] stun/server.go:71: STUN server listening on [::]:3478 +``` + +Other lines appear alongside these, including the Let's Encrypt setup, the health check server and the metrics server. The order changes from one start to the next, because the listeners come up concurrently. What matters is that all four lines are present, not where they sit. + +A missing QUIC line means the relay has no TLS configuration of its own, which is expected only behind a TLS-terminating proxy. + +## Step 5: Check each relay from outside + +Certificates are issued lazily on the first request, so this both provisions and verifies: + +```bash +curl -v https://relay-us.example.com/ +``` + +A `404 page not found` is the correct response. What matters is that the TLS handshake succeeded: + +``` +* Server certificate: +* subject: CN=relay-us.example.com +* issuer: C=US; O=Let's Encrypt; CN=E8 +* SSL certificate verify ok. +``` + +The issuer's `CN` names whichever intermediate signed your certificate, so yours will often differ. + +Two things can go wrong here, and they look different. If the first attempt fails with a TLS error such as `SSL_ERROR_SYSCALL`, wait a few seconds and run it again: that first request is what triggers issuance, and it can time out while that happens. + +If instead the command hangs with no output and no error, issuance is stuck rather than slow. The relay reports the reason and `curl` cannot: + +```bash +docker compose logs relay | grep -i acme +``` + +A rate limit is the likeliest cause if you have rebuilt the same relay host several times. Let's Encrypt allows five certificates per week for one exact set of domain names, and the error names the time it frees up again. + +Repeat Steps 3 to 5 on every relay server, using the **same** `NB_AUTH_SECRET` and that host's own domain name. + +Do not continue until every relay answers. Step 6 is the cutover. + +## Step 6: Point the main server at the relays + +On the main server, take a copy of the current configuration first, then edit it: + +```bash +cd ~/netbird-enterprise +cp config.yaml config.yaml.bak +nano config.yaml +``` + +On a migrated deployment that has `config.yaml.enterprise`, copy and edit that file instead. It is the one the server reads, and changes made to `config.yaml` never reach it. + +The backup matters because a mistake here does not announce itself: the deployment keeps working and only traffic event logging stops. + +Add `stuns` and `relays` inside the existing `server:` block, at the same indentation as the keys already there. Anywhere inside that block works; the example below puts them after `authSecret` to keep the relay settings together. Everything else already under `server:` stays exactly as it is, including the `auth`, `reverseProxy`, `store`, `activityStore` and `trafficFlow` sections a `getting-started-enterprise.sh` deployment carries. Setting `relays.addresses` is what turns the embedded relay off, and it turns the embedded STUN server off too, which is why `stuns` is required: + +```yaml +server: + listenAddress: ":80" + exposedAddress: "https://netbird.example.com:443" + + # Leave authSecret exactly as it is. relays.addresses below is what + # disables the embedded relay. + authSecret: "your-existing-secret" + + # External STUN servers, one per relay host + stuns: + - uri: "stun:relay-us.example.com:3478" + proto: "udp" + - uri: "stun:relay-eu.example.com:3478" + proto: "udp" + + # External relay servers + relays: + addresses: + - "rels://relay-us.example.com:443" + - "rels://relay-eu.example.com:443" + secret: "the-value-from-step-2" + credentialsTTL: "24h" + + # ... the rest of your existing configuration is unchanged +``` + + +`relays.secret` must be the value you read in Step 2. It must match `NB_AUTH_SECRET` on every relay host and, if your deployment has a traffic-flow receiver, the receiver's `NB_FLOW_AUTH_SECRET`. The two mismatches fail differently: + +- **With a relay:** that relay rejects peers, and its log says `invalid signature` (see [Troubleshooting](#troubleshooting)). +- **With the receiver:** relaying keeps working, and traffic events stop. Only the receiver's log shows it. Step 8 checks for exactly this. + + +Leave `server.authSecret` in place. It is required only when the embedded relay is running, and removing it gains you nothing. + +`stuns` and `relays` are available since **v0.65.0**. + +## Step 7: Restart the server + +`config.yaml` is a bind mount, so editing it changes nothing Docker Compose compares. `docker compose up -d` reports the service as already up to date and keeps serving the old configuration. Restart it explicitly: + +```bash +cd ~/netbird-enterprise +docker compose restart netbird-server +``` + + +Do not run `docker compose down` to apply this change. On an Enterprise deployment that also stops Traefik, PostgreSQL, NATS and the traffic-flow services. Restarting the one container you changed is enough. + + +Confirm the embedded relay is off and your addresses are reaching clients: + +```bash +docker compose logs --tail 400 netbird-server | grep -E 'Relay:|Relay addresses' | tail -3 +``` + +If the server has been restarted before, its log holds a block like this for every restart. The `tail` keeps you looking at the most recent one, which is the only one that reflects the change you just made. + +``` +INFO combined/cmd/root.go:691: Relay: false (log level: ) +INFO combined/cmd/root.go:731: Relay addresses: [rels://relay-us.example.com:443 rels://relay-eu.example.com:443] +INFO combined/cmd/config.go:806: Relay addresses: [rels://relay-us.example.com:443 rels://relay-eu.example.com:443] +``` + +The address list is logged twice, from two different places in the server. That is normal. + + +Then check a peer: + +```bash +netbird status -d +``` + +``` +Relays: + [stun:relay-us.example.com:3478] is Available + [stun:relay-eu.example.com:3478] is Available + [rels://relay-eu.example.com:443] is Available via ws +``` + +Every STUN server you configured appears. Relays behave differently: the client dials them in parallel and keeps the first to answer, normally the nearest, and that one becomes its **home relay**. A second `rels://` line is also normal, because a client connects to another peer's home relay when that peer picked a different one. + +The suffix names the transport that won the race, and it is `ws` or `quic` depending on which answered first. Both are normal, the winner can differ between two peers on the same deployment, and neither is a sign of a problem. What 443/udp buys you is that the QUIC attempt can win or lose on merit rather than sitting until it times out. + +## Step 8: Confirm traffic events survived + +**Do not skip this.** It is the check that belongs to Enterprise, and the failure it catches is invisible everywhere else: peers stay connected, the dashboard shows no error, and only the traffic event stream stops. + +It applies only to a deployment with a traffic-flow receiver. Without one there are no traffic events to lose, and you are done. + +Generate a little traffic between two peers again, `ping` being enough, wait about 90 seconds for the client's reporting interval, then look for rejected tokens on the main server: + +```bash +cd ~/netbird-enterprise +docker compose logs --since 120s receiver | grep -c 'invalid signature' +``` + +On a migrated deployment the service is `flow-receiver`, here and in the next command. + +This must be `0`. + +If it is not, `relays.secret` and the receiver's `NB_FLOW_AUTH_SECRET` do not match. Either side can be the stale one: + +- **`relays.secret` is wrong.** Go back to Step 2, re-read the value from the server configuration, and correct `relays.secret` and `NB_AUTH_SECRET` on every relay host to match it. +- **The receiver still holds an old value**, typically after a rotation applied with `restart`. Check the receiver's value as shown in [Rotating to a new secret](#rotating-to-a-new-secret). Make sure `.env` holds the same value as `relays.secret`, then recreate the receiver with `docker compose up -d receiver` (`flow-receiver` on a migrated deployment). + +While they differ, the receiver log shows: + +``` +ERRO server/auth.go:127: invalid token validation: invalid signature +``` + +and your peers retry in a log nobody watches: + +``` +ERRO flow/client/client.go:287: flow receiver sent no headers +ERRO flow/client/client.go:143: failed to establish flow stream, retrying: check header: should have headers +ERRO client/internal/netflow/manager.go:228: failed to send flow event to server: stream not initialized +``` + +For a positive signal, count the events the receiver accepted: + +```bash +docker compose logs --since 120s receiver | grep -c 'sending event' +``` + +Greater than zero confirms events are arriving. The two checks answer different questions. `0` for `invalid signature` proves the secret is right, which is the part this page can break, but not that events arrive: the receiver can stop accepting them for other reasons. Only a count above zero proves that. The receiver logs these lines at debug level, which is its default (`NB_FLOW_LOG_LEVEL`). If you have raised its log level, this count stays at `0` even when events arrive, so the check is not complete until the events show up on the Traffic Events page in the dashboard. + + +Check the receiver's log rather than the Traffic Events page in the dashboard. Events pass through the enricher and PostgreSQL before they appear there, so the dashboard lags well behind and cannot tell you whether events are arriving right now. + + +## Testing failover + +Stop the relay a peer is actually using, which is the one on its `rels://` line, otherwise nothing observable changes. Then check that peer again: + +```bash +netbird status -d +``` + +**Read the `rels://` line, not the `stun:` lines.** It is the one that shows failover working: the stopped relay drops out of it and the surviving relay carries the traffic. + +``` + [rels://relay-eu.example.com:443] is Available via ws +``` + +The `stun:` entries are not a dependable failover signal. Depending on circumstances they either report the stopped server as `Unavailable` with a reason, or sit at `Checking...` for every configured STUN server, including ones that are perfectly healthy, without clearing on their own or after a client restart. Both have been observed. Either way they return to `Available` once the stopped relay is back, so do not read `Checking...` as a second failure. + +To prove relayed traffic end to end, force a peer to relay instead of connecting directly: + +```bash +sudo netbird service reconfigure --service-env NB_FORCE_RELAY=true +``` + +Test connectivity to another peer, then put it back: + +```bash +sudo netbird service reconfigure --service-env NB_FORCE_RELAY=false +``` + +## Rotating to a new secret + +If you would rather not reuse the existing secret, generate one and set it **everywhere that value appears**, not only where the relay needs it: + +```bash +openssl rand -base64 32 +``` + +1. `NB_AUTH_SECRET` in `relay.env` on every relay host. +2. `relays.secret` in the server configuration on the main server (`config.yaml`, or `config.yaml.enterprise` on a migrated deployment that has it). +3. `server.authSecret` in the same file. +4. In `.env` on the main server, under whichever name your deployment uses: `NETBIRD_RELAY_AUTH_SECRET` on a fresh install, `NB_FLOW_AUTH_SECRET` on one migrated from the community edition with traffic flow. That is the value that reaches the traffic-flow receiver. A migrated deployment without traffic flow has no entry here. + +All four must end up holding the same value. Relaying needs 1 and 2 to match, and traffic events need 2 and 4 to match. Entry 3 is not used by either, but the product's own migration tooling treats `server.authSecret` as the value the receiver's secret must equal, so leaving it stale stores up a failure for a later upgrade. + +Then apply the change. On every relay host: + +```bash +cd ~/netbird-relay +docker compose up -d +``` + +and on the main server, before re-running Step 8: + +```bash +cd ~/netbird-enterprise +docker compose restart netbird-server +docker compose up -d receiver +``` + +Use `flow-receiver` on a migrated deployment. With no receiver, the first command is all you need. + +The main server's two commands differ on purpose. `netbird-server` reads `config.yaml` through a bind mount, so restarting it is enough. The receiver takes its secret from `.env`, and a container's environment is fixed when the container is created, exactly as the relay's `relay.env` is: `docker compose restart receiver` brings back the **old** value and reports nothing wrong. Only `up -d`, which recreates the container, picks up the new one. + +Get that second command wrong and the symptom is exactly the one Step 8 is designed to catch, which will send you back to re-check four values that are already correct. If Step 8 still shows `invalid signature` after a rotation, confirm the receiver really took the new value: + +```bash +docker exec netbird-receiver env | grep NB_FLOW_AUTH_SECRET +``` + +On a migrated deployment the container is `netbird-flow-receiver`. + +Miss the fourth entry, or apply it with `restart`, and you are back to silently losing traffic events. + +## Troubleshooting + +**A relay shows as `Unavailable` on a peer.** The reason string tells you which problem you have. + +``` +[rels://relay-us.example.com:443] is Unavailable, reason: failed to get reader: failed to read frame header: EOF +``` + +That one is a secret mismatch. Over QUIC the same rejection reads `reason: closed by server` instead, and you will see it on a peer only once every relay is rejecting: a peer lists only the relays it is using, so one bad relay among several is simply absent from the list rather than shown as failed. The relay host says so outright either way, in `docker compose logs relay`: + +``` +ERRO relay/server/relay.go:141: failed to handshake: validate sha-... (203.0.113.10:50907): invalid signature +``` + +Make `relays.secret` and every host's `NB_AUTH_SECRET` identical. If the relay logs no connection attempt at all, the problem is reachability instead: confirm the domain resolves, test with `nc -zv relay-us.example.com 443`, and check the certificate is valid. + +Other reason strings point away from the secret and towards reachability or the certificate. `relay client not connected` and `connect to relay server: context deadline exceeded` both mean the peer never completed a TLS session with the relay, so check Step 5 again on that host before touching any secret. + +`x509: certificate signed by unknown authority` in the reason means the peer does not trust the relay's certificate, which happens with a private CA the device has not been given. See [If you already have TLS certificates](#if-you-already-have-tls-certificates). + +When every relay is failing, each relay's entry can carry the errors of all of them, so a reason quoting one host's error may appear against another. The relay host's own log is the reliable record of which host rejected what. + +Either way the `stun:` entry for the same host stays `Available`, because STUN is unauthenticated, so it cannot tell you anything here. + +**STUN is not working.** Confirm 3478/udp is open and that `NB_ENABLE_STUN=true` is set. Some networks block outbound UDP, so try from a different network before changing the relay. + +**Traffic events stopped.** That is the secret mismatch in Step 8, not a relay fault. + +## When not to do this + +Splitting relays out costs hosts to patch, certificates to renew, and a secret to keep in step across machines. Do it when relayed traffic is loading your server, or when users are far enough away that a nearer relay is a real improvement. Do not do it just to make the deployment look tidier. + +If your aim is surviving the loss of the main server rather than spreading relay load, external relays do not get you there: management and signal are still on one host. That is what [Running a Highly Available Self-Hosted Deployment](/selfhosted/maintenance/scaling/high-availability) is for. + +## Summary + +Read the secret your deployment already has, build each relay host with it, prove every relay answers from outside, then point the server configuration at them and restart `netbird-server` alone. Finally, if your deployment has a traffic-flow receiver, check that it is still accepting flow events, because your peers will look perfectly healthy even when it is not. diff --git a/src/pages/selfhosted/maintenance/scaling/scaling-your-self-hosted-deployment.mdx b/src/pages/selfhosted/maintenance/scaling/scaling-your-self-hosted-deployment.mdx index aa1fc07a..a8b539cb 100644 --- a/src/pages/selfhosted/maintenance/scaling/scaling-your-self-hosted-deployment.mdx +++ b/src/pages/selfhosted/maintenance/scaling/scaling-your-self-hosted-deployment.mdx @@ -23,11 +23,12 @@ The default single-server deployment runs all services on one machine: **Traefik and a **combined netbird-server** container that includes Management, Signal, and Relay + STUN as components. Traefik handles TLS termination on ports 80/443, while STUN listens on UDP port 3478. The Management server uses a **SQLite** database by default. After splitting, the **main server** keeps Traefik, Dashboard, Management, and optionally Signal. -The **relay servers** run independently on different machines, each handling relay (port 443) and STUN (port 3478) traffic. Peers receive relay addresses from the Management server and connect to them directly. Optionally, the SQLite database can be migrated to **PostgreSQL** on a dedicated server, and Signal can also be extracted to its own machine. +The **relay servers** run independently on different machines, each handling relay (port 443, TCP for WebSocket and UDP for QUIC) and STUN (port 3478/udp) traffic. Peers receive relay addresses from the Management server and connect to them directly. Optionally, the SQLite database can be migrated to **PostgreSQL** on a dedicated server, and Signal can also be extracted to its own machine. ## Guides - [Set Up External Relay Servers](/selfhosted/maintenance/scaling/set-up-external-relays) — Deploy relay and STUN servers on separate machines and configure the main server to use them +- [External Relays on a Commercial License](/selfhosted/enterprise/external-relays) — What differs on NetBird Enterprise, where the shared secret is also used by traffic flow - [Migrate from SQLite to PostgreSQL](/selfhosted/maintenance/scaling/migrate-sqlite-to-postgresql) — Move the Management database to a dedicated PostgreSQL instance (optional) - [Set Up External Signal Server](/selfhosted/maintenance/scaling/set-up-external-signal) — Extract the Signal server to its own machine (optional) @@ -46,6 +47,7 @@ The **relay servers** run independently on different machines, each handling rel | `NB_LETSENCRYPT_EMAIL` | No | Email for Let's Encrypt notifications | | `NB_TLS_CERT_FILE` | No | Path to TLS certificate (alternative to Let's Encrypt) | | `NB_TLS_KEY_FILE` | No | Path to TLS private key | +| `NB_TRUSTED_PROXIES` | No | Comma-separated IPs or CIDRs whose `X-Real-Ip` and `X-Real-Port` headers the relay trusts. Since v0.75.0 | | `NB_LOG_LEVEL` | No | Log level: `debug`, `info`, `warn`, `error` | ### Main Server config.yaml - External Services diff --git a/src/pages/selfhosted/maintenance/scaling/set-up-external-relays.mdx b/src/pages/selfhosted/maintenance/scaling/set-up-external-relays.mdx index f714e040..4333a7ce 100644 --- a/src/pages/selfhosted/maintenance/scaling/set-up-external-relays.mdx +++ b/src/pages/selfhosted/maintenance/scaling/set-up-external-relays.mdx @@ -4,15 +4,25 @@ import {Note, Warning} from "@/components/mdx"; This guide is part of the [Splitting Your Self-Hosted Deployment](/selfhosted/maintenance/scaling/scaling-your-self-hosted-deployment) guide. It covers deploying external relay and STUN servers and configuring your main server to use them. + +Running NetBird Enterprise with a commercial license? Use [External Relays on a Commercial License](/selfhosted/enterprise/external-relays) instead. It is the same procedure written for an Enterprise deployment, and it is complete on its own. Following this page would have you generate a new shared secret, which on an Enterprise deployment with traffic flow stops traffic event logging without breaking connectivity. + + For each relay server you want to deploy: ## Server Requirements -- A Linux VM with at least **1 CPU** and **1GB RAM** +- A Linux VM with at least **1 CPU** and **1 GB RAM** - Public IP address - A domain name pointing to the server (e.g., `relay-us.example.com`) - Docker installed -- Firewall ports open: **80/tcp** (Let's Encrypt HTTP challenge), **443/tcp** (relay), and **3478/udp** (STUN). If you configure multiple STUN ports, open all of them +- Firewall ports open: **443/tcp** and **443/udp** (relay), and **3478/udp** (STUN). If you configure multiple STUN ports, open all of them + + +**No inbound port 80 is needed.** A relay proves its domain to Let's Encrypt over 443 itself, using the TLS-ALPN-01 challenge. Your main NetBird server is different and does still need 80, and so may a proxy in front of the relay. + +**Both 443 rules matter.** The relay serves two transports on that port: WebSocket over TCP and QUIC over UDP. Clients race both and use whichever connects first. With 443/udp closed the QUIC attempt runs until it times out, WebSocket carries the connection, and nothing reports an error, so the missing rule is easy to overlook. A relay behind a TLS-terminating reverse proxy is the exception: it cannot serve QUIC, so it does not need 443/udp. See [Running the Relay Behind a Proxy or Load Balancer](#running-the-relay-behind-a-proxy-or-load-balancer). + ## Generate Authentication Secret @@ -23,7 +33,9 @@ All relay servers must share the same authentication secret with your main serve openssl rand -base64 32 ``` -Save this secret - you'll need it for both the relay servers and your main server's config. +Save this secret. The identical value goes in two settings: `NB_AUTH_SECRET` on every relay server, and `relays.secret` on your main server. + +Paste it from one source into each file rather than retyping it. A single wrong character produces a relay that looks healthy and refuses every peer. ## Create Relay Configuration @@ -42,9 +54,6 @@ NB_LISTEN_ADDRESS=:443 NB_EXPOSED_ADDRESS=rels://relay-us.example.com:443 NB_AUTH_SECRET=your-shared-secret-here -# Only when a reverse proxy or load balancer is directly in front of the relay -# NB_TRUSTED_PROXIES=10.20.0.10,10.20.1.0/24 - # TLS via Let's Encrypt (automatic certificate provisioning) NB_LETSENCRYPT_DOMAINS=relay-us.example.com NB_LETSENCRYPT_EMAIL=admin@example.com @@ -59,6 +68,14 @@ NB_STUN_PORTS=3478 Replace `relay-us.example.com` with your relay server's domain and `your-shared-secret-here` with the secret you generated. +The file holds the shared secret, so make it readable by root only: + +```bash +chmod 600 relay.env +``` + +The relay reads `relay.env` when its container is created. If you change the file later on a running relay, apply it with `docker compose up -d`. `docker compose restart` keeps the old values. + Create `docker-compose.yml`: ```yaml @@ -68,7 +85,10 @@ services: container_name: netbird-relay restart: unless-stopped ports: - - '443:443' + # Both relay transports: WebSocket over TCP and QUIC over UDP. + # Docker treats a bare '443:443' as TCP only, so the UDP line is required. + - '443:443/tcp' + - '443:443/udp' # Expose all ports listed in NB_STUN_PORTS - '3478:3478/udp' env_file: @@ -87,31 +107,68 @@ volumes: ## Running the Relay Behind a Proxy or Load Balancer -When a reverse proxy or load balancer terminates the Relay WebSocket connection, set `NB_TRUSTED_PROXIES` to the IP addresses or CIDRs of the proxies that connect **directly** to the relay: +In this shape your proxy holds the certificate and the relay serves plain HTTP behind it. + +Use this shape only because a load balancer or proxy is already in front of your relays. Do not add one just to terminate TLS on the relay host: the relay already does that itself, and a proxy in front of it costs you QUIC. + +**On the relay host**, `relay.env` has no `NB_LETSENCRYPT_*` lines, listens on an internal port, and names the proxy's address: ```bash -NB_TRUSTED_PROXIES=10.20.0.10,10.20.1.0/24 +NB_LOG_LEVEL=info +NB_LISTEN_ADDRESS=:8080 +NB_EXPOSED_ADDRESS=rels://relay-us.example.com:443 +NB_AUTH_SECRET=your-shared-secret-here +NB_TRUSTED_PROXIES=10.20.0.10 +NB_ENABLE_STUN=true +NB_STUN_PORTS=3478 ``` -The proxy must strip any client-supplied `X-Real-Ip` and `X-Real-Port`, then overwrite both headers with the address and source port from the validated client connection. The relay accepts those headers only when the immediate connection source matches the trusted list. If either header is missing, the list is empty, or the immediate source is not trusted, the relay ignores the headers and uses the direct connection address. +Keep `rels://` written out in `NB_EXPOSED_ADDRESS`. With no TLS of its own, the relay would otherwise advertise `rel://`. + +In `docker-compose.yml`, remove the two `443` port lines and keep `3478:3478/udp` for STUN. Make port 8080 reachable from the proxy and nothing else. If the proxy is on another machine, publish the port with `'8080:8080/tcp'` and allow it from the proxy's address only, at a network firewall or cloud security group. A host firewall such as UFW or firewalld does not restrict it, because Docker-published ports bypass those rules. + +**Your proxy must:** + +- hold a certificate for the relay's domain, and accept connections on 443/tcp; +- forward WebSocket upgrades over HTTP/1.1 to the relay's port 8080; +- set both `X-Real-Ip` and `X-Real-Port` to the address and source port of the connection it received, overwriting anything the client sent. + +The relay uses those two headers only when the connection comes from an address in `NB_TRUSTED_PROXIES`, and only when both are present. A proxy that sets only `X-Real-Ip` gets its own address logged for every peer. A proxy that passes the client's headers through lets any client write the address of its choice into your relay logs. + +`NB_TRUSTED_PROXIES` is the address the relay sees the proxy's connections come from, which is not always the proxy's public address. When the proxy and relay share a Docker network, it is the proxy container's address on that network. When the proxy is another machine on the same private network, it is that machine's private address: a published Docker port keeps the original source address. -Trust only the smallest proxy IP ranges you control. Do not trust `0.0.0.0/0`, `::/0`, or a shared network where untrusted systems can connect directly to the relay and forge forwarding headers. Invalid IP or CIDR entries prevent the relay from starting. +Trust only the smallest proxy ranges you control. Never `0.0.0.0/0`, `::/0`, or a shared network where an untrusted system could reach the relay directly and forge these headers. Entries are IP addresses or CIDRs separated by commas. Hostnames are rejected, and a prefix such as `10.20.0.10/24` trusts the whole `/24`. -The CLI equivalent is `--trusted-proxies`. Restart the relay after changing the list and check its startup logs for parsing errors. +These headers decide only which client address the relay records, including in the `invalid signature` lines used for troubleshooting. Authentication does not depend on them. + +Apply a change to `NB_TRUSTED_PROXIES` with `docker compose up -d relay`. An invalid entry stops the relay: it exits with `failed to parse trusted proxies: ...` and restarts in a loop until you fix it. To confirm the setting works, check that the relay logs your peers' addresses rather than the proxy's: + +```bash +docker compose logs relay | grep 'WS client connected from' +``` + +The CLI equivalent of `NB_TRUSTED_PROXIES` is `--trusted-proxies`, available since **v0.75.0**. + +At startup the relay reports that it cannot serve QUIC: + +``` +WARN relay/server/server.go:78: Not starting QUIC listener: valid TLS config is required for QUIC listener +``` + +That is expected here. Peers connect over WebSocket, so neither the proxy nor the relay host needs 443/udp. The proxy needs 443/tcp, plus whatever it uses to obtain its own certificate, often 80/tcp. The relay host needs 3478/udp for STUN. ## Alternative: TLS with Existing Certificates -If you have existing TLS certificates (e.g., from your own CA or a wildcard cert), replace the Let's Encrypt variables in `relay.env` with: +If you have existing TLS certificates (e.g., a wildcard certificate, or one from your own CA), delete the three `NB_LETSENCRYPT_*` lines from `relay.env` and add: ```bash -# Replace the NB_LETSENCRYPT_* lines with: NB_TLS_CERT_FILE=/certs/fullchain.pem NB_TLS_KEY_FILE=/certs/privkey.pem ``` -And add a certificate volume to `docker-compose.yml`: +Then replace the relay service's `volumes:` list in `docker-compose.yml` with this one. Do not add it as a second `volumes:` key, which makes the file invalid: ```yaml volumes: @@ -119,6 +176,13 @@ And add a certificate volume to `docker-compose.yml`: - relay_data:/data ``` +`fullchain.pem` is your certificate followed by any intermediates, and the certificate must cover the relay's domain. Four things catch people out: + +- **Set both variables, and remove every `NB_LETSENCRYPT_*` line.** If a Let's Encrypt line remains, the relay uses Let's Encrypt and ignores your files, and the `curl -v` check below shows a Let's Encrypt issuer instead of yours. If only one `NB_TLS_*` variable is set, the relay starts with no TLS at all, and the sign is a missing QUIC line at startup. +- **Nothing in the startup log names your certificate.** The `curl -v` check below is how you confirm it is the one being served. +- **The files are read once, at startup.** Restart the relay after every renewal: `docker compose restart relay`. +- **A private CA must be trusted by every peer.** NetBird checks the relay's certificate against each device's operating-system trust store, and has no setting for a separate CA. A device that does not trust yours reports the relay as `Unavailable` with `x509: certificate signed by unknown authority`, and cannot relay to other peers. Distribute the CA to every device before you switch. + ## Start the Relay Server ```bash @@ -131,19 +195,26 @@ Verify it's running: docker compose logs -f ``` -You should see: +You should see the relay announce its exposed address, both listeners, and the STUN server: + ``` -level=info msg="Starting relay server on :443" -level=info msg="Starting STUN server on port 3478" +INFO relay/cmd/root.go:242: server will be available on: rels://relay-us.example.com:443 +INFO relay/server/listener/ws/listener.go:51: WS server listening address: :443 +INFO relay/server/listener/quic/listener.go:39: QUIC server listening on address: :443 +INFO [component: stun] stun/server.go:71: STUN server listening on [::]:3478 ``` +Other lines appear alongside these, including the Let's Encrypt setup, the health check server and the metrics server. The order changes from one start to the next, because the listeners come up concurrently. What matters is that all four lines are present, not where they sit. + +A missing QUIC line means the relay has no TLS configuration of its own, which is expected only behind a TLS-terminating proxy. + If you configured Let's Encrypt, the relay generates TLS certificates lazily on the first incoming request. Trigger certificate provisioning and verify it by running: ```bash curl -v https://relay-us.example.com/ ``` -A `404 page not found` response is expected — what matters is that the TLS handshake succeeds. Look for a valid Let's Encrypt certificate in the output: +A `404 page not found` response is expected. What matters is that the TLS handshake succeeds: ``` * Server certificate: @@ -152,6 +223,18 @@ A `404 page not found` response is expected — what matters is that the TLS han * SSL certificate verify ok. ``` +The issuer's `CN` names whichever intermediate signed your certificate, so yours will often differ. + +Two things can go wrong here, and they look different. If the first attempt fails with a TLS error such as `SSL_ERROR_SYSCALL`, wait a few seconds and run it again: that first request is what triggers issuance, and it can time out while that happens. + +If instead the command hangs with no output and no error, issuance is stuck rather than slow, and only the relay can tell you why: + +```bash +docker compose logs relay | grep -i acme +``` + +A rate limit is the likeliest cause if you have rebuilt the same relay host several times. Let's Encrypt allows five certificates per week for one exact set of domain names. + ## Repeat for Additional Relay Servers If deploying multiple relays (e.g., for different regions), repeat the steps above on each server. Use the **same `NB_AUTH_SECRET`** but update the domain name for each. @@ -169,15 +252,23 @@ cd ~/netbird # or wherever your deployment is nano config.yaml ``` -Remove the `authSecret` from the `server` section and add `relays` and `stuns` sections pointing to your external servers. The presence of the `relays` section disables both the embedded relay and the embedded STUN server, so the `stuns` section is required to provide external STUN addresses: +Add `stuns` and `relays` inside the existing `server:` block, at the same indentation as the keys already there. Anywhere inside that block works. + +This is an addition, not a replacement. Leave every key already under `server:` exactly as it is. A quickstart deployment also carries `auth`, `reverseProxy` and `store` sections there, holding your OIDC redirect URIs, your reverse-proxy trust settings and your datastore encryption key. Overwriting any of them breaks the deployment, so do not paste the example below over your existing block. + +The presence of `relays.addresses` is what disables the embedded relay, and it disables the embedded STUN server too, so the `stuns` section is required to provide external STUN addresses. + +You do not need to remove `server.authSecret`. It is required only when the embedded relay is running. Leaving it in place is also the safer choice on a NetBird Enterprise deployment, where the same value has a second job: see [External Relays on a Commercial License](/selfhosted/enterprise/external-relays). + +Both `relays` and `stuns` are available since **v0.65.0**. ```yaml server: listenAddress: ":80" exposedAddress: "https://netbird.example.com:443" - # Remove authSecret to disable the embedded relay - # authSecret: ... - # Remove or comment out stunPorts since we're using external STUN + # Leave authSecret as it is. relays.addresses below is what disables the embedded relay. + authSecret: "your-existing-secret" + # stunPorts no longer applies once stuns is set, so you can comment it out # stunPorts: # - 3478 metricsPort: 9090 @@ -201,18 +292,20 @@ server: secret: "your-shared-secret-here" credentialsTTL: "24h" - auth: - issuer: "https://netbird.example.com/oauth2" - # ... rest of auth config + # ... the rest of your existing configuration is unchanged ``` -The `secret` under `relays` and the `NB_AUTH_SECRET` on all relay servers **must be identical**. Mismatched secrets will cause relay connections to fail silently. +The `secret` under `relays` and the `NB_AUTH_SECRET` on every relay server **must be identical**. If they differ, peers cannot use that relay. +A mismatch is easiest to spot on the relay host, which logs `failed to handshake: ... invalid signature` for every rejected peer. + +It is harder to spot from a peer, because a peer lists only the relays it is actually using. One bad relay among several never appears as a failure: the peer picks a working one instead, and the broken relay is simply absent. Only when every relay rejects does `netbird status -d` report `Unavailable, reason: failed to get reader: failed to read frame header: EOF`. The `stun:` entry stays `Available` throughout, because STUN is unauthenticated. + ### Update docker-compose.yml (Optional) -If your main server was exposing STUN port 3478, you can remove it since STUN is now handled by external relays: +If your main server was exposing STUN port 3478, you can remove it, since STUN is now handled by external relays. Change only the `ports` entry and leave the rest of the service as it is: ```yaml netbird-server: @@ -227,15 +320,53 @@ If your main server was exposing STUN port 3478, you can remove it since STUN is - netbird_data:/var/lib/netbird - ./config.yaml:/etc/netbird/config.yaml command: ["--config", "/etc/netbird/config.yaml"] + # ... the rest of this service, including its labels, is unchanged ``` ### Restart the Main Server +Which command you need depends on whether you did the optional `docker-compose.yml` step above. + +**If you edited `config.yaml` only**, restart the container: + ```bash -docker compose down -docker compose up -d +docker compose restart netbird-server ``` +`config.yaml` is a bind mount, so editing it does not change anything Docker Compose compares. `docker compose up -d` would report the service as already up to date, and the server would keep serving the old configuration. + +**If you also removed the STUN port from `docker-compose.yml`**, recreate the container instead: + +```bash +docker compose up -d netbird-server +``` + +A published port is fixed when the container is created, so `docker compose restart` cannot remove one: the container comes back with the mapping still in place and nothing reports a problem. Recreating the container applies both files at once, so this one command covers the `config.yaml` change as well. + + +Avoid `docker compose down` here. It stops every other service too, including your reverse proxy and, on larger deployments, the database and the traffic-flow services. + + +## The Embedded Relay Is Now Off + +Setting `relays.addresses` switches the embedded relay off. There is no setting that keeps it running alongside external relays, and listing the main server's own address among them does not bring it back: clients are handed an address that never answers. + +```bash +curl https://netbird.example.com/relay +``` + +``` +Relay service not enabled +``` + +The endpoint answers with HTTP 404 and that single line as its body. Add `-i` to the command if you want to see the status code as well. + +From here on, relay capacity is whatever your external relay hosts provide. Size them accordingly. + + +This is a cutover, not a gradual migration. The restart that applies the change is also the restart that drops every connection the embedded relay was serving, so those peers reconnect straight onto the external relays. Bring the relay hosts up and confirm they answer before you edit `config.yaml`, because there is no period where both are available. + + ## Verify the Configuration ### Check Main Server Logs @@ -256,6 +387,8 @@ INFO combined/cmd/root.go: Relay: false (log level: ) Relay addresses: [rels://relay-us.example.com:443 rels://relay-eu.example.com:443] ``` +The address list is logged twice, from two different places in the server. That is normal. + ### Check Peer Status Connect a NetBird client and verify that both STUN and relay services are available: @@ -264,24 +397,31 @@ Connect a NetBird client and verify that both STUN and relay services are availa netbird status -d ``` -The output should list your external STUN and relay servers. All configured STUN servers will appear, but only one randomly chosen relay is used per client: +The output lists your external STUN and relay servers. Every configured STUN server appears. Relays work differently: the client dials them in parallel and keeps the first to answer, which is normally the nearest. That one becomes its **home relay**, so relays in several regions give each client the nearest one automatically. ``` Relays: [stun:relay-us.example.com:3478] is Available [stun:relay-eu.example.com:3478] is Available - [rels://relay-eu.example.com:443] is Available + [rels://relay-eu.example.com:443] is Available via ws ``` -You can also test failover by stopping one of the relay servers and checking the status again. The client will detect the unavailable server and use the remaining one: +You may see more than one `rels://` line. A client also connects to another peer's home relay when that peer picked a different one, so holding two relay connections at once is normal in a multi-region deployment. + +The suffix names the transport that won the race, and it is `ws` or `quic` depending on which answered first. Both are normal, the winner can differ between two peers on the same deployment, and neither is a sign of a problem. What 443/udp buys you is that the QUIC attempt can win or lose on merit rather than sitting until it times out. + +### Test Failover + +Stop the relay a peer is actually using, which is the one on its `rels://` line, otherwise nothing observable changes. Then check that peer again. + +Read the `rels://` line, not the `stun:` lines. It is the one that shows failover working: the stopped relay drops out of it and the surviving relay carries the traffic. ``` -Relays: - [stun:relay-us.example.com:3478] is Available - [stun:relay-eu.example.com:3478] is Unavailable, reason: stun request: context deadline exceeded - [rels://relay-us.example.com:443] is Available + [rels://relay-us.example.com:443] is Available via ws ``` +The `stun:` entries are not a dependable failover signal. Depending on circumstances they either report the stopped server as `Unavailable` with a reason, or sit at `Checking...` for every configured STUN server, including healthy ones, without clearing on their own or after a client restart. Both have been observed. Either way they return to `Available` once the stopped relay is back, so do not read `Checking...` as a second failure. + ### Test Relay Connectivity You can force all peer connections through relay to verify it works end-to-end. On a client, run: