sendnrw/netbird-docs
2
0
Fork 0
mirror of https://github.com/netbirdio/docs.git synced 2026-08-24 16:51:26 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: grammar and readability pass on Docker Swarm HA guide

Browse Source
This commit is contained in:
Jack Carter
2026-07-06 18:17:59 +02:00
parent fcb76c3558
commit 8bc5c088bd
1 changed files with 23 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
src/pages/selfhosted/enterprise/docker-swarm.mdx
View File

@@ -17,7 +17,7 @@ import {Note, Warning} from "@/components/mdx";
If you're choosing infrastructure fresh, rather than fitting HA onto Swarm you already run, follow the [main HA guide](/selfhosted/maintenance/scaling/high-availability) directly: plain VMs or managed services behind a real load balancer more closely match what NetBird's HA design assumes, and you'll hit fewer of the rough edges this page exists to document. If you need Kubernetes instead, that's a separate deployment path not covered here.
If you do run it here, Swarm buys you: one-unit deploys and updates, Docker secrets distributing the byte-identical keys every Management replica must share, declarative spread across failure domains, and crashed instances replaced in seconds. Its own routing mesh and service load balancing go unused here (Traefik and HAProxy do that work, below), and the PostgreSQL and Redis HA a managed database gives you for free becomes three clusters you operate yourself (Steps 2-4). If that last trade reads as a burden, stay on the main guide's path.
If you do run HA on Swarm, it buys you one-unit deploys and updates, Docker secrets distributing the byte-identical keys every Management replica must share, declarative spread across failure domains, and crashed instances replaced in seconds. Its own routing mesh and service load balancing go unused here (Traefik and HAProxy do that work, below), and the PostgreSQL and Redis HA a managed database gives you for free becomes three clusters you operate yourself (Steps 2-4). If that last trade reads as a burden, stay on the main guide's path.
## The problem with Swarm's own tooling
@@ -28,26 +28,26 @@ Two of Swarm's defaults get in the way of this architecture:
## Architecture
Run one Swarm manager node per failure domain — three nodes across three availability zones is the working example this guide uses throughout, and matches the minimum the main guide asks for (NATS alone needs 3 instances on separate failure domains). Use an odd number of managers: Swarm's own cluster consensus needs a majority to keep working, and 4 managers tolerate the same single failure as 3 while costing more.
Run one Swarm manager node per failure domain: three nodes across three availability zones is the working example this guide uses throughout, and matches the minimum the main guide asks for (NATS alone needs 3 instances on separate failure domains). Use an odd number of managers: Swarm's own cluster consensus needs a majority to keep working, and 4 managers tolerate the same single failure as 3 while costing more.
Each node runs one replica of everything: Traefik, one instance from each of the Management/Signal/Relay/Dashboard pools, one PostgreSQL/Patroni member, one Redis/Sentinel member, and one member each of the NATS and etcd clusters.
<img src="/docs-static/img/selfhosted/enterprise/docker-swarm-ha-topology.png" alt="Three-node NetBird HA topology on Docker Swarm: a load balancer health-checks each node's Traefik ping endpoint and fronts all three nodes; each node runs Traefik in front of Management, Signal, Relay, and Dashboard; Management reaches HAProxy (fronting PostgreSQL/Redis) and NATS directly; Signal reaches NATS directly; etcd (Patroni's coordination store) is never reached by the app tier; Relay and Dashboard stay on the application network only" className="imagewrapper-big"/>
*Three nodes, one per availability zone, behind a load balancer that health-checks each node's Traefik. Only Management and Signal touch the data-tier network — Relay and Dashboard never need to and don't get a network that lets them.*
The one load balancer this guide adds that the main guide doesn't need is HAProxy, in front of PostgreSQL and Redis specifically. On plain VMs you'd point those at your database/cache HA tooling (a managed Postgres, ElastiCache with a primary endpoint, and so on). Running that tooling *inside* Swarm means something must watch for the current primary and route only to it, and Traefik [can't do that for plain TCP](https://github.com/traefik/traefik/issues/1657). HAProxy does. NATS and etcd are never behind HAProxy: Management and Signal dial NATS members directly, and only Patroni talks to etcd.
The one load balancer this guide adds that the main guide doesn't need is HAProxy, in front of PostgreSQL and Redis specifically. On plain VMs you'd point those at your database/cache HA tooling (a managed PostgreSQL, ElastiCache with a primary endpoint, and so on). Running that tooling *inside* Swarm means something must watch for the current primary and route only to it, and Traefik [can't do that for plain TCP](https://github.com/traefik/traefik/issues/1657). HAProxy does. NATS and etcd are never behind HAProxy: Management and Signal dial NATS members directly, and only Patroni talks to etcd.
## Prerequisites
Everything the [main guide's prerequisites](/selfhosted/maintenance/scaling/high-availability#prerequisites) lists, plus:
- 3 or more Docker Swarm manager nodes (an odd number), one per failure domain, each reachable from the others on `2377/tcp`, `7946/tcp+udp`, and `4789/udp` (Swarm's own cluster-management and overlay-networking ports).
- A load balancer (or at minimum a DNS setup) in front of the nodes for the three public FQDNs. Prefer a real load balancer with active health checks — Step 7 gives it a per-node health endpoint to target, and it's what removes a dead node from client rotation automatically. Plain round-robin DNS works for a lab but keeps resolving to a dead node's address until someone edits the records. No Elastic/static IP is required on the nodes themselves, but whatever you choose needs to keep tracking the nodes' current addresses.
- A load balancer (or at minimum a DNS setup) in front of the nodes for the three public FQDNs. Prefer a real load balancer with active health checks — Step 7 gives it a per-node health endpoint to target, and it's what removes a dead node from client rotation automatically. Plain round-robin DNS works for a lab but keeps resolving to a dead node's address until someone edits the records. No static IP (such as an AWS Elastic IP) is required on the nodes themselves, but whatever you choose needs to keep tracking the nodes' current addresses.
- The split-component images, not the combined self-hosted image: `ghcr.io/netbirdio/management-cloud`, `ghcr.io/netbirdio/signal-cloud`, `netbirdio/relay`, `ghcr.io/netbirdio/dashboard-cloud`. The combined `netbird-server-cloud` image used by the single-file installer does not support HA mode. `management-cloud`, `signal-cloud`, and `dashboard-cloud` are GHCR-authenticated images; `netbirdio/relay` is public.
- `docker login ghcr.io` on every manager node, or `--with-registry-auth` on every `docker service create`/`docker stack deploy` so credentials propagate to whichever node pulls the image. Without one of these, services scheduled on other nodes fail to pull.
<Warning>
The commands below generate real secrets (database passwords, encryption keys) inline as `-e VAR=value` flags for readability. On a real deployment, use [Docker secrets](https://docs.docker.com/engine/swarm/secrets/) instead: a value passed as `-e` on the command line lands in shell history and is readable by anyone who can run `docker service inspect`.
The commands below pass real secrets (database passwords, encryption keys) inline as `-e VAR=value` flags for readability. On a real deployment, use [Docker secrets](https://docs.docker.com/engine/swarm/secrets/) instead: a value passed as `-e` on the command line lands in shell history and is readable by anyone who can run `docker service inspect`.
</Warning>
## Step 1: Label the nodes and create the overlay networks
@@ -67,9 +67,9 @@ Two networks, not one: Traefik and all four application pools sit on `netbird-ap
## Step 2: Deploy PostgreSQL as a Swarm-native HA cluster
You can point NetBird at any HA PostgreSQL the [main guide describes](/selfhosted/maintenance/scaling/high-availability#step-1-make-postgre-sql-highly-available) — a managed database works exactly as well from inside Swarm as from a plain VM, and is the simpler choice if it's available to you. If you want PostgreSQL running as Swarm services alongside everything else, [Patroni](https://patroni.readthedocs.io/) is the tool for it, using [Zalando's Spilo image](https://github.com/zalando/spilo) (Patroni and PostgreSQL bundled).
You can point NetBird at any HA PostgreSQL the [main guide describes](/selfhosted/maintenance/scaling/high-availability#step-1-make-postgre-sql-highly-available): a managed database works exactly as well from inside Swarm as from a plain VM, and is the simpler choice if it's available to you. If you want PostgreSQL running as Swarm services alongside everything else, [Patroni](https://patroni.readthedocs.io/) is the tool for it, using [Zalando's Spilo image](https://github.com/zalando/spilo) (Patroni and PostgreSQL bundled).
Run a small [etcd](https://etcd.io/) cluster first — Patroni needs a distributed consensus store for leader election, and etcd is the most common choice. Create three separate single-replica services, one per node — a replicated service would get a single Swarm-managed virtual address shared across replicas, and etcd members need to dial each other's specific address to form a cluster, which a shared address can't provide. The same one-service-per-member pattern applies to every clustered component in this guide.
Run a small [etcd](https://etcd.io/) cluster first: Patroni needs a distributed consensus store for leader election, and etcd is the most common choice. Create three separate single-replica services, one per node. A replicated service would get a single Swarm-managed virtual address shared across replicas, and etcd members need to dial each other's specific address to form a cluster, something a shared address can't provide. The same one-service-per-member pattern applies to every clustered component in this guide.
```sh
docker service create --name etcd-1 --network netbird-data \
@@ -87,7 +87,7 @@ docker service create --name etcd-1 --network netbird-data \
quay.io/coreos/etcd:v3.5.17
```
Repeat for `etcd-2`/`etcd-3` on the other two nodes: change the `--constraint`, the volume source, `ETCD_NAME`, and both `*_ADVERTISE_*` URLs to match (`etcd-2`/`etcd-3` throughout); `ETCD_INITIAL_CLUSTER` stays identical on all three, since it lists the whole cluster. Use `quay.io/coreos/etcd`, the image the etcd project itself publishes (some third-party etcd images use tags that don't exist). Mount a volume at `ETCD_DATA_DIR`: without persistent storage for its raft state, a restarted etcd member can fail to rejoin cleanly. No authentication is configured between components here — the non-attachable `netbird-data` network is the only security boundary; evaluate etcd's client/peer TLS for anything beyond a lab.
Repeat for `etcd-2`/`etcd-3` on the other two nodes: change the `--constraint`, the volume source, `ETCD_NAME`, and both `*_ADVERTISE_*` URLs to match (`etcd-2`/`etcd-3` throughout); `ETCD_INITIAL_CLUSTER` stays identical on all three, since it lists the whole cluster. Use `quay.io/coreos/etcd`, the image the etcd project itself publishes (some third-party etcd images use tags that don't exist). Mount a volume at `ETCD_DATA_DIR`: without persistent storage for its Raft state, a restarted etcd member can fail to rejoin cleanly. No authentication is configured between components here — the non-attachable `netbird-data` network is the only security boundary; evaluate etcd's client/peer TLS for anything beyond a lab.
Then one Patroni/Spilo service per node:
@@ -113,7 +113,7 @@ A few things about Patroni/Spilo that aren't obvious from its own documentation:
- **Set `SPILO_PROVIDER=local`, even on real cloud VMs.** Spilo auto-detects its cloud provider by querying the instance metadata service, and on AWS specifically, the follow-up metadata call it makes doesn't attach the token its own detection step just fetched — the call fails silently, and Patroni comes up with no node identity and can't elect a leader. Since HAProxy (Step 4) is doing the failover routing job that the cloud-specific auto-detection exists to support, forcing `local` costs you nothing.
- **`ETCD3_HOSTS`, not `ETCD_HOSTS`.** The latter selects Patroni's older etcd v2 client, which can't talk to etcd 3.x at all.
- **Give each replica a real `--hostname`.** Don't rely on a `PATRONI_NAME` environment variable to set Patroni's node identity — Spilo doesn't read it; node identity comes from the container's hostname.
- **Set a memory limit on every Postgres service.** Spilo sizes `shared_buffers` and `max_connections` from the container's memory limit, not from a value you set directly. With no limit, Spilo saw the full host's memory and set `shared_buffers` to roughly a quarter of it, which is a lot of memory to reserve when several other services share the same node. `2GiB` produces a sane 512MB.
- **Set a memory limit on every PostgreSQL service.** Spilo sizes `shared_buffers` and `max_connections` from the container's memory limit, not from a value you set directly. With no limit, Spilo sees the full host's memory and sets `shared_buffers` to roughly a quarter of it, which is a lot of memory to reserve when several other services share the same node. `2GiB` produces a sane 512MB.
## Step 3: Deploy Redis as a Swarm-native HA cluster
@@ -137,7 +137,7 @@ docker service create --name redis-3 --network netbird-data \
```
<Warning>
`--endpoint-mode dnsrr` here isn't optional. Swarm's default networking routes service-to-service traffic through a per-node proxy that rewrites the connection's source address, and Redis replication identifies a replica by exactly that address. The result: the primary reports replica addresses that belong to nothing real, so Sentinel detects the failed primary, reaches quorum, then aborts every failover with "no good slave to promote." `dnsrr` makes each service name resolve to the container's real address, bypassing the proxy. `--replica-announce-ip` reinforces the fix at the application layer (Redis 6.2 and later lets a replica assert its own identity by name); the two were applied together, with `dnsrr` as the confirmed root-cause fix.
`--endpoint-mode dnsrr` here isn't optional. Swarm's default networking routes service-to-service traffic through a per-node proxy that rewrites the connection's source address, and Redis replication identifies a replica by exactly that address. The result: the primary reports replica addresses that belong to nothing real, so Sentinel detects the failed primary, reaches quorum, then aborts every failover with "no good slave to promote." `dnsrr` makes each service name resolve to the container's real address, bypassing the proxy. `--replica-announce-ip` reinforces the fix at the application layer (Redis 6.2 and later lets a replica assert its own identity by name); this guide applies the two together, with `dnsrr` as the confirmed root-cause fix.
</Warning>
Each Sentinel needs a config file, since Sentinel rewrites its own configuration at runtime to persist which instance is currently primary — a plain read-only file mount fails immediately with "Sentinel config file ... is not writable." Give it a writable path instead, seeded from a template on first start:
@@ -168,7 +168,7 @@ docker service create --name sentinel-1 --network netbird-data \
redis:8 sh -c 'test -f /data/sentinel.conf || cp /etc/redis/sentinel.conf.tmpl /data/sentinel.conf; exec redis-sentinel /data/sentinel.conf'
```
Repeat for `sentinel-2`/`sentinel-3`, changing the `--constraint` and volume source. `sentinel monitor mymaster redis-1 6379 2` in the shared config only names the *starting* primary — Sentinel tracks whoever actually holds that role from there on, and all three Sentinels use the identical file.
Repeat for `sentinel-2`/`sentinel-3`, changing the `--constraint` and volume source. `sentinel monitor mymaster redis-1 6379 2` in the shared config only names the *starting* primary — Sentinel tracks whoever actually holds that role from there on, and all three Sentinels use the same file.
### How Management points at Redis
@@ -223,9 +223,9 @@ listen redis
Patroni exposes a REST endpoint on every instance (`/primary` answers `200` only on the current leader), built for exactly this kind of health check. Redis has no equivalent endpoint, so the Redis check logs into each instance directly and asks it whether it currently considers itself the master.
`on-marked-down shutdown-sessions` matters for correctness, not just cleanliness: without it, a connection HAProxy already opened to the old primary/master stays open across a failover, and whatever's using it keeps talking to a node that's no longer current.
`on-marked-down shutdown-sessions` matters for correctness, not just cleanliness: without it, a connection HAProxy already opened to the old primary stays open across a failover, and whatever's using it keeps talking to a node that's no longer current.
The `resolvers` block and `init-addr none` appear on **both** backends, for different reasons. The Redis servers need it because dnsrr names (Step 3) resolve to container addresses that change on restart. The PostgreSQL servers keep Swarm's default networking (Patroni's health check doesn't care about source addresses) and stable addresses — but without runtime re-resolution, HAProxy resolves them exactly once at startup, and an instance starting on a rebooting node can come up with every PostgreSQL backend marked down and *never recover*, even after the database is long healthy. It's nasty to diagnose because it's per-HAProxy-instance: connections reach a healthy or a wedged instance at random, so Management replicas on one node fail against a database that other nodes use fine.
The `resolvers` block and `init-addr none` appear on **both** backends, for different reasons. The Redis servers need them because dnsrr names (Step 3) resolve to container addresses that change on restart. The PostgreSQL servers keep Swarm's default networking (Patroni's health check doesn't care about source addresses) and stable addresses — but without runtime re-resolution, HAProxy resolves them exactly once at startup, and an instance starting on a rebooting node can come up with every PostgreSQL backend marked down and *never recover*, even long after the database is healthy. It's nasty to diagnose because it's per-HAProxy-instance: connections reach a healthy or a wedged instance at random, so Management replicas on one node fail against a database that other nodes use fine.
Deploy HAProxy itself as a global service, one per node, so it isn't a new single point of failure:
@@ -239,7 +239,7 @@ docker service create --name haproxy --network netbird-data --mode global \
## Step 5: Create the NetBird database and role
With PostgreSQL, Redis, and HAProxy all up, create the database Management will use. [Patroni's own `bootstrap.users` config option doesn't create it for you on current Patroni versions](https://patroni.readthedocs.io/en/latest/releases.html) — it logs an error and skips silently rather than failing loudly, so the omission is easy to miss. Create the role and database directly instead, connecting as the Postgres superuser through HAProxy so the connection lands on whichever instance is currently primary:
With PostgreSQL, Redis, and HAProxy all up, create the database Management will use. [Patroni's own `bootstrap.users` config option doesn't create it for you on current Patroni versions](https://patroni.readthedocs.io/en/latest/releases.html): it logs an error and moves on rather than failing loudly, so the omission is easy to miss. Create the role and database directly instead, connecting as the `postgres` superuser through HAProxy so the connection lands on whichever instance is currently primary:
```sh
docker exec -it $(docker ps -q -f name=pg- | head -1) \
@@ -257,7 +257,7 @@ Use this same `<netbird-role-password>` in the DSN below — it's a different cr
### How Management points at PostgreSQL
Unlike the main guide's `config.yaml`-based `server.store.dsn`, the split `management-cloud` image used here reads its store configuration from environment variables — its `--config` file loader is JSON-only and rejects the main guide's YAML schema outright, so the settings below can't go in a config file at all for this image:
Unlike the main guide, where the store DSN lives in `config.yaml` as `server.store.dsn`, the split `management-cloud` image used here reads its store configuration from environment variables — its `--config` file loader is JSON-only and rejects the main guide's YAML schema outright, so the settings below can't go in a config file at all for this image:
```
NB_STORE_ENGINE_POSTGRES_DSN=host=haproxy port=5432 user=netbird password=<netbird-role-password> dbname=netbird sslmode=require
@@ -269,7 +269,7 @@ Spilo's default `pg_hba.conf` requires an encrypted connection for non-local tra
## Step 6: Deploy the NATS cluster
Follow the [main guide's NATS requirements](/selfhosted/maintenance/scaling/high-availability#step-3-set-up-a-nats-ha-cluster) — 3 instances, JetStream enabled, on separate failure domains. On Swarm, deploy it the same way as etcd: three separate single-replica services, one per node, each with a config file listing all three peers' cluster addresses. The main guide's instructions for verifying the cluster apply unchanged; `NB_NATS_ENDPOINTS` (Management) and `NATS_ENDPOINTS` (Signal) both take the same comma-separated `nats://nats-N:4222` list either way.
Follow the [main guide's NATS requirements](/selfhosted/maintenance/scaling/high-availability#step-3-set-up-a-nats-ha-cluster): 3 instances, JetStream enabled, on separate failure domains. On Swarm, deploy it the same way as etcd: three separate single-replica services, one per node, each with a config file listing all three peers' cluster addresses. The main guide's instructions for verifying the cluster apply unchanged; `NB_NATS_ENDPOINTS` (Management) and `NATS_ENDPOINTS` (Signal) both take the same comma-separated `nats://nats-N:4222` list either way.
## Step 7: Deploy Traefik
@@ -316,11 +316,11 @@ A bare `target` name mounts under `/run/secrets/<name>`, matching the paths in `
Traefik v3 removed `--providers.docker.swarmMode` — the Swarm-aware provider is now a separate provider entirely (`--providers.swarm.*`, as above), not an option flag on the Docker provider. Every older example you'll find online uses the removed v2 flag; it fails at startup with `Docker provider "swarmMode" option has been removed in v3, please use the Swarm Provider instead`.
</Warning>
Deploy as a global service (one per node) so ingress survives a node dying, published in host mode so each node's Traefik binds the port directly rather than going through Swarm's own routing mesh — the exact mesh this guide's introduction explains isn't health-aware enough to use here.
Deploy as a global service (one per node) so ingress survives a node dying, published in host mode so each node's Traefik binds the port directly rather than going through Swarm's own routing mesh — the same mesh the introduction called out for having no active health checking.
## Step 8: Deploy the Relay, Signal, Management, and Dashboard pools
Deploy these as ordinary replicated Swarm services, `--replicas-max-per-node 1` so a single node loss can't take out more than one instance of a pool. Management and Signal attach to both networks — Management needs `netbird-data` for PostgreSQL/Redis, Signal needs it for NATS; Relay and Dashboard only need `netbird-app`. Label every service so Traefik discovers it automatically.
Deploy these as ordinary replicated Swarm services with `--replicas-max-per-node 1` so a single node loss can't take out more than one instance of a pool. Management and Signal attach to both networks — Management needs `netbird-data` for PostgreSQL/Redis, Signal needs it for NATS; Relay and Dashboard only need `netbird-app`. Label every service so Traefik discovers it automatically.
### Management's config file
@@ -413,7 +413,7 @@ Management needs two routers because two client protocols share one port: gRPC (
Traefik's Swarm provider reads *service* labels: `docker service create --label`, or `deploy.labels:` in a stack file. Container labels and Compose's top-level `labels:` are silently invisible — the service comes up healthy and Traefik never sees it. `traefik.docker.network=netbird-app` is also required because Management sits on two networks; without it Traefik may route through the wrong one.
</Warning>
`NETBIRD_ENCRYPTION_KEY` is not the environment-variable form of `management.json`'s `DataStoreEncryptionKey` field above; they're two separate settings that happen to need the same value. `DataStoreEncryptionKey` encrypts NetBird's main object store; `NETBIRD_ENCRYPTION_KEY` gates a distinct subsystem NetBird calls the integrations store, and omitting it fails with a different, specific error: `could not initialize integrations store: encryption key must be 32 bytes, got 0`. Generate one key with `openssl rand -base64 32`, use the same output for both settings, store it in your secret manager, and never let the two drift apart or rotate independently: each one encrypts data already written under it, and a mismatched key can't decrypt data written under the other.
`NETBIRD_ENCRYPTION_KEY` is not the environment-variable form of `management.json`'s `DataStoreEncryptionKey` field above; they're two separate settings that happen to need the same value. `DataStoreEncryptionKey` encrypts NetBird's main object store; `NETBIRD_ENCRYPTION_KEY` gates a distinct subsystem NetBird calls the integrations store, and omitting it produces a different, specific error: `could not initialize integrations store: encryption key must be 32 bytes, got 0`. Generate one key with `openssl rand -base64 32`, use the same output for both settings, store it in your secret manager, and never let the two drift apart or rotate independently: each one encrypts data already written under it, and a mismatched key can't decrypt data written under the other.
Signal follows the same shape, on both networks, with `SINGLE_NODE_MODE=false` and its own `NATS_ENDPOINTS`:
@@ -435,7 +435,7 @@ docker service create --name signal --network netbird-app --network netbird-data
Signal only speaks gRPC, so it needs just the one router, but still needs `scheme=h2c` for the same reason Management's gRPC router does, and `traefik.docker.network=netbird-app` for the same dual-network reason.
Relay and Dashboard only need `netbird-app`, and their environment variables follow the [main guide's Relay and Dashboard configuration](/selfhosted/maintenance/scaling/high-availability#step-5-deploy-the-relay-pool) unchanged — Relay in particular uses the same public `netbirdio/relay` image either way, since there's no Enterprise-specific Relay image to begin with. Their Traefik labels still need spelling out, since they follow the same pattern as Management and Signal above:
Relay and Dashboard only need `netbird-app`, and their environment variables follow the [main guide's Relay and Dashboard configuration](/selfhosted/maintenance/scaling/high-availability#step-5-deploy-the-relay-pool) unchanged — Relay in particular uses the same public `netbirdio/relay` image either way, since there's no Enterprise-specific Relay image to begin with. Their Traefik labels still need spelling out; they follow the same pattern as Management's and Signal's above:
```sh
docker service create --name relay --network netbird-app \
@@ -459,7 +459,7 @@ docker service create --name relay --network netbird-app \
Don't gate Traefik's routing on Relay's own health status — that's why Relay's command above has no `--health-cmd`. The `netbirdio/relay` image's health endpoint tests its own reachability *through* your public ingress, so an unhealthy Relay excluded from Traefik's rotation can never pass the very test that would mark it healthy again: it crash-loops forever with nothing actually wrong. Let Swarm's restart policy handle genuinely crashed Relay containers.
</Warning>
Dashboard shares Management's own FQDN (`app.example.com`) rather than getting one of its own — Management serves the API and OAuth endpoints there, and Dashboard serves everything else on that same host:
Dashboard shares Management's FQDN (`app.example.com`) rather than getting one of its own — Management serves the API and OAuth endpoints there, and Dashboard serves everything else on that same host:
```sh
docker service create --name dashboard --network netbird-app \
@@ -498,11 +498,11 @@ The `--health-*` flags on Management and Dashboard set container-level health ch
Run through the [main guide's verification steps](/selfhosted/maintenance/scaling/high-availability#step-8-verify-ha-end-to-end), then add the Swarm-specific ones. The timings quoted below are what a full run of these tests measured in the three-node environment this guide is based on; treat them as expectations to compare against, not guarantees.
- `docker node update --availability drain <node>` and confirm every service reschedules its lost replicas onto the remaining nodes (a service pinned to that exact node, like a PostgreSQL or NATS member, will not reschedule — that's expected; it comes back when the node returns).
- Drain a node with `docker node update --availability drain <node>` and confirm every service reschedules its lost replicas onto the remaining nodes (a service pinned to that exact node, like a PostgreSQL or NATS member, will not reschedule — that's expected; it comes back when the node returns).
- Force a real primary failure with `docker service scale pg-1=0`, not `docker rm -f`: a plain container kill can let Swarm restart the same instance before Patroni finishes electing a new leader, which looks like success without testing cross-node failover. Confirm via `docker exec <a-surviving-pg-container> patronictl list` that a different instance becomes leader (measured: under 10 seconds, with API reads and writes succeeding aside from a request or two at the promotion moment), then scale back up and confirm the old leader rejoins as a replica on its own.
- Do the same for Redis (`docker service scale redis-1=0`), confirming via `docker exec <a-surviving-sentinel-container> redis-cli -p 26379 sentinel master mymaster` that Sentinel elects a new master (measured: about 12 seconds).
- Kill an entire node — the one holding the Patroni leader, the worst case. A hard failure waits out Patroni's failure-detection TTL rather than reacting to a clean shutdown, so expect a longer but bounded window (measured: about 40 seconds of partial API errors, then full reads and writes with the node still gone). When the node returns, confirm *every* service returns to full replicas and the rejoined PostgreSQL member is streaming in `patronictl list`: a replica count that never recovers, or a Management replica crash-looping against a healthy database, is how the Step 8 restart-policy trap and the Step 4 stale-DNS wedge surface.
- With a peer-to-peer connection established between two test peers, stop a majority of nodes at once (2 of 3 here). Expect the control plane to go down: Swarm loses quorum and Patroni deliberately demotes the PostgreSQL leader rather than risk a split brain, so the API errors until a second node returns. That's correct behavior — three nodes tolerate one loss with full function, not two. The established peer-to-peer connection should keep passing traffic regardless; it no longer depends on any of these services (measured: zero dropped pings at a 200ms cadence across the entire test set, this scenario included).
- With a peer-to-peer connection established between two test peers, stop a majority of nodes at once (2 of 3 here). Expect the control plane to go down: Swarm loses quorum and Patroni deliberately demotes the PostgreSQL leader rather than risk a split brain, so the API errors until a second node returns. That's correct behavior — three nodes tolerate the loss of one node with full function, not two. The established peer-to-peer connection should keep passing traffic regardless; it no longer depends on any of these services (measured: zero dropped pings at a 200ms cadence across the entire test set, this scenario included).
## In one breath