Files
docs-v2/self-host/clustering/deploy-a-cluster.mdx
T

556 lines
19 KiB
Plaintext

---
title: "Deploy a Cluster"
description: "Step-by-step walkthrough for deploying a two-node highly available Pangolin cluster"
---
<Note>
Clustering is only available in [Enterprise Edition](/self-host/enterprise-edition).
</Note>
This guide walks through deploying a minimal two-node Pangolin cluster: two Pangolin nodes behind a load balancer, sharing a PostgreSQL database and a Valkey (Redis) server. Read [Understanding Clustering](/self-host/clustering/understanding-clustering) for the architecture and [Requirements](/self-host/clustering/requirements) for the hosts, ports, and DNS records you need before starting.
<Card icon="github" arrow="true" cta="View reference configuration" href="https://github.com/fosrl/pangolin/tree/main/config/ha-reference">
The complete, working set of files used in this guide lives in the Pangolin repository at [`config/ha-reference`](https://github.com/fosrl/pangolin/tree/main/config/ha-reference). Clone it as a starting point instead of assembling files by hand.
</Card>
<Warning>
`server.secret` in `config.yml` must be **identical on every node in the cluster**. It's used to encrypt sensitive data, including the certificates stored in PostgreSQL - if nodes have different secrets, they won't be able to read each other's data.
</Warning>
Throughout this guide, replace the following placeholders with your own values:
| Placeholder | Description |
| --- | --- |
| `NODE1_EXTERNAL_IP` / `NODE2_EXTERNAL_IP` | Public static IP of each Pangolin node |
| `NODE1_INTERNAL_IP` / `NODE2_INTERNAL_IP` | Internal IP each node uses to address the other |
| `POSTGRES_INTERNAL_HOST` | Address of your shared PostgreSQL server |
| `POSTGRES_USERNAME` / `POSTGRES_PASSWORD` | Credentials for that PostgreSQL server |
| `REDIS_INTERNAL_HOST` | Address of your shared Redis-compatible server |
| `CONTACT_EMAIL` | Email address used for Let's Encrypt ACME registration |
| `SECRET` | Shared server secret - identical on every node |
| `LOAD_BALANCER_IP` | IP address of the load balancer in front of the cluster |
| `pangolin.example.com` | Your dashboard domain - DNS points at the load balancer, not at either node |
You need a domain for the Pangolin UI and API (`pangolin.example.com` in this guide), pointed at your load balancer. **The load balancer is responsible for TLS on this domain** - terminate HTTPS there and forward plain HTTP to the nodes' dashboard port. The nodes' built-in ACME client only issues certificates for resource domains under the delegated nameserver zone, not for the dashboard domain itself. See [Requirements](/self-host/clustering/requirements#dashboard-domain).
<Steps>
<Step title="Provision the shared database">
Stand up a PostgreSQL server and a Redis-compatible server that both nodes can reach. They don't need to run together, or even on a dedicated third host - use whatever you already run, including managed cloud offerings. The only hard requirement is that the Redis-compatible server supports **pub/sub**.
For a simple self-hosted starting point:
```yaml title="docker-compose.yml"
services:
postgres:
image: postgres:17
container_name: postgres
environment:
POSTGRES_DB: postgres # Default database name
POSTGRES_USER: postgres # Default user
POSTGRES_PASSWORD: password # Default password (change for production!)
volumes:
- postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"
restart: always
redis:
image: redis:latest
container_name: redis
ports:
- "6379:6379"
restart: always
volumes:
postgres_data:
```
<Warning>
Change the default PostgreSQL password before running this in production. See [Database Options](/self-host/advanced/database-options) for general PostgreSQL configuration.
</Warning>
</Step>
<Step title="Lay out each node's config directory">
On each of Node 1 and Node 2, create the following directory structure:
```
node/
├── docker-compose.yml
└── config/
├── config.yml
├── privateConfig.yml
├── certificates/ # empty, Pangolin/Traefik populate this
├── dynamic/
│ └── dynamic_config.yml
└── traefik/
└── traefik_config.yml
```
`config/certificates` and `config/dynamic` are shared volumes between the `pangolin` and `traefik` containers - Pangolin writes router configuration and certificates there for Traefik to read, since Traefik can only load certificates from files, not from the Pangolin API. This is `traefik.file_mode` in `config.yml`, covered below.
</Step>
<Step title="Write docker-compose.yml">
Both nodes run the same three containers: `pangolin`, `gerbil`, and `traefik`. Gerbil owns the host networking (WireGuard, relay, DNS, resource ports), and Traefik joins its network namespace so its ports appear alongside Gerbil's.
Each node's `--reachableAt` flag must point at **that node's own** internal address, and `--trusted-upstreams` lists the external IPs of every node in the cluster so Gerbil accepts proxied connections from them.
<CodeGroup>
```yaml Node 1
name: pangolin
services:
pangolin:
image: docker.io/fosrl/pangolin:ee-latest
container_name: pangolin
restart: unless-stopped
volumes:
- ./config:/app/config
- ./config/certificates:/var/certificates
- ./config/dynamic:/var/dynamic
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3001/api/v1/"]
interval: "10s"
timeout: "10s"
retries: 15
gerbil:
image: docker.io/fosrl/gerbil:latest
container_name: gerbil
restart: unless-stopped
depends_on:
pangolin:
condition: service_healthy
command:
- --reachableAt=http://<NODE1_INTERNAL_IP>:3004
- --generateAndSaveKeyTo=/var/config/key
- --remoteConfig=http://pangolin:3001/api/v1/
- --trusted-upstreams=<NODE1_EXTERNAL_IP>,<NODE2_EXTERNAL_IP> # All trusted nodes in the cluster
volumes:
- ./config/:/var/config
cap_add:
- NET_ADMIN
- SYS_MODULE
ports:
- 51820:51820/udp # wireguard
- 21820:21820/udp # relay
- 53:53/udp # DNS
- 443:8443 # resources
- 80:80 # web
- 3004:3004 # gerbil api
- 3000:3000 # Pangolin UI
traefik:
image: docker.io/traefik:v3.7.11
container_name: traefik
restart: unless-stopped
network_mode: service:gerbil # Ports appear on the gerbil service
depends_on:
pangolin:
condition: service_healthy
command:
- --configFile=/etc/traefik/traefik_config.yml
volumes:
- ./config/traefik:/etc/traefik:ro
- ./config/traefik/logs:/var/log/traefik
- ./config/certificates:/var/certificates:ro
- ./config/dynamic:/var/dynamic:ro
networks:
default:
driver: bridge
name: pangolin
```
```yaml Node 2
name: pangolin
services:
pangolin:
image: docker.io/fosrl/pangolin:ee-latest
container_name: pangolin
restart: unless-stopped
volumes:
- ./config:/app/config
- ./config/certificates:/var/certificates
- ./config/dynamic:/var/dynamic
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3001/api/v1/"]
interval: "10s"
timeout: "10s"
retries: 15
gerbil:
image: docker.io/fosrl/gerbil:latest
container_name: gerbil
restart: unless-stopped
depends_on:
pangolin:
condition: service_healthy
command:
- --reachableAt=http://<NODE2_INTERNAL_IP>:3004
- --generateAndSaveKeyTo=/var/config/key
- --remoteConfig=http://pangolin:3001/api/v1/
- --trusted-upstreams=<NODE1_EXTERNAL_IP>,<NODE2_EXTERNAL_IP> # All trusted nodes in the cluster
volumes:
- ./config/:/var/config
cap_add:
- NET_ADMIN
- SYS_MODULE
ports:
- 51820:51820/udp # wireguard
- 21820:21820/udp # relay
- 53:53/udp # DNS
- 443:8443 # resources
- 80:80 # web
- 3004:3004 # gerbil api
- 3000:3000 # Pangolin UI
traefik:
image: docker.io/traefik:v3.7.11
container_name: traefik
restart: unless-stopped
network_mode: service:gerbil # Ports appear on the gerbil service
depends_on:
pangolin:
condition: service_healthy
command:
- --configFile=/etc/traefik/traefik_config.yml
volumes:
- ./config/traefik:/etc/traefik:ro
- ./config/traefik/logs:/var/log/traefik
- ./config/certificates:/var/certificates:ro
- ./config/dynamic:/var/dynamic:ro
networks:
default:
driver: bridge
name: pangolin
```
</CodeGroup>
<Tip>
If Pangolin can't reach the local Gerbil at the address in `--reachableAt` (a loopback issue), override it in `privateConfig.yml` - see [Troubleshooting](#troubleshooting) below.
</Tip>
</Step>
<Step title="Write config.yml">
`config.yml` holds the settings that legitimately differ per node - `gerbil.base_endpoint` and `gerbil.exit_node_name` - alongside the shared PostgreSQL connection and site-type restrictions. In clustered deployments, only Pangolin Sites are supported, so local and basic WireGuard sites are disabled.
Set `app.dashboard_url` and `server.cors.origins` to your dashboard domain (the one pointed at your load balancer, not at either node) - both must match on every node.
<CodeGroup>
```yaml Node 1
gerbil:
start_port: 51820
base_endpoint: "<NODE1_EXTERNAL_IP>"
exit_node_name: "node1"
app:
dashboard_url: "https://pangolin.example.com"
log_level: "info"
postgres:
connection_string: postgresql://<POSTGRES_USERNAME>:<POSTGRES_PASSWORD>@<POSTGRES_INTERNAL_HOST>:5432/postgres
traefik:
site_types: ["newt"] # Wireguard and local sites are not supported in clustering
file_mode: true # Pangolin will generate and save yaml files in a shared volume
server:
secret: "<SECRET>" # Must be identical on every node
cors:
origins: ["https://pangolin.example.com"]
methods: ["GET", "POST", "PUT", "DELETE", "PATCH"]
allowed_headers: ["X-CSRF-Token", "Content-Type"]
credentials: false
maxmind_db_path: "./config/GeoLite2-Country.mmdb" # Download and place into the config dir
maxmind_asn_path: "./config/GeoLite2-ASN.mmdb"
flags:
require_email_verification: false
disable_signup_without_invite: true
disable_user_create_org: false
allow_raw_resources: false
enable_acme_cert_sync: false
disable_local_sites: true
disable_basic_wireguard_sites: true
disable_config_managed_domains: true
```
```yaml Node 2
gerbil:
start_port: 51820
base_endpoint: "<NODE2_EXTERNAL_IP>"
exit_node_name: "node2"
app:
dashboard_url: "https://pangolin.example.com"
log_level: "info"
postgres:
connection_string: postgresql://<POSTGRES_USERNAME>:<POSTGRES_PASSWORD>@<POSTGRES_INTERNAL_HOST>:5432/postgres
traefik:
site_types: ["newt"] # Wireguard and local sites are not supported in clustering
file_mode: true # Pangolin will generate and save yaml files in a shared volume
server:
secret: "<SECRET>" # Must be identical on every node
cors:
origins: ["https://pangolin.example.com"]
methods: ["GET", "POST", "PUT", "DELETE", "PATCH"]
allowed_headers: ["X-CSRF-Token", "Content-Type"]
credentials: false
maxmind_db_path: "./config/GeoLite2-Country.mmdb" # Download and place into the config dir
maxmind_asn_path: "./config/GeoLite2-ASN.mmdb"
flags:
require_email_verification: false
disable_signup_without_invite: true
disable_user_create_org: false
allow_raw_resources: false
enable_acme_cert_sync: false
disable_local_sites: true
disable_basic_wireguard_sites: true
disable_config_managed_domains: true
```
</CodeGroup>
See the [full configuration reference](/self-host/advanced/config-file) for every available option.
</Step>
<Step title="Write privateConfig.yml">
`privateConfig.yml` enables Redis-backed cluster sync and Pangolin's built-in DNS and ACME client. **Only one node** - Node 1 in this example - should have `acme.enable_acme_client` set to `true`. That node issues and renews certificates via DNS-01 challenges and stores them encrypted in PostgreSQL; every other node reads the same certificates from the database.
<Warning>
Do not enable `acme.enable_acme_client` on more than one node. Multiple nodes issuing certificates simultaneously will conflict with each other.
</Warning>
<CodeGroup>
```yaml Node 1 (ACME client enabled)
app:
region: "region1"
identity_provider_mode: "org"
redis:
host: "<REDIS_INTERNAL_HOST>"
port: 6379
flags:
enable_redis: true
use_pangolin_dns: true
acme:
cert_mode: "pangolin"
contact_email: "<CONTACT_EMAIL>"
enable_acme_client: true
dns:
enabled: true
nameserver_name: "ns.example.com"
cname_extension: "cname.example.com"
site_extension: "site.example.com" # Optional
```
```yaml Node 2 (ACME client disabled)
app:
region: "region1"
identity_provider_mode: "org"
redis:
host: "<REDIS_INTERNAL_HOST>"
port: 6379
flags:
enable_redis: true
use_pangolin_dns: true
acme:
cert_mode: "pangolin"
dns:
enabled: true
nameserver_name: "ns.example.com"
cname_extension: "cname.example.com"
site_extension: "site.example.com" # Optional
```
</CodeGroup>
See the [private configuration reference](/self-host/advanced/private-config-file) for every available option.
</Step>
<Step title="Write the Traefik configuration">
`traefik/traefik_config.yml` and `dynamic/dynamic_config.yml` are identical on every node - copy them as-is. Traefik loads router and certificate configuration from the shared `dynamic` volume (`file_mode`) instead of Pangolin's API, and exposes a `:53/udp` DNS entry point that forwards to Pangolin's built-in DNS server.
```yaml title="config/traefik/traefik_config.yml"
providers:
file:
directory: "/var/dynamic"
watch: true
experimental:
plugins:
badger:
moduleName: "github.com/fosrl/badger"
version: "v1.7.0"
log:
level: "INFO"
format: "common"
maxSize: 100
maxBackups: 3
maxAge: 3
compress: true
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
proxyProtocol: # We trust gerbil upstream
trustedIPs:
- 0.0.0.0/0
- ::1/128
transport:
respondingTimeouts:
readTimeout: "30m"
http:
encodedCharacters:
allowEncodedSlash: true
allowEncodedQuestionMark: true
dashboard:
address: ":3000"
dns:
address: ":53/udp"
serversTransport:
insecureSkipVerify: true
ping:
entryPoint: "web"
```
```yaml title="config/dynamic/dynamic_config.yml"
http:
middlewares:
badger:
plugin:
badger:
disableForwardAuth: true
routers:
# Next.js router (handles everything except API and WebSocket paths)
next-router:
rule: "!PathPrefix(`/api/v1`)"
service: next-service
entryPoints:
- dashboard
middlewares:
- badger
# API router (handles /api/v1 paths)
api-router:
rule: "PathPrefix(`/api/v1`)"
service: api-service
entryPoints:
- dashboard
middlewares:
- badger
# WebSocket router
ws-router:
rule: "PathPrefix(`/`)"
service: api-service
entryPoints:
- dashboard
middlewares:
- badger
services:
next-service:
loadBalancer:
servers:
- url: "http://pangolin:3002" # Next.js server
api-service:
loadBalancer:
servers:
- url: "http://pangolin:3000" # API/WebSocket server
tcp:
serversTransports:
pp-transport-v1:
proxyProtocol:
version: 1
pp-transport-v2:
proxyProtocol:
version: 2
udp:
routers:
dns-router:
entryPoints:
- dns
service: dns-service
services:
dns-service:
loadBalancer:
servers:
- address: "pangolin:53"
```
This is also where you place the MaxMind databases referenced in `config.yml` - download `GeoLite2-Country.mmdb` and `GeoLite2-ASN.mmdb` into each node's `config/` directory. See [Enable Geo-location](/self-host/advanced/enable-geolocation) and [Enable ASN Lookup](/self-host/advanced/enable-asn-lookup).
</Step>
<Step title="Point your load balancer at both nodes">
Configure your load balancer - a cloud load balancer or a self-hosted one such as Traefik - to:
- Route TCP `3000` to both nodes for your Pangolin domain. For example `pangolin.example.com` should resolve to the load balancer, which routes to either node's `:3000` port.
- Route UDP `53` to both nodes for DNS. For example `ns.example.com` should resolve to the load balancer, which routes to either node's `:53/udp` port.
- Health-check the `:80/ping` endpoint on each node, and stop routing to a node that fails it. `:3000/api/v1/` can also be monitored for the Pangolin UI and API
- **Terminate TLS for the dashboard domain** (`pangolin.example.com`) at the load balancer, then forward plain HTTP to `:3000` on the nodes. Obtain and renew that certificate through the load balancer itself (a cloud provider's managed certificate, its own ACME client, etc.) - the nodes' built-in ACME client only covers resource domains, not the dashboard domain
This is what makes the cluster appear as a single, consistent domain to users, and what drives failover when a node goes down.
</Step>
<Step title="Start the cluster">
Bring the database up first, then start **one** Pangolin node - it initializes the database and prints an init token to its logs.
```bash
# On the database host
docker compose up -d
# On Node 1
docker compose up -d
docker compose logs -f pangolin
```
Use the init token from the logs to visit the dashboard and create the first user. Once Node 1 is healthy and you've logged in, bring up Node 2:
```bash
# On Node 2
docker compose up -d
```
Repeat the Node 2 steps for any additional nodes, incrementing the `exit_node_name` and IP placeholders for each.
</Step>
<Step title="Verify the cluster">
- Confirm both nodes report healthy: `curl http://<NODE1_EXTERNAL_IP>/ping` and the same for Node 2
- Confirm DNS delegation resolves: `dig @ns.example.com ns.example.com`
- Confirm the dashboard is reachable at your `dashboard_url` through the load balancer
- Confirm you can create a site and it will report connected
- Create a resource, ensure the certificate generates, and is accessible
</Step>
</Steps>
## Troubleshooting
### Gerbil loopback addressing
If Pangolin can't reach the local Gerbil instance at the IP configured in `--reachableAt`, force it to address the Docker container directly instead:
```yaml title="privateConfig.yml"
gerbil:
local_exit_node_reachable_at: "http://gerbil:3004"
```
## Reference Configuration
<Card icon="github" arrow="true" cta="View reference configuration" href="https://github.com/fosrl/pangolin/tree/main/config/ha-reference">
A complete, working two-node reference configuration on GitHub - including both nodes' Docker Compose and config files - that you can clone and adapt: [github.com/fosrl/pangolin/tree/main/config/ha-reference](https://github.com/fosrl/pangolin/tree/main/config/ha-reference)
</Card>