diff --git a/src/pages/help/troubleshooting.mdx b/src/pages/help/troubleshooting.mdx index 0f4fa5fc..62a9a447 100644 --- a/src/pages/help/troubleshooting.mdx +++ b/src/pages/help/troubleshooting.mdx @@ -37,7 +37,7 @@ next to the feature they cover, so nothing here is a copy. chips: [ { label: "Management service", href: "/selfhosted/troubleshooting/connectivity#management-service-unreachable" }, { label: "Signal service", href: "/about-netbird/how-netbird-works#signal-service" }, - { label: "Relay / TURN", href: "/selfhosted/troubleshooting/connectivity#debugging-turn-connections" }, + { label: "Relay", href: "/selfhosted/troubleshooting/connectivity#debugging-relay-connections" }, { label: "Dashboard", href: "/selfhosted/troubleshooting/dashboard" }, ], }, diff --git a/src/pages/selfhosted/troubleshooting.mdx b/src/pages/selfhosted/troubleshooting.mdx index 102b4cdd..0cd1bc5f 100644 --- a/src/pages/selfhosted/troubleshooting.mdx +++ b/src/pages/selfhosted/troubleshooting.mdx @@ -43,9 +43,9 @@ This page helps with issues when self-hosting NetBird. Collect the diagnostics b title="Collect diagnostics first" description="Almost every self-hosted issue is faster to resolve with service status and logs in hand. Grab these before you go deeper." steps={[ - { label: "1. Check services", title: "Service status", command: "docker compose ps", hint: "Are management, signal, relay, and the dashboard all Up?" }, - { label: "2. Read the logs", title: "Service logs", command: "docker compose logs -f management", hint: "Most config, auth, and migration errors surface here." }, - { label: "3. Test reachability", title: "Proxy to API", command: "curl -v https://YOUR_DOMAIN/api/health", hint: "Confirms the reverse proxy can reach the Management API." }, + { label: "1. Check services", title: "Service status", command: "docker compose ps", hint: "Are netbird-server and dashboard Up, plus traefik if you chose the built-in Traefik? Installs from before v0.65.0 run separate management, signal, and relay services instead." }, + { label: "2. Read the logs", title: "Service logs", command: "docker compose logs -f netbird-server", hint: "Most config, auth, and migration errors surface here. On installs from before v0.65.0, read the management service logs." }, + { label: "3. Test reachability", title: "Proxy to API", command: "curl -s https://YOUR_DOMAIN/api/instance", hint: "A JSON reply with setup_required confirms the reverse proxy can reach the Management API." }, ]} /> @@ -82,7 +82,7 @@ This page helps with issues when self-hosting NetBird. Collect the diagnostics b title: "Connectivity", href: "/selfhosted/troubleshooting/connectivity", icon: "firewall", - description: "Testing the TURN server, peer-to-peer failures, and an unreachable Management service.", + description: "Checking the relay, peer-to-peer failures, and an unreachable Management service.", }, { title: "Database", diff --git a/src/pages/selfhosted/troubleshooting/certificates.mdx b/src/pages/selfhosted/troubleshooting/certificates.mdx index 6d7bc70e..e061c1ab 100644 --- a/src/pages/selfhosted/troubleshooting/certificates.mdx +++ b/src/pages/selfhosted/troubleshooting/certificates.mdx @@ -25,7 +25,7 @@ Deployments created before NetBird switched to Traefik may still run Caddy, whic **Likely causes and fixes** (most common first): -1. **Let's Encrypt is still enabled, so two components fight over TLS.** Set `NETBIRD_DISABLE_LETSENCRYPT=true` so NetBird stops managing certificates and leaves termination to your proxy. +1. **NetBird is also terminating TLS, so two components fight over it.** Installs from `getting-started.sh` with options 1 to 5 don't do this: netbird-server serves plain HTTP behind your proxy. If you wrote `config.yaml` yourself, leave `server.tls.certFile` empty and `server.tls.letsencrypt.enabled` set to `false`. On older deployments installed with `configure.sh`, set `NETBIRD_DISABLE_LETSENCRYPT=true` in `setup.env`. 2. **Forwarded headers are missing.** Set `X-Forwarded-For` and `X-Forwarded-Proto` on the proxy so NetBird sees the original scheme and client. 3. **gRPC fails without HTTP/2.** The Management gRPC endpoints need HTTP/2; enable it on the proxy. diff --git a/src/pages/selfhosted/troubleshooting/connectivity.mdx b/src/pages/selfhosted/troubleshooting/connectivity.mdx index d471b3ae..00c56d0f 100644 --- a/src/pages/selfhosted/troubleshooting/connectivity.mdx +++ b/src/pages/selfhosted/troubleshooting/connectivity.mdx @@ -1,15 +1,29 @@ export const description = - "Self-hosted NetBird connectivity troubleshooting: testing the TURN server, peer-to-peer connection failures, and an unreachable Management service." + "Self-hosted NetBird connectivity troubleshooting: checking the relay, peer-to-peer connection failures, and an unreachable Management service." # Connectivity issues Peer connectivity and relay problems on a self-hosted deployment. For other areas, start from [Troubleshooting](/selfhosted/troubleshooting). +## Debugging relay connections + +When peers can't establish a direct connection, they fall back to the NetBird relay. In the combined setup, netbird-server runs the relay itself and serves it at `/relay` on TCP/443 through your reverse proxy. It also runs STUN on UDP/3478. There is no TURN server. + +To check the relay, run `netbird status -d` on a client and read the `Relays:` list. A working setup lists your STUN server and relay as available: + +```text +Relays: + [stun:netbird.example.com:3478] is Available + [rels://netbird.example.com:443] is Available via ws +``` + +If it reads `Unavailable, reason: ...`, the reason names the failure. The most common cause is a reverse proxy that doesn't route `/relay` to netbird-server or doesn't allow the WebSocket upgrade. Compare your proxy with the routes in [External Reverse Proxy Setup](/selfhosted/external-reverse-proxy). + ## Debugging TURN connections -When peers can't establish a direct connection, they fall back to the TURN (relay) server. If the connection still fails with relay, confirm your TURN configuration is correct and reachable. +This section applies to older multi-container deployments that run Coturn, configured in the `TURNConfig` section of `management.json`. The combined setup has no TURN server; use [Debugging relay connections](#debugging-relay-connections) instead. -To test it, open the [Trickle ICE test tool](https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice). In the **ICE servers** box, remove the default server and add your TURN server. Replace netbird.DOMAIN.com and PASSWORD with your STUN/TURN details from your configuration (`config.yaml` for the combined setup, or the `TURNConfig` section in `management.json` for older multi-container setups), then click Add server. +To test it, open the [Trickle ICE test tool](https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice). In the **ICE servers** box, remove the default server and add your TURN server. Replace netbird.DOMAIN.com and PASSWORD with your STUN/TURN details from the `TURNConfig` section in `management.json`, then click Add server.

The Trickle ICE test tool with a NetBird STUN/TURN server added to the ICE servers list @@ -33,7 +47,7 @@ The candidate types are `host` (local address), `srflx` (STUN reflexive address) **Likely causes and fixes** (most common first): -1. **STUN/TURN is unreachable.** Confirm UDP/3478 (STUN/TURN) is open, then verify the relay path with the [TURN test above](#debugging-turn-connections). +1. **STUN or the relay is unreachable.** Confirm UDP/3478 (STUN) and TCP/443 are open, then check the relay with [Debugging relay connections](#debugging-relay-connections). 2. **A host firewall is dropping traffic** on one or both peers. Check the local firewall rules on each peer. 3. **The peer itself is unhealthy.** Run `netbird status -d` on both peers and check the connection type and last handshake. @@ -47,6 +61,6 @@ The candidate types are `host` (local address), `srflx` (STUN reflexive address) 1. **TCP/443 is blocked** between the client and the server. Confirm the port is open end to end. 2. **DNS doesn't resolve your domain** to the server's public IP. Check resolution from the client. -3. **Management is down or erroring.** Review `docker compose logs management`, and test the endpoint directly: `curl -v https://YOUR_DOMAIN/api/health`. +3. **Management is down or erroring.** Review `docker compose logs netbird-server` (`management` on installs from before v0.65.0), and test the endpoint through your proxy: `curl -s https://YOUR_DOMAIN/api/instance`. -**Confirm**: `curl https://YOUR_DOMAIN/api/health` returns a healthy response, and `netbird status` shows `Management: Connected`. +**Confirm**: `curl -s https://YOUR_DOMAIN/api/instance` returns JSON with a `setup_required` field, and `netbird status` shows `Management: Connected`. diff --git a/src/pages/selfhosted/troubleshooting/dashboard.mdx b/src/pages/selfhosted/troubleshooting/dashboard.mdx index 742cc144..14c01834 100644 --- a/src/pages/selfhosted/troubleshooting/dashboard.mdx +++ b/src/pages/selfhosted/troubleshooting/dashboard.mdx @@ -11,8 +11,8 @@ Problems with the self-hosted dashboard. For other areas, start from [Troublesho **Likely causes and fixes** (most common first): -1. **The dashboard can't reach the Management API.** Open the browser console (F12 → Console) and look for failed requests. Confirm the dashboard's configured API URL is correct and reachable from the browser. -2. **CORS (Cross-Origin Resource Sharing) is blocking the API behind a custom reverse proxy.** Serve the API on the same origin as the dashboard, or set the correct CORS headers on your proxy. +1. **The dashboard can't reach the Management API.** Open the browser console (F12 → Console) and look for failed requests. Confirm `NETBIRD_MGMT_API_ENDPOINT` in `dashboard.env` is your public NetBird URL and is reachable from the browser. +2. **CORS (Cross-Origin Resource Sharing) errors behind a custom reverse proxy.** netbird-server already sends CORS headers, so don't add them on the proxy: duplicate `Access-Control-Allow-Origin` headers make browsers reject the response. CORS errors can also mean the proxy doesn't route `/api` and `/oauth2` to netbird-server, so the browser gets a response from something else. 3. **A stale cached bundle.** Hard-reload the page or clear the browser cache. **Confirm**: Reload the dashboard. The login or peers view renders, and the console shows no failed API requests. @@ -25,6 +25,6 @@ Problems with the self-hosted dashboard. For other areas, start from [Troublesho 1. **An expired or invalid token.** Re-authenticate. For direct API access, use a valid Personal Access Token (PAT). 2. **The user lacks permission** for the action. Check the user's role in the dashboard. -3. **Management can't validate the token.** Review `docker compose logs management` for token-validation errors. +3. **Management can't validate the token.** Review `docker compose logs netbird-server` (`management` on installs from before v0.65.0) for token-validation errors. **Confirm**: Re-run the action. It succeeds, and the Management logs show no auth errors. diff --git a/src/pages/selfhosted/troubleshooting/database.mdx b/src/pages/selfhosted/troubleshooting/database.mdx index bc223b22..04834c56 100644 --- a/src/pages/selfhosted/troubleshooting/database.mdx +++ b/src/pages/selfhosted/troubleshooting/database.mdx @@ -7,15 +7,15 @@ Database and migration problems on a self-hosted deployment. For other areas, st ## Management service won't start after upgrade -**Symptom**: After an upgrade, the Management container fails to start or crash-loops. +**Symptom**: After an upgrade, the netbird-server container (`management` on installs from before v0.65.0) fails to start or crash-loops. **Likely causes and fixes** (most common first): -1. **A failed schema migration.** Check `docker compose logs management` for migration errors. This is the usual cause and the logs name the failing step. +1. **A failed schema migration.** Check `docker compose logs netbird-server` (`docker compose logs management` on installs from before v0.65.0) for migration errors. This is the usual cause and the logs name the failing step. 2. **A skipped intermediate version.** Major version jumps may require stepping through intermediate releases. Follow the [upgrade path](/selfhosted/maintenance/upgrade) for your version. 3. **A corrupt or partially-migrated database.** If the migration can't be completed, restore from a pre-upgrade backup and retry the upgrade. -**Confirm**: `docker compose ps` shows `management` as `Up`, and its logs end with the service listening rather than a migration error. +**Confirm**: `docker compose ps` shows `netbird-server` (or `management`) as `Up`, and its logs end with the service listening rather than a migration error. ## Data corruption after power loss @@ -23,8 +23,15 @@ Database and migration problems on a self-hosted deployment. For other areas, st **Likely causes and fixes** (most common first): -1. **A stale lock file.** Check for and remove a leftover database lock if the process that held it is gone. -2. **An interrupted write.** Review all service logs to find which store is failing. For SQLite, run an integrity check on the database file. -3. **An unrecoverable file.** If the store can't be repaired, restore from the most recent backup. +1. **An interrupted write.** Review all service logs to find which store is failing. With the default SQLite store, the files are `store.db`, `events.db`, and `idp.db` in the `netbird_data` volume (`/var/lib/netbird` inside the container). The netbird-server image doesn't include `sqlite3`, so stop the stack with `docker compose down` and check the files from a throwaway container. Find the volume name with `docker volume ls` (it ends in `netbird_data`), then run: + + ```bash + docker run --rm -v VOLUME_NAME:/data:ro alpine sh -c \ + "apk add -q sqlite && for f in store.db events.db idp.db; do echo \$f; if [ -f /data/\$f ]; then cp /data/\$f* /tmp/ && sqlite3 /tmp/\$f 'PRAGMA integrity_check;'; else echo MISSING; fi; done" + ``` + + A healthy file prints `ok`. The check runs on copies inside the throwaway container, so the volume isn't changed. If the interrupted write left a `-journal` file next to a store, the check applies it to the copy, the same way netbird-server does when it starts. `MISSING` means that store doesn't exist in the volume. Start the stack again with `docker compose up -d` when you're done. + +2. **An unrecoverable file.** If the store can't be repaired, restore from the most recent backup. **Confirm**: All services report `Up` in `docker compose ps`, and peers reconnect with `netbird status` showing `Management: Connected`. diff --git a/src/pages/selfhosted/troubleshooting/identity-provider.mdx b/src/pages/selfhosted/troubleshooting/identity-provider.mdx index efe266e7..99f39dc3 100644 --- a/src/pages/selfhosted/troubleshooting/identity-provider.mdx +++ b/src/pages/selfhosted/troubleshooting/identity-provider.mdx @@ -13,7 +13,7 @@ Problems with the embedded identity provider (IdP) and SSO (single sign-on) on a 1. **A user already exists.** `/setup` is only available when no users exist. If you've already created one, use the regular login page. 2. **The embedded IdP is disabled.** Confirm it is enabled in your configuration. -3. **Management isn't running.** Check `docker compose logs management`. +3. **Management isn't running.** Check `docker compose logs netbird-server` (`management` on installs from before v0.65.0). **Confirm**: `/setup` loads (first run), or the regular login page loads (setup already done). @@ -29,9 +29,22 @@ Problems with the embedded identity provider (IdP) and SSO (single sign-on) on a **Symptom**: You created a user but the password doesn't work. -**Cause**: Passwords are shown only once at creation. If it wasn't saved, it can't be recovered. +**Cause**: Passwords are shown only once at creation. If it wasn't saved, it can't be recovered, but you can set a new one. -**Fix**: Recreate the user (delete and re-add, or create a new one with a new password via the API). For the owner account with no other admins, you may need to reset the database. +**Fix**: Set a new password with the admin CLI (NetBird v0.75.0 and later). This works for the owner account too: + +```bash +read -rsp 'New password: ' NB_PASSWORD; echo +printf '%s\n' "$NB_PASSWORD" | docker exec -i netbird-server \ + /go/bin/netbird-server --config /etc/netbird/config.yaml admin user change-password \ + --email user@example.com \ + --password-file - +unset NB_PASSWORD +``` + +The password is read from a silent prompt, so it doesn't end up in your shell history. + +See [Change a local user's password](/selfhosted/maintenance/admin-cli#change-a-local-users-password) for the password rules and other options. **Confirm**: You can log in with the new credentials. @@ -43,7 +56,7 @@ Problems with the embedded identity provider (IdP) and SSO (single sign-on) on a 1. **The connector wasn't saved.** Check **Settings → Identity Providers**. 2. **The redirect URI is wrong.** Ensure the IdP application has the exact redirect URI shown by NetBird. -3. **A configuration error.** Review `docker compose logs management` for connector errors. +3. **A configuration error.** Review `docker compose logs netbird-server` for connector errors. **Confirm**: The SSO button appears on the login page and starts the IdP flow. @@ -61,9 +74,9 @@ Problems with the embedded identity provider (IdP) and SSO (single sign-on) on a **Symptom**: The Identity Providers tab is missing from Settings. -**Cause**: The tab only appears when the embedded IdP is enabled. +**Cause**: The tab only appears when the embedded IdP is enabled and your role is allowed to read identity providers. -**Fix**: Check your deployment: quickstart enables it by default, the combined setup always has it on, and older multi-container deployments need `EmbeddedIdP.Enabled` set to `true` in `management.json`. +**Fix**: Check your deployment: quickstart enables it by default, the combined setup always has it on, and older multi-container deployments need `EmbeddedIdP.Enabled` set to `true` in `management.json`. If the embedded IdP is on and the tab is still missing, sign in with a role that can manage identity providers, or ask an administrator to check your role's permissions. **Confirm**: The Identity Providers tab appears under Settings. diff --git a/src/pages/selfhosted/troubleshooting/installation.mdx b/src/pages/selfhosted/troubleshooting/installation.mdx index 96176560..1a6259e9 100644 --- a/src/pages/selfhosted/troubleshooting/installation.mdx +++ b/src/pages/selfhosted/troubleshooting/installation.mdx @@ -9,7 +9,14 @@ Problems running the self-hosted `getting-started.sh` script. For other areas, s **Symptom**: The `getting-started.sh` script stays on the wait line for several minutes, even though netbird-server looks healthy in `docker ps`. -The wait check probes the OIDC endpoint through your reverse proxy, so a healthy server alone isn't enough. Something on the path from the public internet to netbird-server is broken. Check these likely causes in order, most common first: +What the wait checks depends on the reverse proxy option you picked: + +- **Options 0 and 1 (Traefik)**: the script probes the OIDC endpoint at your public hostname through your reverse proxy, so a healthy server alone isn't enough. Something on the path to netbird-server is broken. The probe runs from the installation host itself, so first run the `curl` in step 5 below from both that host and a machine outside your network. If only the host fails, the host can't reach its own public address (some NAT setups don't support this), and external clients may be fine. Otherwise, work through the causes below. +- **Options 2 to 5 (Nginx, Nginx Proxy Manager, Caddy, other)**: the script probes netbird-server directly on `http://127.0.0.1:8081`, with no proxy in the path. A hang here means netbird-server itself isn't healthy, so start with `docker compose logs netbird-server`. + +After about two minutes of waiting, the script prints `Taking too long. Checking logs...` followed by the last 20 log lines from netbird-server, and from Traefik when the script can find its container. Read that output first; it often names the problem. + +Check these likely causes in order, most common first: **1. DNS doesn't point at the host** @@ -27,7 +34,7 @@ For options 0 (built-in Traefik) and 1 (existing Traefik), check that a real cer curl -vI https://netbird.example.com 2>&1 | grep -E "subject|issuer" ``` -A self-signed or default cert means ACME hasn't completed. Check Traefik logs for ACME errors. Common causes: port 80 not reachable from the internet, DNS not propagated yet, or a Let's Encrypt rate limit. +A self-signed or default cert means ACME hasn't completed. Check Traefik logs for ACME errors. Common causes: TCP/443 not reachable from the internet, DNS not propagated yet, or a Let's Encrypt rate limit. The built-in Traefik (option 0) validates with the TLS-ALPN-01 challenge on port 443, so port 80 doesn't affect issuance. With option 1, the challenge depends on your Traefik's certificate resolver. **3. Traefik can't read the Docker socket (option 1 only)** @@ -51,7 +58,7 @@ curl -k https://netbird.example.com/oauth2/.well-known/openid-configuration A valid response is JSON containing `"issuer"`. Anything else points to where to dig: 502 means Traefik can't reach netbird-server, 404 means the labels aren't matched, connection refused means you're not hitting Traefik at all. -**Confirm**: The probe succeeds and the script continues on its own. You can leave it waiting while you debug. To start over instead, stop it with Ctrl+C, run `docker compose down -v`, fix the issue, and re-run. +**Confirm**: The probe succeeds and the script continues on its own. You can leave it waiting while you debug. To start over instead, stop it with Ctrl+C, remove the generated files as shown in [Script fails on existing installation check](#script-fails-on-existing-installation-check), fix the issue, and re-run. `docker compose down -v` on its own leaves `config.yaml` behind, and the script refuses to run while that file exists. ## Script fails on existing installation check @@ -66,6 +73,7 @@ docker compose down --volumes rm -f docker-compose.yml dashboard.env config.yaml proxy.env \ traefik-dynamic.yaml nginx-netbird.conf caddyfile-netbird.txt \ npm-advanced-config.txt +rm -rf crowdsec/ ``` **Confirm**: Re-run the script; it proceeds past the existing-installation check and starts provisioning.