mirror of
https://github.com/netbirdio/docs.git
synced 2026-09-28 17:59:05 +02:00
Restructure Troubleshooting into a hub with per-area pages (#814)
* Restructure Troubleshooting into a hub with per-area pages - Add a Troubleshooting hub (/help/troubleshooting) with icon/chip cards and a "Still stuck?" CTA - Split NetBird Client troubleshooting into an overview + per-OS pages (Linux, Windows, macOS, Android, iOS) - Split Self-hosted troubleshooting into an overview + per-area pages (installation, IdP, dashboard, certificates, connectivity, database) - Split "Report bugs and issues" into Community Support and NetBird Support pages - Add Troubleshooting resource connectivity and a NetBird Cloud pending-approval page - Add DNS troubleshooting Issue 8 (Windows NRPT rule blocked by a lingering GPO) - Cross-reference the new pages from networks, DNS, and reverse-proxy docs; update nav Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: client terminology, dead props, labels, cross-links - Use "client" instead of "agent" across the client troubleshooting pages (headings, prose, anchors) - Remove unused source: props from the Troubleshooting hub tiles - Relabel the "NetBird Cloud" grouping to "Cloud & identity" (SSO/provisioning also apply to self-hosted) - Add a Tiles title on the report-bug landing; add reverse-proxy -> resource-connectivity cross-link - Fix comma splices introduced by the em-dash cleanup in relayed-connections Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add client-side hash redirect for moved self-hosted anchors Old deep links like /selfhosted/troubleshooting#debugging-turn-connections now forward to the per-area page, since next.config redirects can't act on the URL fragment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply docs-skill review: conventions + reshape area pages - "open source" (no hyphen), expand NRPT on first use, descriptive alt text + captions on TURN images - Fix inherited "Netbird" casing in the client glossary - Reshape the six self-hosted area pages to Symptom -> likely causes (ordered) -> Fix -> Confirm, preserving anchored headings Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix two typos in client glossary (CodeRabbit) - "nunning" -> "running" in the glossary - possessive "it's" -> "its" in the routing-table sentence Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix two broken links in troubleshooting pages - database: point the "upgrade path" link at /selfhosted/maintenance/upgrade; selfhosted-quickstart has no #upgrade anchor so the old link landed at page top - client: add HashRedirect so old #net-bird-agent-status deep links forward to the renamed #net-bird-client-status section on the same page * docs: address review follow-ups (deep-link redirects + client casing) - self-hosted troubleshooting: extend the HashRedirect map with the per-issue (###-level) anchors from the old single page, so old deep links land on the exact sub-section of the new area page rather than just the page top - client glossary: lowercase "NetBird client" in the peer-a/peer-b entries (house convention) and fix "linux" -> "Linux" * docs: review polish — fix image class + first-use acronym glosses - connectivity: fix bad CSS class imagewrapper-nig -> imagewrapper on the TURN-test screenshot (the typo'd class matched no style and broke zoom) - gloss acronyms on first use: GPO (DNS Issue 8), IdP/SSO (identity-provider), ACME (certificates), CORS (dashboard) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com>
This commit is contained in:
Bruno Mercier Costa
co-authored by
Claude Opus 4.8
Jack Carter
Claude Opus 4.8
Jack Carter
parent
ffb72cfa34
commit
5729ad035e
@@ -0,0 +1,28 @@
|
||||
import { SupportBanner } from "@/components/SupportBanner"
|
||||
|
||||
export const description =
|
||||
"Community Support for NetBird: Slack and GitHub Discussions for the client, open source self-hosted, and general questions."
|
||||
|
||||
# Community Support
|
||||
|
||||
<SupportBanner
|
||||
tone="community"
|
||||
badge="Free · everyone"
|
||||
description="For everything outside the NetBird Cloud dashboard and managed infrastructure: the NetBird client, open source self-hosted deployments, and general questions. Open to all users."
|
||||
links={[
|
||||
{ label: "Community Slack", href: "#community-slack" },
|
||||
{ label: "GitHub Discussions", href: "#git-hub-discussions" },
|
||||
]}
|
||||
/>
|
||||
|
||||
Whichever channel you choose, include a [debug bundle](/help/troubleshooting-client#debug-bundle), your NetBird version (`netbird version`), and clear steps to reproduce.
|
||||
|
||||
## Community Slack
|
||||
|
||||
Best for quick questions or general configuration help. Join the [NetBird community Slack](/slack-url) to talk with other users and the team, and share debug output in a thread.
|
||||
|
||||
## GitHub Discussions
|
||||
|
||||
Best for bug reports, feature requests, and anything worth a searchable, written record. Open a thread in [GitHub Discussions](https://github.com/netbirdio/netbird/discussions) and include your version, environment, and a debug bundle.
|
||||
|
||||
For Cloud dashboard, billing, or commercial-license issues, use [NetBird Support](/help/netbird-support) instead.
|
||||
@@ -0,0 +1,51 @@
|
||||
import { SupportBanner } from "@/components/SupportBanner"
|
||||
|
||||
export const description =
|
||||
"NetBird Support for (paying) Cloud customers and users, and commercial-license self-hosted deployments: reach the team with a pre-filled report."
|
||||
|
||||
export const reportTemplate = `NetBird issue report
|
||||
=====================
|
||||
|
||||
1) Describe the problem
|
||||
A clear, concise description of what is going wrong.
|
||||
|
||||
2) Steps to reproduce
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
|
||||
3) Expected behavior
|
||||
What you expected to happen instead.
|
||||
|
||||
4) Environment
|
||||
- NetBird Cloud, or self-hosted control plane:
|
||||
- NetBird version (run: netbird version):
|
||||
- Other VPN software installed (if any):
|
||||
|
||||
5) Debug output
|
||||
- Anonymized status (run: netbird status -d):
|
||||
- Debug bundle, share the returned file key (run: netbird debug for 1m -S -U):
|
||||
Uploaded files are auto-deleted after 30 days.
|
||||
|
||||
6) Additional context
|
||||
Screenshots, logs, or anything else relevant.`
|
||||
|
||||
export const supportMailto = `mailto:support@netbird.io?subject=${encodeURIComponent(
|
||||
"NetBird issue report"
|
||||
)}&body=${encodeURIComponent(reportTemplate)}`
|
||||
|
||||
# NetBird Support
|
||||
|
||||
<SupportBanner
|
||||
tone="support"
|
||||
badge="Cloud & commercial"
|
||||
description="Reserved for (paying) Cloud customers and users, and self-hosted deployments on a commercial license. Use it for the Cloud dashboard, the managed control plane, billing, your subscription, or commercial self-hosted support. Open source self-hosted and general client questions belong in Community Support."
|
||||
links={[
|
||||
{ label: "Open a pre-filled support email", href: supportMailto },
|
||||
{ label: "Attach a debug bundle", href: "/help/troubleshooting-client#debug-bundle" },
|
||||
]}
|
||||
/>
|
||||
|
||||
Reach the team by opening a <a href={supportMailto}>pre-filled support email</a>. It drops a ready-to-fill report into your mail client, with the fields we need already laid out, so you only fill in the blanks. Prefer to write it yourself? Email [support@netbird.io](mailto:support@netbird.io) and include a [debug bundle](/help/troubleshooting-client#debug-bundle), your NetBird version, and clear steps to reproduce.
|
||||
|
||||
Not a Cloud or commercial-license customer? [Community Support](/help/community-support), through Slack and GitHub Discussions, is the right place.
|
||||
@@ -1,67 +1,25 @@
|
||||
import { Tiles } from "@/components/Tiles"
|
||||
|
||||
export const description =
|
||||
"How to report NetBird bugs and issues: Community Support for the client and open source self-hosted, and NetBird Support for Cloud customers and users, and commercial-license deployments."
|
||||
|
||||
# Report bugs and issues
|
||||
NetBird offers different ways to report bugs and issues. For prompt and effective assistance, please provide detailed information as outlined in our bug/issue [reporting template](#reporting-template).
|
||||
|
||||
For cloud users, you can report bugs and issues via email by sending an email to [support@netbird.io](mailto:support@netbird.io), via [Github issues](https://github.com/netbirdio/netbird/issues/new/choose) or by joining our [Slack Channel](/slack-url).
|
||||
NetBird offers two ways to get help, depending on what you are running. Pick the one that fits. Whichever you use, include a [debug bundle](/help/troubleshooting-client#debug-bundle), your NetBird version (`netbird version`), and clear steps to reproduce.
|
||||
|
||||
For on-premise users, you can report bugs and issues via [Github issues](https://github.com/netbirdio/netbird/issues/new/choose) or by joining our [Slack Channel](/slack-url).
|
||||
|
||||
## Reporting Template
|
||||
When reporting bugs and issues, please ensure you provide the following information:
|
||||
|
||||
|
||||
**Describe the problem**
|
||||
|
||||
A clear and concise description of what the problem is.
|
||||
|
||||
**To Reproduce**
|
||||
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
|
||||
**Have you performed any debugging steps?**
|
||||
Learn more at [troubleshooting guide](/help/troubleshooting-client)
|
||||
|
||||
**Expected behavior**
|
||||
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Are you using NetBird Cloud?**
|
||||
|
||||
Please specify whether you use NetBird Cloud or self-host NetBird's control plane.
|
||||
|
||||
**NetBird version**
|
||||
|
||||
`netbird version`
|
||||
|
||||
**Is any other VPN software installed?**
|
||||
|
||||
If yes, which one?
|
||||
|
||||
**Debug output**
|
||||
|
||||
To help us resolve the problem, please attach the following anonymized status output
|
||||
|
||||
netbird status -d
|
||||
|
||||
Create and upload a debug bundle, and share the returned file key:
|
||||
|
||||
netbird debug for 1m -S -U
|
||||
|
||||
*Uploaded files are automatically deleted after 30 days.*
|
||||
|
||||
|
||||
Alternatively, create the file only and attach it here manually:
|
||||
|
||||
netbird debug for 1m -S
|
||||
|
||||
|
||||
**Screenshots**
|
||||
|
||||
If applicable, add screenshots to help explain your problem.
|
||||
|
||||
**Additional context**
|
||||
|
||||
Add any other context about the problem here.
|
||||
<Tiles
|
||||
title="Where to report"
|
||||
id="where-to-report"
|
||||
items={[
|
||||
{
|
||||
href: "/help/community-support",
|
||||
name: "Community Support",
|
||||
description: "Free, for everyone. The NetBird client, open source self-hosted deployments, and general questions, on Slack or GitHub Discussions.",
|
||||
},
|
||||
{
|
||||
href: "/help/netbird-support",
|
||||
name: "NetBird Support",
|
||||
description: "For (paying) Cloud customers and users, and commercial-license self-hosted deployments: the dashboard, control plane, billing, and subscriptions.",
|
||||
},
|
||||
]}
|
||||
/>
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import {Note} from "@/components/mdx";
|
||||
|
||||
export const description = "Why your NetBird user might be pending approval, or added to an account you don't recognize, and how to resolve it safely."
|
||||
|
||||
# Pending approval and account invitations
|
||||
|
||||
Signed in and found your user **pending approval**, or landed in a NetBird account you don't recognize? This usually is not a bug. It almost always means you were added to an existing organization automatically, and an admin still has to approve you.
|
||||
|
||||
## Why this happens
|
||||
|
||||
NetBird can add people to an account without a manual, one-by-one invite:
|
||||
|
||||
- **Indirect (domain) invites.** If an account has verified your email domain, new sign-ups with that domain join it automatically. See [Indirect user invites](/manage/team/add-users-to-your-network#indirect-user-invites).
|
||||
- **Identity provider sync.** If your organization connects NetBird to its identity provider, being added to a synced group in the IdP provisions your user in NetBird. See [Provision users and groups from your identity provider](/manage/team/idp-sync).
|
||||
- **Approval is required.** When an account turns on [user approval](/manage/team/approve-users#require-user-approval), new users stay blocked until an admin approves them. That block is the "pending approval" state you see.
|
||||
|
||||
## What to do
|
||||
|
||||
1. **Confirm it is your organization.** Check with your IT or security team whether they run NetBird and expect you on it. The account [Owner](/manage/team/user-roles#owner), of which there is exactly one per account, or an admin can confirm and approve you.
|
||||
2. **Ask an admin to approve your user.** Approval happens in the dashboard from the account's user list. See [Approve or reject user](/manage/team/approve-users#approve-or-reject-user).
|
||||
3. **If something looks off,** for example you don't recognize the account, your domain was claimed unexpectedly, or no one internally owns it, do not approve or accept anything. Treat it as suspicious and verify internally first.
|
||||
|
||||
<Note>
|
||||
Still unsure who owns the account, or think you were added to the wrong one? Reach out through [NetBird Support](/help/netbird-support) and the team can help confirm the account and its owner.
|
||||
</Note>
|
||||
@@ -1,12 +1,86 @@
|
||||
import { TroubleshootingStart } from "@/components/TroubleshootingStart"
|
||||
import { TroubleshootingTiles } from "@/components/TroubleshootingTiles"
|
||||
import { HashRedirect } from "@/components/HashRedirect"
|
||||
|
||||
<HashRedirect
|
||||
map={{
|
||||
"net-bird-agent-status": "/help/troubleshooting-client#net-bird-client-status",
|
||||
}}
|
||||
/>
|
||||
|
||||
# Troubleshooting client issues
|
||||
|
||||
This document offers practical tips and insights to help you debug various problems, ensuring a seamless user
|
||||
experience.
|
||||
experience. The steps here are cross-platform; for OS-specific details, use the platform guides below.
|
||||
|
||||
## NetBird agent status
|
||||
<TroubleshootingStart
|
||||
eyebrow="Start here"
|
||||
title="Collect diagnostics first"
|
||||
id="collect-diagnostics-first"
|
||||
description="Most client issues are quicker to resolve with two things in hand: the current status, and a debug bundle. Grab them before you go deeper. Each step links to the section that explains it."
|
||||
steps={[
|
||||
{ label: "1. Check status", title: "Client status", href: "#net-bird-client-status", command: "netbird status -d", hint: "Is the peer Connected, and are Management, Signal, and the Relays reachable?" },
|
||||
{ label: "2. Collect debug bundle", title: "Debug bundle", href: "#debug-bundle", command: "netbird debug bundle -A -S", hint: "Logs, status, routes, and config in one archive. See what is inside." },
|
||||
]}
|
||||
/>
|
||||
|
||||
The netbird agent is a daemon service that runs in the background; it provides information about peers connected and
|
||||
about the NetBird control services. You can check the status of the agent with the following command:
|
||||
<TroubleshootingTiles
|
||||
title="Platform-specific guides"
|
||||
id="platform-specific-guides"
|
||||
description="OS-specific steps live on their own pages. Everything else on this page applies to all platforms."
|
||||
items={[
|
||||
{
|
||||
title: "Linux",
|
||||
href: "/help/troubleshooting-client/linux",
|
||||
icon: "terminal",
|
||||
description: "Permanent debug log level (systemd, service, Docker) and host-based firewall (UFW, firewalld).",
|
||||
chips: [
|
||||
{ label: "Debug logs", href: "/help/troubleshooting-client/linux#set-the-log-level-permanently" },
|
||||
{ label: "Host firewall", href: "/help/troubleshooting-client/linux#host-based-firewall" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "Windows",
|
||||
href: "/help/troubleshooting-client/windows",
|
||||
icon: "windows",
|
||||
description: "Debug log level, foreground mode via PSExec, Windows Firewall, and Windows DNS (NRPT/GPO, Active Directory).",
|
||||
chips: [
|
||||
{ label: "Foreground (PSExec)", href: "/help/troubleshooting-client/windows#run-the-client-in-foreground-mode" },
|
||||
{ label: "Windows DNS", href: "/help/troubleshooting-client/windows#windows-dns-scenarios" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "macOS",
|
||||
href: "/help/troubleshooting-client/macos",
|
||||
icon: "laptop",
|
||||
description: "Permanent debug log level and host-based firewall on macOS.",
|
||||
chips: [
|
||||
{ label: "Debug logs", href: "/help/troubleshooting-client/macos#set-the-log-level-permanently" },
|
||||
{ label: "Host firewall", href: "/help/troubleshooting-client/macos#host-based-firewall" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "Android",
|
||||
href: "/help/troubleshooting-client/android",
|
||||
icon: "android",
|
||||
description: "Enable trace logs and capture them with ADB.",
|
||||
chips: [
|
||||
{ label: "Trace logs (ADB)", href: "/help/troubleshooting-client/android#enable-trace-logs-and-capture-them-with-adb" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "iOS",
|
||||
href: "/help/troubleshooting-client/ios",
|
||||
icon: "mobile",
|
||||
description: "Capturing logs from the app. On-device debugging is limited.",
|
||||
},
|
||||
]}
|
||||
/>
|
||||
|
||||
## NetBird client status
|
||||
|
||||
The NetBird client is a daemon service that runs in the background; it provides information about peers connected and
|
||||
about the NetBird control services. You can check the status of the client with the following command:
|
||||
|
||||
```shell
|
||||
netbird status --detail
|
||||
@@ -108,7 +182,7 @@ log rotation conflict detected in: "/etc/logrotate.d/netbird", rotation is disab
|
||||
<Note>
|
||||
To use `logrotate`, we require `copytruncate` and not `create` to be set in the config file, otherwise the daemon needs to be restarted for the new log file to be opened after rotation.
|
||||
|
||||
Netbird supports `compress`, `delaycompress` and `nocompress`.
|
||||
NetBird supports `compress`, `delaycompress` and `nocompress`.
|
||||
</Note>
|
||||
|
||||
On macOS and BSD systems, the equivalent of `logrotate` is `newsyslog`, configured via `/etc/newsyslog.conf` or files in `/etc/newsyslog.d/`. NetBird does not auto-detect `newsyslog` configurations, so if you want `newsyslog` to manage `client.log` you must set [`NB_LOG_DISABLE_ROTATION=true`](/client/environment-variables#logging) on the daemon to disable the built-in rotation. An example `/etc/newsyslog.d/netbird.conf` entry:
|
||||
@@ -152,6 +226,22 @@ This will output the path of the generated file. The output file is owned by and
|
||||
NetBird is running as, by default it is: `Administrator` on Windows, `root` on MacOS/Linux or the operating system\'s
|
||||
equivalent.
|
||||
|
||||
#### What's inside the bundle
|
||||
|
||||
The archive collects the most useful diagnostics into one file, and every bundle ships a `README.txt` that documents each entry. The ones you will reach for most often:
|
||||
|
||||
| File | What it holds |
|
||||
|---|---|
|
||||
| `status.txt` | Status output, the same view as `netbird status -d` |
|
||||
| `client.log`, `netbird.err`, `netbird.out` | Recent client logs, plus stderr and stdout |
|
||||
| `routes.txt`, `ip_rules.txt` | System routing table and IP rules (with `--system-info`) |
|
||||
| `iptables.txt`, `nftables.txt`, `ipset.txt` | Firewall rules with packet counters (Linux, with `--system-info`) |
|
||||
| `resolv.conf`, `scutil_dns.txt`, `resolved_domains.txt` | DNS resolver configuration and the domains NetBird resolved |
|
||||
| `network_map.json` | Sync response: peers, routes, DNS settings, and firewall rules |
|
||||
| `config.txt`, `state.json` | Client configuration and internal client state |
|
||||
|
||||
With `--anonymize`, IP addresses, domains, and interface names are replaced consistently across every file, so the bundle stays readable while sensitive values are masked. Private keys and SSH keys are never included, and the packet capture (`capture.pcap`) is left out of anonymized bundles because it holds raw decrypted packets.
|
||||
|
||||
### Debug for a specific time
|
||||
|
||||
To capture logs for a specific time period, you can use the `debug for` command. This will generate a debug bundle after
|
||||
@@ -298,7 +388,7 @@ On Linux, the userspace packet filter is only active when the kernel firewall ba
|
||||
|
||||
The client has environment variables for tuning routing, firewall behavior, ICE connectivity, and WireGuard mode. These can help work around edge cases (e.g. `NB_USE_LEGACY_ROUTING` for routing loop issues, `NB_WG_KERNEL_DISABLED` to force userspace WireGuard, `NB_SKIP_NFTABLES_CHECK` to fall back to iptables). See the full list at [Client Environment Variables](/client/environment-variables).
|
||||
|
||||
## Enabling debug logs on agent
|
||||
## Enabling debug logs on the client
|
||||
|
||||
Logs can be temporarily set using the following command.
|
||||
|
||||
@@ -316,100 +406,29 @@ The next time the daemon is restarted, the log level will return to the configur
|
||||
|
||||
Using `netbird down` and `netbird up` will not reset the log level.
|
||||
|
||||
To permanently set the log level, see the following sections.
|
||||
To set the log level **permanently**, follow the steps for your platform: [Linux](/help/troubleshooting-client/linux#set-the-log-level-permanently), [Windows](/help/troubleshooting-client/windows#set-the-log-level-permanently), [macOS](/help/troubleshooting-client/macos#set-the-log-level-permanently), or [Android](/help/troubleshooting-client/android#enable-trace-logs-and-capture-them-with-adb).
|
||||
|
||||
<Note>
|
||||
The default logging level is `info`. To revert back to the original state, you can repeat the procedure with `info` instead of `debug` or `trace`.
|
||||
</Note>
|
||||
|
||||
### On Linux with systemd
|
||||
## Running the client in foreground mode
|
||||
|
||||
The default systemd unit file reads a set of environment variables from the path `/etc/sysconfig/netbird`.
|
||||
You can add the following line to the file to enable debug logs:
|
||||
|
||||
```shell
|
||||
sudo mkdir -p /etc/sysconfig
|
||||
echo 'NB_LOG_LEVEL=debug' | sudo tee -a /etc/sysconfig/netbird
|
||||
sudo systemctl restart netbird
|
||||
```
|
||||
|
||||
### On Other Linux and MacOS
|
||||
|
||||
```shell
|
||||
sudo netbird service stop
|
||||
sudo netbird service uninstall
|
||||
sudo netbird service install --log-level debug # or trace
|
||||
sudo netbird service start
|
||||
```
|
||||
|
||||
### On Windows
|
||||
|
||||
You need to run the following commands with an elevated PowerShell or `cmd.exe` window.
|
||||
|
||||
```powershell
|
||||
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")
|
||||
netbird service restart
|
||||
```
|
||||
|
||||
### On Docker
|
||||
|
||||
You can set the environment variable `NB_LOG_LEVEL` to `debug` to enable debug logs.
|
||||
|
||||
```shell
|
||||
docker run --rm --name PEER_NAME --hostname PEER_NAME --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -d \
|
||||
-e NB_SETUP_KEY=<SETUP KEY> -e NB_LOG_LEVEL=debug -v netbird-client:/var/lib/netbird netbirdio/netbird:latest
|
||||
```
|
||||
|
||||
### On Android
|
||||
|
||||
Enable the ADB in the developer menu on the Android device.
|
||||
In the app set the the Trace log level setting - it is a checkbox in the advanced menu.
|
||||
With the ADB tool, you can get the logs from your device. The ADB is part of the SDK platform tools pack (zip file).
|
||||
You can download it from [here](https://developer.android.com/tools/releases/platform-tools).
|
||||
Please extract it and run the next command in the case of Linux:
|
||||
|
||||
```shell
|
||||
sudo adb logcat -v time | grep GoLog
|
||||
```
|
||||
|
||||
## Running the agent in foreground mode
|
||||
|
||||
You can run the agent in foreground mode to see the logs in the terminal. This is useful to debugging issues with the
|
||||
agent.
|
||||
|
||||
### Linux and MacOS
|
||||
You can run the client in foreground mode to see the logs in the terminal. This is useful when debugging issues with the client. On Linux and macOS:
|
||||
|
||||
```shell
|
||||
sudo netbird service stop
|
||||
sudo netbird up -F
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
On Windows, the agent depends on the Wireguard's `wintun.dll` and can only be executed as a system account.
|
||||
To run the agent in foreground mode, you need to use a tool
|
||||
called [PSExec](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec).
|
||||
|
||||
Once you have downloaded and extracted `psexec` open an elevated Powershell window:
|
||||
|
||||
```shell
|
||||
netbird service stop
|
||||
.\PsExec64.exe -s cmd.exe /c "netbird up -F --log-level debug > c:\windows\temp\netbird.out.log 2>&1"
|
||||
```
|
||||
|
||||
In case you need to configure environment variables, you need to add them as system variables so they get picked up by
|
||||
the agent on the next psexec run:
|
||||
|
||||
```powershell
|
||||
[Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")
|
||||
````
|
||||
On Windows, foreground mode needs PSExec because the client runs as the system account. See [Run the client in foreground mode](/help/troubleshooting-client/windows#run-the-client-in-foreground-mode) on the Windows page.
|
||||
|
||||
## Enabling WireGuard in user space
|
||||
|
||||
Sometimes, you want to test NetBird running on userspace mode instead of a kernel module. That can be a check to see if
|
||||
there is a problem with NetBird's firewall management in kernel mode.
|
||||
|
||||
You must run the agent in foreground mode and set the environment variable `NB_WG_KERNEL_DISABLED` to `true`.
|
||||
You must run the client in foreground mode and set the environment variable `NB_WG_KERNEL_DISABLED` to `true`.
|
||||
|
||||
```shell
|
||||
sudo netbird service stop
|
||||
@@ -418,7 +437,7 @@ sudo bash -c 'NB_WG_KERNEL_DISABLED=true netbird up -F' > /tmp/netbird.log
|
||||
|
||||
## Debugging GRPC
|
||||
|
||||
The NetBird agent communicates with the Management and Signal servers using the GRPC framework. With these parameters,
|
||||
The NetBird client communicates with the Management and Signal servers using the GRPC framework. With these parameters,
|
||||
you can
|
||||
set verbose logging for this service.
|
||||
|
||||
@@ -429,7 +448,7 @@ sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info net
|
||||
|
||||
## Debugging ICE connections
|
||||
|
||||
The Netbird agent communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol
|
||||
The NetBird client communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol
|
||||
described in the [RFC 8445](https://datatracker.ietf.org/doc/html/rfc8445). To debug the connection procedure,
|
||||
set verbose logging for the the [Pion/ICE](https://github.com/pion/ice) library with the `PIONS_LOG_DEBUG` or
|
||||
`PIONS_LOG_TRACE` variable.
|
||||
@@ -444,10 +463,6 @@ sudo netbird service stop
|
||||
sudo bash -c 'PIONS_LOG_DEBUG=all NB_LOG_LEVEL=debug netbird up -F' > /tmp/netbird.log
|
||||
```
|
||||
|
||||
## Host-based firewall issues
|
||||
|
||||
NetBird automatically manages host-based firewall rules, but conflicts can occur with other firewall tools or security software. See [Ports & Firewalls — Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls) for symptoms, platform-specific remediation (UFW, firewalld, Windows Firewall), and diagnostic commands.
|
||||
|
||||
## Client login failures
|
||||
|
||||
A single machine can only connect to one NetBird account as the same user/login method throughout the lifetime of
|
||||
@@ -490,37 +505,34 @@ Key while the NetBird client daemon is stopped:
|
||||
|
||||
## Debugging access to network resources
|
||||
|
||||
In this section we will be presenting methodology of troubleshooting access issues involving Netbird.
|
||||
This section is the hands-on, command-level playbook for the case where a peer is `Connected` but a service behind a routing peer is unreachable. For the conceptual model first (where the traffic stops, and the TCP handshake as the dividing line between a NetBird problem and an application problem), start with [Troubleshooting resource connectivity](/help/troubleshooting-resource-connectivity). This section then walks the same checks with concrete commands.
|
||||
|
||||
We will start by presenting a glossary of all machines and services involved.
|
||||
A sub-section will describe a specific use case.
|
||||
Each will start with a concise summary of usual troubleshooting steps then expand into more detailed step-by-step
|
||||
guides.
|
||||
It uses a glossary of the machines and services involved, followed by a specific use case that opens with a concise summary of the usual steps and expands into a detailed, step-by-step walkthrough.
|
||||
|
||||
### Glossary
|
||||
|
||||
We will be using the following names for resources outside the Netbird network:
|
||||
We will be using the following names for resources outside the NetBird network:
|
||||
|
||||
- `int-net1`: an internal network `10.123.45.0/24`,
|
||||
- `srv-c`: an internal HTTP server running at `10.123.45.17`,
|
||||
- `int-dns1`: an internal DNS server running at `10.123.45.6`,
|
||||
- `int-dns2`: an internal DNS server nunning at `10.7.8.9`,
|
||||
- `int-dns2`: an internal DNS server running at `10.7.8.9`,
|
||||
- `cf-dns`: an Internet-accessible CloudFlare DNS server at `1.1.1.1` and `1.0.0.1`,
|
||||
|
||||
and following Netbird network resources:
|
||||
and following NetBird network resources:
|
||||
|
||||
- `peer-a`: end user's device running Netbird Client,
|
||||
- `peer-b`: a linux server inside the internal network running Netbird Client,
|
||||
- `peer-a`: end user's device running the NetBird client,
|
||||
- `peer-b`: a Linux server inside the internal network running the NetBird client,
|
||||
- it has direct access to the whole `int-net1` IP range,
|
||||
- `users:employees`: a Netbird Group containing `peer-a`,
|
||||
- `routers:int-net1`: a Netbird Group containing `peer-b`,
|
||||
- `access:srv-c`: a Netbird Groups used as a target of ACL rules for `srv-c` only,
|
||||
- `access:int-net1`: a Netbird Groups used as a target of ACL rules for the whole subnet,
|
||||
- `net-a`: a Netbird Network
|
||||
- `users:employees`: a NetBird Group containing `peer-a`,
|
||||
- `routers:int-net1`: a NetBird Group containing `peer-b`,
|
||||
- `access:srv-c`: a NetBird Groups used as a target of ACL rules for `srv-c` only,
|
||||
- `access:int-net1`: a NetBird Groups used as a target of ACL rules for the whole subnet,
|
||||
- `net-a`: a NetBird Network
|
||||
- `net-a:srv-c`: a Network Resource handling traffic to `10.123.45.17/32` (`srv-c`),
|
||||
- `net-a:int-net1`: a Network Resource handling traffic to `10.123.45.0/24` (`int-net1`),
|
||||
- `route:int-net1`: a Netbird Network Route handling traffic to `10.123.45.0/24` (`int-net1`),
|
||||
- `route:srv-c`: a Netbird Network Route handling traffic to `10.123.45.17/32` (`srv-c`),
|
||||
- `route:int-net1`: a NetBird Network Route handling traffic to `10.123.45.0/24` (`int-net1`),
|
||||
- `route:srv-c`: a NetBird Network Route handling traffic to `10.123.45.17/32` (`srv-c`),
|
||||
|
||||
### Access from `peer-a` to `srv-c`
|
||||
|
||||
@@ -528,8 +540,8 @@ In short:
|
||||
|
||||
1. Does `peer-b` have direct access to `srv-c`'s port `80`?
|
||||
2. Can a routing peer `peer-b` forward traffic to `srv-c`?
|
||||
3. Are Netbird's network routing resources configured?
|
||||
4. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group?
|
||||
3. Are NetBird's network routing resources configured?
|
||||
4. Do NetBird's Access Control rules allow access from `peer-a` to the target's ACL Group?
|
||||
5. Is `peer-a`'s operating system configured to use the route?
|
||||
|
||||
Access Control rule is not required for connectivity from `peer-a` to `peer-b`
|
||||
@@ -562,7 +574,7 @@ Linux operating system:
|
||||
net.ipv4.ip_forward = 1
|
||||
```
|
||||
|
||||
It should be set up automatically by the Netbird client unless it runs inside a container (which would not be able
|
||||
It should be set up automatically by the NetBird client unless it runs inside a container (which would not be able
|
||||
to modify `sysctl`), then it requires manual setup.
|
||||
|
||||
For setting up the value persistently (across reboots) please consult your operating system's documentation.
|
||||
@@ -574,9 +586,9 @@ Testing the functionality in practice involves:
|
||||
- adding a routing table entry to route `int-net1` (`10.123.45.0/24`) traffic through it,
|
||||
- trying to at least `ping 10.123.45.17` (`srv-c`)
|
||||
|
||||
#### Are Netbird's network routing resources configured?
|
||||
#### Are NetBird's network routing resources configured?
|
||||
|
||||
For Netbird network routing resources configurations you can use either (new) _Networks_ or (old) _Routes_.
|
||||
For NetBird network routing resources configurations you can use either (new) _Networks_ or (old) _Routes_.
|
||||
|
||||
A Network `net-a` should have at minimum:
|
||||
|
||||
@@ -597,7 +609,7 @@ You can loosen the rules and replace following to grant access to the whole `int
|
||||
- _Address_: `10.123.45.17/32` -> `10.123.45.0/24`,
|
||||
- _Assigned Groups_ / _Access Control Groups_: `access:srv-c` -> `access:int-net1`
|
||||
|
||||
#### Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group?
|
||||
#### Do NetBird's Access Control rules allow access from `peer-a` to the target's ACL Group?
|
||||
|
||||
You can skip this check, when you are using (old) Network Route feature without filling in _Access Control Groups (
|
||||
optional)_ section.
|
||||
@@ -634,10 +646,10 @@ Just like with the previous section you can loosen the above example by:
|
||||
|
||||
#### Is `peer-a`'s operating system configured to use the route?
|
||||
|
||||
After all resources are configured in the Netbird management you should check whether they are
|
||||
After all resources are configured in the NetBird management you should check whether they are
|
||||
properly registered with your operating system.
|
||||
|
||||
You can start by checking Netbird client's configuration with `netbird status -d` command:
|
||||
You can start by checking NetBird client's configuration with `netbird status -d` command:
|
||||
|
||||
```shell
|
||||
% netbird status -d
|
||||
@@ -732,7 +744,7 @@ your specific subnet's clamped IP ranges (`10.123.45` in case of `int-net1`) and
|
||||
Depending on specifics of your Linux distribution (or even your configuration of it) you should be able to use either
|
||||
`iproute2` or `net-tools` family of network commands.
|
||||
|
||||
Netbird client stores it's custom routes in the routing table `7120` (or `0x1BD0`) when it's available (through
|
||||
NetBird client stores its custom routes in the routing table `7120` (or `0x1BD0`) when it's available (through
|
||||
`iproute2` interface).
|
||||
|
||||
For `iproute2` (`ip`, `ss` tools):
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
export const description =
|
||||
"Android-specific NetBird client troubleshooting: enabling trace logs and capturing them with ADB."
|
||||
|
||||
# NetBird client on Android
|
||||
|
||||
Android-specific steps for the NetBird client. For everything cross-platform (client status, connectivity, login, and DNS), start from [Troubleshooting client issues](/help/troubleshooting-client).
|
||||
|
||||
## Enable trace logs and capture them with ADB
|
||||
|
||||
1. Enable **ADB** in the device's developer options.
|
||||
2. In the NetBird app, set the **Trace** log level (a checkbox in the advanced menu).
|
||||
3. Install the ADB platform tools (part of the [SDK platform-tools](https://developer.android.com/tools/releases/platform-tools) pack), then capture the logs. On Linux:
|
||||
|
||||
```shell
|
||||
sudo adb logcat -v time | grep GoLog
|
||||
```
|
||||
@@ -0,0 +1,14 @@
|
||||
import {Note} from "@/components/mdx"
|
||||
|
||||
export const description =
|
||||
"iOS-specific NetBird client troubleshooting and how to capture logs from the app."
|
||||
|
||||
# NetBird client on iOS
|
||||
|
||||
iOS-specific steps for the NetBird client. Most troubleshooting is cross-platform, so start from [Troubleshooting client issues](/help/troubleshooting-client).
|
||||
|
||||
On-device debugging on iOS is more limited than on desktop platforms. To share diagnostics, capture logs from the NetBird iOS app and attach them to your report.
|
||||
|
||||
<Note>
|
||||
There are no iOS-specific debug commands yet. For status, connectivity, and DNS issues, follow the cross-platform [Troubleshooting client issues](/help/troubleshooting-client) and [DNS Troubleshooting](/manage/dns/troubleshooting) guides. If you can reproduce a problem, report it through [Community Support](/help/community-support).
|
||||
</Note>
|
||||
@@ -0,0 +1,42 @@
|
||||
export const description =
|
||||
"Linux-specific NetBird client troubleshooting: setting the debug log level (systemd, service, Docker) and host-based firewalls."
|
||||
|
||||
# NetBird client on Linux
|
||||
|
||||
Linux-specific steps for the NetBird client. For everything cross-platform (client status, the debug bundle, GRPC and ICE debugging, login failures, and reaching resources), start from [Troubleshooting client issues](/help/troubleshooting-client).
|
||||
|
||||
## Set the log level permanently
|
||||
|
||||
The [temporary log level](/help/troubleshooting-client#enabling-debug-logs-on-the-client) resets when the daemon restarts. To make it permanent on Linux, use one of the following.
|
||||
|
||||
### systemd
|
||||
|
||||
The default systemd unit reads environment variables from `/etc/sysconfig/netbird`:
|
||||
|
||||
```shell
|
||||
sudo mkdir -p /etc/sysconfig
|
||||
echo 'NB_LOG_LEVEL=debug' | sudo tee -a /etc/sysconfig/netbird
|
||||
sudo systemctl restart netbird
|
||||
```
|
||||
|
||||
### Other init systems
|
||||
|
||||
```shell
|
||||
sudo netbird service stop
|
||||
sudo netbird service uninstall
|
||||
sudo netbird service install --log-level debug # or trace
|
||||
sudo netbird service start
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
Set `NB_LOG_LEVEL=debug` on the container:
|
||||
|
||||
```shell
|
||||
docker run --rm --name PEER_NAME --hostname PEER_NAME --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -d \
|
||||
-e NB_SETUP_KEY=<SETUP KEY> -e NB_LOG_LEVEL=debug -v netbird-client:/var/lib/netbird netbirdio/netbird:latest
|
||||
```
|
||||
|
||||
## Host-based firewall
|
||||
|
||||
NetBird manages its own rules, but UFW, firewalld, or endpoint security software can conflict and silently drop traffic. See [Ports & Firewalls: Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls) for UFW and firewalld symptoms, remediation, and diagnostic commands.
|
||||
@@ -0,0 +1,21 @@
|
||||
export const description =
|
||||
"macOS-specific NetBird client troubleshooting: setting the debug log level and host-based firewalls."
|
||||
|
||||
# NetBird client on macOS
|
||||
|
||||
macOS-specific steps for the NetBird client. For everything cross-platform (client status, the debug bundle, GRPC and ICE debugging, login failures, and reaching resources), start from [Troubleshooting client issues](/help/troubleshooting-client).
|
||||
|
||||
## Set the log level permanently
|
||||
|
||||
The [temporary log level](/help/troubleshooting-client#enabling-debug-logs-on-the-client) resets when the service restarts. To make it permanent on macOS, reinstall the service with the level set:
|
||||
|
||||
```shell
|
||||
sudo netbird service stop
|
||||
sudo netbird service uninstall
|
||||
sudo netbird service install --log-level debug # or trace
|
||||
sudo netbird service start
|
||||
```
|
||||
|
||||
## Host-based firewall
|
||||
|
||||
The built-in macOS application firewall or third-party endpoint security software can block NetBird traffic before it leaves the machine. If connectivity works with that software temporarily disabled, add an exception for the NetBird process. See [Ports & Firewalls: Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls) for the general approach.
|
||||
@@ -0,0 +1,47 @@
|
||||
import {Note} from "@/components/mdx"
|
||||
|
||||
export const description =
|
||||
"Windows-specific NetBird client troubleshooting: debug log level, foreground mode via PSExec, host-based firewall, and Windows DNS scenarios."
|
||||
|
||||
# NetBird client on Windows
|
||||
|
||||
Windows-specific steps for the NetBird client. For everything cross-platform (client status, the debug bundle, GRPC and ICE debugging, login failures, and reaching resources), start from [Troubleshooting client issues](/help/troubleshooting-client).
|
||||
|
||||
## Set the log level permanently
|
||||
|
||||
The [temporary log level](/help/troubleshooting-client#enabling-debug-logs-on-the-client) resets when the service restarts. To make it permanent, run an elevated PowerShell or `cmd.exe` window:
|
||||
|
||||
```powershell
|
||||
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")
|
||||
netbird service restart
|
||||
```
|
||||
|
||||
## Run the client in foreground mode
|
||||
|
||||
On Windows the client depends on WireGuard's `wintun.dll` and can only run as the system account. To run it in foreground mode, use [PSExec](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec). In an elevated PowerShell window:
|
||||
|
||||
```shell
|
||||
netbird service stop
|
||||
.\PsExec64.exe -s cmd.exe /c "netbird up -F --log-level debug > c:\windows\temp\netbird.out.log 2>&1"
|
||||
```
|
||||
|
||||
To pass environment variables, set them as machine-level variables so the client picks them up on the next PSExec run:
|
||||
|
||||
```powershell
|
||||
[Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")
|
||||
```
|
||||
|
||||
## Host-based firewall
|
||||
|
||||
Windows Firewall or endpoint security software can block NetBird traffic before it leaves the machine. See [Ports & Firewalls: Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls) for Windows Firewall symptoms, remediation, and diagnostic commands.
|
||||
|
||||
## Windows DNS scenarios
|
||||
|
||||
DNS on Windows has a few platform-specific failure modes worth checking separately:
|
||||
|
||||
- **Match-domain names don't resolve, even though the NRPT (Name Resolution Policy Table) rule was written.** A lingering Group Policy `DnsPolicyConfig` container can stop NetBird's rule from taking effect on an off-domain machine. See [DNS Troubleshooting: Issue 8 (lingering GPO)](/manage/dns/troubleshooting#issue-8-windows-nrpt-rule-is-written-but-never-takes-effect-lingering-gpo).
|
||||
- **Active Directory login, mapped drives, or DFS fail** while a file share by IP works. This is usually a DC-locator (`SRV` record) problem. See [Domain Controllers as routing peers](/manage/dns/internal-dns-servers#domain-controllers-as-routing-peers).
|
||||
|
||||
<Note>
|
||||
For the full DNS diagnostic flow on any platform, see [DNS Troubleshooting](/manage/dns/troubleshooting).
|
||||
</Note>
|
||||
@@ -5,9 +5,9 @@ export const description = "Learn why a NetBird connection is Relayed instead of
|
||||
|
||||
# Troubleshooting relayed connections
|
||||
|
||||
NetBird always prefers a direct peer-to-peer (P2P) connection and falls back to a relay server when a direct path can't be established. A relayed connection works — it just adds latency and shares the relay's bandwidth, because traffic travels through an intermediary instead of flowing directly between the peers. This page teaches you to find out *why* a connection is relayed, fix it when it's fixable, and recognize the cases where relay is the correct outcome rather than a fault.
|
||||
NetBird always prefers a direct peer-to-peer (P2P) connection and falls back to a relay server when a direct path can't be established. A relayed connection works. It just adds latency and shares the relay's bandwidth, because traffic travels through an intermediary instead of flowing directly between the peers. This page teaches you to find out *why* a connection is relayed, fix it when it's fixable, and recognize the cases where relay is the correct outcome rather than a fault.
|
||||
|
||||
The endpoints on this page are for **NetBird Cloud**. The flow is identical for self-hosted deployments — substitute your own Signal, STUN, and Relay endpoints from the [self-hosted port requirements](/selfhosted/selfhosted-guide#port-requirements).
|
||||
The endpoints on this page are for **NetBird Cloud**. The flow is identical for self-hosted deployments. Substitute your own Signal, STUN, and Relay endpoints from the [self-hosted port requirements](/selfhosted/selfhosted-guide#port-requirements).
|
||||
|
||||
<Note>
|
||||
A relayed connection is just as secure as a direct one. Traffic is encrypted end to end with WireGuard before it leaves the device, and the relay only forwards packets it cannot read. The trade-off is latency and throughput, never confidentiality.
|
||||
@@ -41,15 +41,15 @@ Find the peer in question and look at the **Connection type** field:
|
||||
| Field | What it tells you |
|
||||
|---|---|
|
||||
| `Connection type: Relayed` | Traffic flows through a relay server instead of directly between the peers |
|
||||
| `ICE candidate (Local/Remote)` | How each side is connecting — the key diagnostic, explained below |
|
||||
| `ICE candidate (Local/Remote)` | How each side is connecting, the key diagnostic, explained below |
|
||||
| `Relay server address` | Which relay server carries the connection |
|
||||
| `Last WireGuard handshake` | A recent handshake means the tunnel itself is healthy, relayed or not |
|
||||
|
||||
If the connection type is `P2P` but the link is slow, stop here — that's a different problem (path quality, MTU, or load), not a relay issue. Start from the [general client troubleshooting page](/help/troubleshooting-client#net-bird-agent-status) instead.
|
||||
If the connection type is `P2P` but the link is slow, stop here. That's a different problem (path quality, MTU, or load), not a relay issue. Start from the [general client troubleshooting page](/help/troubleshooting-client#net-bird-client-status) instead.
|
||||
|
||||
## The mental model
|
||||
|
||||
If you remember one thing, remember this: **a relayed connection is not a failure — it's the safety net after a failed hole punch.** Your job is to find out which of two worlds you're in:
|
||||
If you remember one thing, remember this: **a relayed connection is not a failure; it's the safety net after a failed hole punch.** Your job is to find out which of two worlds you're in:
|
||||
|
||||
```
|
||||
Why is this connection relayed?
|
||||
@@ -58,23 +58,23 @@ If you remember one thing, remember this: **a relayed connection is not a failur
|
||||
│ │
|
||||
A fixable blocker An unfixable NAT
|
||||
(Signal, STUN, or UDP is (both sides scramble ports
|
||||
blocked somewhere — find per destination — relay is
|
||||
blocked somewhere, find per destination, relay is
|
||||
it and remove it) doing its designed job)
|
||||
```
|
||||
|
||||
You can't ask NetBird to measure NAT behavior directly, so you work by **elimination**: first triage for environments that are known to defeat hole punching, then clear the fixable blockers one by one. If every check passes on both peers and the connection is still relayed, you have proven the NAT is the cause — and the relay is the designed answer, not a problem left to fix.
|
||||
You can't ask NetBird to measure NAT behavior directly, so you work by **elimination**: first triage for environments that are known to defeat hole punching, then clear the fixable blockers one by one. If every check passes on both peers and the connection is still relayed, you have proven the NAT is the cause, and the relay is the designed answer, not a problem left to fix.
|
||||
|
||||
## The four players
|
||||
|
||||
Four things decide whether a connection goes direct. Each one maps to exactly one check in the flow below.
|
||||
|
||||
**NAT — the obstacle.** Routers rewrite addresses, so a peer behind NAT can't receive unsolicited traffic. Most NATs hand out a *predictable* public address that hole punching can use; symmetric NATs and carrier-grade NAT (CGNAT) hand out a *different* one per destination, which defeats hole punching entirely. The theory lives in [Understanding NAT and Connectivity](/about-netbird/understanding-nat-and-connectivity).
|
||||
**NAT, the obstacle.** Routers rewrite addresses, so a peer behind NAT can't receive unsolicited traffic. Most NATs hand out a *predictable* public address that hole punching can use; symmetric NATs and carrier-grade NAT (CGNAT) hand out a *different* one per destination, which defeats hole punching entirely. The theory lives in [Understanding NAT and Connectivity](/about-netbird/understanding-nat-and-connectivity).
|
||||
|
||||
**Signal — the messenger.** Peers exchange their candidate addresses through the Signal service (`signal.netbird.io`, TCP/443). If Signal is unreachable, the peers can't even compare notes, and the connection silently lands on the relay.
|
||||
**Signal, the messenger.** Peers exchange their candidate addresses through the Signal service (`signal.netbird.io`, TCP/443). If Signal is unreachable, the peers can't even compare notes, and the connection silently lands on the relay.
|
||||
|
||||
**STUN — the mirror.** A peer discovers its own public address by asking a STUN server (`stun.netbird.io`, UDP 80, 443, 3478, 5555). If outbound UDP to STUN is blocked, the peer never learns a public candidate and hole punching never starts.
|
||||
**STUN, the mirror.** A peer discovers its own public address by asking a STUN server (`stun.netbird.io`, UDP 80, 443, 3478, 5555). If outbound UDP to STUN is blocked, the peer never learns a public candidate and hole punching never starts.
|
||||
|
||||
**Relay — the safety net.** When no direct path works, both peers connect outbound to a relay (`*.relay.netbird.io`, TCP/443) and traffic flows through it, still end-to-end encrypted.
|
||||
**Relay, the safety net.** When no direct path works, both peers connect outbound to a relay (`*.relay.netbird.io`, TCP/443) and traffic flows through it, still end-to-end encrypted.
|
||||
|
||||
The authoritative endpoint and port list is in [Ports & Firewalls](/about-netbird/ports-and-firewalls).
|
||||
|
||||
@@ -99,7 +99,7 @@ Work through these steps in order. Each one tells you when to continue and when
|
||||
```
|
||||
Confirm it's Relayed
|
||||
│
|
||||
1. Environment triage — are BOTH peers on known-symmetric networks?
|
||||
1. Environment triage: are BOTH peers on known-symmetric networks?
|
||||
│ yes → relay is expected, stop here
|
||||
│ no / unsure
|
||||
2. Control plane reachable? (Signal + Management, TCP/443)
|
||||
@@ -112,34 +112,34 @@ Confirm it's Relayed
|
||||
relay is the designed path
|
||||
```
|
||||
|
||||
### Step 1 — Environment triage
|
||||
### Step 1: Environment triage
|
||||
|
||||
Some networks are known to defeat hole punching, no matter how clean the firewall config is. Before checking anything else, ask where each peer sits:
|
||||
|
||||
- **Mobile and cellular connections** — carriers use CGNAT, which usually behaves symmetrically.
|
||||
- **Cloud NAT gateways** (AWS NAT Gateway, GCP Cloud NAT) — symmetric by design for instances without a public IP.
|
||||
- **Enterprise firewalls in strict mode** — Cisco ASA, Palo Alto, Fortinet and similar devices often default to symmetric NAT, sometimes labeled "strict NAT" in their settings.
|
||||
- **Mobile and cellular connections**: carriers use CGNAT, which usually behaves symmetrically.
|
||||
- **Cloud NAT gateways** (AWS NAT Gateway, GCP Cloud NAT): symmetric by design for instances without a public IP.
|
||||
- **Enterprise firewalls in strict mode**: Cisco ASA, Palo Alto, Fortinet and similar devices often default to symmetric NAT, sometimes labeled "strict NAT" in their settings.
|
||||
|
||||
If **both** peers sit on networks like these, hole punching can't succeed and no amount of firewall tuning will change that — the relay is the expected outcome, and you can stop here (see [when relay is the right answer](#when-relay-is-the-right-answer)). If only one side does, or you're not sure, keep going: one predictable side is usually enough for P2P.
|
||||
If **both** peers sit on networks like these, hole punching can't succeed and no amount of firewall tuning will change that. The relay is the expected outcome, and you can stop here (see [when relay is the right answer](#when-relay-is-the-right-answer)). If only one side does, or you're not sure, keep going: one predictable side is usually enough for P2P.
|
||||
|
||||
<Note>
|
||||
A CGNAT tell: the public address your network presents is in `100.64.0.0/10`, a range reserved for carrier-grade NAT. Don't confuse it with your own NetBird IP — NetBird intentionally uses the same range for its overlay network, so only the address your *ISP-facing* connection shows counts.
|
||||
A CGNAT tell: the public address your network presents is in `100.64.0.0/10`, a range reserved for carrier-grade NAT. Don't confuse it with your own NetBird IP. NetBird intentionally uses the same range for its overlay network, so only the address your *ISP-facing* connection shows counts.
|
||||
</Note>
|
||||
|
||||
### Step 2 — Is the control plane reachable?
|
||||
### Step 2: Is the control plane reachable?
|
||||
|
||||
If a peer can't reach the Signal service, candidates are never exchanged and the connection goes straight to relay — a common silent cause. From the peer, confirm outbound TCP/443 to both control-plane endpoints:
|
||||
If a peer can't reach the Signal service, candidates are never exchanged and the connection goes straight to relay, a common silent cause. From the peer, confirm outbound TCP/443 to both control-plane endpoints:
|
||||
|
||||
```bash
|
||||
curl -sf https://api.netbird.io/api > /dev/null && echo "management: OK"
|
||||
nc -zv signal.netbird.io 443
|
||||
```
|
||||
|
||||
Both must succeed. If they don't, fix outbound TCP/443 to these endpoints first — nothing else matters until the peers can talk to the control plane.
|
||||
Both must succeed. If they don't, fix outbound TCP/443 to these endpoints first, nothing else matters until the peers can talk to the control plane.
|
||||
|
||||
### Step 3 — Is STUN reachable?
|
||||
### Step 3: Is STUN reachable?
|
||||
|
||||
Hole punching starts with STUN, and STUN runs over UDP. The best evidence is already in `netbird status -d` — the `Relays:` section near the bottom reports reachability of every STUN, TURN, and relay endpoint:
|
||||
Hole punching starts with STUN, and STUN runs over UDP. The best evidence is already in `netbird status -d`. The `Relays:` section near the bottom reports reachability of every STUN, TURN, and relay endpoint:
|
||||
|
||||
```
|
||||
Relays:
|
||||
@@ -148,19 +148,19 @@ Relays:
|
||||
[rels://us-nyc-2.relay.netbird.io:443] is Available
|
||||
```
|
||||
|
||||
Any `Unavailable` entry for a `stun:` or `turn:` endpoint means outbound UDP is being dropped on the path — typically by the site's egress firewall. Ask whoever runs it to allow outbound UDP on ports 80, 443, 3478, and 5555 to `stun.netbird.io` and `turn.netbird.io`; the exact list and example rules are in [Ports & Firewalls](/about-netbird/ports-and-firewalls#outgoing-ports).
|
||||
Any `Unavailable` entry for a `stun:` or `turn:` endpoint means outbound UDP is being dropped on the path, typically by the site's egress firewall. Ask whoever runs it to allow outbound UDP on ports 80, 443, 3478, and 5555 to `stun.netbird.io` and `turn.netbird.io`; the exact list and example rules are in [Ports & Firewalls](/about-netbird/ports-and-firewalls#outgoing-ports).
|
||||
|
||||
<Warning>
|
||||
Every fix on this page is an **outbound** firewall rule or an allowance on the host's `wt0` interface. NetBird never needs an inbound port opened on your perimeter firewall.
|
||||
</Warning>
|
||||
|
||||
### Step 4 — Is a host firewall in the way?
|
||||
### Step 4: Is a host firewall in the way?
|
||||
|
||||
The host's own firewall or security software can block UDP before it ever leaves the machine. Telltale symptoms: peers show `Connected` but can't be pinged, or two peers on the *same office LAN* connect relayed because a host firewall drops their unsolicited direct packets. Both cases, with platform-specific checks and fixes for UFW, firewalld, and Windows Firewall, are covered in [Ports & Firewalls — Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls).
|
||||
The host's own firewall or security software can block UDP before it ever leaves the machine. Telltale symptoms: peers show `Connected` but can't be pinged, or two peers on the *same office LAN* connect relayed because a host firewall drops their unsolicited direct packets. Both cases, with platform-specific checks and fixes for UFW, firewalld, and Windows Firewall, are covered in [Ports & Firewalls: Host-based firewalls](/about-netbird/ports-and-firewalls#host-based-firewalls).
|
||||
|
||||
Endpoint protection software (CrowdStrike, ESET, Sophos and similar) often ships its own firewall that overrides OS rules — if connectivity works with it temporarily disabled, add an exception for the NetBird process.
|
||||
Endpoint protection software (CrowdStrike, ESET, Sophos and similar) often ships its own firewall that overrides OS rules, if connectivity works with it temporarily disabled, add an exception for the NetBird process.
|
||||
|
||||
### Step 5 — Repeat on the other peer
|
||||
### Step 5: Repeat on the other peer
|
||||
|
||||
A connection has two ends, and **both** must pass steps 2–4 for hole punching to work. A perfectly clean laptop still gets a relayed connection if the server's egress firewall silently drops UDP. Run the same checks on the second peer before drawing any conclusion.
|
||||
|
||||
@@ -168,9 +168,9 @@ A connection has two ends, and **both** must pass steps 2–4 for hole punching
|
||||
Can't shell into the far peer? Administrators can trigger a debug bundle remotely from the dashboard (**Peers → select peer → Run Remote Job**). See [remote debug bundle generation](/help/troubleshooting-client#remote-debug-bundle-generation).
|
||||
</Note>
|
||||
|
||||
### Step 6 — Conclude, or escalate
|
||||
### Step 6: Conclude, or escalate
|
||||
|
||||
If every check passes on both peers and the connection is still relayed, you've proven by elimination that a symmetric NAT is in the path. Accept the relay — it's the [designed behavior for exactly this case](#when-relay-is-the-right-answer), and it costs latency, not security.
|
||||
If every check passes on both peers and the connection is still relayed, you've proven by elimination that a symmetric NAT is in the path. Accept the relay. It's the [designed behavior for exactly this case](#when-relay-is-the-right-answer), and it costs latency, not security.
|
||||
|
||||
If instead something looks wrong but you can't place it, collect evidence and escalate:
|
||||
|
||||
@@ -182,8 +182,8 @@ If instead something looks wrong but you can't place it, collect evidence and es
|
||||
|
||||
A remote engineer's laptop reaches `build-server` in the office, but `netbird status -d` on the laptop shows the connection is relayed and latency is poor. Working the flow:
|
||||
|
||||
1. **Confirm.** The laptop shows `Connection type: Relayed` and `ICE candidate (Local/Remote): srflx/relay`. The laptop's own side reached STUN fine (`srflx`) — the weak side is the server.
|
||||
2. **Triage.** The laptop is on home fiber, the server on the office LAN. Neither is mobile, CGNAT, or behind a cloud NAT gateway — so this should be fixable. Continue.
|
||||
1. **Confirm.** The laptop shows `Connection type: Relayed` and `ICE candidate (Local/Remote): srflx/relay`. The laptop's own side reached STUN fine (`srflx`); the weak side is the server.
|
||||
2. **Triage.** The laptop is on home fiber, the server on the office LAN. Neither is mobile, CGNAT, or behind a cloud NAT gateway, so this should be fixable. Continue.
|
||||
3. **Control plane, on the server.** `curl` to the Management API and `nc -zv signal.netbird.io 443` both succeed.
|
||||
4. **STUN, on the server.** The `Relays:` section shows `[stun:stun.netbird.io:3478] is Unavailable, reason: stun request: context deadline exceeded`. The office egress firewall is dropping outbound UDP.
|
||||
5. **Fix and verify.** IT allows outbound UDP on ports 80, 443, 3478, 5555 to `stun.netbird.io` and `turn.netbird.io`. On the server, restart the connection with `netbird down && netbird up`, then re-check from the laptop.
|
||||
@@ -195,7 +195,7 @@ A remote engineer's laptop reaches `build-server` in the office, but `netbird st
|
||||
```
|
||||
|
||||
<Success>
|
||||
Both sides now discover their public addresses, hole punching succeeds, and traffic flows directly — latency drops from ~90 ms to ~15 ms with no relay in the path.
|
||||
Both sides now discover their public addresses, hole punching succeeds, and traffic flows directly; latency drops from ~90 ms to ~15 ms with no relay in the path.
|
||||
</Success>
|
||||
|
||||
## When relay is the right answer
|
||||
@@ -204,13 +204,13 @@ Stop troubleshooting and accept the relay when:
|
||||
|
||||
- **Both peers are on mobile/CGNAT connections.** The carrier's NAT is symmetric and outside anyone's control.
|
||||
- **Corporate policy blocks outbound UDP and won't change.** Relay over TCP/443 is the designed path through such networks.
|
||||
- **A cloud NAT gateway can't be re-architected.** If the instance *can* get a public IP (an Elastic IP on AWS), that restores P2P without opening anything inbound — security groups still only need outbound rules, and the gateway's symmetric NAT drops out of the path. If it can't, relay it is.
|
||||
- **The NAT device belongs to someone else** — a hotel, a café, a customer site.
|
||||
- **A cloud NAT gateway can't be re-architected.** If the instance *can* get a public IP (an Elastic IP on AWS), that restores P2P without opening anything inbound, security groups still only need outbound rules, and the gateway's symmetric NAT drops out of the path. If it can't, relay it is.
|
||||
- **The NAT device belongs to someone else**: a hotel, a café, a customer site.
|
||||
|
||||
Some teams even prefer relayed connections in locked-down networks, because the only flows leaving the perimeter are outbound TCP/443. That's a legitimate posture: the cost is latency, never confidentiality. In all of these cases, relay is NetBird working as designed, not a fault.
|
||||
|
||||
<Note>
|
||||
Guides elsewhere sometimes suggest forwarding a UDP port to a peer to force P2P past a symmetric NAT. It can work — but it gives up NetBird's core promise that you never open an inbound port on your network. Accept the relayed connection instead; carrying traffic past unfixable NATs without exposing anything is exactly what it's for.
|
||||
Guides elsewhere sometimes suggest forwarding a UDP port to a peer to force P2P past a symmetric NAT. It can work, but it gives up NetBird's core promise that you never open an inbound port on your network. Accept the relayed connection instead; carrying traffic past unfixable NATs without exposing anything is exactly what it's for.
|
||||
</Note>
|
||||
|
||||
## Rollout checklist
|
||||
@@ -219,13 +219,13 @@ To keep a whole fleet on direct connections rather than fixing peers one at a ti
|
||||
|
||||
- **Allow outbound UDP to STUN/TURN** (`stun.netbird.io`, `turn.netbird.io`, ports 80, 443, 3478, 5555) at every site's egress firewall.
|
||||
- **Wildcard `*.relay.netbird.io` on TCP/443** so the relay fallback survives rotation of the geo-distributed relay pool.
|
||||
- **Watch the `Relays:` section** of `netbird status -d` during rollout — fix `Unavailable` entries before users report slowness.
|
||||
- **Watch the `Relays:` section** of `netbird status -d` during rollout, fix `Unavailable` entries before users report slowness.
|
||||
- **Bake the `wt0` allowance into host-firewall baselines** (UFW/firewalld/Windows images), so host firewalls never silently block decrypted traffic.
|
||||
- **Decide per site whether relay is acceptable**, and document it — a deliberate relay is fine; a surprising one costs a support ticket.
|
||||
- **Decide per site whether relay is acceptable**, and document it, a deliberate relay is fine; a surprising one costs a support ticket.
|
||||
|
||||
## Recap
|
||||
|
||||
In one breath: **NAT** is the obstacle, **Signal** is the messenger, **STUN** is the mirror, and **Relay** is the safety net. A relayed connection means hole punching failed — either because something fixable blocks Signal, STUN, or UDP (find it: control plane → STUN → host firewall → both ends), or because both peers sit behind symmetric NAT, which you prove by elimination. Every fix is an outbound rule; nothing is ever opened inbound. And when the NAT itself is the cause, the relay is doing exactly the job it was built for — keeping peers connected without exposing anything, with end-to-end encryption intact.
|
||||
In one breath: **NAT** is the obstacle, **Signal** is the messenger, **STUN** is the mirror, and **Relay** is the safety net. A relayed connection means hole punching failed, either because something fixable blocks Signal, STUN, or UDP (find it: control plane → STUN → host firewall → both ends), or because both peers sit behind symmetric NAT, which you prove by elimination. Every fix is an outbound rule; nothing is ever opened inbound. And when the NAT itself is the cause, the relay is doing exactly the job it was built for, keeping peers connected without exposing anything, with end-to-end encryption intact.
|
||||
|
||||
<Tiles
|
||||
title="Go deeper"
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
import {Note, Warning, Success} from "@/components/mdx";
|
||||
import {Tiles} from "@/components/Tiles";
|
||||
|
||||
export const description = "Your peer is Connected but a service behind a routing peer is unreachable. Find out whether NetBird failed to deliver the traffic, or the resource itself rejected it."
|
||||
|
||||
# Troubleshooting resource connectivity
|
||||
|
||||
Your peer shows `Connected` and the tunnel is healthy, yet a service behind a routing peer is out of reach: RDP won't open, a web app times out, a database refuses the connection. This page teaches you to find out *where* the traffic stops, and in particular to tell apart the two cases that look identical from the client. Either NetBird isn't delivering the packets, or NetBird delivers them and the resource itself rejects the session.
|
||||
|
||||
If the connection is *relayed* or slow rather than unreachable, that is a transport-layer question, not a resource one. Start from [Troubleshooting relayed connections](/help/troubleshooting-relayed-connections) instead.
|
||||
|
||||
<Note>
|
||||
NetBird's job ends at the resource's `IP:port`. Once the TCP handshake there completes, NetBird has delivered the traffic. A reset, an empty reply, or a TLS or auth error after that point lives in the application or the host, not in NetBird.
|
||||
</Note>
|
||||
|
||||
## First, confirm where it stops
|
||||
|
||||
On the client, check the routing peer and the route in detail:
|
||||
|
||||
```bash
|
||||
netbird status -d
|
||||
```
|
||||
|
||||
Find the routing peer, confirm the tunnel is healthy, and check that the resource address falls inside a range the peer actually routes:
|
||||
|
||||
```
|
||||
routing-peer-1.netbird.cloud:
|
||||
NetBird IP: 100.92.0.12
|
||||
Status: Connected
|
||||
-- detail --
|
||||
Connection type: P2P
|
||||
Last WireGuard handshake: 12 seconds ago
|
||||
Networks: 10.0.50.0/24
|
||||
```
|
||||
|
||||
| Field | What it tells you |
|
||||
|---|---|
|
||||
| `Status: Connected` | The tunnel to the routing peer is up, so the problem is past the peer |
|
||||
| `Connection type` | If `Relayed`, fix transport first (see the relayed-connections page) |
|
||||
| `Networks` | The ranges this peer routes; your resource's address must fall inside one |
|
||||
| `Last WireGuard handshake` | A recent handshake means the tunnel itself is healthy |
|
||||
|
||||
If the peer is not `Connected`, stop here. That is a transport problem, not a resource one, so start from the [general client troubleshooting page](/help/troubleshooting-client#net-bird-client-status).
|
||||
|
||||
## The mental model
|
||||
|
||||
If you remember one thing, remember this: **NetBird carries traffic through the tunnel to the resource's `IP:port`, and nothing past it.** Your job is to find out which of two worlds you are in:
|
||||
|
||||
```
|
||||
Can the client reach the resource?
|
||||
│
|
||||
┌───────────────────┴───────────────────┐
|
||||
│ │
|
||||
NetBird isn't delivering NetBird delivered fine
|
||||
(peer, DNS, route, ACL, or (the handshake to IP:port
|
||||
forwarding on the peer; find completed; a reset or error
|
||||
it and fix it) after that is the app or host)
|
||||
```
|
||||
|
||||
The dividing line is the **TCP handshake to the resource's `IP:port`**. Before it completes, the problem is somewhere in NetBird's path. Once it completes, NetBird has done its job, and a reset, an immediate close, an empty response, a TLS or cert error, an auth failure, or a wrong upstream target lives in the application or the host. You work this by **elimination**, top to bottom: confirm the peer is connected, that the name resolves, that a route and policy exist, that the peer forwards the traffic, and finally where the handshake succeeds or dies.
|
||||
|
||||
## The five layers
|
||||
|
||||
Five things sit between the client and the resource. Each one maps to exactly one check in the flow below.
|
||||
|
||||
**Peer connectivity.** The client needs a healthy tunnel to the routing peer. If it doesn't, nothing downstream matters, and if the link is merely *relayed*, that belongs on a separate page.
|
||||
|
||||
**DNS.** For a domain resource, the name resolves **on the routing peer**, using the peer's own resolver. The peer has to be able to resolve it. See [DNS Troubleshooting](/manage/dns/troubleshooting).
|
||||
|
||||
**Route and ACL.** The resolved address has to be installed as a route on the client, and a policy has to permit the exact **protocol and port**. Policies are per-protocol, so a TCP-only policy silently drops the UDP half of a protocol that uses both. See [Access Control](/manage/access-control) and the forward-chain vs input-chain note in [Networks](/manage/networks).
|
||||
|
||||
**Forwarding on the peer.** The routing peer has to put the packet onto its LAN toward the resource. One subtlety causes most of these cases. If the resource resolves to the **routing peer's own IP** (a service running on the peer itself), that is the *input chain*, not the *forward chain*. It needs a peer-to-peer policy, and for userspace peers it also needs `NB_ENABLE_LOCAL_FORWARDING=true` (see [Environment Variables](/client/environment-variables)). A subnet or host resource on a *different* LAN machine uses the forward chain and masquerade instead.
|
||||
|
||||
**The handoff.** Once a TCP handshake to `IP:port` completes, NetBird is done.
|
||||
|
||||
## Reading where the handshake dies
|
||||
|
||||
The fastest way to assign ownership is to capture the attempt and read the result:
|
||||
|
||||
| What you see | Owner |
|
||||
|---|---|
|
||||
| Peer not `Connected`, stale WireGuard handshake | NetBird transport (see relayed-connections page) |
|
||||
| Resource resolves to the wrong or empty IP | NetBird DNS, or the peer's upstream resolver |
|
||||
| Route or allowed-IP missing for the resolved address | NetBird route |
|
||||
| Right port blocked, or the UDP half dropped | NetBird ACL (protocol and port) |
|
||||
| `SYN` leaves the peer's LAN NIC, no `SYN-ACK`, fails on-LAN too | Resource host, service, or its firewall |
|
||||
| Handshake completes, then `RST`, `FIN`, empty, auth, or cert error | Application or gateway config |
|
||||
| Works from the resource's own LAN, fails only via the tunnel | NetBird forwarding (escalate) |
|
||||
|
||||
## The decision flow
|
||||
|
||||
Work through these steps in order. Each one tells you when to continue and when to stop.
|
||||
|
||||
```
|
||||
Peer Connected? ── no ──▶ fix transport (relayed-connections page)
|
||||
│ yes
|
||||
▼
|
||||
1. Does the name resolve to the expected IP? (on the peer, for domain resources)
|
||||
2. Is a route installed, and does a policy allow this protocol and port?
|
||||
3. Does the peer forward it? (forward chain vs self-targeted local forwarding)
|
||||
4. Does the handshake to the resource IP:port complete?
|
||||
├─ yes ──▶ NetBird is done; investigate the application or host
|
||||
└─ no ──▶ run the LAN bypass test:
|
||||
├─ fails on-LAN too ──▶ not NetBird (service, host, or firewall)
|
||||
└─ works on-LAN ──▶ NetBird forwarding; escalate with captures
|
||||
```
|
||||
|
||||
<Note>
|
||||
For a hands-on, command-by-command version of these checks, with a worked example tracing a client to an internal server, see [Debugging access to network resources](/help/troubleshooting-client#debugging-access-to-network-resources).
|
||||
</Note>
|
||||
|
||||
### Step 1: Does the name resolve?
|
||||
|
||||
For a domain resource, the name has to resolve to the expected address **on the routing peer**, because the peer is what resolves and forwards it. A name that resolves on your client but not on the peer, or that resolves to a stale or empty address, sends the traffic nowhere. Confirm the resolved address, then check it against the `Networks` ranges from the status output above. For resolver and routing problems, see [DNS Troubleshooting](/manage/dns/troubleshooting).
|
||||
|
||||
### Step 2: Is there a route and a matching policy?
|
||||
|
||||
The resolved address has to be installed as a route on the client, and a policy has to allow the **exact protocol and port**. A policy that allows TCP but not UDP silently drops the UDP half of a protocol that needs both, which looks like a partial outage.
|
||||
|
||||
<Note>
|
||||
`netbird debug trace` simulates a packet through the firewall rules without sending real traffic, so you can confirm an ACL verdict directly. For example: `netbird debug trace in 100.64.1.1 self -p tcp --dport 3389`. It is most useful on macOS and Windows. See [tracing firewall rules](/help/troubleshooting-client#tracing-firewall-rules).
|
||||
</Note>
|
||||
|
||||
### Step 3: Does the peer forward it?
|
||||
|
||||
The routing peer has to put the packet onto its LAN. The common trap is a resource that resolves to the routing peer's own IP: that path is the *input chain*, not the *forward chain*. It needs a peer-to-peer policy, and on userspace peers it also needs `NB_ENABLE_LOCAL_FORWARDING=true` ([Environment Variables](/client/environment-variables)). A resource on a different LAN machine uses the forward chain and masquerade, and is covered by the forward-chain vs input-chain note in [Networks](/manage/networks).
|
||||
|
||||
### Step 4: Where does the handshake die?
|
||||
|
||||
This is the dividing line. Capture the connection attempt and watch the TCP handshake to the resource's `IP:port`.
|
||||
|
||||
- If the handshake **completes**, NetBird delivered the traffic. A reset, an empty reply, or a TLS or auth error after that belongs to the application or host.
|
||||
- If there is **no `SYN-ACK`**, run the LAN bypass test below to decide whether NetBird or the resource owns it.
|
||||
|
||||
<Note>
|
||||
The LAN bypass test: connect to the resource's `IP:port` from another host on its own subnet, outside NetBird entirely. If it fails there too, it was never NetBird. If it works there but not through the tunnel, the problem is NetBird forwarding, so escalate.
|
||||
</Note>
|
||||
|
||||
### Step 5: Conclude, or escalate
|
||||
|
||||
If the handshake completes, or the resource fails on its own LAN too, the answer is in the application or host, not in NetBird. If the resource is reachable on its own LAN but never through the tunnel, you have a NetBird forwarding problem worth escalating. Collect evidence before you do:
|
||||
|
||||
- A [debug bundle](/help/troubleshooting-client#debug-bundle) from the client and, where possible, the routing peer: `netbird debug bundle --system-info`
|
||||
- The `netbird status -d` output showing the peer and its `Networks`
|
||||
- A packet capture from the routing peer's LAN NIC showing the `SYN` leaving and no `SYN-ACK` returning
|
||||
|
||||
## Walkthrough: a gateway that closes the connection
|
||||
|
||||
A client reaches a Windows routing peer running a remote-desktop gateway, but the RDP session never opens. The mirror setup on an adjacent gateway works with an identical policy. Working the flow:
|
||||
|
||||
1. **Peer and DNS.** The peer is `Connected`, and the resource resolves to the peer's own LAN IP, so the gateway runs *on* the routing peer.
|
||||
2. **Route and ACL.** The resolved `/32` is installed on the client and the policy permits the gateway's ports. `netbird debug trace` confirms ACCEPT.
|
||||
3. **Forwarding.** `NB_ENABLE_LOCAL_FORWARDING=true` is set, matching the working mirror, so local delivery is enabled.
|
||||
4. **The handshake.** A client-side capture shows the TCP handshake to the gateway `IP:port` completing, the client sending its first message, and the gateway immediately closing the connection with zero bytes returned.
|
||||
|
||||
The handshake completed, so NetBird delivered the traffic correctly. The gateway itself rejected the session.
|
||||
|
||||
<Success>
|
||||
Root cause: an incorrect upstream destination was registered on the gateway. Nothing in the NetBird path was at fault, which is why an identical policy on the mirror behaved differently.
|
||||
</Success>
|
||||
|
||||
## When it's not NetBird
|
||||
|
||||
Stop looking at NetBird and investigate the resource side when:
|
||||
|
||||
- **The handshake to `IP:port` completes** but the application resets, closes, returns nothing, or errors on TLS or auth. That is the service.
|
||||
- **The LAN bypass test fails too.** The service, its bind address, or the host firewall is the problem, independent of NetBird.
|
||||
- **A proxy or gateway on the routing peer points at a wrong or dead upstream.** That is its own configuration, not the tunnel.
|
||||
|
||||
NetBird supports the overlay: transport, routing, ACLs, and DNS. The proxies, firewalls, and applications behind your resources sit outside it, and are almost always verifiable locally on the resource side. The [Reverse Proxy troubleshooting](/manage/reverse-proxy/troubleshooting) page states the same boundary for proxy targets.
|
||||
|
||||
## Checklist
|
||||
|
||||
To resolve these quickly and avoid surprises across a fleet:
|
||||
|
||||
- **Confirm the peer is `Connected` and not `Relayed`** before looking any further downstream.
|
||||
- **Resolve the name on the routing peer**, not only on the client, for domain resources.
|
||||
- **Match the policy to the exact protocol and port**, and remember that TCP-only policies drop the UDP half.
|
||||
- **Know whether the resource is self-targeted** (input chain, local forwarding) or on a separate LAN machine (forward chain, masquerade).
|
||||
- **Use the handshake as the verdict.** If it completes, hand the issue to the application or host owner with the capture attached.
|
||||
|
||||
## Recap
|
||||
|
||||
In one breath: **NetBird carries traffic to the resource's `IP:port`, and the handshake there is the dividing line.** Work top to bottom: peer connected, then name resolves, then route and policy exist, then the peer forwards it, then the handshake. If the handshake never completes and the resource is reachable on its own LAN, it is a NetBird delivery problem worth escalating. If the handshake completes, or the resource fails on its own LAN too, the answer is in the application or host, and that is where to look.
|
||||
|
||||
<Tiles
|
||||
title="Go deeper"
|
||||
description="The layers underneath this page"
|
||||
items={[
|
||||
{
|
||||
href: '/help/troubleshooting-relayed-connections',
|
||||
name: 'Troubleshooting relayed connections',
|
||||
description: 'The transport-layer sibling: why a connection is relayed instead of direct.',
|
||||
},
|
||||
{
|
||||
href: '/manage/dns/troubleshooting',
|
||||
name: 'DNS Troubleshooting',
|
||||
description: 'Name resolution on the routing peer and management DNS configuration.',
|
||||
},
|
||||
{
|
||||
href: '/manage/networks',
|
||||
name: 'Networks',
|
||||
description: 'Routing peers, and the forward chain vs input chain distinction.',
|
||||
},
|
||||
{
|
||||
href: '/manage/reverse-proxy/troubleshooting',
|
||||
name: 'Reverse Proxy troubleshooting',
|
||||
description: 'The demarcation line for proxy and gateway targets behind a peer.',
|
||||
},
|
||||
{
|
||||
href: '/client/environment-variables',
|
||||
name: 'Environment Variables',
|
||||
description: 'NB_ENABLE_LOCAL_FORWARDING and other client daemon settings.',
|
||||
},
|
||||
{
|
||||
href: '/help/troubleshooting-client',
|
||||
name: 'Troubleshooting client issues',
|
||||
description: 'Debug trace, debug bundles, and log levels on the client.',
|
||||
},
|
||||
]}
|
||||
/>
|
||||
@@ -0,0 +1,96 @@
|
||||
import { TroubleshootingTiles } from "@/components/TroubleshootingTiles"
|
||||
import { StillStuck } from "@/components/StillStuck"
|
||||
|
||||
export const description =
|
||||
"Start here to troubleshoot NetBird. Find your issue by area and jump straight to the relevant guide."
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
Something not working as expected? Pick the area closest to your problem and jump to the
|
||||
guide and section that covers it. Each section links to troubleshooting docs that live
|
||||
next to the feature they cover, so nothing here is a copy.
|
||||
|
||||
<TroubleshootingTiles
|
||||
title="Find your issue by area"
|
||||
id="find-your-issue-by-area"
|
||||
description="Pick the surface you are working with. Links open the existing troubleshooting docs in place."
|
||||
items={[
|
||||
{
|
||||
title: "NetBird Client",
|
||||
href: "/help/troubleshooting-client",
|
||||
icon: "laptop",
|
||||
description:
|
||||
"Desktop, CLI, and mobile client problems on a peer machine.",
|
||||
chips: [
|
||||
{ label: "Connection issues", href: "/help/troubleshooting-client#debugging-ice-connections" },
|
||||
{ label: "Login & SSO", href: "/help/troubleshooting-client#client-login-failures" },
|
||||
{ label: "DNS resolution", href: "/manage/dns/troubleshooting" },
|
||||
{ label: "Routes & exit nodes", href: "/help/troubleshooting-client#debugging-access-to-network-resources" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "Self-hosted",
|
||||
href: "/selfhosted/troubleshooting",
|
||||
icon: "server",
|
||||
description:
|
||||
"Your own Management, Signal, Relay, and Dashboard services.",
|
||||
chips: [
|
||||
{ label: "Management service", href: "/selfhosted/troubleshooting/connectivity#management-service-unreachable" },
|
||||
{ label: "Signal service", href: "/about-netbird/how-netbird-works#signal-service" },
|
||||
{ label: "Relay / TURN", href: "/selfhosted/troubleshooting/connectivity#debugging-turn-connections" },
|
||||
{ label: "Dashboard", href: "/selfhosted/troubleshooting/dashboard" },
|
||||
{ label: "Reverse proxy", href: "/manage/reverse-proxy/troubleshooting" },
|
||||
],
|
||||
},
|
||||
{
|
||||
// No dedicated troubleshooting page, so the title is plain text;
|
||||
// chips point to the relevant management docs.
|
||||
title: "Cloud & identity",
|
||||
icon: "cloud",
|
||||
description:
|
||||
"SSO and user provisioning (Cloud and self-hosted), account approvals, and Cloud plans and quotas.",
|
||||
chips: [
|
||||
{ label: "Pending approval", href: "/help/troubleshooting-account-access" },
|
||||
{ label: "IdP & SSO setup", href: "/manage/team/single-sign-on" },
|
||||
{ label: "User provisioning", href: "/manage/team/idp-sync" },
|
||||
{ label: "Plan limits & quotas", href: "/manage/settings/plans-and-billing" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "Connectivity & networking",
|
||||
href: "/help/troubleshooting-relayed-connections",
|
||||
icon: "firewall",
|
||||
description:
|
||||
"Cross-cutting network issues regardless of how you deploy.",
|
||||
chips: [
|
||||
{ label: "Relayed connections", href: "/help/troubleshooting-relayed-connections" },
|
||||
{ label: "Resource connectivity", href: "/help/troubleshooting-resource-connectivity" },
|
||||
{ label: "NAT & firewall ports", href: "/about-netbird/ports-and-firewalls" },
|
||||
{ label: "DNS", href: "/manage/dns/troubleshooting" },
|
||||
{ label: "Network routes", href: "/manage/network-routes" },
|
||||
],
|
||||
},
|
||||
{
|
||||
title: "Access control",
|
||||
href: "/manage/access-control",
|
||||
icon: "shield",
|
||||
description:
|
||||
"Traffic being allowed or denied unexpectedly.",
|
||||
chips: [
|
||||
{ label: "Policies", href: "/manage/access-control" },
|
||||
{ label: "Posture checks", href: "/manage/access-control/posture-checks" },
|
||||
{ label: "Groups", href: "/manage/access-control#understanding-groups" },
|
||||
],
|
||||
},
|
||||
]}
|
||||
/>
|
||||
|
||||
<StillStuck
|
||||
title="Still stuck?"
|
||||
description="Bring your debug bundle. The community and the team can help from there."
|
||||
separator="or"
|
||||
actions={[
|
||||
{ label: "Contact our Community", href: "/help/community-support", primary: true },
|
||||
{ label: "Contact NetBird Support", href: "/help/netbird-support", primary: true },
|
||||
]}
|
||||
/>
|
||||
Reference in New Issue
Block a user