sendnrw/netbird-docs
2
0
Fork 0
mirror of https://github.com/netbirdio/docs.git synced 2026-04-16 15:36:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0080ae97df655efdc748eeb42bebf640cde30987
netbird-docs/src/pages/how-to/troubleshooting-client.mdx
Brandon Hopkins 0080ae97df Restructuring Phase 3 (#492)
2025-11-24 18:25:44 +01:00

1021 lines
39 KiB
Plaintext
Raw Blame History

# Troubleshooting client issues
This document offers practical tips and insights to help you debug various problems, ensuring a seamless user
experience.
## NetBird agent status
The netbird agent is a daemon service that runs in the background; it provides information about peers connected and
about the NetBird control services. You can check the status of the agent with the following command:
```shell
netbird status --detail
````
This will output the following information:
```shell
Peers detail:
server-a.netbird.cloud:
NetBird IP: 100.75.232.118/32
Public key: kndklnsakldvnsld+XeRF4CLr/lcNF+DSdkd/t0nZHDqmE=
Status: Connected
-- detail --
Connection type: P2P
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 10.128.0.35:51820/10.128.0.54:51820
Last connection update: 20 seconds ago
Last Wireguard handshake: 19 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
Quantum resistance: false
Routes: 10.0.0.0/24
Latency: 37.503682ms
server-b.netbird.cloud:
NetBird IP: 100.75.226.48/32
Public key: Mi6jtrK5Tokndklnsakldvnsld+XeRF4CLr/lcNF+DSdkd=
Status: Connected
-- detail --
Connection type: Relayed
Direct: false
ICE candidate (Local/Remote): relay/host
ICE candidate endpoints (Local/Remote): 108.54.10.33:60434/10.128.0.12:51820
Last connection update: 20 seconds ago
Last Wireguard handshake: 18 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
Quantum resistance: false
Routes: -
Latency: 37.503682ms
OS: darwin/amd64
Daemon version: 0.27.4
CLI version: 0.27.4
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
[stun:turn.netbird.io:5555] is Available
[turns:turn.netbird.io:443?transport=tcp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: maycons-mbp-2.netbird.cloud
NetBird IP: 100.75.143.239/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 2/2 Connected
```
As you can see, the output shows the peers connected, the NetBird IP address, the public key, the connection status, and
the connection type. The status will also report if there is an issue connecting to the relay servers,
the management server, or the signal server.
As for Peers, the status will show the following information:
* `Connection type`: P2P, Relayed, where relayed connections indicate a limitation in the network that prevents a direct
connection between the peers.
* `Direct`: true/false, where true indicates a direct connection between the peers without a local proxy. This case is
common when the local peer is allocating the relay connection.
* `ICE candidate (Local/Remote)`: relay/host, where relay indicates that the local peer is using a relay connection and
host indicates that the remote peer is using a direct connection.
* `Last Wireguard handshake`: Indicating the last time the Wireguard handshake was performed. Usually, this is performed
every 2 minutes, and if you don\'t see an update here or if the value is empty, that indicates that the connection
wasn\'t possible yet.
* `Transfer status (received/sent)`: Indicating the amount of data received and sent by the peer. This is useful to
check if the connection is being used.
See more details about the status command [here](/get-started/cli#status).
## Getting client logs
By default, client logs are located in the `/var/log/netbird/client.log` file on macOS and Linux and in the
`C:\ProgramData\netbird\client.log` file on Windows.
You can analyze the logs to identify the root cause of the problem. If you need help, open
a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and attach the logs.
### Debug bundle
A debug archive containing the recent logs and the status at the time of execution can be generated with the following
command.
Adding the `--anonymize (-A)` flag will anonymize the logs, removing sensitive information such as public IP addresses
and domain
names. In case you have tunneling issues, omitting the `--anonymize` flag might help our analysis.
Adding the `--system-info (-S)` flag will add system information like network routes and interfaces
```shell
netbird debug bundle --anonymize --system-info
```
This will output the path of the generated file. The output file is owned by and can only be accessed by the user
NetBird is running as, by default it is: `Administrator` on Windows, `root` on MacOS/Linux or the operating system\'s
equivalent.
### Debug for a specific time
To capture logs for a specific time period, you can use the `debug for` command. This will generate a debug bundle after
the specified time has elapsed.
```shell
netbird debug for 5m --system-info
```
<Note>
The flag `--anonymize (-A)` can be used to anonymize IP addresses and non-netbird.io domains in logs and status output when needed.
</Note>
To capture any issues arising during the `up` and `down` processes, this will set the log level to `TRACE` and bring
netbird `up` and `down` up to a few times.
After 5 minutes the netbird status will be restored to the previous state and the debug bundle will be generated.
### Debug bundle uploads
Since version `0.43.1`, you can share debug bundle with the NetBird development team without local administrative
privileges
by using the `--upload-bundle (-U)` flag.
It will securely generate and upload the debug bundle to our servers for access by the NetBird development team. See
examples below:
Run debug for a specific time and upload the bundle:
```shell
netbird debug for 1m --system-info --upload-bundle
```
To generate a bundle without restarting the client and then uploading:
```shell
netbird debug bundle --system-info --upload-bundle
```
This will output an `Upload file key`, which is effectively a random filename in our internal storage system
and can be safely shared with us through public channels such as GitHub Issues or Slack.
```shell
netbird debug bundle --system-info --upload-bundle
Local file:
/tmp/netbird.debug.2611377582.zip
Upload file key:
1234567890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/12345678-90ab-cdef-1234-567890abcdef
```
<Note>
The flag `--anonymize` can be used to anonymize IP addresses and non-netbird.io domains in logs and status output when needed.
</Note>
### Debug bundle uploads with GUI
Since version `0.43.2` users can upload their debug bundle via the GUI client.
To generate a bundle via GUI, you can access the application then go to `Settings` > `Create Debug Bundle` and follow
the wizard to upload the bundle:
<p>
<img src="/docs-static/img/troubleshooting-client/ui-settings.png" alt="service-user-overview" className="imagewrapper-big"/>
</p>
<Note>
If needed, you can update the upload URL and select to anonymize sensitive information like IP addresses and non-netbird.io domains in logs and status output.
</Note>
<p>
<img src="/docs-static/img/troubleshooting-client/ui-bundle-wizard.png" alt="service-user-overview" className="imagewrapper-big"/>
</p>
By default running with trace log enable before generating the bundle is selected. This will restart the client connections and provide a `disconnect to connected` information for our engineers.
If you uncheck this option, a bundle will be generated without running this step. Which is very useful when you have an
issue that recovers when restarting the client.
<p>
<img src="/docs-static/img/troubleshooting-client/ui-bundle-success.png" alt="service-user-overview" className="imagewrapper-big"/>
</p>
Once the bundle generation is complete, you can click on `Copy Key` to get the uploaded key and share with NetBird\'s team.
## Enabling debug logs on agent
Logs can be temporarily set using the following command.
```shell
netbird debug log level debug
```
or
```shell
netbird debug log level trace
```
The next time the daemon is restarted, the log level will return to the configured level.
Using `netbird down` and `netbird up` will not reset the log level.
To permanently set the log level, see the following sections.
<Note>
The default logging level is `info`. To revert back to the original state, you can repeat the procedure with `info` instead of `debug` or `trace`.
</Note>
### On Linux with systemd
The default systemd unit file reads a set of environment variables from the path `/etc/sysconfig/netbird`.
You can add the following line to the file to enable debug logs:
```shell
sudo mkdir -p /etc/sysconfig
echo 'NB_LOG_LEVEL=debug' | sudo tee -a /etc/sysconfig/netbird
sudo systemctl restart netbird
```
### On Other Linux and MacOS
```shell
sudo netbird service stop
sudo netbird service uninstall
sudo netbird service install --log-level debug # or trace
sudo netbird service start
```
### On Windows
You need to run the following commands with an elevated PowerShell or `cmd.exe` window.
```shell
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")
netbird service restart
```
### On Docker
You can set the environment variable `NB_LOG_LEVEL` to `debug` to enable debug logs.
```shell
docker run --rm --name PEER_NAME --hostname PEER_NAME --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -d \
-e NB_SETUP_KEY=<SETUP KEY> -e NB_LOG_LEVEL=debug -v netbird-client:/var/lib/netbird netbirdio/netbird:latest
```
### On Android
Enable the ADB in the developer menu on the Android device.
In the app set the the Trace log level setting - it is a checkbox in the advanced menu.
With the ADB tool, you can get the logs from your device. The ADB is part of the SDK platform tools pack (zip file).
You can download it from [here](https://developer.android.com/tools/releases/platform-tools).
Please extract it and run the next command in the case of Linux:
```shell
sudo adb logcat -v time | grep GoLog
```
## Running the agent in foreground mode
You can run the agent in foreground mode to see the logs in the terminal. This is useful to debugging issues with the
agent.
### Linux and MacOS
```shell
sudo netbird service stop
sudo netbird up -F
```
### Windows
On Windows, the agent depends on the Wireguard's `wintun.dll` and can only be executed as a system account.
To run the agent in foreground mode, you need to use a tool
called [PSExec](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec).
Once you have downloaded and extracted `psexec` open an elevated Powershell window:
```shell
netbird service stop
.\PsExec64.exe -s cmd.exe /c "netbird up -F --log-level debug > c:\windows\temp\netbird.out.log 2>&1"
```
In case you need to configure environment variables, you need to add them as system variables so they get picked up by
the agent on the next psexec run:
```shell
[Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")
````
## Enabling WireGuard in user space
Sometimes, you want to test NetBird running on userspace mode instead of a kernel module. That can be a check to see if
there is a problem with NetBird's firewall management in kernel mode.
You must run the agent in foreground mode and set the environment variable `NB_WG_KERNEL_DISABLED` to `true`.
```shell
sudo netbird service stop
sudo bash -c 'NB_WG_KERNEL_DISABLED=true netbird up -F' > /tmp/netbird.log
```
## Debugging GRPC
The NetBird agent communicates with the Management and Signal servers using the GRPC framework. With these parameters,
you can
set verbose logging for this service.
```shell
sudo netbird service stop
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F' > /tmp/netbird.log
```
## Debugging ICE connections
The Netbird agent communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol
described in the [RFC 8445](https://datatracker.ietf.org/doc/html/rfc8445). To debug the connection procedure,
set verbose logging for the the [Pion/ICE](https://github.com/pion/ice) library with the `PIONS_LOG_DEBUG` or
`PIONS_LOG_TRACE` variable.
```shell {{ title: 'Environment variable' }}
PIONS_LOG_DEBUG=all
NB_LOG_LEVEL=debug
```
```shell
sudo netbird service stop
sudo bash -c 'PIONS_LOG_DEBUG=all NB_LOG_LEVEL=debug netbird up -F' > /tmp/netbird.log
```
## Client login failures
A single machine can only connect to one NetBird account as the same user/login method throughout the lifetime of
the `config.json` file:
- `/var/lib/netbird/default.json` for Linux/MacOS
- `C:\ProgramData\netbird\default.json` for Windows
You might get errors like below when trying to use Setup Key/different SSO user account during login:
```
2025-04-08T15:03:04+01:00 ERRO management/client/grpc.go:351: failed to login to Management Service: rpc error: code = PermissionDenied desc = peer login has expired, please log in once more
2025-04-08T15:03:04+01:00 ERRO management/client/grpc.go:351: failed to login to Management Service: rpc error: code = PermissionDenied desc = invalid user
2025-04-08T15:03:04+01:00 ERRO client/internal/login.go:145: failed registering peer rpc error: code = PermissionDenied desc = invalid user,00000000-0000-0000-0000-000000000000
2025-04-08T15:03:04+01:00 WARN client/server/server.go:267: failed login: rpc error: code = PermissionDenied desc = invalid user
```
<Note>
Starting with the release `0.50.0` the `invalid user` message is more descriptive:
`peer is already registered by a different User or a Setup Key`
</Note>
The most notable examples of encountering the issue are:
- a shared machine and/or machine previously logged in by somebody else,
- a machine that was previously logged in using Setup Key, but now attempts SSO login,
- the user makes a mistake and selects
- the user uses different browser/profile or selects the wrong account during SSO login at the start of the workday,
If you know the exact previous Peer which was logged in, you can just delete it from Dashboard without doing anything
else and attempt login again.
Otherwise, to resolve the issue, you will need to remove the file manually to use the machine as a different user/Setup
Key while the NetBird client daemon is stopped:
1. `netbird service stop`
2. `sudo rm /var/lib/netbird/default.json` (*nix) or `rm C:\ProgramData\netbird\config.json` (Windows)
3. `netbird service start`
## Debugging access to network resources
In this section we will be presenting methodology of troubleshooting access issues involving Netbird.
We will start by presenting a glossary of all machines and services involved.
A sub-section will describe a specific use case.
Each will start with a concise summary of usual troubleshooting steps then expand into more detailed step-by-step
guides.
### Glossary
We will be using the following names for resources outside the Netbird network:
- `int-net1`: an internal network `10.123.45.0/24`,
- `srv-c`: an internal HTTP server running at `10.123.45.17`,
- `int-dns1`: an internal DNS server running at `10.123.45.6`,
- `int-dns2`: an internal DNS server nunning at `10.7.8.9`,
- `cf-dns`: an Internet-accessible CloudFlare DNS server at `1.1.1.1` and `1.0.0.1`,
and following Netbird network resources:
- `peer-a`: end user's device running Netbird Client,
- `peer-b`: a linux server inside the internal network running Netbird Client,
- it has direct access to the whole `int-net1` IP range,
- `users:employees`: a Netbird Group containing `peer-a`,
- `routers:int-net1`: a Netbird Group containing `peer-b`,
- `access:srv-c`: a Netbird Groups used as a target of ACL rules for `srv-c` only,
- `access:int-net1`: a Netbird Groups used as a target of ACL rules for the whole subnet,
- `net-a`: a Netbird Network
- `net-a:srv-c`: a Network Resource handling traffic to `10.123.45.17/32` (`srv-c`),
- `net-a:int-net1`: a Network Resource handling traffic to `10.123.45.0/24` (`int-net1`),
- `route:int-net1`: a Netbird Network Route handling traffic to `10.123.45.0/24` (`int-net1`),
- `route:srv-c`: a Netbird Network Route handling traffic to `10.123.45.17/32` (`srv-c`),
### Access from `peer-a` to `srv-c`
In short:
1. Does `peer-b` have direct access to `srv-c`'s port `80`?
2. Can a routing peer `peer-b` forward traffic to `srv-c`?
3. Are Netbird's network routing resources configured?
4. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group?
5. Is `peer-a`'s operating system configured to use the route?
Access Control rule is not required for connectivity from `peer-a` to `peer-b`
#### Does `peer-b` have direct access to `srv-c`'s port `80`?
After logging in to `peer-b` you can confirm/troubleshoot the HTTP port `80` connection by issuing any of the following
commands:
```shell
curl -v "http://10.123.45.17"
curl --fail -v --max-time=2 "http://10.123.45.17:80"
wget -O - --timeout=2 "http://10.123.45.17:80"
nc -nvz -w 2 10.123.45.17 80
```
You can also try `ping` (an ICMP packet), but the firewall might have a different configuration for ICMP and TCP ports:
```shell
ping --numeric --count=1 --timeout=2 10.123.45.17
```
#### Can a routing peer `peer-b` forward traffic to `srv-c`?
This is more complicated to test, but usually boils down to confirming `net.ipv4.ip_forward` is set to `1` on `peer-b`'s
Linux operating system:
```shell
> sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
```
It should be set up automatically by the Netbird client unless it runs inside a container (which would not be able
to modify `sysctl`), then it requires manual setup.
For setting up the value persistently (across reboots) please consult your operating system's documentation.
It is often handled by either `/etc/sysctl.conf` or `/etc/sysctl.d/*.conf` files.
Testing the functionality in practice involves:
- connecting to another machine with direct access to `peer-b`,
- adding a routing table entry to route `int-net1` (`10.123.45.0/24`) traffic through it,
- trying to at least `ping 10.123.45.17` (`srv-c`)
#### Are Netbird's network routing resources configured?
For Netbird network routing resources configurations you can use either (new) _Networks_ or (old) _Network Routes_.
A Network `net-a` should have at minimum:
- _Network Resource_: `net-a:srv-c` with either of:
- an _Address_ set to `10.123.45.17/32` to configure route to `srv-c` exclusively and nothing else,
- _Assigned Groups_ set to `access:srv-c`
- _Routing Peer Group_ assigned to `routers:int-net1`
A _Network Route_ `route:srv-c` should have at least:
- a _Network Range_ set to `10.123.45.17/32` to configure route to `srv-c` exclusively and nothing else,
- _Routing Peer Group_ assigned to `routers:int-net1`,
- _Distribution Group_ assigned to `users:employees`,
- (optional) _Access Control Groups_ assigned to `access:srv-c`,
You can loosen the rules and replace following to grant access to the whole `int-net1` network range:
- _Address_: `10.123.45.17/32` -> `10.123.45.0/24`,
- _Assigned Groups_ / _Access Control Groups_: `access:srv-c` -> `access:int-net1`
#### Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group?
You can skip this check, when you are using (old) Network Route feature without filling in _Access Control Groups (
optional)_ section.
Otherwise, there should be an _Access Control Policy_ present allowing traffic from one of `peer-a`'s Groups to:
- _Networks Resource_'s _Assigned Groups_: `access:srv-c` or `access:int-net1`,
- _Network Route_'s _Access Control Groups_: `access:srv-c` or `access:int-net1`,
You can confirm the _Policy_ is working by:
1. logging in to `peer-a`,
2. issuing `netbird status -d` command,
3. finding `peer-b.netbird.cloud` under `Peers detail`,
4. finding `10.123.45.0/24` or `10.123.45.17/32` under `peer-b.netbird.cloud`'s _Networks_ field,
In the most specific setup it should have at:
- have `TCP` protocol selected,
- a blue arrow should point from left to right and a second right-to-left arrow should be greyed out,
- a _Source group_ set to `users:employees`,
- a _Destination group_ set to `access:srv-c`,
- have `80` in the Ports section,
Just like with the previous section you can loosen the above example by:
- replacing `access:srv-c` _Group_ with `access:int-net1` _Group_,
- allowing `ALL` protocol, _Ports_ will become greyed out because all traffic will be allowed,
- creating a bidirectional rule (both arrows should be green), always true for the protocol `ALL`,
- selecting a different source group from the pool assigned to `peer-a`,
- it could be built-in `All` group, but it is discouraged,
- selecting a different destination group from the pool assigned to `peer-b`,
- it could be built-in `All` group, but it is discouraged,
#### Is `peer-a`'s operating system configured to use the route?
After all resources are configured in the Netbird management you should check whether they are
properly registered with your operating system.
You can start by checking Netbird client's configuration with `netbird status -d` command:
```shell
% netbird status -d
Peers detail:
brys-vm-nbt-ubuntu-isolated-01.netbird.cloud:
...
Status: Connected
-- detail --
Connection type: P2P
...
Networks: 10.123.45.0/24
...
Peers count: 1/1 Connected
```
You should be primarily looking for _Networks_ section under each _Peers detail_, but you can also check:
- _Peer_'s name,
- _Peer_'s _Status_: it should be `Connected`,
- _Peer_'s _Connection type_: it can be either `P2P` (direct) or `Relayed` (over the Internet),
- _Peers count_ near the end of the output,
If it's missing you can search for clues with `netbird networks ls` command:
```shell
% netbird networks ls
Available Networks:
...
- ID: net-a:int-net1
Network: 10.123.45.0/24
Status: Selected
...
```
The _Status_ could be `Not Selected`, which you can fix with `netbird networks select <ID>` or
`netbird networks select all`
##### Verifying routing configuration on the Windows operating system
Below commands assume running a PowerShell prompt with administrator's privileges.
The easiest way is to read output of `Get-NetRoute` command:
```shell
PS C:\Users\user> Get-NetRoute
ifIndex DestinationPrefix NextHop RouteMetric ifMetric PolicyStore
------- ----------------- ------- ----------- -------- -----------
...
17 10.123.45.255/32 0.0.0.0 256 5 ActiveStore
17 10.123.45.0/24 0.0.0.0 1 5 ActiveStore
...
17 100.83.255.255/32 0.0.0.0 256 5 ActiveStore
17 100.83.183.133/32 0.0.0.0 256 5 ActiveStore
17 100.83.0.0/16 0.0.0.0 256 5 ActiveStore
...
```
You should be looking for your specific subnet's IP ranges (`10.123.45.0/24` in case of `int-net1`) and anything from
`100.*.0.0/16` range.
Some other alternatives are `route print` & `Get-NetIPConfiguration`.
##### Verifying routing configuration on the MacOS operating system
The easiest way to verify system configuration is `netstat -nr` command:
```shell
% netstat -nr
Routing tables
Internet:
Destination Gateway Flags Netif Expire
...
100.83/16 utun100 USc utun100
100.83.19.63 100.83.19.63 UH utun100
...
10.123.45 utun100 USc utun100
...
Internet6:
Destination Gateway Flags Netif Expire
...
```
You should be looking for `utun*` interface in 4th column and searching the rows for
your specific subnet's clamped IP ranges (`10.123.45` in case of `int-net1`) and anything from `100.*/16` range.
##### Verifying routing configuration on the Linux operating system
Depending on specifics of your Linux distribution (or even your configuration of it) you should be able to use either
`iproute2` or `net-tools` family of network commands.
Netbird client stores it's custom routes in the routing table `7120` (or `0x1BD0`) when it's available (through
`iproute2` interface).
For `iproute2` (`ip`, `ss` tools):
- `ip route` to find built-in `100.*.0.0/16` route,
- `ip route show table 7120` or `ip route show table all` to find the specific routed networks,
For `net-tools` (`ifconfig`, `route`, `netstat` tools):
- `route -n` to find built-in `100.*.0.0/16` route,
- neither `route` nor `netstat` support viewing content of custom routing tables,
### Public nameservers
When you configure a _Nameserver_ accessible from the Internet without a VPN, the Netbird client acts as a proxy
to the public nameserver.
There are really just two things you can check:
1. Confirm Netbird client picked up the nameserver,
2. Confirm the operating system is configured to use Netbird client's proxy nameserver,
You can check the first one in operating system independent manner by:
1. running `netbird status -d`,
2. locating the _Nameserver_'s IP address
3. confirming it _is Available_ (it could also be timed out or in other state)
```
...
Nameservers:
[1.1.1.1:53, 1.0.0.1:53] for [.] is Available
...
```
#### Verifying the DNS names resolve properly in practice
Here is a short summary of commands querying nameservers for `name.at.example.com` in different operating systems.
The `.` at the end makes sure you are querying a fully-qualified names independent of your local network's configuration
(specifically search domains):
```shell
# MacOS
dscacheutil -q host -a name name.at.example.com.
# Windows PowerShell
Resolve-DnsName -Name name.at.example.com.
# Linux/UNIX
dig name.at.example.com.
nslookup name.at.example.com.
# Linux with systemd-resolved
resolvectl query name.at.example.com.
```
#### Verifying the nameservers are properly registered in Windows operating system
To confirm the nameservers are properly registered in Windows operating system using PowerShell:
```shell
PS C:\Users\kdn> Get-DnsClientNrptRule
Name : NetBird-Match
Version : 2
Namespace : {.netbird.cloud, .83.100.in-addr.arpa}
...
NameServers : 100.83.255.254
...
PS C:\Users\kdn> Get-DnsClientNrptPolicy
Namespace : .83.100.in-addr.arpa
...
NameServers : 100.83.255.254
...
Namespace : .netbird.cloud
...
NameServers : 100.83.255.254
...
PS C:\Users\kdn> ipconfig /all
...
Unknown adapter wt0:
Connection-specific DNS Suffix . : netbird.cloud
Description . . . . . . . . . . . : WireGuard Tunnel
...
Connection-specific DNS Suffix Search List :
netbird.cloud
83.100.in-addr.arpa
...
```
You should be searching for following in the outputs of above commands:
- the `100.XXX.255.254` under _Nameservers_ (a local proxy address of the Netbird client)
- `.netbird.cloud` and `.XXX.100.in-addr.arpa` under matching _Namespace_ for built-in entries,
- `.your.custom.domain.example.com` under matching _Namespace_ for your custom domains,
#### Verifying the nameservers are properly registered in MacOS operating system
To confirm the nameservers are properly registered in MacOS operating system using terminal:
```shell
> scutil --dns
...
resolver #2
domain : netbird.cloud
nameserver[0] : 100.83.255.254
port : 53
flags : Supplemental, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 101200
...
resolver #8
domain : 83.100.in-addr.arpa
nameserver[0] : 100.83.255.254
port : 53
flags : Supplemental, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 102402
...
```
You should be searching for following in the outputs of above commands:
- the `100.XXX.255.254` under _nameserver[N]_ (a local proxy address of the Netbird client)
- `netbird.cloud` and `.XXX.100.in-addr.arpa` under matching _domain_ for built-in entries,
- `.your.custom.domain.example.com` under matching _domain_ for your custom domains,
- `Reachable` under `reach` field,
##### MacOS DNS caching issues
MacOS might have cached the result from a previous attempt (since it's a public record) and keep serving those.
You can try flushing the cache to fix it using following commands:
```shell
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
```
You can validate whether this is the issue in your setup by performing following steps:
1. `netbird down` / `Disconnect`
2. flush cache (see above)
3. resolve the domain, eg: `dscacheutil -q host -a name <domain>`
4. `netbird up` / `Connect`
5. check whether `dscacheutil -q host -a name <domain>` works
- if it doesn't flush the cache and retry
#### Verifying the nameservers are properly registered in Linux operating system
Nameserver can be configured in different ways depending on your specific distribution's configuration:
For `systemd-resolved`, you can see the config with `resolvectl status`,
For other configuration backends, you should see additional entries in `/etc/resolv.conf`:
- `127.0.0.1` - default address for Netbird DNS proxy listener
- `127.0.0.153` - fallback address for Netbird DNS proxy listener
- value of `$NB_DNS_RESOLVER_ADDRESS` - a custom override for the Netbird DNS proxy listener
You can find the address Netbird client is listening by issuing one of following commands:
```shell
sudo ss -nlptu 'sport = 53' | grep netbird
sudo netstat -ltnup | grep ':53' | grep netbird
```
### Internal nameservers
When you configure an internal _Nameserver_, not accessible from the Internet in addition to steps
described in the previous section _Public nameservers_
you should make sure the _Nameserver_\'s IP addresses are properly routed and accessible.
Please refer to _Access from `peer-a` to `srv-c`_ section above.
To configure `int-dns1`, while following _Access from `peer-a` to `srv-c`_ section you should:
- substitute port `80` for port `53`
- substitute ip address `10.123.45.17` for `10.123.45.6`,
To configure `int-dns2`, while following _Access from `peer-a` to `srv-c`_ section you should:
- substitute port `80` for port `53`
- completely ignore the `10.123.45.0/24` network instructions,
- substitute ip address `10.123.45.17` for `10.7.8.9`,
- create a respective _Network_ (along with _Resources_ and _Routing Peers_) or _Network Route_ for the `10.7.8.9/32` IP
address range,
To test the configuration in practice please refer to previous section _Public nameservers_.
## Debugging access to Domain Resources
While we strive to make them "just work", there still are and will be cases of domain-based Resources not behaving
correctly. It can happen for myriad of reason starting with the client's local device management software or system
firewall, through Routing Peer issues (usually a firewall) and ending with a relatively simple Access Policies
misconfiguration and resulting lack of connectivity establishment.
This section will provide general directions for verifying connectivity on every step involved in handling
the Domain Resources, to better understand where issue might lie.
For in-depth overview of the mechanism please read [Domain Resources](/manage/networks#domain-resources) section.
Analyzing those issues will take a "backwards" approach (based on the most common issues), where we will first confirm
that Routing Peer itself is working as expected and will check the client's operating system configuration as one of the
last steps.
For the remainder of the section let's assume:
- there is a `*.nb.test` Network Resource configured,
- we are trying to access a `srv.nb.test` domain,
- a `zxc.nb.test` domain does not exist, it's used to demonstrate errors,
- the Routing Peer's NetBird address is `100.83.136.209`
- it's named `brys-vm-nbt-ubuntu-isolated-02`, when referred in the outputs
- the client is named `brys-vm-nbt-ubuntu-01`, when referred in the outputs
- the client is running Ubuntu, but a lot of commands used are working uniformly across all platforms,
- it's IP address is `100.83.73.97`,
- on MacOS & Windows you would use `100.83.255.254` to access the local DNS forwarder instead,
- the Resource is running on `brys-vm-nbt-ubuntu-isolated-01`, when referred to in the outputs
- we will only check the new port `22054`, but steps might need repeating for port `5353` for legacy clients,
<Note>
Be aware that the port `5353` is a well known Multicast DNS port (aka Avahi aka Bonjour,
used for: printer sharing, Chromecast etc.) and therefore it might be occupied by another software
running on the machine. As a result (old) Routing Peers might be prevented from routing Domain Resources.
While not an issue in the regular server operations, it might come as a surprise to find that the port `5353`
is occupied by a Chrome (and it's derivatives) Web Browser on your remotely accessible Windows Server machine.
This is the primary reason we have switched to the new port `22054`. We strongly advise you to update your fleet
to the latest version (no older than `0.59.10`) to address this issue.
</Note>
### Is Routing Peer correctly resolving queries?
While in practice it almost never the issue, it is always good to double-check whether the Routing Peer itself is able
to resolve the requested domain as-is and whether it can access the target resource.
Please refer
to [Verifying the DNS names resolve properly in practice](#verifying-the-dns-names-resolve-properly-in-practice)
section for operating-system specific commands while adjusting domain for `srv.nb.test`.
It also would not hurt to check whether the Routing Peer has an actual network access to the routed resource with:
For TCP services you should see something like this:
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ nc -vz -w 1 srv.nb.test 80
Connection to srv.nb.test (192.168.100.10) 80 port [tcp/http] succeeded!
kdn@brys-vm-nbt-ubuntu-01:~$ nc -vz -w 1 srv.nb.test 12345
nc: connect to srv.nb.test (192.168.100.10) port 12345 (tcp) failed: Connection refused
```
For UDP you can use:
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ nc -vz -w 1 -u srv.nb.test 12345 ; echo $?
Connection to srv.nb.test (192.168.100.10) 12345 port [udp/*] succeeded!
0
kdn@brys-vm-nbt-ubuntu-01:~$ nc -vz -w 1 -u srv.nb.test 12347 ; echo $?
1
```
### Is the remote DNS resolver accessible to the client?
We want to confirm that a client Peer can reach and use the Routing Peer's DNS resolver, this step will rule out any
firewall-related issues with the Routing Peer. If the following command fails you will need to open up a port `22054`
in the Routing Peer's firewall software.
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ nslookup -timeout=1 -port=22054 srv.nb.test 100.83.136.209
Server: 100.83.136.209
Address: 100.83.136.209#22054
Non-authoritative answer:
Name: srv.nb.test
Address: 192.168.100.10
kdn@brys-vm-nbt-ubuntu-01:~$ nslookup -timeout=1 -port=22054 zxc.nb.test 100.83.136.209
Server: 100.83.136.209
Address: 100.83.136.209#22054
** server can't find zxc.nb.test: NXDOMAIN
```
### Trigger the Domain Resource
I have yet to see a local DNS forwarder fail, but using it is a good way of forcing the NetBird client to set up
routing for the domain (see the [Domain Resources](/manage/networks#domain-resources) for explanation).
<Note>
On MacOS & Windows the IP address would always be `100.83.255.254` instead of `100.83.73.97`.
</Note>
Take a note of the IP addresses being initially missing from the routing table (`ip route show` on Linux), but
get added after resolving the domain for the first time using the local DNS Forwarder.
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ netbird networks ls
Available Networks:
- ID: *.nb.test
Domains: *.nb.test
Status: Selected
Resolved IPs: -
kdn@brys-vm-nbt-ubuntu-01:~$ ip route show table all | grep 192.168.100
kdn@brys-vm-nbt-ubuntu-01:~$ nslookup -timeout=1 srv.nb.test 100.83.73.97
Server: 100.83.73.97
Address: 100.83.73.97#53
Non-authoritative answer:
Name: srv.nb.test
Address: 192.168.100.10
kdn@brys-vm-nbt-ubuntu-01:~$ ip route show table all | grep 192.168.100
192.168.100.10 dev wt0 table 7120
kdn@brys-vm-nbt-ubuntu-01:~$ netbird networks ls
Available Networks:
- ID: *.nb.test
Domains: *.nb.test
Status: Selected
Resolved IPs:
[srv.nb.test.]: 192.168.100.10
```
### Verifying the Domain Resource registration with the Operating System
After we have confirmed **everything** is working within NetBird's scope of operation, let's restart NetBird and
check whether the Operating System's default DNS resolver is resolving the Domain Resource correctly.
<Note>
See [Debugging access to network resources > Public nameservers](#public-nameservers) for the equivalent
MacOS and Windows debugging steps.
</Note>
<Note>
You might be surprised by a simple `netbird down` followed by `netbird up` not clearing the `Resolved IPs`:
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ sudo netbird down
Disconnected
kdn@brys-vm-nbt-ubuntu-01:~$ sudo netbird up
Connected
kdn@brys-vm-nbt-ubuntu-01:~$ netbird networks ls
Available Networks:
- ID: *.nb.test
Domains: *.nb.test
Status: Selected
Resolved IPs:
[srv.nb.test.]: 192.168.100.10
```
Don't be alarmed, this is working as expected (the results are simply stored within the client daemon's
in-memory cache), but routing rules are still properly cleared:
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ ip route show table all | grep 192.168.100
kdn@brys-vm-nbt-ubuntu-01:~$
```
</Note>
We will start "from scratch", by restarting the whole NetBird service to purge all caches and proceed with the tests:
```shell
kdn@brys-vm-nbt-ubuntu-01:~$ sudo netbird service restart
NetBird service has been restarted
kdn@brys-vm-nbt-ubuntu-01:~$ netbird networks ls
Available Networks:
- ID: *.nb.test
Domains: *.nb.test
Status: Selected
Resolved IPs: -
kdn@brys-vm-nbt-ubuntu-01:~$ ip route show table all | grep 192.168.100
kdn@brys-vm-nbt-ubuntu-01:~$ resolvectl query srv.nb.test
srv.nb.test: 192.168.100.10 -- link: wt0
-- Information acquired via protocol DNS in 8.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network
kdn@brys-vm-nbt-ubuntu-01:~$ ip route show table all | grep 192.168.100
192.168.100.10 dev wt0 table 7120
kdn@brys-vm-nbt-ubuntu-01:~$ netbird networks ls
Available Networks:
- ID: *.nb.test
Domains: *.nb.test
Status: Selected
Resolved IPs:
[srv.nb.test.]: 192.168.100.10
```
<Note>
Be aware that operating system resolver might not be the only source of domains, but querying through it is
a hard requirement for getting Domain Resources to start working.
Different applications (most notably web browsers) can cache this information internally and therefore never
activate the Domain Resource routing.
While we can (and do successfully) clear the operating system resolver's caches, there is unfortunately no way to
instruct regular applications to do the same.
</Note>
Reference in New Issue View Git Blame Copy Permalink