From 493803e20c87fafa62b7a4a684fa4ba25ee20cbb Mon Sep 17 00:00:00 2001
From: Viktor Liu
Date: Wed, 15 May 2024 18:09:29 +0200
Subject: [PATCH] Add client cli debug (#186)
Add cli debug command and update outputs
---
src/pages/how-to/cli.mdx | 200 ++++++++++++++++--
...ng-default-routes-for-internet-traffic.mdx | 2 +-
src/pages/how-to/report-bug-issues.mdx | 3 +-
.../how-to/resolve-overlapping-routes.mdx | 11 +
src/pages/how-to/troubleshooting-client.mdx | 75 ++++++-
5 files changed, 263 insertions(+), 28 deletions(-)
diff --git a/src/pages/how-to/cli.mdx b/src/pages/how-to/cli.mdx
index de26b8e9..0994ad85 100644
--- a/src/pages/how-to/cli.mdx
+++ b/src/pages/how-to/cli.mdx
@@ -24,6 +24,7 @@ netbird [command] [subcommand] [flags]
Below is the list of global flags:
```shell
--admin-url string Admin Panel URL [http|https]://[host]:[port] (default "https://app.netbird.io")
+ -A, --anonymize anonymize IP addresses and non-netbird.io domains in logs and status output
-c, --config string Netbird config file location (default "/etc/netbird/config.json")
--daemon-addr string Daemon service address to serve CLI requests [unix|tcp]://[path|host:port] (default "unix:///var/run/netbird.sock")
--log-file string sets NetBird log path. If console is specified the the log will be output to stdout (default "/var/log/netbird/client.log")
@@ -122,6 +123,7 @@ Retrieves the peer status from the daemon service.
--filter-by-ips strings filters the detailed output by a list of one or more IPs, e.g., --filter-by-ips 100.64.0.100,100.64.0.200
--filter-by-names strings filters the detailed output by a list of one or more peer FQDN or hostnames, e.g., --filter-by-names peer-a,peer-b.netbird.cloud
--filter-by-status string filters the detailed output by connection status(connected|disconnected), e.g., --filter-by-status connected
+ -A, --anonymize anonymize IP addresses and non-netbird.io domains in the status output
-h, --help help for status
--ipv4 display only NetBird IPv4 of this peer, e.g., --ipv4 will output 100.64.0.33
--json display detailed status information in json format
@@ -134,12 +136,17 @@ netbird status
```
This will output:
```shell
-Daemon status: Connected
+OS: linux/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected
Signal: Connected
Relays: 2/2 Available
+Nameservers: 1/1 Available
NetBird IP: 100.119.62.6/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/3 Connected
```
@@ -159,9 +166,12 @@ Peers detail:
Direct: false
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:31
+ Last connection update: 26 seconds ago
Last Wireguard handshake: -
Transfer status (received/sent) 0 B/0 B
+ Quantum resistance: false
+ Routes: -
+ Latency: 10.74ms
Peer:
NetBird IP: 100.119.201.225
@@ -172,9 +182,12 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:32
- Last Wireguard handshake: 2022-07-07 12:21:33
+ Last connection update: 26 seconds ago
+ Last Wireguard handshake: 25 seconds ago
Transfer status (received/sent) 2.0 KiB/355 B
+ Quantum resistance: false
+ Routes: 10.0.0.0/24
+ Latency: 20.14ms
Peer:
NetBird IP: 100.119.230.104
@@ -185,18 +198,27 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:33
- Last Wireguard handshake: 2022-07-07 12:21:34
+ Last connection update: 26 seconds ago
+ Last Wireguard handshake: 24 seconds ago
Transfer status (received/sent) 2.4 MiB/532 KiB
+ Quantum resistance: false
+ Routes: -
+ Latency: 16.24ms
-Daemon status: Connected
+OS: linux/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected to https://api.netbird.io:33073
Signal: Connected to https://signal2.wiretrustee.com:10000
Relays:
[stun:stun.netbird.io:5555] is Available
[turns:turn.netbird.io:443?transport=tcp] is Available
+Nameservers:
+ [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
NetBird IP: 100.119.62.6/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/3 Connected
```
To filter the peers' output by connection status, you can use the `--filter-by-status` flag with either "connected" or "disconnected" as value:
@@ -215,9 +237,12 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:32
- Last Wireguard handshake: 2022-07-07 12:21:33
+ Last connection update: 28 seconds ago
+ Last Wireguard handshake: 27 seconds ago
Transfer status (received/sent) 2.0 KiB/355 B
+ Quantum resistance: false
+ Routes: 10.0.0.0/24
+ Latency: 20.14ms
Peer:
NetBird IP: 100.119.230.104
@@ -228,18 +253,27 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:33
- Last Wireguard handshake: 2022-07-07 12:21:34
+ Last connection update: 28 seconds ago
+ Last Wireguard handshake: 26 seconds ago
Transfer status (received/sent) 2.4 MiB/532 KiB
+ Quantum resistance: false
+ Routes: -
+ Latency: 16.24ms
-Daemon status: Connected
+OS: linux/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected to https://api.netbird.io:33073
Signal: Connected to https://signal2.wiretrustee.com:10000
Relays:
[stun:stun.netbird.io:5555] is Available
[turns:turn.netbird.io:443?transport=tcp] is Available
+Nameservers:
+ [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
NetBird IP: 100.119.62.6/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/3 Connected
```
To filter the peers' output by peer IP addresses, you can use the `--filter-by-ips` flag with one or more IPs separated by a comma as a value:
@@ -258,18 +292,27 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:32
- Last Wireguard handshake: 2022-07-07 12:21:33
+ Last connection update: 32 seconds ago
+ Last Wireguard handshake: 30 seconds ago
Transfer status (received/sent) 2.0 KiB/355 B
+ Quantum resistance: false
+ Routes: 10.0.0.0/24
+ Latency: 20.14ms
-Daemon status: Connected
+OS: linux/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected to https://api.netbird.io:33073
Signal: Connected to https://signal2.wiretrustee.com:10000
Relays:
[stun:stun.netbird.io:5555] is Available
[turns:turn.netbird.io:443?transport=tcp] is Available
+Nameservers:
+ [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
NetBird IP: 100.119.62.6/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/3 Connected
```
You can combine both filters and get the peers that are both connected and with specific IPs:
@@ -289,18 +332,27 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): -/-
- Last connection update: 2022-07-07 12:21:33
- Last Wireguard handshake: 2022-07-07 12:21:34
+ Last connection update: 35 seconds ago
+ Last Wireguard handshake: 33 seconds ago
Transfer status (received/sent) 2.4 MiB/532 KiB
+ Quantum resistance: false
+ Routes: -
+ Latency: 16.24ms
-Daemon status: Connected
+OS: linux/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected to https://api.netbird.io:33073
Signal: Connected to https://signal2.wiretrustee.com:10000
Relays:
[stun:stun.netbird.io:5555] is Available
[turns:turn.netbird.io:443?transport=tcp] is Available
+Nameservers:
+ [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
NetBird IP: 100.119.62.6/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/3 Connected
```
@@ -385,3 +437,115 @@ The minimal form of running the command is:
```shell
sudo netbird service stop
```
+
+### debug
+The `debug` command provides tools for diagnosing and understanding the internal operations of the NetBird daemon.
+
+#### Usage
+To access debugging options:
+```shell
+netbird debug [command]
+```
+#### Subcommands
+- `bundle`: Create a debug bundle that includes logs and system information for troubleshooting.
+- `for`: Run the daemon with trace logging for a specified duration and create a debug bundle.
+- `log`: Manage logging levels for the NetBird daemon.
+
+#### Flags
+```shell
+ -h, --help help for debug
+```
+
+### debug bundle
+Generates a compressed archive containing diagnostic information, which can be used for troubleshooting.
+The file will be generated in the a temporary directory and the path will be printed to the console.
+The file is only accessible as root/Administrator.
+
+#### Usage
+To create a debug bundle:
+```shell
+netbird debug bundle -A
+```
+
+#### Examples
+Create a debug bundle:
+```shell
+netbird debug bundle
+```
+
+This will output:
+```
+/tmp/netbird.debug.676945815.zip
+```
+
+#### Flags
+```shell
+ -h, --help help for bundle
+ -A, --anonymize anonymize IP addresses and non-netbird.io domains in the debug output
+```
+
+### debug for
+Sets the logging level to trace, runs for the specified duration, and then generates a debug bundle.
+This is useful for capturing detailed logs over a period where issues are occurring.
+
+#### Usage
+To run debugging for a specific time period:
+```shell
+netbird debug for
+### Enabling All Routes
+
+When using the command `netbird routes select all` in the CLI or the button in the GUI,
+all currently available routes are selected. This action includes any new routes that become available in the future.
+
+This basically restores the default behavior of the NetBird client, where all routes are selected by default.
+
+### Disabling All Routes
+
+When using the command `netbird routes deselect all` in the CLI or the button GUI, all routes are deselected.
+This applies not only to the currently available routes but also to any routes that might be added in the future.
## Get started
diff --git a/src/pages/how-to/troubleshooting-client.mdx b/src/pages/how-to/troubleshooting-client.mdx
index 46452c65..4869b5f5 100644
--- a/src/pages/how-to/troubleshooting-client.mdx
+++ b/src/pages/how-to/troubleshooting-client.mdx
@@ -21,9 +21,12 @@ Peers detail:
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 10.128.0.35:51820/10.128.0.54:51820
- Last connection update: 2024-01-30 08:52:51
- Last Wireguard handshake: 2024-01-30 10:58:13
+ Last connection update: 20 seconds ago
+ Last Wireguard handshake: 19 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
+ Quantum resistance: false
+ Routes: 10.0.0.0/24
+ Latency: 37.503682ms
server-b.netbird.cloud:
NetBird IP: 100.75.226.48/32
@@ -34,20 +37,28 @@ Peers detail:
Direct: false
ICE candidate (Local/Remote): relay/host
ICE candidate endpoints (Local/Remote): 108.54.10.33:60434/10.128.0.12:51820
- Last connection update: 2024-01-30 08:52:51
- Last Wireguard handshake: 2024-01-30 10:58:13
+ Last connection update: 20 seconds ago
+ Last Wireguard handshake: 18 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
+ Quantum resistance: false
+ Routes: -
+ Latency: 37.503682ms
-Daemon version: 0.25.5
-CLI version: 0.25.5
+OS: darwin/amd64
+Daemon version: 0.27.4
+CLI version: 0.27.4
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
[stun:turn.netbird.io:5555] is Available
[turns:turn.netbird.ioo:443?transport=tcp] is Available
+Nameservers:
+ [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: maycons-mbp-2.netbird.cloud
NetBird IP: 100.75.143.239/16
Interface type: Kernel
+Quantum resistance: false
+Routes: -
Peers count: 2/2 Connected
```
As you can see, the output shows the peers connected, the NetBird IP address, the public key, the connection status, and the connection type.
@@ -63,14 +74,62 @@ As for Peers, the status will show the following information:
See more details about the status command [here](/how-to/cli#status).
## Getting client logs
-By default, client logs are located in the `/var/log/netbird/client.log` folder on macOS and Linux and in the `C:\ProgramData\netbird\client.log` folder on Windows.
+By default, client logs are located in the `/var/log/netbird/client.log` file on macOS and Linux and in the `C:\ProgramData\netbird\client.log` file on Windows.
You can analyze the logs to identify the root cause of the problem. If you need help, open a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and attach the logs.
+### Debug bundle
+
+A debug archive containing the recent logs and the status at the time of execution can be generated with the following command.
+
+Adding the `-A` flag will anonymize the logs, removing sensitive information such as public IP addresses and domain names.
+
+```shell
+netbird debug bundle -A
+```
+
+This will output the path of the generated file, which can be accesses with administrative privileges.
+
+### Debug for a specific time
+
+To capture logs for a specific time, you can use the `debug for` command. This will generate a debug bundle after the specified time has elapsed.
+
+```shell
+netbird debug for 5m -A
+```
+
+To capture any issues arising during a full `up`/`down` cycle, this will bring netbird `down`, set the log level to `trace`, and bring it back `up`.
+After 5 minutes netbird will be brought down again and the debug bundle will be generated.
+
+
+
+ Netbird won't be automatically brought back up after the debug bundle is generated.
+
+
+
## Enabling debug logs on agent
+Logs can be temporarily set using the following command.
+
+```shell
+netbird debug log level debug
+````
+
+or
+
+```shell
+netbird debug log level trace
+```
+
+The next time the daemon is restarted, the log level will return to the configured level.
+
+Using `netbird down` and `netbird up` will not reset the log level.
+
+To permanently set the log level, see the following sections.
+
### On Linux with systemd
-The default systemd unit file reads a set of environment variables from the path `/etc/sysconfig/netbird`. You can add the following line to the file to enable debug logs:
+The default systemd unit file reads a set of environment variables from the path `/etc/sysconfig/netbird`.
+You can add the following line to the file to enable debug logs:
```shell
sudo mkdir -p /etc/sysconfig