pangolin_mirror/newt
1
0
Fork 0
mirror of https://github.com/fosrl/newt.git synced 2026-03-26 20:46:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(readme): enhance clarity and structure of installation and documentation sections

Browse Source
This commit is contained in:
Marc Schäfer
2025-10-10 14:49:14 +02:00
parent 3cd7329d8b
commit c32828128f
1 changed files with 94 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files

192
README.md
View File

@@ -1,16 +1,18 @@
<!-- markdownlint-disable MD033 -->
# Newt # Newt
[![PkgGoDev](https://pkg.go.dev/badge/github.com/fosrl/newt)](https://pkg.go.dev/github.com/fosrl/newt) [![PkgGoDev](https://pkg.go.dev/badge/github.com/fosrl/newt)](https://pkg.go.dev/github.com/fosrl/newt)
[![GitHub License](https://img.shields.io/github/license/fosrl/newt)](https://github.com/fosrl/newt/blob/main/LICENSE) [![GitHub License](https://img.shields.io/github/license/fosrl/newt)](https://github.com/fosrl/newt/blob/main/LICENSE)
[![Go Report Card](https://goreportcard.com/badge/github.com/fosrl/newt)](https://goreportcard.com/report/github.com/fosrl/newt) [![Go Report Card](https://goreportcard.com/badge/github.com/fosrl/newt)](https://goreportcard.com/report/github.com/fosrl/newt)
Newt is a fully user space [WireGuard](https://www.wireguard.com/) tunnel client and TCP/UDP proxy, designed to securely expose private resources controlled by Pangolin. By using Newt, you don't need to manage complex WireGuard tunnels and NATing. Newt is a fully user space [WireGuard](https://www.wireguard.com/) tunnel client and TCP/UDP proxy, designed to securely expose private resources controlled by Pangolin. By using Newt, you don't need to manage complex WireGuard tunnels and NATing.
### Installation and Documentation ## Installation and Documentation
Newt is used with Pangolin and Gerbil as part of the larger system. See documentation below: Newt is used with Pangolin and Gerbil as part of the larger system. See documentation below:
- [Full Documentation](https://docs.fossorial.io) - [Full Documentation](https://docs.fossorial.io)
- Observability Quickstart: see docs/observability.md (Prometheus/OTel Collector setup, smoke tests) - Observability Quickstart: see `docs/observability.md` — canonical Prometheus/OTel Collector quickstart and smoke tests
## Preview ## Preview
@@ -34,63 +36,63 @@ When Newt receives WireGuard control messages, it will use the information encod
## CLI Args ## CLI Args
- `id`: Newt ID generated by Pangolin to identify the client. - `id`: Newt ID generated by Pangolin to identify the client.
- `secret`: A unique secret (not shared and kept private) used to authenticate the client ID with the websocket in order to receive commands. - `secret`: A unique secret (not shared and kept private) used to authenticate the client ID with the websocket in order to receive commands.
- `endpoint`: The endpoint where both Gerbil and Pangolin reside in order to connect to the websocket. - `endpoint`: The endpoint where both Gerbil and Pangolin reside in order to connect to the websocket.
- `mtu` (optional): MTU for the internal WG interface. Default: 1280 - `mtu` (optional): MTU for the internal WG interface. Default: 1280
- `dns` (optional): DNS server to use to resolve the endpoint. Default: 9.9.9.9 - `dns` (optional): DNS server to use to resolve the endpoint. Default: 9.9.9.9
- `log-level` (optional): The log level to use (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO - `log-level` (optional): The log level to use (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO
- `enforce-hc-cert` (optional): Enforce certificate validation for health checks. Default: false (accepts any cert) - `enforce-hc-cert` (optional): Enforce certificate validation for health checks. Default: false (accepts any cert)
- `docker-socket` (optional): Set the Docker socket to use the container discovery integration - `docker-socket` (optional): Set the Docker socket to use the container discovery integration
- `ping-interval` (optional): Interval for pinging the server. Default: 3s - `ping-interval` (optional): Interval for pinging the server. Default: 3s
- `ping-timeout` (optional): Timeout for each ping. Default: 5s - `ping-timeout` (optional): Timeout for each ping. Default: 5s
- `updown` (optional): A script to be called when targets are added or removed. - `updown` (optional): A script to be called when targets are added or removed.
- `tls-client-cert` (optional): Client certificate (p12 or pfx) for mTLS. See [mTLS](#mtls) - `tls-client-cert` (optional): Client certificate (p12 or pfx) for mTLS. See [mTLS](#mtls)
- `tls-client-cert` (optional): Path to client certificate (PEM format, optional if using PKCS12). See [mTLS](#mtls) - `tls-client-cert` (optional): Path to client certificate (PEM format, optional if using PKCS12). See [mTLS](#mtls)
- `tls-client-key` (optional): Path to private key for mTLS (PEM format, optional if using PKCS12) - `tls-client-key` (optional): Path to private key for mTLS (PEM format, optional if using PKCS12)
- `tls-ca-cert` (optional): Path to CA certificate to verify server (PEM format, optional if using PKCS12) - `tls-ca-cert` (optional): Path to CA certificate to verify server (PEM format, optional if using PKCS12)
- `docker-enforce-network-validation` (optional): Validate the container target is on the same network as the newt process. Default: false - `docker-enforce-network-validation` (optional): Validate the container target is on the same network as the newt process. Default: false
- `health-file` (optional): Check if connection to WG server (pangolin) is ok. creates a file if ok, removes it if not ok. Can be used with docker healtcheck to restart newt - `health-file` (optional): Check if connection to WG server (pangolin) is ok. creates a file if ok, removes it if not ok. Can be used with docker healtcheck to restart newt
- `accept-clients` (optional): Enable WireGuard server mode to accept incoming newt client connections. Default: false - `accept-clients` (optional): Enable WireGuard server mode to accept incoming newt client connections. Default: false
- `generateAndSaveKeyTo` (optional): Path to save generated private key - `generateAndSaveKeyTo` (optional): Path to save generated private key
- `native` (optional): Use native WireGuard interface when accepting clients (requires WireGuard kernel module and Linux, must run as root). Default: false (uses userspace netstack) - `native` (optional): Use native WireGuard interface when accepting clients (requires WireGuard kernel module and Linux, must run as root). Default: false (uses userspace netstack)
- `interface` (optional): Name of the WireGuard interface. Default: newt - `interface` (optional): Name of the WireGuard interface. Default: newt
- `keep-interface` (optional): Keep the WireGuard interface. Default: false - `keep-interface` (optional): Keep the WireGuard interface. Default: false
## Environment Variables ## Environment Variables
All CLI arguments can be set using environment variables as an alternative to command line flags. Environment variables are particularly useful when running Newt in containerized environments. All CLI arguments can be set using environment variables as an alternative to command line flags. Environment variables are particularly useful when running Newt in containerized environments.
- `PANGOLIN_ENDPOINT`: Endpoint of your pangolin server (equivalent to `--endpoint`) - `PANGOLIN_ENDPOINT`: Endpoint of your pangolin server (equivalent to `--endpoint`)
- `NEWT_ID`: Newt ID generated by Pangolin (equivalent to `--id`) - `NEWT_ID`: Newt ID generated by Pangolin (equivalent to `--id`)
- `NEWT_SECRET`: Newt secret for authentication (equivalent to `--secret`) - `NEWT_SECRET`: Newt secret for authentication (equivalent to `--secret`)
- `MTU`: MTU for the internal WG interface. Default: 1280 (equivalent to `--mtu`) - `MTU`: MTU for the internal WG interface. Default: 1280 (equivalent to `--mtu`)
- `DNS`: DNS server to use to resolve the endpoint. Default: 9.9.9.9 (equivalent to `--dns`) - `DNS`: DNS server to use to resolve the endpoint. Default: 9.9.9.9 (equivalent to `--dns`)
- `LOG_LEVEL`: Log level (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO (equivalent to `--log-level`) - `LOG_LEVEL`: Log level (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO (equivalent to `--log-level`)
- `DOCKER_SOCKET`: Path to Docker socket for container discovery (equivalent to `--docker-socket`) - `DOCKER_SOCKET`: Path to Docker socket for container discovery (equivalent to `--docker-socket`)
- `PING_INTERVAL`: Interval for pinging the server. Default: 3s (equivalent to `--ping-interval`) - `PING_INTERVAL`: Interval for pinging the server. Default: 3s (equivalent to `--ping-interval`)
- `PING_TIMEOUT`: Timeout for each ping. Default: 5s (equivalent to `--ping-timeout`) - `PING_TIMEOUT`: Timeout for each ping. Default: 5s (equivalent to `--ping-timeout`)
- `UPDOWN_SCRIPT`: Path to updown script for target add/remove events (equivalent to `--updown`) - `UPDOWN_SCRIPT`: Path to updown script for target add/remove events (equivalent to `--updown`)
- `TLS_CLIENT_CERT`: Path to client certificate for mTLS (equivalent to `--tls-client-cert`) - `TLS_CLIENT_CERT`: Path to client certificate for mTLS (equivalent to `--tls-client-cert`)
- `TLS_CLIENT_CERT`: Path to client certificate for mTLS (equivalent to `--tls-client-cert`) - `TLS_CLIENT_CERT`: Path to client certificate for mTLS (equivalent to `--tls-client-cert`)
- `TLS_CLIENT_KEY`: Path to private key for mTLS (equivalent to `--tls-client-key`) - `TLS_CLIENT_KEY`: Path to private key for mTLS (equivalent to `--tls-client-key`)
- `TLS_CA_CERT`: Path to CA certificate to verify server (equivalent to `--tls-ca-cert`) - `TLS_CA_CERT`: Path to CA certificate to verify server (equivalent to `--tls-ca-cert`)
- `DOCKER_ENFORCE_NETWORK_VALIDATION`: Validate container targets are on same network. Default: false (equivalent to `--docker-enforce-network-validation`) - `DOCKER_ENFORCE_NETWORK_VALIDATION`: Validate container targets are on same network. Default: false (equivalent to `--docker-enforce-network-validation`)
- `ENFORCE_HC_CERT`: Enforce certificate validation for health checks. Default: false (equivalent to `--enforce-hc-cert`) - `ENFORCE_HC_CERT`: Enforce certificate validation for health checks. Default: false (equivalent to `--enforce-hc-cert`)
- `HEALTH_FILE`: Path to health file for connection monitoring (equivalent to `--health-file`) - `HEALTH_FILE`: Path to health file for connection monitoring (equivalent to `--health-file`)
- `ACCEPT_CLIENTS`: Enable WireGuard server mode. Default: false (equivalent to `--accept-clients`) - `ACCEPT_CLIENTS`: Enable WireGuard server mode. Default: false (equivalent to `--accept-clients`)
- `GENERATE_AND_SAVE_KEY_TO`: Path to save generated private key (equivalent to `--generateAndSaveKeyTo`) - `GENERATE_AND_SAVE_KEY_TO`: Path to save generated private key (equivalent to `--generateAndSaveKeyTo`)
- `USE_NATIVE_INTERFACE`: Use native WireGuard interface (Linux only). Default: false (equivalent to `--native`) - `USE_NATIVE_INTERFACE`: Use native WireGuard interface (Linux only). Default: false (equivalent to `--native`)
- `INTERFACE`: Name of the WireGuard interface. Default: newt (equivalent to `--interface`) - `INTERFACE`: Name of the WireGuard interface. Default: newt (equivalent to `--interface`)
- `KEEP_INTERFACE`: Keep the WireGuard interface after shutdown. Default: false (equivalent to `--keep-interface`) - `KEEP_INTERFACE`: Keep the WireGuard interface after shutdown. Default: false (equivalent to `--keep-interface`)
- `CONFIG_FILE`: Load the config json from this file instead of in the home folder. - `CONFIG_FILE`: Load the config json from this file instead of in the home folder.
## Loading secrets from files ## Loading secrets from files
You can use `CONFIG_FILE` to define a location of a config file to store the credentials between runs. You can use `CONFIG_FILE` to define a location of a config file to store the credentials between runs.
``` ```sh
$ cat ~/.config/newt-client/config.json $ cat ~/.config/newt-client/config.json
{ {
"id": "spmzu8rbpzj1qq6", "id": "spmzu8rbpzj1qq6",
@@ -100,26 +102,21 @@ $ cat ~/.config/newt-client/config.json
} }
``` ```
This file is also written to when newt first starts up. So you do not need to run every time with --id and secret if you have run it once! This file is also written to when newt first starts up. So you do not need to run every time with --id and secret if you have run it once!
Default locations: Default locations:
- **macOS**: `~/Library/Application Support/newt-client/config.json` - **macOS**: `~/Library/Application Support/newt-client/config.json`
- **Windows**: `%PROGRAMDATA%\newt\newt-client\config.json` - **Windows**: `%PROGRAMDATA%\newt\newt-client\config.json`
- **Linux/Others**: `~/.config/newt-client/config.json` - **Linux/Others**: `~/.config/newt-client/config.json`
## Observability Quickstart <!-- Observability Quickstart moved to docs/observability.md (canonical). -->
For a quick start with Prometheus scraping and smoke checks, read the step-by-step guide in docs/observability.md. It includes:
- docker-compose.metrics.yml for direct /metrics scraping (recommended)
- docker-compose.metrics.collector.yml for the OTLP → Collector → Prometheus exporter path (no double-scrape)
- scripts/smoke-metrics.sh for basic verification
## Examples ## Examples
**Note**: When both environment variables and CLI arguments are provided, CLI arguments take precedence. **Note**: When both environment variables and CLI arguments are provided, CLI arguments take precedence.
- Example: - Example:
```bash ```bash
newt \ newt \
@@ -166,16 +163,16 @@ When the `--accept-clients` flag is enabled (or `ACCEPT_CLIENTS=true` environmen
In client acceptance mode, Newt: In client acceptance mode, Newt:
- **Creates a WireGuard service** that can accept incoming connections from other WireGuard clients - **Creates a WireGuard service** that can accept incoming connections from other WireGuard clients
- **Starts a connection testing server** (WGTester) that responds to connectivity checks from remote clients - **Starts a connection testing server** (WGTester) that responds to connectivity checks from remote clients
- **Manages peer configurations** dynamically based on Pangolin's instructions - **Manages peer configurations** dynamically based on Pangolin's instructions
- **Enables bidirectional communication** between the Newt instance and connected clients - **Enables bidirectional communication** between the Newt instance and connected clients
### Use Cases ### Use Cases
- **Site-to-site connectivity**: Connect multiple locations through a central Newt instance - **Site-to-site connectivity**: Connect multiple locations through a central Newt instance
- **Client access to private networks**: Allow remote clients to access resources behind the Newt instance - **Client access to private networks**: Allow remote clients to access resources behind the Newt instance
- **Development environments**: Provide developers secure access to internal services - **Development environments**: Provide developers secure access to internal services
### Client Tunneling Modes ### Client Tunneling Modes
@@ -185,11 +182,11 @@ Newt supports two WireGuard tunneling modes:
By default, Newt uses a fully userspace WireGuard implementation using [netstack](https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go). This mode: By default, Newt uses a fully userspace WireGuard implementation using [netstack](https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go). This mode:
- **Does not require root privileges** - **Does not require root privileges**
- **Works on all supported platforms** (Linux, Windows, macOS) - **Works on all supported platforms** (Linux, Windows, macOS)
- **Does not require WireGuard kernel module** to be installed - **Does not require WireGuard kernel module** to be installed
- **Runs entirely in userspace** - no system network interface is created - **Runs entirely in userspace** - no system network interface is created
- **Is containerization-friendly** - works seamlessly in Docker containers - **Is containerization-friendly** - works seamlessly in Docker containers
This is the recommended mode for most deployments, especially containerized environments. This is the recommended mode for most deployments, especially containerized environments.
@@ -199,11 +196,11 @@ In this mode, TCP and UDP is proxied out of newt from the remote client using TC
When using the `--native` flag or setting `USE_NATIVE_INTERFACE=true`, Newt uses the native WireGuard kernel module. This mode: When using the `--native` flag or setting `USE_NATIVE_INTERFACE=true`, Newt uses the native WireGuard kernel module. This mode:
- **Requires root privileges** to create and manage network interfaces - **Requires root privileges** to create and manage network interfaces
- **Only works on Linux** with the WireGuard kernel module installed - **Only works on Linux** with the WireGuard kernel module installed
- **Creates a real network interface** (e.g., `newt0`) on the system - **Creates a real network interface** (e.g., `newt0`) on the system
- **May offer better performance** for high-throughput scenarios - **May offer better performance** for high-throughput scenarios
- **Requires proper network permissions** and may conflict with existing network configurations - **Requires proper network permissions** and may conflict with existing network configurations
In this mode it functions like a traditional VPN interface - all data arrives on the interface and you must get it to the destination (or access things locally). In this mode it functions like a traditional VPN interface - all data arrives on the interface and you must get it to the destination (or access things locally).
@@ -235,10 +232,10 @@ services:
When client acceptance is enabled: When client acceptance is enabled:
- **WGTester Server**: Runs on `port + 1` (e.g., if WireGuard uses port 51820, WGTester uses 51821) - **WGTester Server**: Runs on `port + 1` (e.g., if WireGuard uses port 51820, WGTester uses 51821)
- **Connection Testing**: Responds to UDP packets with magic header `0xDEADBEEF` for connectivity verification - **Connection Testing**: Responds to UDP packets with magic header `0xDEADBEEF` for connectivity verification
- **Dynamic Configuration**: Peer configurations are managed remotely through Pangolin - **Dynamic Configuration**: Peer configurations are managed remotely through Pangolin
- **Proxy Integration**: Can work with both userspace (netstack) and native WireGuard modes - **Proxy Integration**: Can work with both userspace (netstack) and native WireGuard modes
**Note**: Client acceptance mode requires coordination with Pangolin for peer management and configuration distribution. **Note**: Client acceptance mode requires coordination with Pangolin for peer management and configuration distribution.
@@ -252,24 +249,23 @@ You can specify the Docker socket path using the `--docker-socket` CLI argument
Supported values include: Supported values include:
- Local UNIX socket (default): - Local UNIX socket (default):
>You must mount the socket file into the container using a volume, so Newt can access it. >You must mount the socket file into the container using a volume, so Newt can access it.
`unix:///var/run/docker.sock` `unix:///var/run/docker.sock`
- TCP socket (e.g., via Docker Socket Proxy): - TCP socket (e.g., via Docker Socket Proxy):
`tcp://localhost:2375` `tcp://localhost:2375`
- HTTP/HTTPS endpoints (e.g., remote Docker APIs): - HTTP/HTTPS endpoints (e.g., remote Docker APIs):
`http://your-host:2375` `http://your-host:2375`
- SSH connections (experimental, requires SSH setup): - SSH connections (experimental, requires SSH setup):
`ssh://user@host` `ssh://user@host`
```yaml ```yaml
services: services:
newt: newt:
@@ -284,16 +280,17 @@ services:
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2 - NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- DOCKER_SOCKET=unix:///var/run/docker.sock - DOCKER_SOCKET=unix:///var/run/docker.sock
``` ```
>If you previously used just a path like `/var/run/docker.sock`, it still works — Newt assumes it is a UNIX socket by default. >If you previously used just a path like `/var/run/docker.sock`, it still works — Newt assumes it is a UNIX socket by default.
#### Hostnames vs IPs #### Hostnames vs IPs
When the Docker Socket Integration is used, depending on the network which Newt is run with, either the hostname (generally considered the container name) or the IP address of the container will be sent to Pangolin. Here are some of the scenarios where IPs or hostname of the container will be utilised: When the Docker Socket Integration is used, depending on the network which Newt is run with, either the hostname (generally considered the container name) or the IP address of the container will be sent to Pangolin. Here are some of the scenarios where IPs or hostname of the container will be utilised:
- **Running in Network Mode 'host'**: IP addresses will be used - **Running in Network Mode 'host'**: IP addresses will be used
- **Running in Network Mode 'bridge'**: IP addresses will be used - **Running in Network Mode 'bridge'**: IP addresses will be used
- **Running in docker-compose without a network specification**: Docker compose creates a network for the compose by default, hostnames will be used - **Running in docker-compose without a network specification**: Docker compose creates a network for the compose by default, hostnames will be used
- **Running on docker-compose with defined network**: Hostnames will be used - **Running on docker-compose with defined network**: Hostnames will be used
### Docker Enforce Network Validation ### Docker Enforce Network Validation
@@ -329,12 +326,12 @@ Newt supports mutual TLS (mTLS) authentication if the server is configured to re
> This is the original method and still supported. > This is the original method and still supported.
* File must contain: - File must contain:
* Client private key - Client private key
* Public certificate - Public certificate
* CA certificate - CA certificate
* Encrypted `.p12` files are **not supported** - Encrypted `.p12` files are **not supported**
Example: Example:
@@ -350,9 +347,9 @@ newt \
You can now provide separate files for: You can now provide separate files for:
* `--tls-client-cert`: client certificate (`.crt` or `.pem`) - `--tls-client-cert`: client certificate (`.crt` or `.pem`)
* `--tls-client-key`: client private key (`.key` or `.pem`) - `--tls-client-key`: client private key (`.key` or `.pem`)
* `--tls-ca-cert`: CA cert to verify the server - `--tls-ca-cert`: CA cert to verify the server
Example: Example:
@@ -366,7 +363,6 @@ newt \
--tls-ca-cert ./ca.crt --tls-ca-cert ./ca.crt
``` ```
```yaml ```yaml
services: services:
newt: newt: