mirror of
https://github.com/netbirdio/docs.git
synced 2026-10-17 02:59:04 +02:00
Add MySQL to PostgreSQL migration guide (#1024)
This commit is contained in:
Bethuel Mmbaga
@@ -900,6 +900,10 @@ export const docsNavigation = [
|
||||
title: 'External IdP to Embedded IdP',
|
||||
href: '/selfhosted/migration/external-to-embedded-idp',
|
||||
},
|
||||
{
|
||||
title: 'MySQL to PostgreSQL',
|
||||
href: '/selfhosted/migration/mysql-to-postgresql',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
|
||||
@@ -54,9 +54,9 @@ The default quickstart deployment uses the relay service's embedded STUN server.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `NETBIRD_STORE_CONFIG_ENGINE` | `sqlite` | Store engine: `sqlite`, `postgres`, or `mysql` |
|
||||
| `NETBIRD_STORE_CONFIG_ENGINE` | `sqlite` | Store engine: `sqlite`, `postgres`, or `mysql` (deprecated) |
|
||||
| `NETBIRD_STORE_ENGINE_POSTGRES_DSN` | - | PostgreSQL connection string |
|
||||
| `NETBIRD_STORE_ENGINE_MYSQL_DSN` | - | MySQL connection string |
|
||||
| `NETBIRD_STORE_ENGINE_MYSQL_DSN` | - | MySQL connection string. Deprecated, see [Migrate from MySQL to PostgreSQL](/selfhosted/migration/mysql-to-postgresql) |
|
||||
| `NETBIRD_DATASTORE_ENC_KEY` | Auto-generated | Encryption key for sensitive data |
|
||||
|
||||
<Note>
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
# Activity Events MySQL store
|
||||
|
||||
## Using MySQL for fresh installations
|
||||
|
||||
To enable MySQL add to your management service the following environmental variable:
|
||||
```bash
|
||||
NB_ACTIVITY_EVENT_STORE_ENGINE=mysql
|
||||
NB_ACTIVITY_EVENT_MYSQL_DSN="mysql://<username>:<password>@<host>:<port>/<db_name>"
|
||||
```
|
||||
|
||||
You can switch back to sqlite storage by setting the `NB_ACTIVITY_EVENT_STORE_ENGINE` variable to `sqlite`.
|
||||
<Note>
|
||||
Switching between storage options requires migration steps to prevent data loss.
|
||||
</Note>
|
||||
|
||||
@@ -333,10 +333,10 @@ Configures the database backend for storing all NetBird management data includin
|
||||
|
||||
<Properties>
|
||||
<Property name="server.store.engine" type="string">
|
||||
Database engine. Options: `sqlite`, `postgres`, `mysql`. Default: `sqlite`.
|
||||
Database engine. Options: `sqlite`, `postgres`, `mysql`. Default: `sqlite`. `mysql` is deprecated, see [Migrate from MySQL to PostgreSQL](/selfhosted/migration/mysql-to-postgresql).
|
||||
</Property>
|
||||
<Property name="server.store.dsn" type="string">
|
||||
Connection string for postgres or mysql engines. For postgres: `host=localhost user=netbird password=secret dbname=netbird port=5432`. Alternatively, use the `NETBIRD_STORE_ENGINE_POSTGRES_DSN` or `NETBIRD_STORE_ENGINE_MYSQL_DSN` environment variables.
|
||||
Connection string for the postgres engine, or the deprecated mysql engine. For postgres: `host=localhost user=netbird password=secret dbname=netbird port=5432`. Alternatively, use the `NETBIRD_STORE_ENGINE_POSTGRES_DSN` environment variable (`NETBIRD_STORE_ENGINE_MYSQL_DSN` for mysql).
|
||||
</Property>
|
||||
<Property name="server.store.encryptionKey" type="string">
|
||||
base64 string of a 32-byte (256-bit) encryption key for sensitive data at rest. Used to encrypt setup keys, API tokens, and other secrets stored in the database. Auto-generated by the setup script.
|
||||
@@ -358,9 +358,9 @@ Configures the database backend for storing all NetBird management data includin
|
||||
|--------|---------|-------|
|
||||
| SQLite (default) | `/var/lib/netbird/` volume | File-based database stored in the `netbird_data` Docker volume. Zero configuration required, but does not support concurrent writes or running multiple management instances. Best for testing or small deployments. |
|
||||
| PostgreSQL | External database server | Recommended for production deployments. Supports concurrent access, enabling multiple management instances for high availability. |
|
||||
| MySQL | External database server | Alternative to PostgreSQL for organizations that have standardized on MySQL/MariaDB. Provides similar benefits including concurrent access. |
|
||||
| MySQL | External database server | Deprecated and will be removed in an upcoming release. See [Migrate from MySQL to PostgreSQL](/selfhosted/migration/mysql-to-postgresql). |
|
||||
|
||||
For PostgreSQL or MySQL, set the connection string via the `server.store.dsn` field in `config.yaml` or environment variables on the `netbird-server` container. See [Using an External Database](#using-an-external-database) below.
|
||||
For PostgreSQL, or the deprecated MySQL engine, set the connection string via the `server.store.dsn` field in `config.yaml` or environment variables on the `netbird-server` container. See [Using an External Database](#using-an-external-database) below.
|
||||
|
||||
See [Management Postgres Store](/selfhosted/maintenance/postgres-store) for detailed PostgreSQL setup.
|
||||
|
||||
@@ -499,8 +499,6 @@ Alternatively, you can use an environment variable instead of putting the DSN in
|
||||
netbird-server:
|
||||
environment:
|
||||
- NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=db-server user=netbird password=secret dbname=netbird port=5432
|
||||
# Or for MySQL:
|
||||
# - NETBIRD_STORE_ENGINE_MYSQL_DSN=user:password@tcp(host:3306)/netbird
|
||||
```
|
||||
|
||||
See [Management Postgres Store](/selfhosted/maintenance/postgres-store) for detailed setup.
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
import {Note} from "@/components/mdx"
|
||||
|
||||
export const description = 'Move a self-hosted NetBird deployment from MySQL to PostgreSQL before MySQL support is removed.'
|
||||
|
||||
# Migrate from MySQL to PostgreSQL
|
||||
|
||||
MySQL support for the Management store is deprecated and will be removed in an upcoming release. This guide moves an existing MySQL deployment to PostgreSQL with the `netbird-mysql-migrate` tool.
|
||||
|
||||
The tool copies the following into PostgreSQL:
|
||||
- the Management store, from MySQL
|
||||
- the activity events (`events.db`) and embedded IdP (`idp.db`) stores, from their SQLite files
|
||||
|
||||
The tool only reads your MySQL database and doesn't change it.
|
||||
|
||||
## Before You Begin
|
||||
|
||||
- Upgrade the management server to `v0.81.0` or later and start it once on MySQL, so the database schema is current. That's the `netbird-server` service, or `management` on the older multi-container setup, not the NetBird client. Use the tool from the same release.
|
||||
- Prepare an empty PostgreSQL database. To run one in Docker, see [Set Up PostgreSQL](/selfhosted/maintenance/scaling/migrate-sqlite-to-postgresql#set-up-postgre-sql). The examples in this guide connect with `sslmode=require`; use `verify-full` to also verify the server certificate. Use `sslmode=disable` only when PostgreSQL runs on the same host or Docker network as the management server.
|
||||
- Run the tool on a Linux machine that can reach both MySQL and PostgreSQL. The tool is Linux-only, built for `amd64`, `arm64`, and `arm`.
|
||||
|
||||
## Step 1: Get the Migration Tool
|
||||
|
||||
```bash
|
||||
VERSION=0.81.0
|
||||
curl -L -o netbird-mysql-migrate.tar.gz \
|
||||
https://github.com/netbirdio/netbird/releases/download/v${VERSION}/netbird-mysql-migrate_${VERSION}_linux_amd64.tar.gz
|
||||
tar xzf netbird-mysql-migrate.tar.gz
|
||||
chmod +x netbird-mysql-migrate
|
||||
```
|
||||
|
||||
On `arm64` or `arm`, replace `linux_amd64` with `linux_arm64` or `linux_arm`.
|
||||
|
||||
## Step 2: Stop the Management Server and Back Up
|
||||
|
||||
Stop the management server and copy its data directory, which holds `events.db` and `idp.db`:
|
||||
|
||||
```bash
|
||||
docker compose stop netbird-server
|
||||
mkdir backup
|
||||
docker compose cp -a netbird-server:/var/lib/netbird/. backup/
|
||||
chmod 700 backup
|
||||
```
|
||||
|
||||
On the older multi-container setup, use the `management` service instead of `netbird-server`.
|
||||
|
||||
Back up MySQL as well:
|
||||
|
||||
```bash
|
||||
mysqldump --no-tablespaces -h <MYSQL_HOST> -u <MYSQL_USER> -p <MYSQL_DB> > backup/netbird-mysql.sql
|
||||
```
|
||||
|
||||
The backup holds password hashes from `idp.db` and a full database dump, so `chmod 700` keeps it readable only by you. `--no-tablespaces` avoids a `PROCESS` privilege error when you dump as the NetBird database user.
|
||||
|
||||
## Step 3: Run the Migration
|
||||
|
||||
```bash
|
||||
./netbird-mysql-migrate \
|
||||
--mysql-dsn '<MYSQL_USER>:<MYSQL_PASSWORD>@tcp(<MYSQL_HOST>:3306)/<MYSQL_DB>' \
|
||||
--postgres-dsn 'host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require' \
|
||||
--events-db backup/events.db \
|
||||
--auth-db backup/idp.db
|
||||
```
|
||||
|
||||
When it finishes, you see:
|
||||
|
||||
```
|
||||
management: 1520 rows from 39 tables
|
||||
activity: 8230 rows from 2 tables
|
||||
auth: 41 rows from 12 tables
|
||||
Done. Point the store, activity and auth store settings at Postgres before starting the server.
|
||||
```
|
||||
|
||||
<Note>
|
||||
- All stores go into the database given by `--postgres-dsn`. To keep the activity or auth store in a separate database, add `--events-postgres-dsn` or `--auth-postgres-dsn`.
|
||||
- A missing `events.db` or `idp.db` is skipped. For example, deployments on an external IdP have no `idp.db`.
|
||||
- MySQL timestamps are read as UTC. If your management server runs in another time zone, for example with `TZ` set on the container, add `--mysql-timezone`, such as `--mysql-timezone Europe/Berlin`.
|
||||
- The tool stops if the PostgreSQL database already holds NetBird data.
|
||||
</Note>
|
||||
|
||||
## Step 4: Point the Management Server at PostgreSQL
|
||||
|
||||
Change only the store settings. Keep the encryption key (`server.store.encryptionKey` in `config.yaml`, `DataStoreEncryptionKey` in `management.json`) as it is: the tool copies encrypted values as they are, so the management server needs the same key to read them.
|
||||
|
||||
### Combined setup (config.yaml)
|
||||
|
||||
Set all three stores to PostgreSQL in `config.yaml`, using the DSNs you migrated into:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
store:
|
||||
engine: "postgres"
|
||||
dsn: "host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require"
|
||||
activityStore:
|
||||
engine: "postgres"
|
||||
dsn: "host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require"
|
||||
authStore:
|
||||
engine: "postgres"
|
||||
dsn: "host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require"
|
||||
```
|
||||
|
||||
Remove any `NETBIRD_STORE_ENGINE_MYSQL_DSN` or `NB_STORE_ENGINE_MYSQL_DSN` variable from `docker-compose.yml`.
|
||||
|
||||
### Older multi-container setup (management.json)
|
||||
|
||||
In `management.json`, set the store engine and, if the file has an `EmbeddedIdP` section, its storage:
|
||||
|
||||
```json
|
||||
"StoreConfig": {
|
||||
"Engine": "postgres"
|
||||
},
|
||||
"EmbeddedIdP": {
|
||||
"Storage": {
|
||||
"Type": "postgres",
|
||||
"Config": {
|
||||
"DSN": "host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then, on the `management` service in `docker-compose.yml`, replace `NETBIRD_STORE_ENGINE_MYSQL_DSN` with:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require
|
||||
- NB_ACTIVITY_EVENT_STORE_ENGINE=postgres
|
||||
- NB_ACTIVITY_EVENT_POSTGRES_DSN=host=<PG_HOST> port=5432 user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB> sslmode=require
|
||||
```
|
||||
|
||||
## Step 5: Start and Verify
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Log in to the dashboard and confirm that your peers, users, and activity events are there. Then delete the `backup` directory.
|
||||
|
||||
## Roll Back
|
||||
|
||||
The tool doesn't change MySQL, so to roll back, revert the store settings from Step 4 and restart the management server. Changes made while running on PostgreSQL aren't carried back to MySQL.
|
||||
|
||||
A failed run leaves no data in PostgreSQL, so you can fix the problem and run the tool again.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tool exits with "target already holds data"
|
||||
|
||||
The PostgreSQL database for the store named in the error already holds NetBird data. If an earlier run of the tool finished, your data is already there, so continue with [Step 4](#step-4-point-the-management-server-at-postgre-sql). Otherwise, point the tool at an empty database.
|
||||
|
||||
### Tool exits with "tls error: server refused TLS connection"
|
||||
|
||||
The DSN requires TLS (`sslmode=require` or `verify-full`), but PostgreSQL doesn't accept TLS connections. Enable TLS on PostgreSQL. If PostgreSQL runs on the same host or Docker network as the management server, you can use `sslmode=disable` instead, in the tool's DSNs and in the Step 4 settings.
|
||||
|
||||
### Tool exits with "source lacks columns"
|
||||
|
||||
Your data was last written by an older release than the tool. Start the management server once against MySQL on the same release as the tool, then repeat [Step 2](#step-2-stop-the-management-server-and-back-up) so your backup holds the updated `events.db` and `idp.db`, and run the tool again.
|
||||
|
||||
### Tool exits with "no NetBird tables found in the source"
|
||||
|
||||
The error names the store it came from. For `management`, check the database name in `--mysql-dsn`. For `activity` or `auth`, check that `--events-db` or `--auth-db` points to the `events.db` or `idp.db` in your backup.
|
||||
|
||||
### Management server doesn't start: "activityStore.dsn is required when activityStore.engine is postgres"
|
||||
|
||||
The `activityStore` block in `config.yaml` sets `engine: "postgres"` without a `dsn`. Add the DSN as shown in [Step 4](#step-4-point-the-management-server-at-postgre-sql). You can see this error with `docker compose logs netbird-server`.
|
||||
Reference in New Issue
Block a user