Add experimental postgresql store engine support (#188)

* postgres store engine

* add steps to rollback postgres data

* add steps to rollback postgres data

* add missing steps

* fix migration errors

src/components/NavigationDocs.jsx

@@ -138,6 +138,7 @@ export const docsNavigation = [
       { title: 'Quickstart guide', href: '/selfhosted/selfhosted-quickstart' },
       { title: 'Advanced guide', href: '/selfhosted/selfhosted-guide' },
       { title: 'Management SQLite Store', href: '/selfhosted/sqlite-store'},
+      { title: 'Management Postgres Store', href: '/selfhosted/postgres-store'},
       { title: 'Supported IdPs', href: '/selfhosted/identity-providers' },
       { title: 'Management geolocation', href: '/selfhosted/geo-support' },
       { title: 'Troubleshooting', href: '/selfhosted/troubleshooting' },

src/pages/selfhosted/postgres-store.mdx

@@ -0,0 +1,197 @@
+# Management Postgresql store
+
+<Note>
+This feature provides experimental support for using Postgres as the storage engine. Please use with caution and ensure proper testing in your environment.
+</Note>
+
+## Using Postgres for fresh installations
+
+As of version 0.26.0, the default configuration for fresh installations is SQLite storage.
+However, users have the option to leverage the benefits of Postgres for new instances beginning from version `<version>`.
+To enable Postgres, add to your `setup.env` the following variable:
+```bash
+NETBIRD_STORE_CONFIG_ENGINE=postgres
+```
+This will result in a configuration similar to the following in your `management.json` file:
+```json
+    "StoreConfig": {
+        "Engine": "postgres"
+    }
+```
+You can switch back to sqlite storage by setting the `NETBIRD_STORE_CONFIG_ENGINE` variable to `sqlite`.
+<Note>
+    Switching between storage options requires migration steps to prevent data loss.
+</Note>
+
+
+## Migrating from SQLite store to Postgres store
+This migration process allows users to seamlessly transition between storage options while maintaining data integrity.
+
+<Note>
+    The following commands assume you use the latest docker version with the compose plugin. If you have docker-compose installed as a standalone, please use docker-compose as a command.
+</Note>
+
+1. Backup your data store (`store.db` in `datadir` - default `/var/lib/netbird/`)
+```bash
+mkdir backup
+docker compose cp -a management:/var/lib/netbird/. backup/
+```
+
+2. Import Sqlite data to Postgres
+
+For migrating the Sqlite data we rely on the [pgloader](https://github.com/dimitri/pgloader) tool. You can install it by running
+`sudo apt-get install pgloader` on debian or `brew install pgloader` on MacOS.
+
+```bash
+pgloader --type sqlite backup/store.db "postgresql://<PG_USER>:<PG_PASSWORD>@<PG_HOST>:<PG_PORT>/<PG_DB_NAME>"
+```
+
+3. Resolve any compatibility issues that may arise due to the migration process.
+
+<Note>
+As of version 0.26.0, NetBird used SQLite as its default database store. However, SQLite lacked support for cascading deletions,
+which means that related entries were not automatically removed when their parent entries were deleted.
+ With the migration to PostgreSQL, foreign key constraints are enforced, including cascading deletes, to ensure referential integrity.
+</Note>
+
+To address this, you can execute the following SQL commands in your PostgreSQL database:
+
+```sql
+DELETE FROM policy_rules WHERE policy_rules.policy_id NOT IN (SELECT id FROM policies);
+```
+
+```sql
+DELETE FROM routes WHERE routes.account_id NOT IN (SELECT id FROM accounts);
+```
+
+```sql
+DELETE FROM name_server_groups WHERE name_server_groups.account_id NOT IN (SELECT id FROM accounts);
+```
+
+4. Enable Postgres by updating the `management.json` file and setting the `Engine` field to `postgres` as the following example:
+```json
+"StoreConfig": {
+    "Engine": "postgres"
+}
+```
+
+5. Create `.env` file with the following content:
+```bash
+NETBIRD_STORE_ENGINE_POSTGRES_DSN="host=<PG_HOST> user=<PG_USER> password=<PG_PASSWORD> dbname=<PG_DB_NAME> port=<PG_PORT>"
+```
+
+6. Update `docker-compose.yml` file to  pass the postgres configuration to the `management` container
+```yaml
+environment:
+  - NETBIRD_STORE_ENGINE_POSTGRES_DSN=${NETBIRD_STORE_ENGINE_POSTGRES_DSN}
+env_file:
+  - .env
+```
+
+7. Restart the management service
+```bash
+docker compose restart management
+```
+
+8. Check logs to confirm the store switch:
+```bash
+docker compose logs management
+```
+You should see an entry similar to:
+```
+2024-05-10T15:09:34Z INFO management/server/store.go:109: using Postgres store engine
+```
+
+## Rollback to Sqlite store
+To rollback to the Sqlite  store, follow these steps:
+<Note>
+    The following commands assume you use the latest docker version with the compose plugin. If you have docker-compose installed as a standalone, please use docker-compose as a command.
+</Note>
+
+1. Restore `store.db` backup
+```bash
+docker compose cp backup/. management:/var/lib/netbird/
+```
+
+2. Enable SQLite by updating the `management.json` file and setting the `Engine` field to `sqlite` as the following example:
+```json
+"StoreConfig": {
+    "Engine": "sqlite"
+}
+```
+
+3. Restart the Management service.
+```bash
+docker compose restart management
+```
+
+4. Check logs to confirm the store switch:
+```bash
+docker compose logs management
+```
+
+You should see an entry similar to:
+```
+2024-05-10T15:09:34Z INFO management/server/store.go:109: using SQLite file store engine
+```
+
+## Optional Rollback Postgres data to Sqlite
+
+This is optional and should be used only if you want to rollback the data from Postgres to SQLite while running the same NetBird version.
+For migrating the Postgres data, we rely on the `pg_dump`, `sed`, and `sqlite3` tools. Make sure these are installed before proceeding
+
+1. Export Postgres data
+```bash
+pg_dump --data-only --inserts "postgresql://<PG_USER>:<PG_PASSWORD>@<PG_HOST>:<PG_PORT>/<PG_DB_NAME>" > data.sql
+```
+
+2. Convert exported Postgres data sql to Sqlite format
+```bash
+sed \
+-e 's/\\\\:/\:/g' \
+-e 's/\\\\//g' \
+-e 's/\\\\;/;/g' \
+-e '/^SET /d' \
+-e '/setval/d' \
+-e "s/'true'/1/g" \
+-e "s/'false'/0/g" \
+-e 's/public\.//' \
+-e '/^[[:space:]]*SELECT/d' data.sql > data.sql
+```
+
+3. Generate database schema from Sqlite backup
+```bash
+sqlite3 backup/store.db '.schema' > schema.sql
+````
+
+4. Create Sqlite database with Postgres exported data
+```bash
+sqlite3 store.db '.read schema.sql' && sqlite3 store.db '.read data.sql'
+```
+
+5. Copy db to the management container
+```bash
+docker compose cp store.db management:/var/lib/netbird/store.db
+```
+
+6. Enable SQLite by updating the `management.json` file and setting the `Engine` field to `sqlite` as the following example:
+```json
+"StoreConfig": {
+    "Engine": "sqlite"
+}
+```
+
+7. Restart the Management service.
+```bash
+docker compose restart management
+```
+
+8. Check logs to confirm the store switch:
+```bash
+docker compose logs management
+```
+
+You should see an entry similar to:
+```
+2024-05-10T15:09:34Z INFO management/server/store.go:109: using SQLite file store engine
+```