diff --git a/src/components/NavigationDocs.jsx b/src/components/NavigationDocs.jsx index b03f4205..15384a75 100644 --- a/src/components/NavigationDocs.jsx +++ b/src/components/NavigationDocs.jsx @@ -138,6 +138,7 @@ export const docsNavigation = [ { title: 'Quickstart guide', href: '/selfhosted/selfhosted-quickstart' }, { title: 'Advanced guide', href: '/selfhosted/selfhosted-guide' }, { title: 'Management SQLite Store', href: '/selfhosted/sqlite-store'}, + { title: 'Management Postgres Store', href: '/selfhosted/postgres-store'}, { title: 'Supported IdPs', href: '/selfhosted/identity-providers' }, { title: 'Management geolocation', href: '/selfhosted/geo-support' }, { title: 'Troubleshooting', href: '/selfhosted/troubleshooting' }, diff --git a/src/pages/selfhosted/postgres-store.mdx b/src/pages/selfhosted/postgres-store.mdx new file mode 100644 index 00000000..333a28da --- /dev/null +++ b/src/pages/selfhosted/postgres-store.mdx @@ -0,0 +1,197 @@ +# Management Postgresql store + + +This feature provides experimental support for using Postgres as the storage engine. Please use with caution and ensure proper testing in your environment. + + +## Using Postgres for fresh installations + +As of version 0.26.0, the default configuration for fresh installations is SQLite storage. +However, users have the option to leverage the benefits of Postgres for new instances beginning from version ``. +To enable Postgres, add to your `setup.env` the following variable: +```bash +NETBIRD_STORE_CONFIG_ENGINE=postgres +``` +This will result in a configuration similar to the following in your `management.json` file: +```json + "StoreConfig": { + "Engine": "postgres" + } +``` +You can switch back to sqlite storage by setting the `NETBIRD_STORE_CONFIG_ENGINE` variable to `sqlite`. + + Switching between storage options requires migration steps to prevent data loss. + + + +## Migrating from SQLite store to Postgres store +This migration process allows users to seamlessly transition between storage options while maintaining data integrity. + + + The following commands assume you use the latest docker version with the compose plugin. If you have docker-compose installed as a standalone, please use docker-compose as a command. + + +1. Backup your data store (`store.db` in `datadir` - default `/var/lib/netbird/`) +```bash +mkdir backup +docker compose cp -a management:/var/lib/netbird/. backup/ +``` + +2. Import Sqlite data to Postgres + +For migrating the Sqlite data we rely on the [pgloader](https://github.com/dimitri/pgloader) tool. You can install it by running +`sudo apt-get install pgloader` on debian or `brew install pgloader` on MacOS. + +```bash +pgloader --type sqlite backup/store.db "postgresql://:@:/" +``` + +3. Resolve any compatibility issues that may arise due to the migration process. + + +As of version 0.26.0, NetBird used SQLite as its default database store. However, SQLite lacked support for cascading deletions, +which means that related entries were not automatically removed when their parent entries were deleted. + With the migration to PostgreSQL, foreign key constraints are enforced, including cascading deletes, to ensure referential integrity. + + +To address this, you can execute the following SQL commands in your PostgreSQL database: + +```sql +DELETE FROM policy_rules WHERE policy_rules.policy_id NOT IN (SELECT id FROM policies); +``` + +```sql +DELETE FROM routes WHERE routes.account_id NOT IN (SELECT id FROM accounts); +``` + +```sql +DELETE FROM name_server_groups WHERE name_server_groups.account_id NOT IN (SELECT id FROM accounts); +``` + +4. Enable Postgres by updating the `management.json` file and setting the `Engine` field to `postgres` as the following example: +```json +"StoreConfig": { + "Engine": "postgres" +} +``` + +5. Create `.env` file with the following content: +```bash +NETBIRD_STORE_ENGINE_POSTGRES_DSN="host= user= password= dbname= port=" +``` + +6. Update `docker-compose.yml` file to pass the postgres configuration to the `management` container +```yaml +environment: + - NETBIRD_STORE_ENGINE_POSTGRES_DSN=${NETBIRD_STORE_ENGINE_POSTGRES_DSN} +env_file: + - .env +``` + +7. Restart the management service +```bash +docker compose restart management +``` + +8. Check logs to confirm the store switch: +```bash +docker compose logs management +``` +You should see an entry similar to: +``` +2024-05-10T15:09:34Z INFO management/server/store.go:109: using Postgres store engine +``` + +## Rollback to Sqlite store +To rollback to the Sqlite store, follow these steps: + + The following commands assume you use the latest docker version with the compose plugin. If you have docker-compose installed as a standalone, please use docker-compose as a command. + + +1. Restore `store.db` backup +```bash +docker compose cp backup/. management:/var/lib/netbird/ +``` + +2. Enable SQLite by updating the `management.json` file and setting the `Engine` field to `sqlite` as the following example: +```json +"StoreConfig": { + "Engine": "sqlite" +} +``` + +3. Restart the Management service. +```bash +docker compose restart management +``` + +4. Check logs to confirm the store switch: +```bash +docker compose logs management +``` + +You should see an entry similar to: +``` +2024-05-10T15:09:34Z INFO management/server/store.go:109: using SQLite file store engine +``` + +## Optional Rollback Postgres data to Sqlite + +This is optional and should be used only if you want to rollback the data from Postgres to SQLite while running the same NetBird version. +For migrating the Postgres data, we rely on the `pg_dump`, `sed`, and `sqlite3` tools. Make sure these are installed before proceeding + +1. Export Postgres data +```bash +pg_dump --data-only --inserts "postgresql://:@:/" > data.sql +``` + +2. Convert exported Postgres data sql to Sqlite format +```bash +sed \ +-e 's/\\\\:/\:/g' \ +-e 's/\\\\//g' \ +-e 's/\\\\;/;/g' \ +-e '/^SET /d' \ +-e '/setval/d' \ +-e "s/'true'/1/g" \ +-e "s/'false'/0/g" \ +-e 's/public\.//' \ +-e '/^[[:space:]]*SELECT/d' data.sql > data.sql +``` + +3. Generate database schema from Sqlite backup +```bash +sqlite3 backup/store.db '.schema' > schema.sql +```` + +4. Create Sqlite database with Postgres exported data +```bash +sqlite3 store.db '.read schema.sql' && sqlite3 store.db '.read data.sql' +``` + +5. Copy db to the management container +```bash +docker compose cp store.db management:/var/lib/netbird/store.db +``` + +6. Enable SQLite by updating the `management.json` file and setting the `Engine` field to `sqlite` as the following example: +```json +"StoreConfig": { + "Engine": "sqlite" +} +``` + +7. Restart the Management service. +```bash +docker compose restart management +``` + +8. Check logs to confirm the store switch: +```bash +docker compose logs management +``` + +You should see an entry similar to: +``` +2024-05-10T15:09:34Z INFO management/server/store.go:109: using SQLite file store engine +``` \ No newline at end of file