netbird/tools/idp-migrate/DEVELOPMENT.md

IdP Migration Tool — Developer Guide

Overview

This tool migrates NetBird deployments from an external IdP (Auth0, Zitadel, Okta, etc.) to the embedded Dex IdP introduced in v0.62.0. It does two things:

1. DB migration — Re-encodes every user ID from {original_id} to Dex's protobuf-encoded format base64(proto{original_id, connector_id}).
2. Config generation — Transforms management.json: removes IdpManagerConfig, PKCEAuthorizationFlow, and DeviceAuthorizationFlow; strips HttpConfig to only CertFile/CertKey; adds EmbeddedIdP with the static connector configuration.

Code Layout

tools/idp-migrate/
├── config.go            # migrationConfig struct, CLI flags, env vars, validation
├── main.go              # CLI entry point, migration phases, config generation
├── main_test.go         # 8 test functions (18 subtests) covering config, connector, URL builder, config generation
└── DEVELOPMENT.md       # this file

management/server/idp/migration/
├── migration.go         # Server interface, MigrateUsersToStaticConnectors(), PopulateUserInfo(), migrateUser(), reconcileActivityStore()
├── migration_test.go    # 6 top-level tests (with subtests) using hand-written mocks
└── store.go             # Store, EventStore interfaces, SchemaCheck, RequiredSchema, SchemaError types

management/server/store/
└── sql_store_idp_migration.go   # CheckSchema(), ListUsers(), UpdateUserInfo(), UpdateUserID(), txDeferFKConstraints() on SqlStore

management/server/activity/store/
├── sql_store_idp_migration.go      # UpdateUserID() on activity Store
└── sql_store_idp_migration_test.go # 5 subtests for activity UpdateUserID

Release / Distribution

The tool is included in .goreleaser.yaml as the netbird-idp-migrate build target. Each NetBird release produces pre-built archives for Linux (amd64, arm64, arm) that are uploaded to GitHub Releases. The archive naming convention is:

netbird-idp-migrate_<version>_linux_<arch>.tar.gz

The build requires CGO_ENABLED=1 because it links the SQLite driver used by SqlStore. The cross-compilation setup (CC env for arm64/arm) mirrors the netbird-mgmt build.

CLI Flags

Flag	Type	Default	Description
--config	string	(required)	Path to management.json
--datadir	string	(required)	Data directory (containing store.db / events.db)
--idp-seed-info	string	(required)	Base64-encoded connector JSON
--domain	string	""	Sets both dashboard and API domain (convenience shorthand)
--dashboard-domain	string	(required)	Dashboard domain (for redirect URIs)
--api-domain	string	(required)	API domain (for Dex issuer and callback URLs)
--dry-run	bool	false	Preview changes without writing
--force	bool	false	Skip interactive confirmation prompt
--skip-config	bool	false	Skip config generation (DB-only migration)
--skip-populate-user-info	bool	false	Skip populating user info (user ID migration only)
--log-level	string	"info"	Log level (debug, info, warn, error)

Environment Variables

All flags can be overridden via environment variables. Env vars take precedence over flags.

Env Var	Overrides
NETBIRD_DOMAIN	Sets both --dashboard-domain and --api-domain
NETBIRD_API_URL	--api-domain
NETBIRD_DASHBOARD_URL	--dashboard-domain
NETBIRD_CONFIG_PATH	--config
NETBIRD_DATA_DIR	--datadir
NETBIRD_IDP_SEED_INFO	--idp-seed-info
NETBIRD_DRY_RUN	--dry-run (set to "true")
NETBIRD_FORCE	--force (set to "true")
NETBIRD_SKIP_CONFIG	--skip-config (set to "true")
NETBIRD_SKIP_POPULATE_USER_INFO	--skip-populate-user-info (set to "true")
NETBIRD_LOG_LEVEL	--log-level

Resolution order: CLI flags are parsed first, then --domain sets both URLs, then NETBIRD_DOMAIN overrides both, then NETBIRD_API_URL / NETBIRD_DASHBOARD_URL override individually. After all resolution, validateConfig() ensures all required fields are set.

Migration Flow

Phase 0: Schema Validation

validateSchema() opens the store and calls CheckSchema(RequiredSchema) to verify that all tables and columns required by the migration exist in the database. If anything is missing, the tool exits with a descriptive error instructing the operator to start the management server (v0.66.4+) at least once so that automatic GORM migrations create the required schema.

Phase 1: Populate User Info

Unless --skip-populate-user-info is set, populateUserInfoFromIDP() runs before connector resolution:

1. Creates an IDP manager from the existing IdpManagerConfig in management.json.
2. Calls idpManager.GetAllAccounts() to fetch email and name for all users from the external IDP.
3. Calls migration.PopulateUserInfo() which iterates over all store users, skipping service users and users that already have both email and name populated. For Dex-encoded user IDs, it decodes back to the original IDP ID for lookup.
4. Updates the store with any missing email/name values.

This ensures user contact info is preserved before the ID migration makes the original IDP IDs inaccessible.

Phase 2: Connector Decoding

decodeConnectorConfig() base64-decodes and JSON-unmarshals the connector JSON provided via --idp-seed-info (or NETBIRD_IDP_SEED_INFO). It validates that the connector ID is non-empty. There is no auto-detection or fallback — the operator must provide the full connector configuration.

Phase 3: DB Migration

migrateDB() orchestrates the database migration:

1. openStores() opens the main store (SqlStore) and activity store (non-fatal if missing).
2. Type-asserts both to migration.Store / migration.EventStore.
3. previewUsers() scans all users — counts pending vs already-migrated (using DecodeDexUserID).
4. confirmPrompt() asks for interactive confirmation (unless --force or --dry-run).
5. Calls migration.MigrateUsersToStaticConnectors(srv, conn):
   • Reconciliation pass: fixes activity store references for users already migrated in the main DB but whose events still reference old IDs (from a previous partial failure).
   • Main loop: for each non-migrated user, calls migrateUser() which atomically updates the user ID in both the main store and activity store.
   • Dry-run: logs what would happen, skips all writes.

SqlStore.UpdateUserID() atomically updates the user's primary key and all foreign key references (peers, PATs, groups, policies, jobs, etc.) in a single transaction.

Phase 4: Config Generation

Unless --skip-config is set, generateConfig() runs:

1. Read — loads existing management.json as raw JSON to preserve unknown fields.

2. Strip — removes keys that are no longer needed:

   • IdpManagerConfig
   • PKCEAuthorizationFlow
   • DeviceAuthorizationFlow
   • All HttpConfig fields except CertFile and CertKey

3. Add EmbeddedIdP — inserts a minimal section with:

   • Enabled: true
   • Issuer built from --api-domain + /oauth2
   • DashboardRedirectURIs built from --dashboard-domain + /nb-auth and /nb-silent-auth
   • StaticConnectors containing the decoded connector, with redirectURI overridden to --api-domain + /oauth2/callback

4. Write — backs up original as management.json.bak, writes new config. In dry-run mode, prints to stdout instead.

Interface Decoupling

Migration methods (ListUsers, UpdateUserID) are not on the core store.Store or activity.Store interfaces. Instead, they're defined in migration/store.go:

type Store interface {
    ListUsers(ctx context.Context) ([]*types.User, error)
    UpdateUserID(ctx context.Context, accountID, oldUserID, newUserID string) error
    UpdateUserInfo(ctx context.Context, userID, email, name string) error
    CheckSchema(checks []SchemaCheck) []SchemaError
}

type EventStore interface {
    UpdateUserID(ctx context.Context, oldUserID, newUserID string) error
}

A Server interface wraps both stores for dependency injection:

type Server interface {
    Store() Store
    EventStore() EventStore // may return nil
}

The concrete SqlStore types already have these methods (in their respective sql_store_idp_migration.go files), so they satisfy the interfaces via Go's structural typing — zero changes needed on the core store interfaces. At runtime, the standalone tool type-asserts:

migStore, ok := mainStore.(migration.Store)

This keeps migration concerns completely separate from the core store contract.

Dex User ID Encoding

EncodeDexUserID(userID, connectorID) produces a manually-encoded protobuf with two string fields, then base64-encodes the result (raw, no padding). DecodeDexUserID reverses this. The migration loop uses DecodeDexUserID to detect already-migrated users (decode succeeds → skip).

See idp/dex/provider.go for the implementation.

Standalone Tool

The standalone tool (tools/idp-migrate/main.go) is the primary migration entry point. It opens stores directly, runs schema validation, populates user info from the external IDP, migrates user IDs, and generates the new config — then exits. Configuration is handled entirely through config.go which parses CLI flags and environment variables.

Running Tests

# Migration library
go test -v ./management/server/idp/migration/...

# Standalone tool
go test -v ./tools/idp-migrate/...

# Activity store migration tests
go test -v -run TestUpdateUserID ./management/server/activity/store/...

# Build locally
go build ./tools/idp-migrate/

Clean Removal

When migration tooling is no longer needed, delete:

1. tools/idp-migrate/ — entire directory
2. management/server/idp/migration/ — entire directory
3. management/server/store/sql_store_idp_migration.go — migration methods on main SqlStore
4. management/server/activity/store/sql_store_idp_migration.go — migration method on activity Store
5. management/server/activity/store/sql_store_idp_migration_test.go — tests for the above
6. In .goreleaser.yaml:
   • Remove the netbird-idp-migrate build entry
   • Remove the netbird-idp-migrate archive entry
7. Run go mod tidy

No core interfaces or mocks need editing — that's the point of the decoupling.