[management] Optimize network map updates (#2718)

* Skip peer update on unchanged network map (#2236) * Enhance network updates by skipping unchanged messages Optimizes the network update process by skipping updates where no changes in the peer update message received. * Add unit tests * add locks * Improve concurrency and update peer message handling * Refactor account manager network update tests * fix test * Fix inverted network map update condition * Add default group and policy to test data * Run peer updates in a separate goroutine * Refactor * Refactor lock * Fix peers update by including NetworkMap and posture Checks * go mod tidy * fix merge Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix merge Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * [management] Skip account peers update if no changes affect peers (#2310) * Remove incrementing network serial and updating peers after group deletion * Update account peer if posture check is linked to policy * Remove account peers update on saving setup key * Refactor group link checking into re-usable functions * Add HasPeers function to group * Refactor group management * Optimize group change effects on account peers * Update account peers if ns group has peers * Refactor group changes * Optimize account peers update in DNS settings * Optimize update of account peers on jwt groups sync * Refactor peer account updates for efficiency * Optimize peer update on user deletion and changes * Remove condition check for network serial update * Optimize account peers updates on route changes * Remove UpdatePeerSSHKey method * Remove unused isPolicyRuleGroupsEmpty * Add tests for peer update behavior on posture check changes * Add tests for peer update behavior on policy changes * Add tests for peer update behavior on group changes * Add tests for peer update behavior on dns settings changes * Refactor * Add tests for peer update behavior on name server changes * Add tests for peer update behavior on user changes * Add tests for peer update behavior on route changes * fix tests * Add tests for peer update behavior on setup key changes * Add tests for peer update behavior on peers changes * fix merge * Fix tests * go mod tidy * Add NameServer and Route comparators * Update network map diff logic with custom comparators * Add tests * Refactor duplicate diff handling logic * fix linter * fix tests * Refactor policy group handling and update logic. Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Update route check by checking if group has peers Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Refactor posture check policy linking logic Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Simplify peer update condition in DNS management Refactor the condition for updating account peers to remove redundant checks Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix merge Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add policy tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add posture checks tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix user and setup key tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix account and route tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix typo Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix nameserver tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix routes tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix group tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * upgrade diff package Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix nameserver tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * use generic differ for netip.Addr and netip.Prefix Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * go mod tidy Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add peer tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix merge Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix management suite tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * fix postgres tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * enable diff nil structs comparison Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * skip the update only last sent the serial is larger Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * refactor peer and user Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * skip spell check for groupD Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Refactor group, ns group, policy and posture checks Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * skip spell check for GroupD Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * update account policy check before verifying policy status Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Update management/server/route_test.go Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> * Update management/server/route_test.go Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> * Update management/server/route_test.go Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> * Update management/server/route_test.go Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> * Update management/server/route_test.go Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> * add tests missing tests for dns setting groups Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add tests for posture checks changes Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add ns group and policy tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add route and group tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * increase Linux test timeout to 10 minutes Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Run diff for client posture checks only Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * add panic recovery and detailed logging in peer update comparison Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> * Fix tests Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> --------- Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> Co-authored-by: Maycon Santos <mlsmaycon@gmail.com> --------- Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com> Co-authored-by: Maycon Santos <mlsmaycon@gmail.com>
2026-04-18 08:16:39 +00:00 · 2024-10-23 13:05:02 +03:00
parent 30ebcf38c7
commit 7bda385e1b
32 changed files with 3485 additions and 280 deletions
--- a/management/server/posture_checks_test.go
+++ b/management/server/posture_checks_test.go
@@ -3,7 +3,10 @@ package server
 import (
 	"context"
 	"testing"
+	"time"

+	"github.com/netbirdio/netbird/management/server/group"
+	"github.com/rs/xid"
 	"github.com/stretchr/testify/assert"

 	"github.com/netbirdio/netbird/management/server/posture"
@@ -118,3 +121,458 @@ func initTestPostureChecksAccount(am *DefaultAccountManager) (*Account, error) {

 	return am.Store.GetAccount(context.Background(), account.Id)
 }
+
+func TestPostureCheckAccountPeersUpdate(t *testing.T) {
+	manager, account, peer1, peer2, peer3 := setupNetworkMapTest(t)
+
+	err := manager.SaveGroups(context.Background(), account.Id, userID, []*group.Group{
+		{
+			ID:    "groupA",
+			Name:  "GroupA",
+			Peers: []string{peer1.ID, peer2.ID, peer3.ID},
+		},
+		{
+			ID:    "groupB",
+			Name:  "GroupB",
+			Peers: []string{},
+		},
+		{
+			ID:    "groupC",
+			Name:  "GroupC",
+			Peers: []string{},
+		},
+	})
+	assert.NoError(t, err)
+
+	updMsg := manager.peersUpdateManager.CreateChannel(context.Background(), peer1.ID)
+	t.Cleanup(func() {
+		manager.peersUpdateManager.CloseChannel(context.Background(), peer1.ID)
+	})
+
+	postureCheck := posture.Checks{
+		ID:        "postureCheck",
+		Name:      "postureCheck",
+		AccountID: account.Id,
+		Checks: posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.28.0",
+			},
+		},
+	}
+
+	// Saving unused posture check should not update account peers and not send peer update
+	t.Run("saving unused posture check", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	// Updating unused posture check should not update account peers and not send peer update
+	t.Run("updating unused posture check", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		postureCheck.Checks = posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.29.0",
+			},
+		}
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	policy := Policy{
+		ID:      "policyA",
+		Enabled: true,
+		Rules: []*PolicyRule{
+			{
+				ID:            xid.New().String(),
+				Enabled:       true,
+				Sources:       []string{"groupA"},
+				Destinations:  []string{"groupA"},
+				Bidirectional: true,
+				Action:        PolicyTrafficActionAccept,
+			},
+		},
+		SourcePostureChecks: []string{postureCheck.ID},
+	}
+
+	// Linking posture check to policy should trigger update account peers and send peer update
+	t.Run("linking posture check to policy with peers", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		err := manager.SavePolicy(context.Background(), account.Id, userID, &policy, false)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldReceiveUpdate")
+		}
+	})
+
+	// Updating linked posture checks should update account peers and send peer update
+	t.Run("updating linked to posture check with peers", func(t *testing.T) {
+		postureCheck.Checks = posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.29.0",
+			},
+			ProcessCheck: &posture.ProcessCheck{
+				Processes: []posture.Process{
+					{LinuxPath: "/usr/bin/netbird", MacPath: "/usr/local/bin/netbird"},
+				},
+			},
+		}
+
+		done := make(chan struct{})
+		go func() {
+			peerShouldReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldReceiveUpdate")
+		}
+	})
+
+	// Saving unchanged posture check should not trigger account peers update and not send peer update
+	// since there is no change in the network map
+	t.Run("saving unchanged posture check", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	// Removing posture check from policy should trigger account peers update and send peer update
+	t.Run("removing posture check from policy", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		policy.SourcePostureChecks = []string{}
+
+		err := manager.SavePolicy(context.Background(), account.Id, userID, &policy, true)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldReceiveUpdate")
+		}
+	})
+
+	// Deleting unused posture check should not trigger account peers update and not send peer update
+	t.Run("deleting unused posture check", func(t *testing.T) {
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		err := manager.DeletePostureChecks(context.Background(), account.Id, "postureCheck", userID)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	err = manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+	assert.NoError(t, err)
+
+	// Updating linked posture check to policy with no peers should not trigger account peers update and not send peer update
+	t.Run("updating linked posture check to policy with no peers", func(t *testing.T) {
+		policy = Policy{
+			ID:      "policyB",
+			Enabled: true,
+			Rules: []*PolicyRule{
+				{
+					ID:            xid.New().String(),
+					Enabled:       true,
+					Sources:       []string{"groupB"},
+					Destinations:  []string{"groupC"},
+					Bidirectional: true,
+					Action:        PolicyTrafficActionAccept,
+				},
+			},
+			SourcePostureChecks: []string{postureCheck.ID},
+		}
+		err = manager.SavePolicy(context.Background(), account.Id, userID, &policy, false)
+		assert.NoError(t, err)
+
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		postureCheck.Checks = posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.29.0",
+			},
+		}
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	// Updating linked posture check to policy where destination has peers but source does not
+	// should trigger account peers update and send peer update
+	t.Run("updating linked posture check to policy where destination has peers but source does not", func(t *testing.T) {
+		updMsg1 := manager.peersUpdateManager.CreateChannel(context.Background(), peer2.ID)
+		t.Cleanup(func() {
+			manager.peersUpdateManager.CloseChannel(context.Background(), peer2.ID)
+		})
+		policy = Policy{
+			ID:      "policyB",
+			Enabled: true,
+			Rules: []*PolicyRule{
+				{
+					ID:            xid.New().String(),
+					Enabled:       true,
+					Sources:       []string{"groupB"},
+					Destinations:  []string{"groupA"},
+					Bidirectional: true,
+					Action:        PolicyTrafficActionAccept,
+				},
+			},
+			SourcePostureChecks: []string{postureCheck.ID},
+		}
+
+		err = manager.SavePolicy(context.Background(), account.Id, userID, &policy, true)
+		assert.NoError(t, err)
+
+		done := make(chan struct{})
+		go func() {
+			peerShouldReceiveUpdate(t, updMsg1)
+			close(done)
+		}()
+
+		postureCheck.Checks = posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.29.0",
+			},
+		}
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldReceiveUpdate")
+		}
+	})
+
+	// Updating linked posture check to policy where source has peers but destination does not,
+	// should not trigger account peers update or send peer update
+	t.Run("updating linked posture check to policy where source has peers but destination does not", func(t *testing.T) {
+		policy = Policy{
+			ID:      "policyB",
+			Enabled: true,
+			Rules: []*PolicyRule{
+				{
+					Enabled:       true,
+					Sources:       []string{"groupA"},
+					Destinations:  []string{"groupB"},
+					Bidirectional: true,
+					Action:        PolicyTrafficActionAccept,
+				},
+			},
+			SourcePostureChecks: []string{postureCheck.ID},
+		}
+		err = manager.SavePolicy(context.Background(), account.Id, userID, &policy, true)
+		assert.NoError(t, err)
+
+		done := make(chan struct{})
+		go func() {
+			peerShouldNotReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		postureCheck.Checks = posture.ChecksDefinition{
+			NBVersionCheck: &posture.NBVersionCheck{
+				MinVersion: "0.29.0",
+			},
+		}
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldNotReceiveUpdate")
+		}
+	})
+
+	// Updating linked client posture check to policy where source has peers but destination does not,
+	// should trigger account peers update and send peer update
+	t.Run("updating linked client posture check to policy where source has peers but destination does not", func(t *testing.T) {
+		policy = Policy{
+			ID:      "policyB",
+			Enabled: true,
+			Rules: []*PolicyRule{
+				{
+					Enabled:       true,
+					Sources:       []string{"groupA"},
+					Destinations:  []string{"groupB"},
+					Bidirectional: true,
+					Action:        PolicyTrafficActionAccept,
+				},
+			},
+			SourcePostureChecks: []string{postureCheck.ID},
+		}
+		err = manager.SavePolicy(context.Background(), account.Id, userID, &policy, true)
+		assert.NoError(t, err)
+
+		done := make(chan struct{})
+		go func() {
+			peerShouldReceiveUpdate(t, updMsg)
+			close(done)
+		}()
+
+		postureCheck.Checks = posture.ChecksDefinition{
+			ProcessCheck: &posture.ProcessCheck{
+				Processes: []posture.Process{
+					{
+						LinuxPath: "/usr/bin/netbird",
+					},
+				},
+			},
+		}
+		err := manager.SavePostureChecks(context.Background(), account.Id, userID, &postureCheck)
+		assert.NoError(t, err)
+
+		select {
+		case <-done:
+		case <-time.After(time.Second):
+			t.Error("timeout waiting for peerShouldReceiveUpdate")
+		}
+	})
+}
+
+func TestArePostureCheckChangesAffectingPeers(t *testing.T) {
+	account := &Account{
+		Policies: []*Policy{
+			{
+				ID: "policyA",
+				Rules: []*PolicyRule{
+					{
+						Enabled:      true,
+						Sources:      []string{"groupA"},
+						Destinations: []string{"groupA"},
+					},
+				},
+				SourcePostureChecks: []string{"checkA"},
+			},
+		},
+		Groups: map[string]*group.Group{
+			"groupA": {
+				ID:    "groupA",
+				Peers: []string{"peer1"},
+			},
+			"groupB": {
+				ID:    "groupB",
+				Peers: []string{},
+			},
+		},
+		PostureChecks: []*posture.Checks{
+			{
+				ID: "checkA",
+			},
+			{
+				ID: "checkB",
+			},
+		},
+	}
+
+	t.Run("posture check exists and is linked to policy with peers", func(t *testing.T) {
+		result := arePostureCheckChangesAffectingPeers(account, "checkA", true)
+		assert.True(t, result)
+	})
+
+	t.Run("posture check exists but is not linked to any policy", func(t *testing.T) {
+		result := arePostureCheckChangesAffectingPeers(account, "checkB", true)
+		assert.False(t, result)
+	})
+
+	t.Run("posture check does not exist", func(t *testing.T) {
+		result := arePostureCheckChangesAffectingPeers(account, "unknown", false)
+		assert.False(t, result)
+	})
+
+	t.Run("posture check is linked to policy with no peers in source groups", func(t *testing.T) {
+		account.Policies[0].Rules[0].Sources = []string{"groupB"}
+		account.Policies[0].Rules[0].Destinations = []string{"groupA"}
+		result := arePostureCheckChangesAffectingPeers(account, "checkA", true)
+		assert.True(t, result)
+	})
+
+	t.Run("posture check is linked to policy with no peers in destination groups", func(t *testing.T) {
+		account.Policies[0].Rules[0].Sources = []string{"groupA"}
+		account.Policies[0].Rules[0].Destinations = []string{"groupB"}
+		result := arePostureCheckChangesAffectingPeers(account, "checkA", true)
+		assert.True(t, result)
+	})
+
+	t.Run("posture check is linked to policy with non-existent group", func(t *testing.T) {
+		account.Policies[0].Rules[0].Sources = []string{"nonExistentGroup"}
+		account.Policies[0].Rules[0].Destinations = []string{"nonExistentGroup"}
+		result := arePostureCheckChangesAffectingPeers(account, "checkA", true)
+		assert.False(t, result)
+	})
+
+	t.Run("posture check is linked to policy but no peers in groups", func(t *testing.T) {
+		account.Groups["groupA"].Peers = []string{}
+		result := arePostureCheckChangesAffectingPeers(account, "checkA", true)
+		assert.False(t, result)
+	})
+}