mirror of
https://github.com/netbirdio/netbird.git
synced 2026-09-20 05:39:07 +02:00
[client] warm lazy connections from the DNS resolver (#6854)
## Describe your changes Supersedes #6767 (same change, moved to an unprefixed branch; review feedback from there is addressed here). With lazy connections enabled, a peer is not dialed until on-demand traffic arrives, so the first request to a peer resolved by name (e.g. the agent-network reverse-proxy) races — or loses to — the WireGuard handshake. Activation was previously reactive only: a data-path packet or an inbound signal. This adds a proactive, DNS-time trigger. When the local resolver answers for an overlay name, it now warms the lazy connection to the peer(s) the answer points at and waits briefly for one to connect before returning the response — so by the time the client sends its first packet the tunnel is already up. - `dns/local`: new `PeerActivator` capability + `SetPeerActivator` setter (mirrors the existing `PeerConnectivity`/`SetPeerConnectivity` injection). `ServeDNS` warms on the pre-filter answer, so activating a lazily-idle peer also lets it survive the disconnected-peer filter. Warm-up is scoped to match-only (non-authoritative) zones — the synthesized private-service zones and user-created zones — so plain peer-name lookups in the account's authoritative peer zone never wake idle peers. No-op when no activator is wired (lazy off) or the answer carries no peer IPs. Budget is `NB_DNS_LAZY_WARMUP_TIMEOUT` (default 2s, parsed once at construction, invalid values logged); on timeout the answer is returned anyway (never SERVFAIL). - The resolver-facing interfaces (`PeerActivator`, `PeerConnectivity`) take `netip.Addr` instead of string IPs; record addresses are extracted as `netip.Addr` (v4-mapped forms unmapped) and converted to string only at the `peer.Status` boundary. - `SetPeerActivator` is part of the `dns.Server` interface (no-op on the mock), so the engine wires it without a type assertion. - `client/internal`: a small engine-side adapter (`dnsPeerActivator`) resolves answer IPs to peers via `Status.PeerStateByIP`, activates them through `ConnMgr.ActivatePeer` (HA fan-out included), and polls `PeerStateByIP` until one is connected. The activation dial is tied to the engine's long-lived context so a handshake that outlasts the per-query wait still completes in the background. `ConnMgr.ActivatePeer` is safe for concurrent use (the lazy manager pointer is guarded by a dedicated RWMutex and the manager is internally synchronized), so the DNS path never contends with network-map processing on `syncMsgMux`. Scope is overlay-only for free: the trigger lives in the local resolver, which only answers for NetBird-managed names; public/upstream DNS is unaffected. Already-connected peers short-circuit, so steady-state DNS latency is unchanged. ## Issue ticket number and link N/A — follow-up to the lazy-connection rollout; fixes the agent-network cold-start observed in the e2e (proxy peer stuck disconnected until traffic). ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) - [x] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) Internal client behavior; the only knob is the optional `NB_DNS_LAZY_WARMUP_TIMEOUT` tuning env var with a safe default. ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: https://github.com/netbirdio/docs/pull/__ ## Tests - `dns/local`: warm-up invokes the activator with the answer's peer address in match-only zones; authoritative-zone answers never trigger warm-up; no-activator path unchanged; no-answer queries don't invoke the activator; `NB_DNS_LAZY_WARMUP_TIMEOUT` parsing (valid/invalid/non-positive); `extractRecordAddr` unmaps v4-mapped record data. - `client/internal`: `dnsPeerActivator` skips connected/unknown/conn-less peers with no wait, returns as soon as a pending peer connects, and releases the DNS response at the budget when the peer stays idle; `ConnMgr.ActivatePeer` races the manager lifecycle cleanly under `-race`. - Agent-network e2e green on this change (with lazy connections enabled): https://github.com/netbirdio/netbird/actions/runs/29891467544
This commit is contained in:
@@ -91,7 +91,14 @@ func availableProviders() []providerCase {
|
||||
if region == "" {
|
||||
region = "us-east-1"
|
||||
}
|
||||
ps = append(ps, providerCase{name: "bedrock", catalogID: "bedrock_api", upstream: "https://bedrock-runtime." + region + ".amazonaws.com", apiKey: k, model: "us.anthropic.claude-haiku-4-5", kind: harness.WireBedrock})
|
||||
// A valid Bedrock inference-profile id (region prefix + date + version),
|
||||
// overridable per account. `global.` profiles can be invoked from any
|
||||
// region; set AWS_BEDROCK_MODEL to match the enabled profile for the token.
|
||||
model := os.Getenv("AWS_BEDROCK_MODEL")
|
||||
if model == "" {
|
||||
model = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
}
|
||||
ps = append(ps, providerCase{name: "bedrock", catalogID: "bedrock_api", upstream: "https://bedrock-runtime." + region + ".amazonaws.com", apiKey: k, model: model, kind: harness.WireBedrock})
|
||||
}
|
||||
return ps
|
||||
}
|
||||
@@ -108,8 +115,16 @@ func providerRequest(pc providerCase) api.AgentNetworkProviderRequest {
|
||||
Enabled: ptr(true),
|
||||
}
|
||||
if pc.kind != harness.WireVertex {
|
||||
// The router matches the normalized catalog id. Bedrock's request model
|
||||
// travels as a region-prefixed inference-profile id in the URL path
|
||||
// (us.anthropic...), which the router strips before matching, so register
|
||||
// the normalized form here or routing fails as model_not_routable.
|
||||
modelID := pc.model
|
||||
if pc.kind == harness.WireBedrock {
|
||||
modelID = catalogModel(pc)
|
||||
}
|
||||
req.Models = &[]api.AgentNetworkProviderModel{
|
||||
{Id: pc.model, InputPer1k: 0.001, OutputPer1k: 0.002},
|
||||
{Id: modelID, InputPer1k: 0.001, OutputPer1k: 0.002},
|
||||
}
|
||||
}
|
||||
return req
|
||||
@@ -201,11 +216,13 @@ func TestProvidersMatrix(t *testing.T) {
|
||||
t.Cleanup(func() { _ = cl.Terminate(context.Background()) })
|
||||
|
||||
require.NoError(t, cl.WaitConnected(ctx, 90*time.Second), "client must connect to management")
|
||||
// Resolve first: the DNS lookup triggers the lazy-connection warm-up, waking
|
||||
// the proxy peer so WaitProxyPeer then observes it connected.
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve agent-network endpoint to proxy IP")
|
||||
if err := cl.WaitProxyPeer(ctx, 180*time.Second); err != nil {
|
||||
t.Fatalf("client did not see the proxy peer: %v\n=== proxy logs ===\n%s", err, px.Logs(context.Background()))
|
||||
}
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve agent-network endpoint to proxy IP")
|
||||
|
||||
for _, pc := range matrix {
|
||||
pc := pc
|
||||
|
||||
@@ -4,6 +4,7 @@ package agentnetwork
|
||||
|
||||
import (
|
||||
"context"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -15,13 +16,29 @@ import (
|
||||
"github.com/netbirdio/netbird/shared/management/http/api"
|
||||
)
|
||||
|
||||
// bedrockRegionPrefixes and bedrockVersionSuffix mirror the proxy's Bedrock
|
||||
// model normalization (region/inference-profile prefix + version suffix) so the
|
||||
// provider is registered under the same catalog key the router matches against.
|
||||
var (
|
||||
bedrockRegionPrefixes = []string{"us.", "eu.", "apac.", "global."}
|
||||
bedrockVersionSuffix = regexp.MustCompile(`-(\d{8}-)?v\d+(:\d+)?$`)
|
||||
)
|
||||
|
||||
// catalogModel returns the normalized catalog id the proxy stamps for a
|
||||
// path-routed provider's configured model — the form the guardrail allowlist is
|
||||
// compared against (region prefix / @version stripped).
|
||||
// path-routed provider's configured model — the form the router and guardrail
|
||||
// allowlist compare against (Bedrock region prefix + version stripped, Vertex
|
||||
// @version stripped).
|
||||
func catalogModel(pc providerCase) string {
|
||||
switch pc.kind {
|
||||
case harness.WireBedrock:
|
||||
return strings.TrimPrefix(pc.model, "us.")
|
||||
m := pc.model
|
||||
for _, p := range bedrockRegionPrefixes {
|
||||
if strings.HasPrefix(m, p) {
|
||||
m = m[len(p):]
|
||||
break
|
||||
}
|
||||
}
|
||||
return bedrockVersionSuffix.ReplaceAllString(m, "")
|
||||
case harness.WireVertex:
|
||||
return strings.SplitN(pc.model, "@", 2)[0]
|
||||
default:
|
||||
@@ -147,11 +164,13 @@ func TestModelAllowlistEnforced(t *testing.T) {
|
||||
t.Cleanup(func() { _ = cl.Terminate(context.Background()) })
|
||||
|
||||
require.NoError(t, cl.WaitConnected(ctx, 90*time.Second), "client must connect to management")
|
||||
// Resolve first: the DNS lookup triggers the lazy-connection warm-up, waking
|
||||
// the proxy peer so WaitProxyPeer then observes it connected.
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve agent-network endpoint to proxy IP")
|
||||
if err := cl.WaitProxyPeer(ctx, 180*time.Second); err != nil {
|
||||
t.Fatalf("client did not see the proxy peer: %v\n=== proxy logs ===\n%s", err, px.Logs(context.Background()))
|
||||
}
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve agent-network endpoint to proxy IP")
|
||||
|
||||
for _, pc := range providers {
|
||||
pc := pc
|
||||
|
||||
@@ -104,11 +104,13 @@ func TestProviderSkipTLSVerification(t *testing.T) {
|
||||
t.Cleanup(func() { _ = cl.Terminate(context.Background()) })
|
||||
|
||||
require.NoError(t, cl.WaitConnected(ctx, 90*time.Second), "client must connect to management")
|
||||
// Resolve first: the DNS lookup triggers the lazy-connection warm-up, waking
|
||||
// the proxy peer so WaitProxyPeer then observes it connected.
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve endpoint to proxy IP")
|
||||
if err := cl.WaitProxyPeer(ctx, 180*time.Second); err != nil {
|
||||
t.Fatalf("client did not see the proxy peer: %v\n=== proxy logs ===\n%s", err, px.Logs(context.Background()))
|
||||
}
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve endpoint to proxy IP")
|
||||
|
||||
// Positive: skip=true reaches the self-signed upstream. Retry to absorb
|
||||
// tunnel/DNS jitter on the first call; success also proves the path works.
|
||||
|
||||
@@ -106,11 +106,13 @@ func TestVLLMProvider(t *testing.T) {
|
||||
t.Cleanup(func() { _ = cl.Terminate(context.Background()) })
|
||||
|
||||
require.NoError(t, cl.WaitConnected(ctx, 90*time.Second), "client must connect to management")
|
||||
// Resolve first: the DNS lookup triggers the lazy-connection warm-up, waking
|
||||
// the proxy peer so WaitProxyPeer then observes it connected.
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve endpoint to proxy IP")
|
||||
if err := cl.WaitProxyPeer(ctx, 180*time.Second); err != nil {
|
||||
t.Fatalf("client did not see the proxy peer: %v\n=== proxy logs ===\n%s", err, px.Logs(context.Background()))
|
||||
}
|
||||
proxyIP, err := cl.ResolveProxyIP(ctx, settings.Endpoint)
|
||||
require.NoError(t, err, "resolve endpoint to proxy IP")
|
||||
|
||||
before, _ := srv.ListAccessLogs(ctx)
|
||||
sessionID := "e2e-session-vllm"
|
||||
|
||||
Reference in New Issue
Block a user