client/server: throttle and single-flight health probes

Status(GetFullPeerStatus=true) RPCs trigger a full health probe
(network round-trips to management, signal and the relays). The
desktop UI issues these frequently and concurrently, and a burst of
parallel Get() calls each fired its own probe — the lastProbe guard
was unprotected against concurrent access and only advanced when every
component was healthy, so a sustained unhealthy state (e.g. relay down)
disabled the throttle entirely and let every call re-probe.

Extract the throttle/single-flight policy into probeThrottle:
  - single-flight: only one probe runs at a time; concurrent callers
    that piled up while it ran share its result instead of each
    launching another, even when that probe failed.
  - throttle: lastOK only advances on a fully successful probe, so
    while anything is unhealthy callers keep probing frequently and
    notice recovery quickly (preserved from the original design).

RunHealthProbes now takes a context so a caller that gives up (e.g. a
Status RPC whose client disconnected) cancels the in-flight STUN/TURN
probe instead of letting it run to its per-component timeout. The
engine's own lifetime ctx still applies independently.
This commit is contained in:
Zoltán Papp
2026-06-01 21:07:12 +02:00
parent 4cee07bef5
commit 60c86c63aa
7 changed files with 229 additions and 22 deletions

View File

@@ -1173,7 +1173,7 @@ func (e *Engine) handleBundle(params *mgmProto.BundleParameters) (*mgmProto.JobR
TempDir: e.config.TempDir,
ClientMetrics: e.clientMetrics,
RefreshStatus: func() {
e.RunHealthProbes(true)
e.RunHealthProbes(e.ctx, true)
},
}
@@ -2058,7 +2058,20 @@ func (e *Engine) getRosenpassAddr() string {
// RunHealthProbes executes health checks for Signal, Management, Relay, and WireGuard services
// and updates the status recorder with the latest states.
func (e *Engine) RunHealthProbes(waitForResult bool) bool {
//
// ctx scopes the (potentially slow) STUN/TURN probing: a caller that gives up —
// e.g. a Status RPC whose client disconnected — cancels its ctx and the probe
// returns instead of running to its per-component timeout. The engine's own
// lifetime ctx still applies independently, so an engine shutdown aborts the
// probe even if the caller's ctx is context.Background().
func (e *Engine) RunHealthProbes(ctx context.Context, waitForResult bool) bool {
// Tie the caller's ctx to the engine lifetime: either cancelling aborts
// the probe below.
ctx, cancel := context.WithCancel(ctx)
defer cancel()
stop := context.AfterFunc(e.ctx, cancel)
defer stop()
e.syncMsgMux.Lock()
signalHealthy := e.signal.IsHealthy()
@@ -2081,9 +2094,9 @@ func (e *Engine) RunHealthProbes(waitForResult bool) bool {
if runtime.GOOS != "js" {
var results []relay.ProbeResult
if waitForResult {
results = e.probeStunTurn.ProbeAllWaitResult(e.ctx, stuns, turns)
results = e.probeStunTurn.ProbeAllWaitResult(ctx, stuns, turns)
} else {
results = e.probeStunTurn.ProbeAll(e.ctx, stuns, turns)
results = e.probeStunTurn.ProbeAll(ctx, stuns, turns)
}
e.statusRecorder.UpdateRelayStates(results)