Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.6 KiB
title
| title |
|---|
| WindowsNetworkUtilizationHigh |
Meaning
Received or sent throughput on the interface nic exceeds the configured share
of its link bandwidth for 15 minutes. The default is 90%. The recordings
windows:net_received_utilization:ratio and
windows:net_sent_utilization:ratio divide the byte rate of each direction by
windows_net_current_bandwidth_bytes.
Impact
The link is close to saturation in at least one direction. Further traffic queues at the adapter, which raises latency and causes discards and TCP retransmissions for every application on the host.
Diagnosis
- Inspect both utilization ratios to find the direction, and
windows:net_received_bytes:rateandwindows:net_sent_bytes:ratefor the absolute volume. - Check
windows_net_current_bandwidth_bytes. A link that negotiated a lower speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high utilization at normal traffic; compare it withGet-NetAdapter. - Correlate with
windows:net_received_discarded:rate,windows:net_outbound_discarded:rateandwindows_net_output_queue_length_packetsto see whether the saturation already drops packets. - Identify the traffic with Resource Monitor,
Get-NetTCPConnection, or switch-port statistics: backups, replication, file copies and monitoring bursts are common causes.
Mitigation
Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that negotiated the wrong speed. Confirm utilization returns below the threshold and discards stop. Tune the threshold for interfaces that run near line rate by design, such as storage replication links.