Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.7 KiB
title
| title |
|---|
| WindowsCPUSaturation |
Meaning
The number of threads waiting for a processor, averaged over the rate window
and divided by the number of logical processors, exceeds the configured
threshold for 15 minutes. The default is two waiting threads per logical
processor. The metric is windows:cpu_queue_per_core:ratio, built from
windows_system_processor_queue_length and windows:cpu_cores:count.
Impact
Runnable threads wait for CPU time, so applications respond slowly even when their own CPU usage looks moderate. A long queue with CPU usage below 100% points at a few busy cores, a single-threaded bottleneck, or processor scheduling problems such as parked cores or hypervisor contention.
Diagnosis
- Inspect
windows:cpu_queue_per_core:rationext towindows:cpu_usage:ratiofor the host. Saturation with high usage is a capacity problem; saturation with low usage points at scheduling. - Compare the per-core rates of
windows_cpu_time_seconds_total. One or two cores at 100% while others idle indicate a thread that cannot spread. - Check
windows_cpu_parking_status,windows_cpu_core_frequency_hertzand the power plan. On virtual machines check the hypervisor for CPU ready time or steal. - Use Task Manager or Performance Monitor (
System\Processor Queue Length,Process\Thread Count) to find the processes with many runnable threads.
Mitigation
Reduce or redistribute the workload, add logical processors, or fix the power plan and parking settings. Give the virtual machine more CPU share or move it to a less loaded host. Tune the threshold for hosts whose workload keeps a long queue by design, such as build servers. Confirm the queue length per core and application latency return to normal.