Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.5 KiB
title
| title |
|---|
| WindowsScrapeQueueTimeout |
Meaning
windows_exporter_scrape_queue_timeout == 1 for ten minutes. Scrapes run one
at a time; a scrape whose timeout ends while it waits for an earlier scrape to
finish runs no collector and reports this metric as 1. On such a scrape
windows_exporter_collector_success is 0 for every collector.
Impact
The affected scrapes return no collector metrics, so the host has gaps in all
of its series and WindowsCollectorFailed can fire for every collector at
once. The exporter itself is reachable.
Diagnosis
-
Confirm the queue timeout and find the scrape that holds the slot:
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1 windows_exporter_collector_timeout{job="windows_exporter"} == 1 scrape_duration_seconds{job="windows_exporter"} -
Check whether more than one Prometheus server or job scrapes the same exporter, or whether a job scrapes it more often than its scrape duration allows. Compare
scrape_duration_secondswith thescrape_interval. -
Identify the slow collector with
windows_exporter_collector_duration_secondsand followWindowsCollectorTimeoutfor it.
Mitigation
Scrape each exporter from one job, or give the jobs a scrape_interval longer
than a full scrape takes. Shorten the slow collector or raise the
scrape_timeout so a scrape finishes before the next one starts. Confirm
windows_exporter_scrape_queue_timeout stays 0 and collector success returns
to 1.