Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.8 KiB
title
| title |
|---|
| WindowsCollectorTimeout |
Meaning
The collector identified by the collector label has reported
windows_exporter_collector_timeout == 1 for ten minutes. The collector did
not finish within the scrape timeout, or its call from an earlier scrape is
still running and later scrapes skip it until it returns.
Impact
The metrics of this collector are missing or stale on every scrape that times out, so dashboards show gaps and alerts on those metrics stop evaluating. A collector that never returns can hold a scrape slot and delay the other collectors of the host.
Diagnosis
-
Find the collector and compare how long it usually takes:
windows_exporter_collector_timeout{job="windows_exporter"} == 1 windows_exporter_collector_duration_seconds{job="windows_exporter"} -
Compare the duration with the scrape timeout of the job. The exporter subtracts
--scrape.timeout-marginfrom thescrape_timeoutthat Prometheus sends. -
Check the exporter log for the collector. WMI and Management Infrastructure queries, the Task Scheduler and Windows Update are slow on hosts with many objects or a busy WMI provider; the collector's documentation lists the expensive options.
-
Check whether the host was under CPU, disk or memory pressure at the same time, and whether a Windows service the collector depends on hangs, for example
Winmgmtfor WMI-based collectors.
Mitigation
Raise the scrape_timeout of the job within its scrape_interval, narrow the
collector with its include and exclude flags, or move the collector to a
separate job with a longer timeout. Restart a hanging provider service when the
log identifies it. Disable the collector on hosts where its data is not needed.
Confirm windows_exporter_collector_timeout returns to 0 and the collector's
metrics are published again.