mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-17 18:19:04 +02:00
mixin: group alerts by collector area and add tier 1 alerts
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
@@ -8,5 +8,7 @@ the affected host. Disk alerts also identify a `volume`; network alerts identify
|
||||
a `nic`. Adjust example PromQL selectors to match your scrape configuration.
|
||||
|
||||
The default thresholds are starting points. Check the generated alert rules for
|
||||
your configured thresholds and durations. Active Directory and NTP clock alerts
|
||||
are opt-in and require their collectors to be enabled.
|
||||
your configured thresholds and durations. The rules are grouped by collector
|
||||
area, and each alert is generated only when the collectors it reads are enabled
|
||||
in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
|
||||
and `time` collectors, which are off by default.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: WindowsCollectorTimeout
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The collector identified by the `collector` label has reported
|
||||
`windows_exporter_collector_timeout == 1` for ten minutes. The collector did
|
||||
not finish within the scrape timeout, or its call from an earlier scrape is
|
||||
still running and later scrapes skip it until it returns.
|
||||
|
||||
## Impact
|
||||
|
||||
The metrics of this collector are missing or stale on every scrape that times
|
||||
out, so dashboards show gaps and alerts on those metrics stop evaluating. A
|
||||
collector that never returns can hold a scrape slot and delay the other
|
||||
collectors of the host.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Find the collector and compare how long it usually takes:
|
||||
|
||||
```promql
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
windows_exporter_collector_duration_seconds{job="windows_exporter"}
|
||||
```
|
||||
|
||||
2. Compare the duration with the scrape timeout of the job. The exporter
|
||||
subtracts `--scrape.timeout-margin` from the `scrape_timeout` that
|
||||
Prometheus sends.
|
||||
3. Check the exporter log for the collector. WMI and Management Infrastructure
|
||||
queries, the Task Scheduler and Windows Update are slow on hosts with many
|
||||
objects or a busy WMI provider; the collector's documentation lists the
|
||||
expensive options.
|
||||
4. Check whether the host was under CPU, disk or memory pressure at the same
|
||||
time, and whether a Windows service the collector depends on hangs, for
|
||||
example `Winmgmt` for WMI-based collectors.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Raise the `scrape_timeout` of the job within its `scrape_interval`, narrow the
|
||||
collector with its include and exclude flags, or move the collector to a
|
||||
separate job with a longer timeout. Restart a hanging provider service when the
|
||||
log identifies it. Disable the collector on hosts where its data is not needed.
|
||||
Confirm `windows_exporter_collector_timeout` returns to `0` and the collector's
|
||||
metrics are published again.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: WindowsCPUSaturation
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The number of threads waiting for a processor, averaged over the rate window
|
||||
and divided by the number of logical processors, exceeds the configured
|
||||
threshold for 15 minutes. The default is two waiting threads per logical
|
||||
processor. The metric is `windows:cpu_queue_per_core:ratio`, built from
|
||||
`windows_system_processor_queue_length` and `windows:cpu_cores:count`.
|
||||
|
||||
## Impact
|
||||
|
||||
Runnable threads wait for CPU time, so applications respond slowly even when
|
||||
their own CPU usage looks moderate. A long queue with CPU usage below 100%
|
||||
points at a few busy cores, a single-threaded bottleneck, or processor
|
||||
scheduling problems such as parked cores or hypervisor contention.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows:cpu_queue_per_core:ratio` next to `windows:cpu_usage:ratio`
|
||||
for the host. Saturation with high usage is a capacity problem; saturation
|
||||
with low usage points at scheduling.
|
||||
2. Compare the per-core rates of `windows_cpu_time_seconds_total`. One or two
|
||||
cores at 100% while others idle indicate a thread that cannot spread.
|
||||
3. Check `windows_cpu_parking_status`, `windows_cpu_core_frequency_hertz` and
|
||||
the power plan. On virtual machines check the hypervisor for CPU ready time
|
||||
or steal.
|
||||
4. Use Task Manager or Performance Monitor (`System\Processor Queue Length`,
|
||||
`Process\Thread Count`) to find the processes with many runnable threads.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Reduce or redistribute the workload, add logical processors, or fix the power
|
||||
plan and parking settings. Give the virtual machine more CPU share or move it to
|
||||
a less loaded host. Tune the threshold for hosts whose workload keeps a long
|
||||
queue by design, such as build servers. Confirm the queue length per core and
|
||||
application latency return to normal.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: WindowsHostRebooted
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The boot time of the host, `windows_system_boot_time_timestamp`, is less than
|
||||
ten minutes old and differs from the boot time known ten minutes earlier. The
|
||||
alert fires once for about ten minutes after a reboot and then resolves by
|
||||
itself. It has the `info` severity and is meant for a dashboard, a ticket or a
|
||||
chat notification, not for a page.
|
||||
|
||||
## Impact
|
||||
|
||||
Every service of the host restarted, connections to it were dropped and
|
||||
caches are cold. A reboot outside a maintenance window can mean a crash, an
|
||||
unattended Windows Update restart or a power event.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Compare the boot time with the maintenance calendar and the Windows Update
|
||||
schedule. With the `update` collector enabled, `windows_update_pending_info`
|
||||
shows updates that were pending before the reboot.
|
||||
2. Check the System event log: event 1074 names the user or process that
|
||||
requested the shutdown, event 6008 marks an unexpected shutdown and event 41
|
||||
a reset without a clean shutdown. Check for a memory dump when it was a
|
||||
crash.
|
||||
3. Confirm the host came back with the expected services; the
|
||||
`WindowsServiceNotRunning` alert covers automatic services that did not
|
||||
start.
|
||||
|
||||
## Mitigation
|
||||
|
||||
No action is needed for a planned reboot. For an unplanned one, find the cause
|
||||
in the event log and fix the driver, update setting or power problem. Turn the
|
||||
alert off with `alerts+: { WindowsHostRebooted: false }` when reboots are not
|
||||
worth a notification.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: WindowsMemoryPagingHigh
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The rate of pages read from or written to disk to resolve hard page faults,
|
||||
`windows:memory_swap_page_operations:rate`, exceeds the configured threshold
|
||||
for 15 minutes. The default is 1000 pages per second. The counter counts
|
||||
pagefile traffic and memory-mapped file I/O alike.
|
||||
|
||||
## Impact
|
||||
|
||||
The host is short of physical memory and uses the disk as backing store, so
|
||||
every affected process slows down to disk speed. Sustained paging also
|
||||
competes with application I/O on the system disk.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect the paging rate together with `windows:memory_usage:ratio`,
|
||||
`windows_memory_available_bytes` and `windows:memory_committed:ratio`. Low
|
||||
available memory confirms memory pressure; a high rate with ample memory
|
||||
points at a process reading a large memory-mapped file, such as a backup or
|
||||
an index rebuild.
|
||||
2. Split the rate into reads and writes with
|
||||
`windows_memory_swap_page_reads_total` and
|
||||
`windows_memory_swap_page_writes_total`. Reads dominate when the working
|
||||
sets of processes do not fit; writes dominate when the system trims and
|
||||
writes modified pages.
|
||||
3. Find the processes with the largest working sets and page fault rates in
|
||||
Resource Monitor or, with the `process` collector enabled, with
|
||||
`windows_process_working_set_private_bytes` and
|
||||
`windows_process_page_faults_total`.
|
||||
4. Check `windows_memory_standby_cache_lifetime_seconds`; a low lifetime means
|
||||
cached pages are repurposed quickly, another sign of memory pressure.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Stop or move the process that uses the memory, fix a memory leak, or add
|
||||
physical memory. Move large sequential file jobs off peak hours when they cause
|
||||
the paging. Confirm the paging rate drops and available memory recovers. Raise
|
||||
the threshold on hosts whose workload maps large files by design.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: WindowsNetworkInterfaceFlapping
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The operational status of the interface `nic` changed more than twice within
|
||||
the rate window, and did so for two minutes. The alert counts the changes of
|
||||
`windows_net_nic_operation_status{status="up"}` between `1` and `0`.
|
||||
|
||||
## Impact
|
||||
|
||||
Every transition drops the connections that use the interface and triggers
|
||||
route, DHCP and DNS registration changes. A flapping uplink on a server causes
|
||||
intermittent application failures that are hard to attribute.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_net_nic_operation_status` for the interface to see the
|
||||
status it flips to; `down` and `lower layer down` point at the link,
|
||||
`dormant` at authentication such as 802.1X.
|
||||
2. Check the System event log for NDIS, the adapter driver and
|
||||
`Microsoft-Windows-NetworkProfile` events, and the switch port for link
|
||||
state changes, errors and port security actions.
|
||||
3. Check the cable, SFP, speed and duplex negotiation, and whether a teaming or
|
||||
failover configuration switches between members.
|
||||
4. On virtual machines, check the hypervisor for virtual switch changes, live
|
||||
migrations, or a host adapter that flaps.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Replace the cable or transceiver, fix the speed and duplex settings, update the
|
||||
adapter driver, or correct the switch port and authentication configuration.
|
||||
Exclude an interface that flaps by design, such as a Wi-Fi or VPN adapter,
|
||||
with the `nicSelector`. Confirm the status stays `up`.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: WindowsNetworkUtilizationHigh
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Received or sent throughput on the interface `nic` exceeds the configured share
|
||||
of its link bandwidth for 15 minutes. The default is 90%. The recordings
|
||||
`windows:net_received_utilization:ratio` and
|
||||
`windows:net_sent_utilization:ratio` divide the byte rate of each direction by
|
||||
`windows_net_current_bandwidth_bytes`.
|
||||
|
||||
## Impact
|
||||
|
||||
The link is close to saturation in at least one direction. Further traffic
|
||||
queues at the adapter, which raises latency and causes discards and TCP
|
||||
retransmissions for every application on the host.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect both utilization ratios to find the direction, and
|
||||
`windows:net_received_bytes:rate` and `windows:net_sent_bytes:rate` for the
|
||||
absolute volume.
|
||||
2. Check `windows_net_current_bandwidth_bytes`. A link that negotiated a lower
|
||||
speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high
|
||||
utilization at normal traffic; compare it with `Get-NetAdapter`.
|
||||
3. Correlate with `windows:net_received_discarded:rate`,
|
||||
`windows:net_outbound_discarded:rate` and
|
||||
`windows_net_output_queue_length_packets` to see whether the saturation
|
||||
already drops packets.
|
||||
4. Identify the traffic with Resource Monitor, `Get-NetTCPConnection`, or
|
||||
switch-port statistics: backups, replication, file copies and monitoring
|
||||
bursts are common causes.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that
|
||||
negotiated the wrong speed. Confirm utilization returns below the threshold and
|
||||
discards stop. Tune the threshold for interfaces that run near line rate by
|
||||
design, such as storage replication links.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: WindowsScrapeQueueTimeout
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
`windows_exporter_scrape_queue_timeout == 1` for ten minutes. Scrapes run one
|
||||
at a time; a scrape whose timeout ends while it waits for an earlier scrape to
|
||||
finish runs no collector and reports this metric as `1`. On such a scrape
|
||||
`windows_exporter_collector_success` is `0` for every collector.
|
||||
|
||||
## Impact
|
||||
|
||||
The affected scrapes return no collector metrics, so the host has gaps in all
|
||||
of its series and `WindowsCollectorFailed` can fire for every collector at
|
||||
once. The exporter itself is reachable.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Confirm the queue timeout and find the scrape that holds the slot:
|
||||
|
||||
```promql
|
||||
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
scrape_duration_seconds{job="windows_exporter"}
|
||||
```
|
||||
|
||||
2. Check whether more than one Prometheus server or job scrapes the same
|
||||
exporter, or whether a job scrapes it more often than its scrape duration
|
||||
allows. Compare `scrape_duration_seconds` with the `scrape_interval`.
|
||||
3. Identify the slow collector with `windows_exporter_collector_duration_seconds`
|
||||
and follow `WindowsCollectorTimeout` for it.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Scrape each exporter from one job, or give the jobs a `scrape_interval` longer
|
||||
than a full scrape takes. Shorten the slow collector or raise the
|
||||
`scrape_timeout` so a scrape finishes before the next one starts. Confirm
|
||||
`windows_exporter_scrape_queue_timeout` stays `0` and collector success returns
|
||||
to `1`.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: WindowsServiceCrashLooping
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The service `service` has run as more processes in the last 15 minutes than
|
||||
the configured number of restarts allows. The default is two restarts. Every
|
||||
service process has its own `windows_service_process` series, identified by
|
||||
`process_id`, so the series seen within 15 minutes count the restarts.
|
||||
|
||||
## Impact
|
||||
|
||||
The service crashes or is stopped and started repeatedly. It is unavailable
|
||||
during every restart, and the recovery options of the Service Control Manager
|
||||
give up after a configurable number of failures, after which
|
||||
`WindowsServiceNotRunning` follows.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. List the processes the service ran as:
|
||||
|
||||
```promql
|
||||
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
|
||||
```
|
||||
|
||||
The value of each series is the start time of the process.
|
||||
2. Check the Application log for crashes of the service executable (events
|
||||
1000 and 1026, .NET Runtime errors) and the System log for Service Control
|
||||
Manager events 7031 and 7034, which also show the recovery action.
|
||||
3. Check whether a deployment, configuration push, scheduled task or another
|
||||
automation restarts the service on purpose.
|
||||
4. Check for resources the service fails to acquire at start, such as a port
|
||||
that is in use, a certificate, a database or a license.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the crash or the start failure, roll back the deployment that caused it, or
|
||||
stop the automation that restarts the service. Keep the service stopped until
|
||||
the cause is known if every restart has side effects. Confirm the service keeps
|
||||
one process and stays running. Raise the threshold for services that restart by
|
||||
design, for example worker services that recycle themselves.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: WindowsServiceNotRunning
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The service `service` has the start mode `auto` but has not been in the
|
||||
`running` state for 15 minutes. Trigger-start services, which Windows stops by
|
||||
design when their trigger condition ends, are left out through the
|
||||
`trigger_start` label of `windows_service_info`. Delayed-start services are
|
||||
included; the 15 minute duration covers their later start after a reboot.
|
||||
|
||||
## Impact
|
||||
|
||||
A service that should always run is stopped, so the function it provides is
|
||||
unavailable, for example printing, time synchronization, a web site or a
|
||||
database instance. Windows restarts a crashed service only when its recovery
|
||||
options say so.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect the current state of the service and when it changed:
|
||||
|
||||
```promql
|
||||
windows_service_state{job="windows_exporter", service="<service>"} == 1
|
||||
```
|
||||
|
||||
2. Check the System event log for Service Control Manager events 7000, 7009,
|
||||
7031 and 7034, and the Application log for crashes of the service process.
|
||||
3. Check `windows_service_info` for the account in `run_as`; an expired
|
||||
password or a removed right to log on as a service stops the service at the
|
||||
next start. `windows_service_process` shows whether a process is running.
|
||||
4. Check whether the service was stopped on purpose for maintenance or
|
||||
disabled by policy, and whether it depends on another service that is also
|
||||
stopped.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Start the service after fixing the cause: the account, a missing dependency, a
|
||||
configuration error or a crash. Set recovery options so Windows restarts the
|
||||
service itself. Change the start mode to `manual` or `disabled` for services
|
||||
that should not run, or exclude them with the `serviceSelector` when the mixin
|
||||
should not watch them. Confirm the service stays running.
|
||||
Reference in New Issue
Block a user