sendnrw/windows_exporter
2
0
Fork 0
mirror of https://github.com/prometheus-community/windows_exporter.git synced 2026-10-17 10:09:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

mixin: group alerts by collector area and add tier 1 alerts

Browse Source 
The alerts of the mixin now come in one rule group per collector area
(windows-exporter, windows-cpu, windows-memory, windows-disk,
windows-network, windows-service, windows-system, windows-active-directory,
windows-time). Every alert names the collectors it reads and is generated
only when _config.collectors enables them, the same switch the dashboard
uses; enableActiveDirectory and enableTime set collectors.ad and
collectors.time. _config.alerts switches single alerts off by name, and an
unknown name fails the build.

New alerts for the default collectors: WindowsCollectorTimeout,
WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core),
WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh,
WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic
services without triggers), WindowsServiceCrashLooping (processes per
service in 15 minutes) and WindowsHostRebooted (info, fires once per
reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error
and discard rates with the packet rate (networkErrorRatioThreshold and
networkDiscardRatioThreshold, in percent) instead of an absolute packet
rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new
infoSeverity and serviceSelector knobs follow the existing ones.

New recordings: windows:cpu_queue_per_core:ratio,
windows:net_received_packets:rate and windows:net_sent_packets:rate.

Each new alert has a runbook, a README row and a unit test;
tests/alerts.test.jsonnet checks the collector gating, the alert switches
and the group layout. The runbook check renders the alerts of every
collector. contrib/mixin/AGENTS.md documents the checklist for new alerts.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-16 23:06:59 +00:00
parent da7bebc6b3
commit ae5d0cca2a
23 changed files with 1210 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/mixin/AGENTS.md
+51
View File
@@ -0,0 +1,51 @@
# Mixin instructions
Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers
`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources.
[`README.md`](README.md) documents the configuration and the generated rules;
keep it in sync with every change.
## Alerts
`alerts/alerts.libsonnet` renders one rule group per collector area. Every
alert names the collectors it reads (`collectors=[...]`) and is generated only
with those enabled in `_config.collectors`; `_config.alerts` switches single
alerts off by name. Threshold alerts get `keep_firing_for` from
`keepFiringFor`; pass `threshold=false` for state alerts (`== 0`, `== 1`) and
one-shot alerts, which must resolve at once.
A new alert touches five places, and CI checks all of them:
1. The rule in `alerts/alerts.libsonnet`, in the group of its area, with
`{{ $labels.<key> }}` templates that use the 0.32 label keys (`service`,
`volume`, `nic`, `task`, `device_id`, never `name`).
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
fails when the map and the generated alerts differ. Thresholds are
`_config` knobs named `<area><Condition>Threshold[Unit]`.
3. A unit test in `tests/alerts.test.yaml` that checks the first firing
evaluation, the labels and annotations, and the recovery. With
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
condition clears. `rate()` extrapolates while a window has few samples, so
ratios of two rates activate earlier than a single rate; probe the
activation time with `ALERTS_FOR_STATE` instead of guessing it.
4. A runbook `docs/runbooks/<lowercase-alert-name>.md` with the `Meaning`,
`Impact`, `Diagnosis` and `Mitigation` sections.
5. A row in the alerts table of `README.md`.
Metrics with a state label (`windows_service_state{state="running"} == 0`)
keep that label in the alert; drop it with `max without (state) (...)` so the
alert does not say `state="running"` for a stopped service. Join a state metric
with its info metric `on (job, instance, <key>)`.
## Recording rules
New recordings go in `rules/rules.libsonnet` with the shared expressions in
`lib/queries.libsonnet`, a row in the README table and a test in
`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero
capacities produce no sample instead of `NaN` or `+Inf`.
## Checks
`mise run test` runs everything CI runs. Without mise, the tools from
`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
the `lint`, `check-runbooks` and `test` tasks do the same.
contrib/mixin/README.md
+87 -40
View File
@@ -29,10 +29,12 @@ to generate rules. Generated rule files are ignored by Git:
- `windows_alerts.yaml`: alerting rules.
The default selector is `job="windows_exporter"`. Change it to match your scrape
job before generating rules. Resource rules use the default `cpu`, `memory`,
`logical_disk`, and `net` collectors. Enable `ad` on domain controllers when
opting into Active Directory alerts. Clock alerts require the `time` collector
with its `ntp` subcollector and are intended for hosts using NTP synchronization.
job before generating rules. The rules read the default `cpu`, `memory`,
`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors`
lists which collectors the targets run, and the alerts of a collector that is
turned off are not generated. Enable `ad` on domain controllers when opting
into Active Directory alerts. Clock alerts require the `time` collector with
its `ntp` subcollector and are intended for hosts using NTP synchronization.
Copy both generated files into your Prometheus rule directory and reference
them in `prometheus.yml`:
@@ -62,6 +64,9 @@ modifying its source. For example, create `custom.libsonnet` in this directory:
cpuHighUsageThreshold: 95,
enableActiveDirectory: true,
enableTime: true,
collectors+: { pagefile: true },
alerts+: { WindowsHostRebooted: false },
keepFiringFor: '10m',
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
},
}
@@ -77,9 +82,11 @@ jsonnet -S -e 'std.manifestYamlDoc((import "custom.libsonnet").prometheusAlerts)
Selectors are comma-separated Prometheus label matchers without braces. They
apply to both recording rules and alerts. All target labels, including `job`,
`instance`, `cluster`, and custom selector labels, are preserved. CPU usage
averages idle rates across cores and removes only `core` and `mode`. Disk and
network recordings also retain `volume` and `nic` respectively. The CPU mode
recording retains `mode` and removes only `core`.
averages idle rates across cores and removes only `core` and `mode`. Disk,
network and service rules also retain `volume`, `nic` and `service`
respectively, filtered by `volumeSelector`, `nicSelector` and
`serviceSelector`. The CPU mode recording retains `mode` and removes only
`core`.
`rateInterval` is the fixed window for recording rules and Active Directory
failure rates; keep it at least four times your scrape interval. Grafana's
@@ -93,6 +100,7 @@ rules.
| `windows:cpu_usage:ratio` | Non-idle CPU time averaged across cores | Ratio, 0–1 |
| `windows:cpu_cores:count` | Logical processors observed in idle CPU metrics | Count |
| `windows:cpu_time:ratio_rate` | CPU time averaged across cores, retaining `mode` | Seconds/second |
| `windows:cpu_queue_per_core:ratio` | Threads waiting for a processor, averaged over `rateInterval`, per logical processor | Threads/processor |
| `windows:memory_usage:ratio` | Physical memory in use, based on available bytes | Ratio, 0–1 |
| `windows:memory_used:bytes` | Total physical memory minus available memory | Bytes |
| `windows:memory_cached:bytes` | Cache, modified pages, and three standby lists | Bytes |
@@ -109,6 +117,8 @@ rules.
| `windows:logical_disk_write_latency:seconds` | Write-time rate divided by write-operation rate | Seconds/operation |
| `windows:net_received_bytes:rate` | Received network traffic per interface | Bytes/second |
| `windows:net_sent_bytes:rate` | Sent network traffic per interface | Bytes/second |
| `windows:net_received_packets:rate` | Received packets per interface | Packets/second |
| `windows:net_sent_packets:rate` | Sent packets per interface | Packets/second |
| `windows:net_received_utilization:ratio` | Received throughput divided by interface bandwidth | Ratio |
| `windows:net_sent_utilization:ratio` | Sent throughput divided by interface bandwidth | Ratio |
| `windows:net_received_errors:rate` | Inbound packet errors | Packets/second |
@@ -127,46 +137,81 @@ which can exceed wall time for concurrent I/O. Latency is an average and is
omitted when there are no operations. Memory, disk space, commit, and network
utilization ratios are omitted when their capacity is zero or missing. Network
bandwidth is already in bytes/second; receive and send utilization are separate
for full-duplex links.
for full-duplex links. The processor queue length is an instantaneous sample,
so the queue recording averages it over `rateInterval` before dividing by the
core count.
The cache recording includes modified pages and is not a measure of immediately
reclaimable memory. Paging operations can include file-backed pages as well as
pagefile traffic.
## Alerts
Threshold configuration values below are percentages except where noted.
The alerts are grouped by collector area. Each group lists the collectors its
rules read; a group is generated only with those collectors enabled in
`_config.collectors`, and a rule that needs two collectors, such as the CPU
saturation alert, only with both. Threshold configuration values below are
percentages except where noted.
| Alert | Default condition | Duration | Severity |
| --- | --- | --- | --- |
| `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
| `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
| `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
| `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
| `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
| `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
| `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
| `WindowsNetworkErrors` | Combined receive/outbound errors above `networkErrorRateThreshold` (1 packet/second) | 15m | Warning |
| `WindowsNetworkDiscards` | Combined receive/outbound discards above `networkDiscardRateThreshold` (1 packet/second) | 15m | Warning |
| `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
| `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
| `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
| `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
| `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
| `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
| `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
| Group | Collectors | Alert | Default condition | Duration | Severity |
| --- | --- | --- | --- | --- | --- |
| `windows-exporter` | | `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
| `windows-exporter` | | `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
| `windows-exporter` | | `WindowsCollectorTimeout` | `windows_exporter_collector_timeout == 1` | 10m | Warning |
| `windows-exporter` | | `WindowsScrapeQueueTimeout` | `windows_exporter_scrape_queue_timeout == 1` | 10m | Warning |
| `windows-cpu` | `cpu` | `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
| `windows-cpu` | `cpu`, `system` | `WindowsCPUSaturation` | Waiting threads per logical processor above `cpuQueuePerCoreThreshold` (2) | 15m | Warning |
| `windows-memory` | `memory` | `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
| `windows-memory` | `memory` | `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
| `windows-memory` | `memory` | `WindowsMemoryPagingHigh` | Pages read or written per second above `memoryPagingThreshold` (1000) | 15m | Warning |
| `windows-disk` | `logical_disk` | `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
| `windows-disk` | `logical_disk` | `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
| `windows-disk` | `logical_disk` | `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
| `windows-network` | `net` | `WindowsNetworkErrors` | Receive and outbound errors above `networkErrorRatioThreshold` (1%) of all packets | 15m | Warning |
| `windows-network` | `net` | `WindowsNetworkDiscards` | Receive and outbound discards above `networkDiscardRatioThreshold` (1%) of all packets | 15m | Warning |
| `windows-network` | `net` | `WindowsNetworkUtilizationHigh` | Received or sent throughput above `networkUtilizationThreshold` (90%) of the link bandwidth | 15m | Warning |
| `windows-network` | `net` | `WindowsNetworkInterfaceFlapping` | Operational status changed more than twice within `rateInterval` | 2m | Warning |
| `windows-service` | `service` | `WindowsServiceNotRunning` | Automatic service without triggers not running | 15m | Warning |
| `windows-service` | `service` | `WindowsServiceCrashLooping` | More than `serviceRestartThreshold` (2) restarts in 15 minutes | 0m | Warning |
| `windows-system` | `system` | `WindowsHostRebooted` | Boot time less than ten minutes old and changed | 0m | Info |
| `windows-active-directory` | `ad` | `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
Active Directory alerts are generated only when `enableActiveDirectory` is
`true`. `criticalSeverity` defaults to `critical` and can be overridden to match
your routing labels. Below the critical disk threshold, both disk severity
levels fire; configure Alertmanager inhibition if only the critical
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
build, and a name missing from a replaced object stays enabled. A group whose
alerts are all turned off is not generated.
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
value that oscillates around a threshold does not flap. State alerts such as
`WindowsExporterDown` and `WindowsServiceNotRunning` resolve as soon as their
condition clears. `keep_firing_for` needs Prometheus 2.42 or later; set
`keepFiringFor: ''` for older servers.
`criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override
them to match your routing labels. Below the critical disk threshold, both disk
severity levels fire; configure Alertmanager inhibition if only the critical
notification should be sent.
Clock alerts are generated only when `enableTime` is `true`. A zero NTP source
count may be expected on hosts using another time provider. The offset compares
the host to its chosen source and does not independently verify that source.
Disk latency and network error/discard thresholds should be tuned for the
Active Directory alerts need `collectors.ad`, which `enableActiveDirectory`
sets. Clock alerts need `collectors.time`, which `enableTime` sets. A zero NTP
source count may be expected on hosts using another time provider. The offset
compares the host to its chosen source and does not independently verify that
source. Disk latency, paging and network thresholds should be tuned for the
storage tier, traffic volume, and application requirements.
`WindowsServiceNotRunning` watches services with the `auto` start mode and
skips trigger-start services, which stop by design, through the
`trigger_start` label of `windows_service_info`. Delayed-start services are
included; the 15 minute duration covers their later start after a reboot.
`WindowsServiceCrashLooping` counts the `windows_service_process` series of a
service within 15 minutes, one per process the service ran as. Exclude services
with `serviceSelector`. `WindowsHostRebooted` fires once for about ten minutes
after a reboot and resolves by itself.
Windows disk size/free-space counters may lag by 10–15 minutes. Predictions
depend on a sustained trend and are intentionally delayed for an hour. A target
removed from service discovery has no `up` series and will not trigger
@@ -366,11 +411,13 @@ find unused variables and references to missing fields, checks that the
committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
`promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic
time series.
Tests cover all 25 recordings, alert timing and recovery, threshold
boundaries, counter resets, zero denominators, disk forecasts, target-label
preservation, and custom selectors/thresholds with Active Directory and clock
alerts enabled. Runbook checks include opt-in alerts and verify that their pages
and URLs exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`,
threshold boundaries, counter resets, zero denominators, disk forecasts,
target-label preservation, and custom selectors/thresholds with Active
Directory and clock alerts enabled. `tests/alerts.test.jsonnet` checks the
collector gating, the per-alert switches and the group layout. Runbook checks
render the alerts of every collector and verify that their pages and URLs
exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
`mise install`.
To list the available tasks:
contrib/mixin/alerts/alerts.libsonnet
+288 -149
View File
@@ -1,158 +1,297 @@
{
local c = self._config,
local q = (import '../lib/queries.libsonnet')(c),
local alert(name, expr, duration, severity, summary, description) = {
alert: name,
expr: expr,
'for': duration,
labels: { severity: severity },
annotations: { summary: summary, description: description },
},
prometheusAlerts+:: {
groups: [
local known = std.objectFields((import '../config.libsonnet')._config.alerts),
local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
// A collector missing from a replaced collectors object counts as disabled;
// an alert missing from a replaced alerts object counts as enabled.
local collectorEnabled(collector) = std.get(c.collectors, collector, false),
local alertEnabled(name) = std.get(c.alerts, name, true),
local keepFiring = if c.keepFiringFor != '' then { keep_firing_for: c.keepFiringFor } else {},
// alert renders a rule when every collector it needs is enabled and the
// alert is not turned off. Threshold alerts set keepFiringFor so a value
// oscillating around the threshold does not flap; state alerts and
// one-shot alerts leave it out.
local alert(name, expr, duration, severity, summary, description, collectors=[], threshold=true) =
if std.all([collectorEnabled(collector) for collector in collectors]) && alertEnabled(name) then [
{
name: 'windows-exporter-alerts',
rules: [
alert(
'WindowsExporterDown',
q.metric('up') + ' == 0',
'5m',
c.criticalSeverity,
'Windows exporter is unreachable',
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).'
),
alert(
'WindowsCollectorFailed',
q.metric('windows_exporter_collector_success') + ' == 0',
'5m',
'warning',
'Windows collector failed',
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.'
),
alert(
'WindowsCPUHighUsage',
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
'15m',
'warning',
'Windows CPU usage is high',
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold
),
alert(
'WindowsMemoryHighUsage',
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
'15m',
'warning',
'Windows memory usage is high',
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold
),
alert(
'WindowsMemoryCommitHighUsage',
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
'15m',
'warning',
'Windows memory commit usage is high',
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold
),
] + [
alert(
'WindowsDisk%sLatencyHigh' % direction[0],
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
'15m',
'warning',
'Windows disk %s latency is high' % direction[1],
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]]
)
for direction in [['Read', 'read'], ['Write', 'write']]
] + [
alert(
'WindowsNetwork%s' % outcome[0],
'(%s + %s) > %g' % [
alert: name,
expr: expr,
labels: { severity: severity },
annotations: { summary: summary, description: description },
} + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
] else [],
local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
local onService = 'on (job, instance, service)',
prometheusAlerts+:: {
groups:
group('windows-exporter', std.flattenArrays([
alert(
'WindowsExporterDown',
q.metric('up') + ' == 0',
'5m',
c.criticalSeverity,
'Windows exporter is unreachable',
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
threshold=false
),
alert(
'WindowsCollectorFailed',
q.metric('windows_exporter_collector_success') + ' == 0',
'5m',
'warning',
'Windows collector failed',
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
threshold=false
),
alert(
'WindowsCollectorTimeout',
q.metric('windows_exporter_collector_timeout') + ' == 1',
'10m',
'warning',
'Windows collector exceeds the scrape timeout',
'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
threshold=false
),
alert(
'WindowsScrapeQueueTimeout',
q.metric('windows_exporter_scrape_queue_timeout') + ' == 1',
'10m',
'warning',
'Windows exporter scrapes time out in the queue',
'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
threshold=false
),
]))
+ group('windows-cpu', std.flattenArrays([
alert(
'WindowsCPUHighUsage',
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
'15m',
'warning',
'Windows CPU usage is high',
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold,
collectors=['cpu']
),
alert(
'WindowsCPUSaturation',
q.metric('windows:cpu_queue_per_core:ratio') + ' > %s' % c.cpuQueuePerCoreThreshold,
'15m',
'warning',
'Windows processor queue is long',
'More than %s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c.cpuQueuePerCoreThreshold,
collectors=['cpu', 'system']
),
]))
+ group('windows-memory', std.flattenArrays([
alert(
'WindowsMemoryHighUsage',
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
'15m',
'warning',
'Windows memory usage is high',
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold,
collectors=['memory']
),
alert(
'WindowsMemoryCommitHighUsage',
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
'15m',
'warning',
'Windows memory commit usage is high',
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold,
collectors=['memory']
),
alert(
'WindowsMemoryPagingHigh',
q.metric('windows:memory_swap_page_operations:rate') + ' > %s' % c.memoryPagingThreshold,
'15m',
'warning',
'Windows memory paging is high',
'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold,
collectors=['memory']
),
]))
+ group('windows-disk', std.flattenArrays([
alert(
'WindowsDisk%sLatencyHigh' % direction[0],
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
'15m',
'warning',
'Windows disk %s latency is high' % direction[1],
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]],
collectors=['logical_disk']
)
for direction in [['Read', 'read'], ['Write', 'write']]
] + [
alert(
'WindowsDiskAlmostFull',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
'15m',
entry[0],
'Windows disk space is low',
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1],
collectors=['logical_disk']
)
for entry in [
['warning', c.diskFreeWarningThreshold],
[c.criticalSeverity, c.diskFreeCriticalThreshold],
]
] + [
alert(
'WindowsDiskFillingUp',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
c.diskFreeWarningThreshold,
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
c.diskPredictionWindow,
c.diskPredictionHours * 3600,
],
'1h',
'warning',
'Windows disk is predicted to fill up',
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours,
collectors=['logical_disk']
),
]))
+ group('windows-network', std.flattenArrays([
alert(
'WindowsNetwork%s' % outcome[0],
q.ratio(
'(%s + %s)' % [
q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]),
q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]),
outcome[2],
],
'15m',
'warning',
'Windows network packet %s are high' % outcome[3],
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g per second for 15 minutes.' % [outcome[3], outcome[2]]
)
for outcome in [
['Errors', 'errors', c.networkErrorRateThreshold, 'errors'],
['Discards', 'discarded', c.networkDiscardRateThreshold, 'discards'],
]
] + [
alert(
'WindowsDiskAlmostFull',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
'15m',
entry[0],
'Windows disk space is low',
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1]
)
for entry in [
['warning', c.diskFreeWarningThreshold],
[c.criticalSeverity, c.diskFreeCriticalThreshold],
]
] + [
alert(
'WindowsDiskFillingUp',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
c.diskFreeWarningThreshold,
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
c.diskPredictionWindow,
c.diskPredictionHours * 3600,
],
'1h',
'warning',
'Windows disk is predicted to fill up',
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours
),
],
},
] + (if c.enableActiveDirectory then [
{
name: 'windows-active-directory',
rules: [
alert(
'WindowsADPendingReplication',
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
'15m',
'warning',
'Active Directory replication queue is high',
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold
),
alert(
'WindowsADReplicationFailures',
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
'15m',
'warning',
'Active Directory replication requests are failing',
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.'
),
],
},
] else []) + if c.enableTime then [
{
name: 'windows-time',
rules: [
alert(
'WindowsClockNotSynchronising',
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
'10m',
'warning',
'Windows clock has no NTP time source',
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.'
),
alert(
'WindowsClockSkewDetected',
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
'10m',
'warning',
'Windows clock offset is high',
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds
),
],
},
] else [],
'(%s + %s)' % [
q.metric('windows:net_received_packets:rate', [c.nicSelector]),
q.metric('windows:net_sent_packets:rate', [c.nicSelector]),
]
) + ' * 100 > %g' % outcome[2],
'15m',
'warning',
'Windows network packet %s are high' % outcome[3],
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g%% of all packets for 15 minutes.' % [outcome[3], outcome[2]],
collectors=['net']
)
for outcome in [
['Errors', 'errors', c.networkErrorRatioThreshold, 'errors'],
['Discards', 'discarded', c.networkDiscardRatioThreshold, 'discards'],
]
] + [
alert(
'WindowsNetworkUtilizationHigh',
'(%s * 100 > %s) or (%s * 100 > %s)' % [
q.metric('windows:net_received_utilization:ratio', [c.nicSelector]),
c.networkUtilizationThreshold,
q.metric('windows:net_sent_utilization:ratio', [c.nicSelector]),
c.networkUtilizationThreshold,
],
'15m',
'warning',
'Windows network interface utilization is high',
'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %s%% of its bandwidth in at least one direction for 15 minutes.' % c.networkUtilizationThreshold,
collectors=['net']
),
alert(
'WindowsNetworkInterfaceFlapping',
'max without (status) (changes(%s[%s])) > 2' % [
q.metric('windows_net_nic_operation_status', [c.nicSelector, 'status="up"']),
c.rateInterval,
],
'2m',
'warning',
'Windows network interface is flapping',
'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %s.' % c.rateInterval,
collectors=['net']
),
]))
+ group('windows-service', std.flattenArrays([
alert(
'WindowsServiceNotRunning',
std.join(' ', [
'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']),
'and %s (%s == 1)' % [onService, q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"'])],
'unless %s (%s == 1)' % [onService, q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"'])],
]),
'15m',
'warning',
'Windows automatic service is not running',
'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
collectors=['service'],
threshold=false
),
alert(
'WindowsServiceCrashLooping',
// Every service process has its own series, so the series seen in the
// window count the processes the service ran as.
'count without (process_id) (count_over_time(%s[15m])) - 1 > %s' % [
q.metric('windows_service_process', [c.serviceSelector]),
c.serviceRestartThreshold,
],
'0m',
'warning',
'Windows service is restarting repeatedly',
'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %s times in 15 minutes.' % c.serviceRestartThreshold,
collectors=['service']
),
]))
+ group('windows-system', std.flattenArrays([
alert(
'WindowsHostRebooted',
// Fires once per reboot: for the first ten minutes after the boot,
// while the boot time known ten minutes ago was an older one.
'(time() - %s) < 600 and (time() - %s offset 10m) > 600' % [
q.metric('windows_system_boot_time_timestamp'),
q.metric('windows_system_boot_time_timestamp'),
],
'0m',
c.infoSeverity,
'Windows host has rebooted',
'{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
collectors=['system'],
threshold=false
),
]))
+ group('windows-active-directory', std.flattenArrays([
alert(
'WindowsADPendingReplication',
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
'15m',
'warning',
'Active Directory replication queue is high',
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold,
collectors=['ad']
),
alert(
'WindowsADReplicationFailures',
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
'15m',
'warning',
'Active Directory replication requests are failing',
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
collectors=['ad']
),
]))
+ group('windows-time', std.flattenArrays([
alert(
'WindowsClockNotSynchronising',
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
'10m',
'warning',
'Windows clock has no NTP time source',
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
collectors=['time'],
threshold=false
),
alert(
'WindowsClockSkewDetected',
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
'10m',
'warning',
'Windows clock offset is high',
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds,
collectors=['time']
),
])),
},
}
contrib/mixin/config.libsonnet
+54 -5
View File
@@ -4,11 +4,12 @@
windowsExporterSelector: 'job="windows_exporter"',
volumeSelector: 'volume!=""',
nicSelector: 'nic!=""',
serviceSelector: 'service!=""',
rateInterval: '5m',
// Opt in only for targets with the ad collector enabled.
// Opt in only for targets with the ad collector enabled. Sets collectors.ad.
enableActiveDirectory: false,
// Requires the time collector with its ntp subcollector enabled.
// Requires the time collector with its ntp subcollector enabled. Sets collectors.time.
enableTime: false,
// Grafana dashboard metadata. A UID is the v2 metadata.name; change the
@@ -37,8 +38,10 @@
// Collectors enabled on the windows_exporter targets, by collector name.
// The dashboard only renders the tabs, rows and panels of enabled
// collectors. The defaults match the exporter's default collectors.
// collectors, and the alerts only the rules whose collectors are enabled.
// The defaults match the exporter's default collectors.
collectors: {
ad: $._config.enableActiveDirectory,
cpu: true,
diskdrive: false,
gpu: false,
@@ -62,20 +65,66 @@
update: false,
},
// Alerts by name. Set one to false to leave it out of the generated
// rules, for example alerts+: { WindowsHostRebooted: false }. Names that
// are not listed here fail the build; names left out of a replaced object
// stay enabled.
alerts: {
WindowsExporterDown: true,
WindowsCollectorFailed: true,
WindowsCollectorTimeout: true,
WindowsScrapeQueueTimeout: true,
WindowsCPUHighUsage: true,
WindowsCPUSaturation: true,
WindowsMemoryHighUsage: true,
WindowsMemoryCommitHighUsage: true,
WindowsMemoryPagingHigh: true,
WindowsDiskReadLatencyHigh: true,
WindowsDiskWriteLatencyHigh: true,
WindowsDiskAlmostFull: true,
WindowsDiskFillingUp: true,
WindowsNetworkErrors: true,
WindowsNetworkDiscards: true,
WindowsNetworkUtilizationHigh: true,
WindowsNetworkInterfaceFlapping: true,
WindowsServiceNotRunning: true,
WindowsServiceCrashLooping: true,
WindowsHostRebooted: true,
WindowsADPendingReplication: true,
WindowsADReplicationFailures: true,
WindowsClockNotSynchronising: true,
WindowsClockSkewDetected: true,
},
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
// Threshold alerts keep firing this long after their condition clears,
// so a value oscillating around a threshold does not flap. Needs
// Prometheus 2.42 or later; set it to '' to leave keep_firing_for out.
keepFiringFor: '5m',
cpuHighUsageThreshold: 90,
// Threads waiting for a processor, per logical processor.
cpuQueuePerCoreThreshold: 2,
memoryHighUsageThreshold: 90,
memoryCommitHighUsageThreshold: 90,
// Pages read from or written to disk per second.
memoryPagingThreshold: 1000,
diskLatencyThresholdSeconds: 0.05,
networkErrorRateThreshold: 1,
networkDiscardRateThreshold: 1,
// Percent of all packets on the interface.
networkErrorRatioThreshold: 1,
networkDiscardRatioThreshold: 1,
// Percent of the interface bandwidth, in either direction.
networkUtilizationThreshold: 90,
clockOffsetThresholdSeconds: 0.05,
diskFreeWarningThreshold: 10,
diskFreeCriticalThreshold: 5,
diskPredictionWindow: '6h',
diskPredictionHours: 24,
// Restarts of one service within 15 minutes.
serviceRestartThreshold: 2,
criticalSeverity: 'critical',
infoSeverity: 'info',
adPendingReplicationThreshold: 50,
adReplicationFailureRateThreshold: 0,
},
contrib/mixin/lib/queries.libsonnet
+7
View File
@@ -7,6 +7,13 @@ function(c)
ratio(numerator, denominator): '(%s / (%s > 0))' % [numerator, denominator],
cpuUsage:
'1 - avg without (core, mode) (%s)' % self.rate('windows_cpu_time_seconds_total', ['mode="idle"']),
// Threads waiting for a processor, averaged over the rate interval, per
// logical processor. The queue length is an instantaneous sample.
cpuQueuePerCore:
self.ratio(
'avg_over_time(%s[%s])' % [self.metric('windows_system_processor_queue_length'), c.rateInterval],
self.metric('windows:cpu_cores:count')
),
memoryUsage:
'1 - %s' % self.ratio(
self.metric('windows_memory_available_bytes'),
contrib/mixin/mise.toml
+2 -1
View File
@@ -95,13 +95,14 @@ pint --offline lint tests/custom_rules.yaml tests/custom_alerts.yaml
[tasks.check-runbooks]
description = "Check runbook coverage for all alerts, including opt-in alerts"
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { enableActiveDirectory: true, enableTime: true } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [name]: true for name in std.objectFields(super.collectors) } } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
[tasks.test]
description = "Run all mixin lint, runbook, and rule tests"
depends = ["lint", "check-runbooks", "vendor"]
run = [
"jsonnet tests/runbooks.test.jsonnet",
"jsonnet tests/alerts.test.jsonnet",
"jsonnet -J vendor tests/dashboard.test.jsonnet > /dev/null",
"promtool test rules tests/*.test.yaml",
]
contrib/mixin/rules/rules.libsonnet
+7
View File
@@ -9,6 +9,7 @@
{ record: 'windows:cpu_usage:ratio', expr: q.cpuUsage },
{ record: 'windows:cpu_cores:count', expr: 'count without (core, mode) (%s)' % q.metric('windows_cpu_time_seconds_total', ['mode="idle"']) },
{ record: 'windows:cpu_time:ratio_rate', expr: 'avg without (core) (%s)' % q.rate('windows_cpu_time_seconds_total') },
{ record: 'windows:cpu_queue_per_core:ratio', expr: q.cpuQueuePerCore },
{ record: 'windows:memory_usage:ratio', expr: q.memoryUsage },
{
record: 'windows:memory_used:bytes',
@@ -56,6 +57,12 @@
expr: q.rate('windows_net_bytes_%s_total' % direction, [c.nicSelector]),
}
for direction in ['received', 'sent']
] + [
{
record: 'windows:net_%s_packets:rate' % direction,
expr: q.rate('windows_net_packets_%s_total' % direction, [c.nicSelector]),
}
for direction in ['received', 'sent']
] + [
{ record: 'windows:net_%s_utilization:ratio' % direction, expr: q.networkUtilization(direction) }
for direction in ['received', 'sent']
contrib/mixin/tests/alerts.test.jsonnet
+50
View File
@@ -0,0 +1,50 @@
// Checks the alert configuration: collector gating, per-alert switches,
// keep_firing_for and the group layout.
local mixin = import '../mixin.libsonnet';
local config = (import '../config.libsonnet')._config;
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true } };
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
local groupNames(alerts) = [group.name for group in alerts.groups];
// Every configured switch belongs to a generated alert, and every alert has a switch.
local all = std.set(names(everything.prometheusAlerts));
assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs from generated alerts: %s' % std.manifestJson([std.setDiff(all, std.set(std.objectFields(config.alerts))), std.setDiff(std.set(std.objectFields(config.alerts)), all)]);
// Groups follow the collector areas; opt-in groups appear only when enabled.
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory', 'windows-time'];
// Disabling a collector removes its alerts and an empty group; the alerts of a
// rule that needs two collectors go with either.
local noNet = mixin { _config+:: { collectors+: { net: false } } };
assert !std.member(groupNames(noNet.prometheusAlerts), 'windows-network');
local noSystem = mixin { _config+:: { collectors+: { system: false } } };
assert names(noSystem.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUSaturation', 'WindowsHostRebooted']);
assert std.member(names(noSystem.prometheusAlerts), 'WindowsCPUHighUsage');
// A replaced collectors object counts missing collectors as disabled.
local onlyCPU = mixin { _config+:: { collectors: { cpu: true } } };
assert names(onlyCPU.prometheusAlerts) == ['WindowsCPUHighUsage', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsExporterDown', 'WindowsScrapeQueueTimeout'] : std.manifestJson(names(onlyCPU.prometheusAlerts));
// Single alerts switch off by name, with alerts+: or a replaced object.
local noReboot = mixin { _config+:: { alerts+: { WindowsHostRebooted: false } } };
assert names(noReboot.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsHostRebooted']);
assert !std.member(groupNames(noReboot.prometheusAlerts), 'windows-system');
local replaced = mixin { _config+:: { alerts: { WindowsCPUHighUsage: false } } };
assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUHighUsage']);
// keep_firing_for applies to threshold alerts only and can be turned off.
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
local stateAlerts = ['WindowsExporterDown', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsScrapeQueueTimeout', 'WindowsServiceNotRunning', 'WindowsHostRebooted', 'WindowsClockNotSynchronising'];
assert std.all([
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
for rule in rules(everything.prometheusAlerts)
]) : 'keep_firing_for set on a state alert or missing on a threshold alert';
assert std.all([rule.keep_firing_for == '5m' for rule in rules(everything.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]);
local noKeep = mixin { _config+:: { keepFiringFor: '' } };
assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]) == 0;
// The info severity label is configurable.
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
contrib/mixin/tests/alerts.test.yaml
+224 -17
View File
@@ -4,7 +4,15 @@ rule_files:
evaluation_interval: 1m
group_eval_order:
- windows-exporter-recording
- windows-exporter-alerts
- windows-exporter
- windows-cpu
- windows-memory
- windows-disk
- windows-network
- windows-service
- windows-system
- windows-active-directory
- windows-time
tests:
- name: Commit pressure, disk latency and network failures wait and recover
@@ -30,6 +38,10 @@ tests:
values: '0x31'
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
values: '0+120x20 2520x10'
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
values: '0+3000x31'
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
values: '0+3000x31'
alert_rule_test:
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 14m, exp_alerts: []}
- alertname: WindowsMemoryCommitHighUsage
@@ -40,7 +52,8 @@ tests:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/
summary: Windows memory commit usage is high
description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 21m, exp_alerts: []}
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 25m, exp_alerts: [{exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}, exp_annotations: {runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/, summary: Windows memory commit usage is high, description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'}}]}
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 26m, exp_alerts: []}
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 15m, exp_alerts: []}
- alertname: WindowsDiskReadLatencyHigh
eval_time: 16m
@@ -50,7 +63,7 @@ tests:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/
summary: Windows disk read latency is high
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per read operation for 15 minutes.'
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 25m, exp_alerts: []}
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 30m, exp_alerts: []}
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 15m, exp_alerts: []}
- alertname: WindowsDiskWriteLatencyHigh
eval_time: 16m
@@ -60,27 +73,27 @@ tests:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/
summary: Windows disk write latency is high
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per write operation for 15 minutes.'
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 25m, exp_alerts: []}
- {alertname: WindowsNetworkErrors, eval_time: 17m, exp_alerts: []}
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 30m, exp_alerts: []}
- {alertname: WindowsNetworkErrors, eval_time: 15m, exp_alerts: []}
- alertname: WindowsNetworkErrors
eval_time: 18m
eval_time: 16m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/
summary: Windows network packet errors are high
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
- {alertname: WindowsNetworkErrors, eval_time: 25m, exp_alerts: []}
- {alertname: WindowsNetworkDiscards, eval_time: 17m, exp_alerts: []}
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
- {alertname: WindowsNetworkErrors, eval_time: 30m, exp_alerts: []}
- {alertname: WindowsNetworkDiscards, eval_time: 15m, exp_alerts: []}
- alertname: WindowsNetworkDiscards
eval_time: 18m
eval_time: 16m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/
summary: Windows network packet discards are high
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
- {alertname: WindowsNetworkDiscards, eval_time: 25m, exp_alerts: []}
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
- {alertname: WindowsNetworkDiscards, eval_time: 30m, exp_alerts: []}
- name: New resource alerts stay quiet at exact thresholds and clock alerts are disabled
interval: 1m
@@ -105,6 +118,10 @@ tests:
values: '0+30x20'
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '0+30x20'
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '0+3600x20'
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '0+3600x20'
- series: 'windows_time_ntp_client_time_sources{job="windows_exporter", instance="host:9182"}'
values: '0x20'
- series: 'windows_time_computed_time_offset_seconds{job="windows_exporter", instance="host:9182"}'
@@ -178,7 +195,7 @@ tests:
summary: Windows CPU usage is high
description: 'CPU usage on host:9182 has exceeded 90% for 15 minutes.'
- alertname: WindowsCPUHighUsage
eval_time: 25m
eval_time: 30m
exp_alerts: []
- alertname: WindowsMemoryHighUsage
eval_time: 14m
@@ -192,16 +209,16 @@ tests:
summary: Windows memory usage is high
description: 'Physical memory usage on host:9182 has exceeded 90% for 15 minutes.'
- alertname: WindowsMemoryHighUsage
eval_time: 21m
eval_time: 26m
exp_alerts: []
- name: Disk alerts retain volume labels and fire at both severity levels
interval: 1m
input_series:
- series: 'windows_logical_disk_free_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
values: '3x15 30x5'
values: '3x15 30x10'
- series: 'windows_logical_disk_size_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
values: '100x21'
values: '100x26'
alert_rule_test:
- alertname: WindowsDiskAlmostFull
eval_time: 14m
@@ -220,7 +237,7 @@ tests:
summary: Windows disk space is low
description: 'Volume C: on host:9182 has less than 5% free space. Windows may delay space metrics by 10–15 minutes.'
- alertname: WindowsDiskAlmostFull
eval_time: 16m
eval_time: 21m
exp_alerts: []
- name: Healthy resources at the exact thresholds do not fire
@@ -291,3 +308,193 @@ tests:
- {alertname: WindowsMemoryHighUsage, eval_time: 70m, exp_alerts: []}
- {alertname: WindowsDiskAlmostFull, eval_time: 70m, exp_alerts: []}
- {alertname: WindowsDiskFillingUp, eval_time: 70m, exp_alerts: []}
- name: Collector and scrape queue timeouts wait ten minutes and recover without keep_firing_for
interval: 1m
input_series:
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="service"}'
values: '1x10 0x5'
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="cpu"}'
values: '0x15'
- series: 'windows_exporter_scrape_queue_timeout{job="windows_exporter", instance="host:9182"}'
values: '1x10 0x5'
alert_rule_test:
- {alertname: WindowsCollectorTimeout, eval_time: 9m, exp_alerts: []}
- alertname: WindowsCollectorTimeout
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', collector: service, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/
summary: Windows collector exceeds the scrape timeout
description: 'Collector service on host:9182 has not finished within the scrape timeout for 10 minutes.'
- {alertname: WindowsCollectorTimeout, eval_time: 11m, exp_alerts: []}
- {alertname: WindowsScrapeQueueTimeout, eval_time: 9m, exp_alerts: []}
- alertname: WindowsScrapeQueueTimeout
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/
summary: Windows exporter scrapes time out in the queue
description: 'Scrapes of host:9182 have timed out waiting for an earlier scrape to finish for 10 minutes.'
- {alertname: WindowsScrapeQueueTimeout, eval_time: 11m, exp_alerts: []}
- name: CPU saturation and paging use recordings, wait fifteen minutes and keep firing five
interval: 1m
input_series:
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
values: '9x25 0x10'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
values: '0+60x35'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
values: '0+60x35'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
values: '0+60x35'
- series: 'windows_memory_swap_page_operations_total{job="windows_exporter", instance="host:9182", cluster="a"}'
values: '0+90000x20 1800000x15'
alert_rule_test:
- {alertname: WindowsCPUSaturation, eval_time: 14m, exp_alerts: []}
- alertname: WindowsCPUSaturation
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
summary: Windows processor queue is long
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
- alertname: WindowsCPUSaturation
eval_time: 31m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
summary: Windows processor queue is long
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
- {alertname: WindowsCPUSaturation, eval_time: 32m, exp_alerts: []}
- {alertname: WindowsMemoryPagingHigh, eval_time: 18m, exp_alerts: []}
- alertname: WindowsMemoryPagingHigh
eval_time: 19m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
summary: Windows memory paging is high
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
- {alertname: WindowsMemoryPagingHigh, eval_time: 35m, exp_alerts: []}
- name: Network utilization fires in one direction and interface flapping counts status changes
interval: 1m
input_series:
- series: 'windows_net_bytes_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '0+6000x20 120000x15'
- series: 'windows_net_bytes_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '0x36'
- series: 'windows_net_current_bandwidth_bytes{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
values: '100x36'
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="up"}'
values: '1 0 1 0 1 0 1 0 1 0 1x10'
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="down"}'
values: '0 1 0 1 0 1 0 1 0 1 0x10'
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Wi-Fi", status="up"}'
values: '0x21'
alert_rule_test:
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 19m, exp_alerts: []}
- alertname: WindowsNetworkUtilizationHigh
eval_time: 20m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
summary: Windows network interface utilization is high
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 35m, exp_alerts: []}
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []}
- alertname: WindowsNetworkInterfaceFlapping
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/
summary: Windows network interface is flapping
description: 'Interface Ethernet on host:9182 has changed its operational status more than twice in 5m.'
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 17m, exp_alerts: []}
- name: Service alerts skip trigger-start and manual services and count restarts by process
interval: 1m
input_series:
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}'
values: '0x15 1x5'
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}'
values: '1x20'
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}'
values: '1x20'
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}'
values: '0x20'
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}'
values: '1x20'
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}'
values: '1x20'
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}'
values: '0x20'
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}'
values: '1x20'
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}'
values: '0x20'
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}'
values: '1000x4 _x16'
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}'
values: '_x5 2000x4 _x11'
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}'
values: '_x10 3000x4 _x6'
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}'
values: '_x15 4000x6'
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}'
values: '_x16 5000x5'
alert_rule_test:
- {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []}
- alertname: WindowsServiceNotRunning
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: Spooler, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
summary: Windows automatic service is not running
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
- {alertname: WindowsServiceNotRunning, eval_time: 16m, exp_alerts: []}
- {alertname: WindowsServiceCrashLooping, eval_time: 14m, exp_alerts: []}
- alertname: WindowsServiceCrashLooping
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
summary: Windows service is restarting repeatedly
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
- name: Host reboot fires once for the first ten minutes after a new boot time
interval: 1m
input_series:
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
values: '-100000x10 540x15'
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="steady:9182"}'
values: '-100000x25'
alert_rule_test:
- {alertname: WindowsHostRebooted, eval_time: 10m, exp_alerts: []}
- alertname: WindowsHostRebooted
eval_time: 11m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
summary: Windows host has rebooted
description: 'host:9182 rebooted 2m 0s ago.'
- alertname: WindowsHostRebooted
eval_time: 18m
exp_alerts:
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
exp_annotations:
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
summary: Windows host has rebooted
description: 'host:9182 rebooted 9m 0s ago.'
- {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []}
contrib/mixin/tests/config.libsonnet
+2 -2
View File
@@ -14,8 +14,8 @@
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
memoryCommitHighUsageThreshold: 80,
diskLatencyThresholdSeconds: 0.02,
networkErrorRateThreshold: 0.5,
networkDiscardRateThreshold: 0.5,
networkErrorRatioThreshold: 0.5,
networkDiscardRatioThreshold: 0.5,
clockOffsetThresholdSeconds: 0.1,
adPendingReplicationThreshold: 10,
adReplicationFailureRateThreshold: 0.5,
contrib/mixin/tests/custom.test.yaml
+20 -6
View File
@@ -4,7 +4,13 @@ rule_files:
evaluation_interval: 1m
group_eval_order:
- windows-exporter-recording
- windows-exporter-alerts
- windows-exporter
- windows-cpu
- windows-memory
- windows-disk
- windows-network
- windows-service
- windows-system
- windows-active-directory
- windows-time
@@ -52,6 +58,14 @@ tests:
values: '0+120x20'
- series: 'windows_net_packets_outbound_discarded_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
values: '0+120x20'
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
values: '0+3000x20'
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
values: '0+3000x20'
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
values: '0+120x20'
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
values: '0+120x20'
alert_rule_test:
- alertname: WindowsMemoryCommitHighUsage
eval_time: 20m
@@ -84,7 +98,7 @@ tests:
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsnetworkerrors/
summary: Windows network packet errors are high
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
- alertname: WindowsNetworkDiscards
eval_time: 20m
exp_alerts:
@@ -92,7 +106,7 @@ tests:
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsnetworkdiscards/
summary: Windows network packet discards are high
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
- name: Clock alerts opt in, use absolute offset, custom URLs and recover
interval: 1m
@@ -129,7 +143,7 @@ tests:
runbook_url: https://runbooks.example.com/windows/windowsclockskewdetected/
summary: Windows clock offset is high
description: 'Windows Time on host:9182 reports a clock offset above 0.1 seconds for 10 minutes.'
- {alertname: WindowsClockSkewDetected, eval_time: 11m, exp_alerts: []}
- {alertname: WindowsClockSkewDetected, eval_time: 16m, exp_alerts: []}
- name: Custom selectors, thresholds and critical severity propagate to alerts
interval: 1m
input_series:
@@ -239,7 +253,7 @@ tests:
summary: Active Directory replication queue is high
description: 'Pending replication operations on host:9182 have exceeded 10 for 15 minutes.'
- alertname: WindowsADPendingReplication
eval_time: 21m
eval_time: 26m
exp_alerts: []
- alertname: WindowsADReplicationFailures
eval_time: 16m
@@ -253,5 +267,5 @@ tests:
summary: Active Directory replication requests are failing
description: 'Replication synchronization requests on host:9182 have been failing for 15 minutes.'
- alertname: WindowsADReplicationFailures
eval_time: 23m
eval_time: 28m
exp_alerts: []
contrib/mixin/tests/resources.test.yaml
+9 -1
View File
@@ -4,7 +4,15 @@ rule_files:
evaluation_interval: 1m
group_eval_order:
- windows-exporter-recording
- windows-exporter-alerts
- windows-exporter
- windows-cpu
- windows-memory
- windows-disk
- windows-network
- windows-service
- windows-system
- windows-active-directory
- windows-time
tests:
- name: Memory cache includes all lists, commit is a ratio and paging is a rate
contrib/mixin/tests/rules.test.yaml
+41
View File
@@ -111,3 +111,44 @@ tests:
exp_samples:
- labels: 'windows:net_sent_bytes:rate{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
value: 30
- name: Processor queue per core averages the queue and packet rates keep the nic label
interval: 1m
input_series:
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
values: '12x5 0x5'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
values: '0+60x10'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
values: '0+60x10'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
values: '0+60x10'
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="3", mode="idle"}'
values: '0+60x10'
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="nocpu:9182", cluster="a"}'
values: '12x10'
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
values: '0+600x10'
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
values: '0+1200x10'
promql_expr_test:
- expr: windows:cpu_queue_per_core:ratio
eval_time: 5m
exp_samples:
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
value: 3
- expr: windows:cpu_queue_per_core:ratio
eval_time: 10m
exp_samples:
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
value: 0
- expr: windows:net_received_packets:rate
eval_time: 10m
exp_samples:
- labels: 'windows:net_received_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
value: 10
- expr: windows:net_sent_packets:rate
eval_time: 10m
exp_samples:
- labels: 'windows:net_sent_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
value: 20
docs/runbooks/_index.md
+4 -2
View File
@@ -8,5 +8,7 @@ the affected host. Disk alerts also identify a `volume`; network alerts identify
a `nic`. Adjust example PromQL selectors to match your scrape configuration.
The default thresholds are starting points. Check the generated alert rules for
your configured thresholds and durations. Active Directory and NTP clock alerts
are opt-in and require their collectors to be enabled.
your configured thresholds and durations. The rules are grouped by collector
area, and each alert is generated only when the collectors it reads are enabled
in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
and `time` collectors, which are off by default.
docs/runbooks/windowscollectortimeout.md
+46
View File
@@ -0,0 +1,46 @@
---
title: WindowsCollectorTimeout
---
## Meaning
The collector identified by the `collector` label has reported
`windows_exporter_collector_timeout == 1` for ten minutes. The collector did
not finish within the scrape timeout, or its call from an earlier scrape is
still running and later scrapes skip it until it returns.
## Impact
The metrics of this collector are missing or stale on every scrape that times
out, so dashboards show gaps and alerts on those metrics stop evaluating. A
collector that never returns can hold a scrape slot and delay the other
collectors of the host.
## Diagnosis
1. Find the collector and compare how long it usually takes:
```promql
windows_exporter_collector_timeout{job="windows_exporter"} == 1
windows_exporter_collector_duration_seconds{job="windows_exporter"}
```
2. Compare the duration with the scrape timeout of the job. The exporter
subtracts `--scrape.timeout-margin` from the `scrape_timeout` that
Prometheus sends.
3. Check the exporter log for the collector. WMI and Management Infrastructure
queries, the Task Scheduler and Windows Update are slow on hosts with many
objects or a busy WMI provider; the collector's documentation lists the
expensive options.
4. Check whether the host was under CPU, disk or memory pressure at the same
time, and whether a Windows service the collector depends on hangs, for
example `Winmgmt` for WMI-based collectors.
## Mitigation
Raise the `scrape_timeout` of the job within its `scrape_interval`, narrow the
collector with its include and exclude flags, or move the collector to a
separate job with a longer timeout. Restart a hanging provider service when the
log identifies it. Disable the collector on hosts where its data is not needed.
Confirm `windows_exporter_collector_timeout` returns to `0` and the collector's
metrics are published again.
docs/runbooks/windowscpusaturation.md
+39
View File
@@ -0,0 +1,39 @@
---
title: WindowsCPUSaturation
---
## Meaning
The number of threads waiting for a processor, averaged over the rate window
and divided by the number of logical processors, exceeds the configured
threshold for 15 minutes. The default is two waiting threads per logical
processor. The metric is `windows:cpu_queue_per_core:ratio`, built from
`windows_system_processor_queue_length` and `windows:cpu_cores:count`.
## Impact
Runnable threads wait for CPU time, so applications respond slowly even when
their own CPU usage looks moderate. A long queue with CPU usage below 100%
points at a few busy cores, a single-threaded bottleneck, or processor
scheduling problems such as parked cores or hypervisor contention.
## Diagnosis
1. Inspect `windows:cpu_queue_per_core:ratio` next to `windows:cpu_usage:ratio`
for the host. Saturation with high usage is a capacity problem; saturation
with low usage points at scheduling.
2. Compare the per-core rates of `windows_cpu_time_seconds_total`. One or two
cores at 100% while others idle indicate a thread that cannot spread.
3. Check `windows_cpu_parking_status`, `windows_cpu_core_frequency_hertz` and
the power plan. On virtual machines check the hypervisor for CPU ready time
or steal.
4. Use Task Manager or Performance Monitor (`System\Processor Queue Length`,
`Process\Thread Count`) to find the processes with many runnable threads.
## Mitigation
Reduce or redistribute the workload, add logical processors, or fix the power
plan and parking settings. Give the virtual machine more CPU share or move it to
a less loaded host. Tune the threshold for hosts whose workload keeps a long
queue by design, such as build servers. Confirm the queue length per core and
application latency return to normal.
docs/runbooks/windowshostrebooted.md
+37
View File
@@ -0,0 +1,37 @@
---
title: WindowsHostRebooted
---
## Meaning
The boot time of the host, `windows_system_boot_time_timestamp`, is less than
ten minutes old and differs from the boot time known ten minutes earlier. The
alert fires once for about ten minutes after a reboot and then resolves by
itself. It has the `info` severity and is meant for a dashboard, a ticket or a
chat notification, not for a page.
## Impact
Every service of the host restarted, connections to it were dropped and
caches are cold. A reboot outside a maintenance window can mean a crash, an
unattended Windows Update restart or a power event.
## Diagnosis
1. Compare the boot time with the maintenance calendar and the Windows Update
schedule. With the `update` collector enabled, `windows_update_pending_info`
shows updates that were pending before the reboot.
2. Check the System event log: event 1074 names the user or process that
requested the shutdown, event 6008 marks an unexpected shutdown and event 41
a reset without a clean shutdown. Check for a memory dump when it was a
crash.
3. Confirm the host came back with the expected services; the
`WindowsServiceNotRunning` alert covers automatic services that did not
start.
## Mitigation
No action is needed for a planned reboot. For an unplanned one, find the cause
in the event log and fix the driver, update setting or power problem. Turn the
alert off with `alerts+: { WindowsHostRebooted: false }` when reboots are not
worth a notification.
docs/runbooks/windowsmemorypaginghigh.md
+42
View File
@@ -0,0 +1,42 @@
---
title: WindowsMemoryPagingHigh
---
## Meaning
The rate of pages read from or written to disk to resolve hard page faults,
`windows:memory_swap_page_operations:rate`, exceeds the configured threshold
for 15 minutes. The default is 1000 pages per second. The counter counts
pagefile traffic and memory-mapped file I/O alike.
## Impact
The host is short of physical memory and uses the disk as backing store, so
every affected process slows down to disk speed. Sustained paging also
competes with application I/O on the system disk.
## Diagnosis
1. Inspect the paging rate together with `windows:memory_usage:ratio`,
`windows_memory_available_bytes` and `windows:memory_committed:ratio`. Low
available memory confirms memory pressure; a high rate with ample memory
points at a process reading a large memory-mapped file, such as a backup or
an index rebuild.
2. Split the rate into reads and writes with
`windows_memory_swap_page_reads_total` and
`windows_memory_swap_page_writes_total`. Reads dominate when the working
sets of processes do not fit; writes dominate when the system trims and
writes modified pages.
3. Find the processes with the largest working sets and page fault rates in
Resource Monitor or, with the `process` collector enabled, with
`windows_process_working_set_private_bytes` and
`windows_process_page_faults_total`.
4. Check `windows_memory_standby_cache_lifetime_seconds`; a low lifetime means
cached pages are repurposed quickly, another sign of memory pressure.
## Mitigation
Stop or move the process that uses the memory, fix a memory leak, or add
physical memory. Move large sequential file jobs off peak hours when they cause
the paging. Confirm the paging rate drops and available memory recovers. Raise
the threshold on hosts whose workload maps large files by design.
docs/runbooks/windowsnetworkinterfaceflapping.md
+35
View File
@@ -0,0 +1,35 @@
---
title: WindowsNetworkInterfaceFlapping
---
## Meaning
The operational status of the interface `nic` changed more than twice within
the rate window, and did so for two minutes. The alert counts the changes of
`windows_net_nic_operation_status{status="up"}` between `1` and `0`.
## Impact
Every transition drops the connections that use the interface and triggers
route, DHCP and DNS registration changes. A flapping uplink on a server causes
intermittent application failures that are hard to attribute.
## Diagnosis
1. Inspect `windows_net_nic_operation_status` for the interface to see the
status it flips to; `down` and `lower layer down` point at the link,
`dormant` at authentication such as 802.1X.
2. Check the System event log for NDIS, the adapter driver and
`Microsoft-Windows-NetworkProfile` events, and the switch port for link
state changes, errors and port security actions.
3. Check the cable, SFP, speed and duplex negotiation, and whether a teaming or
failover configuration switches between members.
4. On virtual machines, check the hypervisor for virtual switch changes, live
migrations, or a host adapter that flaps.
## Mitigation
Replace the cable or transceiver, fix the speed and duplex settings, update the
adapter driver, or correct the switch port and authentication configuration.
Exclude an interface that flaps by design, such as a Wi-Fi or VPN adapter,
with the `nicSelector`. Confirm the status stays `up`.
docs/runbooks/windowsnetworkutilizationhigh.md
+40
View File
@@ -0,0 +1,40 @@
---
title: WindowsNetworkUtilizationHigh
---
## Meaning
Received or sent throughput on the interface `nic` exceeds the configured share
of its link bandwidth for 15 minutes. The default is 90%. The recordings
`windows:net_received_utilization:ratio` and
`windows:net_sent_utilization:ratio` divide the byte rate of each direction by
`windows_net_current_bandwidth_bytes`.
## Impact
The link is close to saturation in at least one direction. Further traffic
queues at the adapter, which raises latency and causes discards and TCP
retransmissions for every application on the host.
## Diagnosis
1. Inspect both utilization ratios to find the direction, and
`windows:net_received_bytes:rate` and `windows:net_sent_bytes:rate` for the
absolute volume.
2. Check `windows_net_current_bandwidth_bytes`. A link that negotiated a lower
speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high
utilization at normal traffic; compare it with `Get-NetAdapter`.
3. Correlate with `windows:net_received_discarded:rate`,
`windows:net_outbound_discarded:rate` and
`windows_net_output_queue_length_packets` to see whether the saturation
already drops packets.
4. Identify the traffic with Resource Monitor, `Get-NetTCPConnection`, or
switch-port statistics: backups, replication, file copies and monitoring
bursts are common causes.
## Mitigation
Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that
negotiated the wrong speed. Confirm utilization returns below the threshold and
discards stop. Tune the threshold for interfaces that run near line rate by
design, such as storage replication links.
docs/runbooks/windowsscrapequeuetimeout.md
+40
View File
@@ -0,0 +1,40 @@
---
title: WindowsScrapeQueueTimeout
---
## Meaning
`windows_exporter_scrape_queue_timeout == 1` for ten minutes. Scrapes run one
at a time; a scrape whose timeout ends while it waits for an earlier scrape to
finish runs no collector and reports this metric as `1`. On such a scrape
`windows_exporter_collector_success` is `0` for every collector.
## Impact
The affected scrapes return no collector metrics, so the host has gaps in all
of its series and `WindowsCollectorFailed` can fire for every collector at
once. The exporter itself is reachable.
## Diagnosis
1. Confirm the queue timeout and find the scrape that holds the slot:
```promql
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
windows_exporter_collector_timeout{job="windows_exporter"} == 1
scrape_duration_seconds{job="windows_exporter"}
```
2. Check whether more than one Prometheus server or job scrapes the same
exporter, or whether a job scrapes it more often than its scrape duration
allows. Compare `scrape_duration_seconds` with the `scrape_interval`.
3. Identify the slow collector with `windows_exporter_collector_duration_seconds`
and follow `WindowsCollectorTimeout` for it.
## Mitigation
Scrape each exporter from one job, or give the jobs a `scrape_interval` longer
than a full scrape takes. Shorten the slow collector or raise the
`scrape_timeout` so a scrape finishes before the next one starts. Confirm
`windows_exporter_scrape_queue_timeout` stays `0` and collector success returns
to `1`.
docs/runbooks/windowsservicecrashlooping.md
+42
View File
@@ -0,0 +1,42 @@
---
title: WindowsServiceCrashLooping
---
## Meaning
The service `service` has run as more processes in the last 15 minutes than
the configured number of restarts allows. The default is two restarts. Every
service process has its own `windows_service_process` series, identified by
`process_id`, so the series seen within 15 minutes count the restarts.
## Impact
The service crashes or is stopped and started repeatedly. It is unavailable
during every restart, and the recovery options of the Service Control Manager
give up after a configurable number of failures, after which
`WindowsServiceNotRunning` follows.
## Diagnosis
1. List the processes the service ran as:
```promql
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
```
The value of each series is the start time of the process.
2. Check the Application log for crashes of the service executable (events
1000 and 1026, .NET Runtime errors) and the System log for Service Control
Manager events 7031 and 7034, which also show the recovery action.
3. Check whether a deployment, configuration push, scheduled task or another
automation restarts the service on purpose.
4. Check for resources the service fails to acquire at start, such as a port
that is in use, a certificate, a database or a license.
## Mitigation
Fix the crash or the start failure, roll back the deployment that caused it, or
stop the automation that restarts the service. Keep the service stopped until
the cause is known if every restart has side effects. Confirm the service keeps
one process and stays running. Raise the threshold for services that restart by
design, for example worker services that recycle themselves.
docs/runbooks/windowsservicenotrunning.md
+43
View File
@@ -0,0 +1,43 @@
---
title: WindowsServiceNotRunning
---
## Meaning
The service `service` has the start mode `auto` but has not been in the
`running` state for 15 minutes. Trigger-start services, which Windows stops by
design when their trigger condition ends, are left out through the
`trigger_start` label of `windows_service_info`. Delayed-start services are
included; the 15 minute duration covers their later start after a reboot.
## Impact
A service that should always run is stopped, so the function it provides is
unavailable, for example printing, time synchronization, a web site or a
database instance. Windows restarts a crashed service only when its recovery
options say so.
## Diagnosis
1. Inspect the current state of the service and when it changed:
```promql
windows_service_state{job="windows_exporter", service="<service>"} == 1
```
2. Check the System event log for Service Control Manager events 7000, 7009,
7031 and 7034, and the Application log for crashes of the service process.
3. Check `windows_service_info` for the account in `run_as`; an expired
password or a removed right to log on as a service stops the service at the
next start. `windows_service_process` shows whether a process is running.
4. Check whether the service was stopped on purpose for maintenance or
disabled by policy, and whether it depends on another service that is also
stopped.
## Mitigation
Start the service after fixing the cause: the account, a missing dependency, a
configuration error or a crash. Set recovery options so Windows restarts the
service itself. Change the start mode to `manual` or `disabled` for services
that should not run, or exclude them with the `serviceSelector` when the mixin
should not watch them. Confirm the service stays running.