mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-21 03:59:04 +02:00
mixin: group alerts by collector area and add tier 1 alerts
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
@@ -0,0 +1,51 @@
|
||||
# Mixin instructions
|
||||
|
||||
Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers
|
||||
`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources.
|
||||
[`README.md`](README.md) documents the configuration and the generated rules;
|
||||
keep it in sync with every change.
|
||||
|
||||
## Alerts
|
||||
|
||||
`alerts/alerts.libsonnet` renders one rule group per collector area. Every
|
||||
alert names the collectors it reads (`collectors=[...]`) and is generated only
|
||||
with those enabled in `_config.collectors`; `_config.alerts` switches single
|
||||
alerts off by name. Threshold alerts get `keep_firing_for` from
|
||||
`keepFiringFor`; pass `threshold=false` for state alerts (`== 0`, `== 1`) and
|
||||
one-shot alerts, which must resolve at once.
|
||||
|
||||
A new alert touches five places, and CI checks all of them:
|
||||
|
||||
1. The rule in `alerts/alerts.libsonnet`, in the group of its area, with
|
||||
`{{ $labels.<key> }}` templates that use the 0.32 label keys (`service`,
|
||||
`volume`, `nic`, `task`, `device_id`, never `name`).
|
||||
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
|
||||
fails when the map and the generated alerts differ. Thresholds are
|
||||
`_config` knobs named `<area><Condition>Threshold[Unit]`.
|
||||
3. A unit test in `tests/alerts.test.yaml` that checks the first firing
|
||||
evaluation, the labels and annotations, and the recovery. With
|
||||
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
|
||||
condition clears. `rate()` extrapolates while a window has few samples, so
|
||||
ratios of two rates activate earlier than a single rate; probe the
|
||||
activation time with `ALERTS_FOR_STATE` instead of guessing it.
|
||||
4. A runbook `docs/runbooks/<lowercase-alert-name>.md` with the `Meaning`,
|
||||
`Impact`, `Diagnosis` and `Mitigation` sections.
|
||||
5. A row in the alerts table of `README.md`.
|
||||
|
||||
Metrics with a state label (`windows_service_state{state="running"} == 0`)
|
||||
keep that label in the alert; drop it with `max without (state) (...)` so the
|
||||
alert does not say `state="running"` for a stopped service. Join a state metric
|
||||
with its info metric `on (job, instance, <key>)`.
|
||||
|
||||
## Recording rules
|
||||
|
||||
New recordings go in `rules/rules.libsonnet` with the shared expressions in
|
||||
`lib/queries.libsonnet`, a row in the README table and a test in
|
||||
`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero
|
||||
capacities produce no sample instead of `NaN` or `+Inf`.
|
||||
|
||||
## Checks
|
||||
|
||||
`mise run test` runs everything CI runs. Without mise, the tools from
|
||||
`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
|
||||
the `lint`, `check-runbooks` and `test` tasks do the same.
|
||||
+87
-40
@@ -29,10 +29,12 @@ to generate rules. Generated rule files are ignored by Git:
|
||||
- `windows_alerts.yaml`: alerting rules.
|
||||
|
||||
The default selector is `job="windows_exporter"`. Change it to match your scrape
|
||||
job before generating rules. Resource rules use the default `cpu`, `memory`,
|
||||
`logical_disk`, and `net` collectors. Enable `ad` on domain controllers when
|
||||
opting into Active Directory alerts. Clock alerts require the `time` collector
|
||||
with its `ntp` subcollector and are intended for hosts using NTP synchronization.
|
||||
job before generating rules. The rules read the default `cpu`, `memory`,
|
||||
`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors`
|
||||
lists which collectors the targets run, and the alerts of a collector that is
|
||||
turned off are not generated. Enable `ad` on domain controllers when opting
|
||||
into Active Directory alerts. Clock alerts require the `time` collector with
|
||||
its `ntp` subcollector and are intended for hosts using NTP synchronization.
|
||||
|
||||
Copy both generated files into your Prometheus rule directory and reference
|
||||
them in `prometheus.yml`:
|
||||
@@ -62,6 +64,9 @@ modifying its source. For example, create `custom.libsonnet` in this directory:
|
||||
cpuHighUsageThreshold: 95,
|
||||
enableActiveDirectory: true,
|
||||
enableTime: true,
|
||||
collectors+: { pagefile: true },
|
||||
alerts+: { WindowsHostRebooted: false },
|
||||
keepFiringFor: '10m',
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
},
|
||||
}
|
||||
@@ -77,9 +82,11 @@ jsonnet -S -e 'std.manifestYamlDoc((import "custom.libsonnet").prometheusAlerts)
|
||||
Selectors are comma-separated Prometheus label matchers without braces. They
|
||||
apply to both recording rules and alerts. All target labels, including `job`,
|
||||
`instance`, `cluster`, and custom selector labels, are preserved. CPU usage
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk and
|
||||
network recordings also retain `volume` and `nic` respectively. The CPU mode
|
||||
recording retains `mode` and removes only `core`.
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk,
|
||||
network and service rules also retain `volume`, `nic` and `service`
|
||||
respectively, filtered by `volumeSelector`, `nicSelector` and
|
||||
`serviceSelector`. The CPU mode recording retains `mode` and removes only
|
||||
`core`.
|
||||
|
||||
`rateInterval` is the fixed window for recording rules and Active Directory
|
||||
failure rates; keep it at least four times your scrape interval. Grafana's
|
||||
@@ -93,6 +100,7 @@ rules.
|
||||
| `windows:cpu_usage:ratio` | Non-idle CPU time averaged across cores | Ratio, 0–1 |
|
||||
| `windows:cpu_cores:count` | Logical processors observed in idle CPU metrics | Count |
|
||||
| `windows:cpu_time:ratio_rate` | CPU time averaged across cores, retaining `mode` | Seconds/second |
|
||||
| `windows:cpu_queue_per_core:ratio` | Threads waiting for a processor, averaged over `rateInterval`, per logical processor | Threads/processor |
|
||||
| `windows:memory_usage:ratio` | Physical memory in use, based on available bytes | Ratio, 0–1 |
|
||||
| `windows:memory_used:bytes` | Total physical memory minus available memory | Bytes |
|
||||
| `windows:memory_cached:bytes` | Cache, modified pages, and three standby lists | Bytes |
|
||||
@@ -109,6 +117,8 @@ rules.
|
||||
| `windows:logical_disk_write_latency:seconds` | Write-time rate divided by write-operation rate | Seconds/operation |
|
||||
| `windows:net_received_bytes:rate` | Received network traffic per interface | Bytes/second |
|
||||
| `windows:net_sent_bytes:rate` | Sent network traffic per interface | Bytes/second |
|
||||
| `windows:net_received_packets:rate` | Received packets per interface | Packets/second |
|
||||
| `windows:net_sent_packets:rate` | Sent packets per interface | Packets/second |
|
||||
| `windows:net_received_utilization:ratio` | Received throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_sent_utilization:ratio` | Sent throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_received_errors:rate` | Inbound packet errors | Packets/second |
|
||||
@@ -127,46 +137,81 @@ which can exceed wall time for concurrent I/O. Latency is an average and is
|
||||
omitted when there are no operations. Memory, disk space, commit, and network
|
||||
utilization ratios are omitted when their capacity is zero or missing. Network
|
||||
bandwidth is already in bytes/second; receive and send utilization are separate
|
||||
for full-duplex links.
|
||||
for full-duplex links. The processor queue length is an instantaneous sample,
|
||||
so the queue recording averages it over `rateInterval` before dividing by the
|
||||
core count.
|
||||
The cache recording includes modified pages and is not a measure of immediately
|
||||
reclaimable memory. Paging operations can include file-backed pages as well as
|
||||
pagefile traffic.
|
||||
|
||||
## Alerts
|
||||
|
||||
Threshold configuration values below are percentages except where noted.
|
||||
The alerts are grouped by collector area. Each group lists the collectors its
|
||||
rules read; a group is generated only with those collectors enabled in
|
||||
`_config.collectors`, and a rule that needs two collectors, such as the CPU
|
||||
saturation alert, only with both. Threshold configuration values below are
|
||||
percentages except where noted.
|
||||
|
||||
| Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- |
|
||||
| `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsNetworkErrors` | Combined receive/outbound errors above `networkErrorRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsNetworkDiscards` | Combined receive/outbound discards above `networkDiscardRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
| Group | Collectors | Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `windows-exporter` | | `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `windows-exporter` | | `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `windows-exporter` | | `WindowsCollectorTimeout` | `windows_exporter_collector_timeout == 1` | 10m | Warning |
|
||||
| `windows-exporter` | | `WindowsScrapeQueueTimeout` | `windows_exporter_scrape_queue_timeout == 1` | 10m | Warning |
|
||||
| `windows-cpu` | `cpu` | `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-cpu` | `cpu`, `system` | `WindowsCPUSaturation` | Waiting threads per logical processor above `cpuQueuePerCoreThreshold` (2) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryPagingHigh` | Pages read or written per second above `memoryPagingThreshold` (1000) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkErrors` | Receive and outbound errors above `networkErrorRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkDiscards` | Receive and outbound discards above `networkDiscardRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkUtilizationHigh` | Received or sent throughput above `networkUtilizationThreshold` (90%) of the link bandwidth | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkInterfaceFlapping` | Operational status changed more than twice within `rateInterval` | 2m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceNotRunning` | Automatic service without triggers not running | 15m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceCrashLooping` | More than `serviceRestartThreshold` (2) restarts in 15 minutes | 0m | Warning |
|
||||
| `windows-system` | `system` | `WindowsHostRebooted` | Boot time less than ten minutes old and changed | 0m | Info |
|
||||
| `windows-active-directory` | `ad` | `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
|
||||
Active Directory alerts are generated only when `enableActiveDirectory` is
|
||||
`true`. `criticalSeverity` defaults to `critical` and can be overridden to match
|
||||
your routing labels. Below the critical disk threshold, both disk severity
|
||||
levels fire; configure Alertmanager inhibition if only the critical
|
||||
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
|
||||
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
|
||||
build, and a name missing from a replaced object stays enabled. A group whose
|
||||
alerts are all turned off is not generated.
|
||||
|
||||
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
|
||||
value that oscillates around a threshold does not flap. State alerts such as
|
||||
`WindowsExporterDown` and `WindowsServiceNotRunning` resolve as soon as their
|
||||
condition clears. `keep_firing_for` needs Prometheus 2.42 or later; set
|
||||
`keepFiringFor: ''` for older servers.
|
||||
|
||||
`criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override
|
||||
them to match your routing labels. Below the critical disk threshold, both disk
|
||||
severity levels fire; configure Alertmanager inhibition if only the critical
|
||||
notification should be sent.
|
||||
|
||||
Clock alerts are generated only when `enableTime` is `true`. A zero NTP source
|
||||
count may be expected on hosts using another time provider. The offset compares
|
||||
the host to its chosen source and does not independently verify that source.
|
||||
Disk latency and network error/discard thresholds should be tuned for the
|
||||
Active Directory alerts need `collectors.ad`, which `enableActiveDirectory`
|
||||
sets. Clock alerts need `collectors.time`, which `enableTime` sets. A zero NTP
|
||||
source count may be expected on hosts using another time provider. The offset
|
||||
compares the host to its chosen source and does not independently verify that
|
||||
source. Disk latency, paging and network thresholds should be tuned for the
|
||||
storage tier, traffic volume, and application requirements.
|
||||
|
||||
`WindowsServiceNotRunning` watches services with the `auto` start mode and
|
||||
skips trigger-start services, which stop by design, through the
|
||||
`trigger_start` label of `windows_service_info`. Delayed-start services are
|
||||
included; the 15 minute duration covers their later start after a reboot.
|
||||
`WindowsServiceCrashLooping` counts the `windows_service_process` series of a
|
||||
service within 15 minutes, one per process the service ran as. Exclude services
|
||||
with `serviceSelector`. `WindowsHostRebooted` fires once for about ten minutes
|
||||
after a reboot and resolves by itself.
|
||||
|
||||
Windows disk size/free-space counters may lag by 10–15 minutes. Predictions
|
||||
depend on a sustained trend and are intentionally delayed for an hour. A target
|
||||
removed from service discovery has no `up` series and will not trigger
|
||||
@@ -366,11 +411,13 @@ find unused variables and references to missing fields, checks that the
|
||||
committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
|
||||
`promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic
|
||||
time series.
|
||||
Tests cover all 25 recordings, alert timing and recovery, threshold
|
||||
boundaries, counter resets, zero denominators, disk forecasts, target-label
|
||||
preservation, and custom selectors/thresholds with Active Directory and clock
|
||||
alerts enabled. Runbook checks include opt-in alerts and verify that their pages
|
||||
and URLs exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`,
|
||||
threshold boundaries, counter resets, zero denominators, disk forecasts,
|
||||
target-label preservation, and custom selectors/thresholds with Active
|
||||
Directory and clock alerts enabled. `tests/alerts.test.jsonnet` checks the
|
||||
collector gating, the per-alert switches and the group layout. Runbook checks
|
||||
render the alerts of every collector and verify that their pages and URLs
|
||||
exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
`mise install`.
|
||||
|
||||
To list the available tasks:
|
||||
|
||||
@@ -1,158 +1,297 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local q = (import '../lib/queries.libsonnet')(c),
|
||||
local alert(name, expr, duration, severity, summary, description) = {
|
||||
alert: name,
|
||||
expr: expr,
|
||||
'for': duration,
|
||||
labels: { severity: severity },
|
||||
annotations: { summary: summary, description: description },
|
||||
},
|
||||
prometheusAlerts+:: {
|
||||
groups: [
|
||||
local known = std.objectFields((import '../config.libsonnet')._config.alerts),
|
||||
local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
|
||||
assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
|
||||
// A collector missing from a replaced collectors object counts as disabled;
|
||||
// an alert missing from a replaced alerts object counts as enabled.
|
||||
local collectorEnabled(collector) = std.get(c.collectors, collector, false),
|
||||
local alertEnabled(name) = std.get(c.alerts, name, true),
|
||||
local keepFiring = if c.keepFiringFor != '' then { keep_firing_for: c.keepFiringFor } else {},
|
||||
// alert renders a rule when every collector it needs is enabled and the
|
||||
// alert is not turned off. Threshold alerts set keepFiringFor so a value
|
||||
// oscillating around the threshold does not flap; state alerts and
|
||||
// one-shot alerts leave it out.
|
||||
local alert(name, expr, duration, severity, summary, description, collectors=[], threshold=true) =
|
||||
if std.all([collectorEnabled(collector) for collector in collectors]) && alertEnabled(name) then [
|
||||
{
|
||||
name: 'windows-exporter-alerts',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsExporterDown',
|
||||
q.metric('up') + ' == 0',
|
||||
'5m',
|
||||
c.criticalSeverity,
|
||||
'Windows exporter is unreachable',
|
||||
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).'
|
||||
),
|
||||
alert(
|
||||
'WindowsCollectorFailed',
|
||||
q.metric('windows_exporter_collector_success') + ' == 0',
|
||||
'5m',
|
||||
'warning',
|
||||
'Windows collector failed',
|
||||
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsCPUHighUsage',
|
||||
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows CPU usage is high',
|
||||
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryHighUsage',
|
||||
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory usage is high',
|
||||
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryCommitHighUsage',
|
||||
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory commit usage is high',
|
||||
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold
|
||||
),
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDisk%sLatencyHigh' % direction[0],
|
||||
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows disk %s latency is high' % direction[1],
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]]
|
||||
)
|
||||
for direction in [['Read', 'read'], ['Write', 'write']]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsNetwork%s' % outcome[0],
|
||||
'(%s + %s) > %g' % [
|
||||
alert: name,
|
||||
expr: expr,
|
||||
labels: { severity: severity },
|
||||
annotations: { summary: summary, description: description },
|
||||
} + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
|
||||
] else [],
|
||||
local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
|
||||
local onService = 'on (job, instance, service)',
|
||||
prometheusAlerts+:: {
|
||||
groups:
|
||||
group('windows-exporter', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsExporterDown',
|
||||
q.metric('up') + ' == 0',
|
||||
'5m',
|
||||
c.criticalSeverity,
|
||||
'Windows exporter is unreachable',
|
||||
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
|
||||
threshold=false
|
||||
),
|
||||
alert(
|
||||
'WindowsCollectorFailed',
|
||||
q.metric('windows_exporter_collector_success') + ' == 0',
|
||||
'5m',
|
||||
'warning',
|
||||
'Windows collector failed',
|
||||
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
|
||||
threshold=false
|
||||
),
|
||||
alert(
|
||||
'WindowsCollectorTimeout',
|
||||
q.metric('windows_exporter_collector_timeout') + ' == 1',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows collector exceeds the scrape timeout',
|
||||
'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
|
||||
threshold=false
|
||||
),
|
||||
alert(
|
||||
'WindowsScrapeQueueTimeout',
|
||||
q.metric('windows_exporter_scrape_queue_timeout') + ' == 1',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows exporter scrapes time out in the queue',
|
||||
'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
|
||||
threshold=false
|
||||
),
|
||||
]))
|
||||
+ group('windows-cpu', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsCPUHighUsage',
|
||||
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows CPU usage is high',
|
||||
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold,
|
||||
collectors=['cpu']
|
||||
),
|
||||
alert(
|
||||
'WindowsCPUSaturation',
|
||||
q.metric('windows:cpu_queue_per_core:ratio') + ' > %s' % c.cpuQueuePerCoreThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows processor queue is long',
|
||||
'More than %s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c.cpuQueuePerCoreThreshold,
|
||||
collectors=['cpu', 'system']
|
||||
),
|
||||
]))
|
||||
+ group('windows-memory', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsMemoryHighUsage',
|
||||
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory usage is high',
|
||||
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold,
|
||||
collectors=['memory']
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryCommitHighUsage',
|
||||
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory commit usage is high',
|
||||
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold,
|
||||
collectors=['memory']
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryPagingHigh',
|
||||
q.metric('windows:memory_swap_page_operations:rate') + ' > %s' % c.memoryPagingThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory paging is high',
|
||||
'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold,
|
||||
collectors=['memory']
|
||||
),
|
||||
]))
|
||||
+ group('windows-disk', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsDisk%sLatencyHigh' % direction[0],
|
||||
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows disk %s latency is high' % direction[1],
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]],
|
||||
collectors=['logical_disk']
|
||||
)
|
||||
for direction in [['Read', 'read'], ['Write', 'write']]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskAlmostFull',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
|
||||
'15m',
|
||||
entry[0],
|
||||
'Windows disk space is low',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1],
|
||||
collectors=['logical_disk']
|
||||
)
|
||||
for entry in [
|
||||
['warning', c.diskFreeWarningThreshold],
|
||||
[c.criticalSeverity, c.diskFreeCriticalThreshold],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskFillingUp',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
|
||||
c.diskFreeWarningThreshold,
|
||||
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
|
||||
c.diskPredictionWindow,
|
||||
c.diskPredictionHours * 3600,
|
||||
],
|
||||
'1h',
|
||||
'warning',
|
||||
'Windows disk is predicted to fill up',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours,
|
||||
collectors=['logical_disk']
|
||||
),
|
||||
]))
|
||||
+ group('windows-network', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsNetwork%s' % outcome[0],
|
||||
q.ratio(
|
||||
'(%s + %s)' % [
|
||||
q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
outcome[2],
|
||||
],
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows network packet %s are high' % outcome[3],
|
||||
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g per second for 15 minutes.' % [outcome[3], outcome[2]]
|
||||
)
|
||||
for outcome in [
|
||||
['Errors', 'errors', c.networkErrorRateThreshold, 'errors'],
|
||||
['Discards', 'discarded', c.networkDiscardRateThreshold, 'discards'],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskAlmostFull',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
|
||||
'15m',
|
||||
entry[0],
|
||||
'Windows disk space is low',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1]
|
||||
)
|
||||
for entry in [
|
||||
['warning', c.diskFreeWarningThreshold],
|
||||
[c.criticalSeverity, c.diskFreeCriticalThreshold],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskFillingUp',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
|
||||
c.diskFreeWarningThreshold,
|
||||
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
|
||||
c.diskPredictionWindow,
|
||||
c.diskPredictionHours * 3600,
|
||||
],
|
||||
'1h',
|
||||
'warning',
|
||||
'Windows disk is predicted to fill up',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours
|
||||
),
|
||||
],
|
||||
},
|
||||
] + (if c.enableActiveDirectory then [
|
||||
{
|
||||
name: 'windows-active-directory',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsADPendingReplication',
|
||||
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication queue is high',
|
||||
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsADReplicationFailures',
|
||||
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication requests are failing',
|
||||
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.'
|
||||
),
|
||||
],
|
||||
},
|
||||
] else []) + if c.enableTime then [
|
||||
{
|
||||
name: 'windows-time',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsClockNotSynchronising',
|
||||
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock has no NTP time source',
|
||||
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsClockSkewDetected',
|
||||
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock offset is high',
|
||||
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds
|
||||
),
|
||||
],
|
||||
},
|
||||
] else [],
|
||||
'(%s + %s)' % [
|
||||
q.metric('windows:net_received_packets:rate', [c.nicSelector]),
|
||||
q.metric('windows:net_sent_packets:rate', [c.nicSelector]),
|
||||
]
|
||||
) + ' * 100 > %g' % outcome[2],
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows network packet %s are high' % outcome[3],
|
||||
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g%% of all packets for 15 minutes.' % [outcome[3], outcome[2]],
|
||||
collectors=['net']
|
||||
)
|
||||
for outcome in [
|
||||
['Errors', 'errors', c.networkErrorRatioThreshold, 'errors'],
|
||||
['Discards', 'discarded', c.networkDiscardRatioThreshold, 'discards'],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsNetworkUtilizationHigh',
|
||||
'(%s * 100 > %s) or (%s * 100 > %s)' % [
|
||||
q.metric('windows:net_received_utilization:ratio', [c.nicSelector]),
|
||||
c.networkUtilizationThreshold,
|
||||
q.metric('windows:net_sent_utilization:ratio', [c.nicSelector]),
|
||||
c.networkUtilizationThreshold,
|
||||
],
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows network interface utilization is high',
|
||||
'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %s%% of its bandwidth in at least one direction for 15 minutes.' % c.networkUtilizationThreshold,
|
||||
collectors=['net']
|
||||
),
|
||||
alert(
|
||||
'WindowsNetworkInterfaceFlapping',
|
||||
'max without (status) (changes(%s[%s])) > 2' % [
|
||||
q.metric('windows_net_nic_operation_status', [c.nicSelector, 'status="up"']),
|
||||
c.rateInterval,
|
||||
],
|
||||
'2m',
|
||||
'warning',
|
||||
'Windows network interface is flapping',
|
||||
'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %s.' % c.rateInterval,
|
||||
collectors=['net']
|
||||
),
|
||||
]))
|
||||
+ group('windows-service', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsServiceNotRunning',
|
||||
std.join(' ', [
|
||||
'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']),
|
||||
'and %s (%s == 1)' % [onService, q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"'])],
|
||||
'unless %s (%s == 1)' % [onService, q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"'])],
|
||||
]),
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows automatic service is not running',
|
||||
'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
|
||||
collectors=['service'],
|
||||
threshold=false
|
||||
),
|
||||
alert(
|
||||
'WindowsServiceCrashLooping',
|
||||
// Every service process has its own series, so the series seen in the
|
||||
// window count the processes the service ran as.
|
||||
'count without (process_id) (count_over_time(%s[15m])) - 1 > %s' % [
|
||||
q.metric('windows_service_process', [c.serviceSelector]),
|
||||
c.serviceRestartThreshold,
|
||||
],
|
||||
'0m',
|
||||
'warning',
|
||||
'Windows service is restarting repeatedly',
|
||||
'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %s times in 15 minutes.' % c.serviceRestartThreshold,
|
||||
collectors=['service']
|
||||
),
|
||||
]))
|
||||
+ group('windows-system', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsHostRebooted',
|
||||
// Fires once per reboot: for the first ten minutes after the boot,
|
||||
// while the boot time known ten minutes ago was an older one.
|
||||
'(time() - %s) < 600 and (time() - %s offset 10m) > 600' % [
|
||||
q.metric('windows_system_boot_time_timestamp'),
|
||||
q.metric('windows_system_boot_time_timestamp'),
|
||||
],
|
||||
'0m',
|
||||
c.infoSeverity,
|
||||
'Windows host has rebooted',
|
||||
'{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
|
||||
collectors=['system'],
|
||||
threshold=false
|
||||
),
|
||||
]))
|
||||
+ group('windows-active-directory', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsADPendingReplication',
|
||||
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication queue is high',
|
||||
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold,
|
||||
collectors=['ad']
|
||||
),
|
||||
alert(
|
||||
'WindowsADReplicationFailures',
|
||||
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication requests are failing',
|
||||
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
|
||||
collectors=['ad']
|
||||
),
|
||||
]))
|
||||
+ group('windows-time', std.flattenArrays([
|
||||
alert(
|
||||
'WindowsClockNotSynchronising',
|
||||
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock has no NTP time source',
|
||||
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
|
||||
collectors=['time'],
|
||||
threshold=false
|
||||
),
|
||||
alert(
|
||||
'WindowsClockSkewDetected',
|
||||
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock offset is high',
|
||||
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds,
|
||||
collectors=['time']
|
||||
),
|
||||
])),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -4,11 +4,12 @@
|
||||
windowsExporterSelector: 'job="windows_exporter"',
|
||||
volumeSelector: 'volume!=""',
|
||||
nicSelector: 'nic!=""',
|
||||
serviceSelector: 'service!=""',
|
||||
rateInterval: '5m',
|
||||
|
||||
// Opt in only for targets with the ad collector enabled.
|
||||
// Opt in only for targets with the ad collector enabled. Sets collectors.ad.
|
||||
enableActiveDirectory: false,
|
||||
// Requires the time collector with its ntp subcollector enabled.
|
||||
// Requires the time collector with its ntp subcollector enabled. Sets collectors.time.
|
||||
enableTime: false,
|
||||
|
||||
// Grafana dashboard metadata. A UID is the v2 metadata.name; change the
|
||||
@@ -37,8 +38,10 @@
|
||||
|
||||
// Collectors enabled on the windows_exporter targets, by collector name.
|
||||
// The dashboard only renders the tabs, rows and panels of enabled
|
||||
// collectors. The defaults match the exporter's default collectors.
|
||||
// collectors, and the alerts only the rules whose collectors are enabled.
|
||||
// The defaults match the exporter's default collectors.
|
||||
collectors: {
|
||||
ad: $._config.enableActiveDirectory,
|
||||
cpu: true,
|
||||
diskdrive: false,
|
||||
gpu: false,
|
||||
@@ -62,20 +65,66 @@
|
||||
update: false,
|
||||
},
|
||||
|
||||
// Alerts by name. Set one to false to leave it out of the generated
|
||||
// rules, for example alerts+: { WindowsHostRebooted: false }. Names that
|
||||
// are not listed here fail the build; names left out of a replaced object
|
||||
// stay enabled.
|
||||
alerts: {
|
||||
WindowsExporterDown: true,
|
||||
WindowsCollectorFailed: true,
|
||||
WindowsCollectorTimeout: true,
|
||||
WindowsScrapeQueueTimeout: true,
|
||||
WindowsCPUHighUsage: true,
|
||||
WindowsCPUSaturation: true,
|
||||
WindowsMemoryHighUsage: true,
|
||||
WindowsMemoryCommitHighUsage: true,
|
||||
WindowsMemoryPagingHigh: true,
|
||||
WindowsDiskReadLatencyHigh: true,
|
||||
WindowsDiskWriteLatencyHigh: true,
|
||||
WindowsDiskAlmostFull: true,
|
||||
WindowsDiskFillingUp: true,
|
||||
WindowsNetworkErrors: true,
|
||||
WindowsNetworkDiscards: true,
|
||||
WindowsNetworkUtilizationHigh: true,
|
||||
WindowsNetworkInterfaceFlapping: true,
|
||||
WindowsServiceNotRunning: true,
|
||||
WindowsServiceCrashLooping: true,
|
||||
WindowsHostRebooted: true,
|
||||
WindowsADPendingReplication: true,
|
||||
WindowsADReplicationFailures: true,
|
||||
WindowsClockNotSynchronising: true,
|
||||
WindowsClockSkewDetected: true,
|
||||
},
|
||||
|
||||
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
|
||||
|
||||
// Threshold alerts keep firing this long after their condition clears,
|
||||
// so a value oscillating around a threshold does not flap. Needs
|
||||
// Prometheus 2.42 or later; set it to '' to leave keep_firing_for out.
|
||||
keepFiringFor: '5m',
|
||||
|
||||
cpuHighUsageThreshold: 90,
|
||||
// Threads waiting for a processor, per logical processor.
|
||||
cpuQueuePerCoreThreshold: 2,
|
||||
memoryHighUsageThreshold: 90,
|
||||
memoryCommitHighUsageThreshold: 90,
|
||||
// Pages read from or written to disk per second.
|
||||
memoryPagingThreshold: 1000,
|
||||
diskLatencyThresholdSeconds: 0.05,
|
||||
networkErrorRateThreshold: 1,
|
||||
networkDiscardRateThreshold: 1,
|
||||
// Percent of all packets on the interface.
|
||||
networkErrorRatioThreshold: 1,
|
||||
networkDiscardRatioThreshold: 1,
|
||||
// Percent of the interface bandwidth, in either direction.
|
||||
networkUtilizationThreshold: 90,
|
||||
clockOffsetThresholdSeconds: 0.05,
|
||||
diskFreeWarningThreshold: 10,
|
||||
diskFreeCriticalThreshold: 5,
|
||||
diskPredictionWindow: '6h',
|
||||
diskPredictionHours: 24,
|
||||
// Restarts of one service within 15 minutes.
|
||||
serviceRestartThreshold: 2,
|
||||
criticalSeverity: 'critical',
|
||||
infoSeverity: 'info',
|
||||
adPendingReplicationThreshold: 50,
|
||||
adReplicationFailureRateThreshold: 0,
|
||||
},
|
||||
|
||||
@@ -7,6 +7,13 @@ function(c)
|
||||
ratio(numerator, denominator): '(%s / (%s > 0))' % [numerator, denominator],
|
||||
cpuUsage:
|
||||
'1 - avg without (core, mode) (%s)' % self.rate('windows_cpu_time_seconds_total', ['mode="idle"']),
|
||||
// Threads waiting for a processor, averaged over the rate interval, per
|
||||
// logical processor. The queue length is an instantaneous sample.
|
||||
cpuQueuePerCore:
|
||||
self.ratio(
|
||||
'avg_over_time(%s[%s])' % [self.metric('windows_system_processor_queue_length'), c.rateInterval],
|
||||
self.metric('windows:cpu_cores:count')
|
||||
),
|
||||
memoryUsage:
|
||||
'1 - %s' % self.ratio(
|
||||
self.metric('windows_memory_available_bytes'),
|
||||
|
||||
@@ -95,13 +95,14 @@ pint --offline lint tests/custom_rules.yaml tests/custom_alerts.yaml
|
||||
|
||||
[tasks.check-runbooks]
|
||||
description = "Check runbook coverage for all alerts, including opt-in alerts"
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { enableActiveDirectory: true, enableTime: true } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [name]: true for name in std.objectFields(super.collectors) } } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
|
||||
[tasks.test]
|
||||
description = "Run all mixin lint, runbook, and rule tests"
|
||||
depends = ["lint", "check-runbooks", "vendor"]
|
||||
run = [
|
||||
"jsonnet tests/runbooks.test.jsonnet",
|
||||
"jsonnet tests/alerts.test.jsonnet",
|
||||
"jsonnet -J vendor tests/dashboard.test.jsonnet > /dev/null",
|
||||
"promtool test rules tests/*.test.yaml",
|
||||
]
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
{ record: 'windows:cpu_usage:ratio', expr: q.cpuUsage },
|
||||
{ record: 'windows:cpu_cores:count', expr: 'count without (core, mode) (%s)' % q.metric('windows_cpu_time_seconds_total', ['mode="idle"']) },
|
||||
{ record: 'windows:cpu_time:ratio_rate', expr: 'avg without (core) (%s)' % q.rate('windows_cpu_time_seconds_total') },
|
||||
{ record: 'windows:cpu_queue_per_core:ratio', expr: q.cpuQueuePerCore },
|
||||
{ record: 'windows:memory_usage:ratio', expr: q.memoryUsage },
|
||||
{
|
||||
record: 'windows:memory_used:bytes',
|
||||
@@ -56,6 +57,12 @@
|
||||
expr: q.rate('windows_net_bytes_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{
|
||||
record: 'windows:net_%s_packets:rate' % direction,
|
||||
expr: q.rate('windows_net_packets_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{ record: 'windows:net_%s_utilization:ratio' % direction, expr: q.networkUtilization(direction) }
|
||||
for direction in ['received', 'sent']
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
// Checks the alert configuration: collector gating, per-alert switches,
|
||||
// keep_firing_for and the group layout.
|
||||
local mixin = import '../mixin.libsonnet';
|
||||
local config = (import '../config.libsonnet')._config;
|
||||
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true } };
|
||||
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
|
||||
local groupNames(alerts) = [group.name for group in alerts.groups];
|
||||
|
||||
// Every configured switch belongs to a generated alert, and every alert has a switch.
|
||||
local all = std.set(names(everything.prometheusAlerts));
|
||||
assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs from generated alerts: %s' % std.manifestJson([std.setDiff(all, std.set(std.objectFields(config.alerts))), std.setDiff(std.set(std.objectFields(config.alerts)), all)]);
|
||||
|
||||
// Groups follow the collector areas; opt-in groups appear only when enabled.
|
||||
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
|
||||
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory', 'windows-time'];
|
||||
|
||||
// Disabling a collector removes its alerts and an empty group; the alerts of a
|
||||
// rule that needs two collectors go with either.
|
||||
local noNet = mixin { _config+:: { collectors+: { net: false } } };
|
||||
assert !std.member(groupNames(noNet.prometheusAlerts), 'windows-network');
|
||||
local noSystem = mixin { _config+:: { collectors+: { system: false } } };
|
||||
assert names(noSystem.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUSaturation', 'WindowsHostRebooted']);
|
||||
assert std.member(names(noSystem.prometheusAlerts), 'WindowsCPUHighUsage');
|
||||
// A replaced collectors object counts missing collectors as disabled.
|
||||
local onlyCPU = mixin { _config+:: { collectors: { cpu: true } } };
|
||||
assert names(onlyCPU.prometheusAlerts) == ['WindowsCPUHighUsage', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsExporterDown', 'WindowsScrapeQueueTimeout'] : std.manifestJson(names(onlyCPU.prometheusAlerts));
|
||||
|
||||
// Single alerts switch off by name, with alerts+: or a replaced object.
|
||||
local noReboot = mixin { _config+:: { alerts+: { WindowsHostRebooted: false } } };
|
||||
assert names(noReboot.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsHostRebooted']);
|
||||
assert !std.member(groupNames(noReboot.prometheusAlerts), 'windows-system');
|
||||
local replaced = mixin { _config+:: { alerts: { WindowsCPUHighUsage: false } } };
|
||||
assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUHighUsage']);
|
||||
|
||||
// keep_firing_for applies to threshold alerts only and can be turned off.
|
||||
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
|
||||
local stateAlerts = ['WindowsExporterDown', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsScrapeQueueTimeout', 'WindowsServiceNotRunning', 'WindowsHostRebooted', 'WindowsClockNotSynchronising'];
|
||||
assert std.all([
|
||||
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
|
||||
for rule in rules(everything.prometheusAlerts)
|
||||
]) : 'keep_firing_for set on a state alert or missing on a threshold alert';
|
||||
assert std.all([rule.keep_firing_for == '5m' for rule in rules(everything.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]);
|
||||
local noKeep = mixin { _config+:: { keepFiringFor: '' } };
|
||||
assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]) == 0;
|
||||
|
||||
// The info severity label is configurable.
|
||||
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
|
||||
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
|
||||
|
||||
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
|
||||
@@ -4,7 +4,15 @@ rule_files:
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Commit pressure, disk latency and network failures wait and recover
|
||||
@@ -30,6 +38,10 @@ tests:
|
||||
values: '0x31'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+120x20 2520x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
@@ -40,7 +52,8 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/
|
||||
summary: Windows memory commit usage is high
|
||||
description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 21m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 25m, exp_alerts: [{exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}, exp_annotations: {runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/, summary: Windows memory commit usage is high, description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'}}]}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 26m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskReadLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -50,7 +63,7 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/
|
||||
summary: Windows disk read latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per read operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskWriteLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -60,27 +73,27 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/
|
||||
summary: Windows disk write latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per write operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 17m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkErrors
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 17m, exp_alerts: []}
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 25m, exp_alerts: []}
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 30m, exp_alerts: []}
|
||||
|
||||
- name: New resource alerts stay quiet at exact thresholds and clock alerts are disabled
|
||||
interval: 1m
|
||||
@@ -105,6 +118,10 @@ tests:
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_time_ntp_client_time_sources{job="windows_exporter", instance="host:9182"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_time_computed_time_offset_seconds{job="windows_exporter", instance="host:9182"}'
|
||||
@@ -178,7 +195,7 @@ tests:
|
||||
summary: Windows CPU usage is high
|
||||
description: 'CPU usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsCPUHighUsage
|
||||
eval_time: 25m
|
||||
eval_time: 30m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 14m
|
||||
@@ -192,16 +209,16 @@ tests:
|
||||
summary: Windows memory usage is high
|
||||
description: 'Physical memory usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Disk alerts retain volume labels and fire at both severity levels
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_logical_disk_free_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '3x15 30x5'
|
||||
values: '3x15 30x10'
|
||||
- series: 'windows_logical_disk_size_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '100x21'
|
||||
values: '100x26'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 14m
|
||||
@@ -220,7 +237,7 @@ tests:
|
||||
summary: Windows disk space is low
|
||||
description: 'Volume C: on host:9182 has less than 5% free space. Windows may delay space metrics by 10–15 minutes.'
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 16m
|
||||
eval_time: 21m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Healthy resources at the exact thresholds do not fire
|
||||
@@ -291,3 +308,193 @@ tests:
|
||||
- {alertname: WindowsMemoryHighUsage, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskAlmostFull, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskFillingUp, eval_time: 70m, exp_alerts: []}
|
||||
|
||||
- name: Collector and scrape queue timeouts wait ten minutes and recover without keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="service"}'
|
||||
values: '1x10 0x5'
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="cpu"}'
|
||||
values: '0x15'
|
||||
- series: 'windows_exporter_scrape_queue_timeout{job="windows_exporter", instance="host:9182"}'
|
||||
values: '1x10 0x5'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsCollectorTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', collector: service, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/
|
||||
summary: Windows collector exceeds the scrape timeout
|
||||
description: 'Collector service on host:9182 has not finished within the scrape timeout for 10 minutes.'
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsScrapeQueueTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/
|
||||
summary: Windows exporter scrapes time out in the queue
|
||||
description: 'Scrapes of host:9182 have timed out waiting for an earlier scrape to finish for 10 minutes.'
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 11m, exp_alerts: []}
|
||||
|
||||
- name: CPU saturation and paging use recordings, wait fifteen minutes and keep firing five
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '9x25 0x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_memory_swap_page_operations_total{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '0+90000x20 1800000x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 31m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 32m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 18m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryPagingHigh
|
||||
eval_time: 19m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
|
||||
summary: Windows memory paging is high
|
||||
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 35m, exp_alerts: []}
|
||||
|
||||
- name: Network utilization fires in one direction and interface flapping counts status changes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_net_bytes_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+6000x20 120000x15'
|
||||
- series: 'windows_net_bytes_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0x36'
|
||||
- series: 'windows_net_current_bandwidth_bytes{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '100x36'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="up"}'
|
||||
values: '1 0 1 0 1 0 1 0 1 0 1x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="down"}'
|
||||
values: '0 1 0 1 0 1 0 1 0 1 0x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Wi-Fi", status="up"}'
|
||||
values: '0x21'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 19m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkUtilizationHigh
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
|
||||
summary: Windows network interface utilization is high
|
||||
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 35m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkInterfaceFlapping
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/
|
||||
summary: Windows network interface is flapping
|
||||
description: 'Interface Ethernet on host:9182 has changed its operational status more than twice in 5m.'
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 17m, exp_alerts: []}
|
||||
|
||||
- name: Service alerts skip trigger-start and manual services and count restarts by process
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}'
|
||||
values: '0x15 1x5'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}'
|
||||
values: '1000x4 _x16'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}'
|
||||
values: '_x5 2000x4 _x11'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}'
|
||||
values: '_x10 3000x4 _x6'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}'
|
||||
values: '_x15 4000x6'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}'
|
||||
values: '_x16 5000x5'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceNotRunning
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: Spooler, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
|
||||
summary: Windows automatic service is not running
|
||||
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 16m, exp_alerts: []}
|
||||
- {alertname: WindowsServiceCrashLooping, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceCrashLooping
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
|
||||
summary: Windows service is restarting repeatedly
|
||||
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
|
||||
|
||||
- name: Host reboot fires once for the first ten minutes after a new boot time
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
|
||||
values: '-100000x10 540x15'
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="steady:9182"}'
|
||||
values: '-100000x25'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHostRebooted, eval_time: 10m, exp_alerts: []}
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 11m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 2m 0s ago.'
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 18m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 9m 0s ago.'
|
||||
- {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []}
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
memoryCommitHighUsageThreshold: 80,
|
||||
diskLatencyThresholdSeconds: 0.02,
|
||||
networkErrorRateThreshold: 0.5,
|
||||
networkDiscardRateThreshold: 0.5,
|
||||
networkErrorRatioThreshold: 0.5,
|
||||
networkDiscardRatioThreshold: 0.5,
|
||||
clockOffsetThresholdSeconds: 0.1,
|
||||
adPendingReplicationThreshold: 10,
|
||||
adReplicationFailureRateThreshold: 0.5,
|
||||
|
||||
@@ -4,7 +4,13 @@ rule_files:
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
@@ -52,6 +58,14 @@ tests:
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
eval_time: 20m
|
||||
@@ -84,7 +98,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
@@ -92,7 +106,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
|
||||
- name: Clock alerts opt in, use absolute offset, custom URLs and recover
|
||||
interval: 1m
|
||||
@@ -129,7 +143,7 @@ tests:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclockskewdetected/
|
||||
summary: Windows clock offset is high
|
||||
description: 'Windows Time on host:9182 reports a clock offset above 0.1 seconds for 10 minutes.'
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 16m, exp_alerts: []}
|
||||
- name: Custom selectors, thresholds and critical severity propagate to alerts
|
||||
interval: 1m
|
||||
input_series:
|
||||
@@ -239,7 +253,7 @@ tests:
|
||||
summary: Active Directory replication queue is high
|
||||
description: 'Pending replication operations on host:9182 have exceeded 10 for 15 minutes.'
|
||||
- alertname: WindowsADPendingReplication
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 16m
|
||||
@@ -253,5 +267,5 @@ tests:
|
||||
summary: Active Directory replication requests are failing
|
||||
description: 'Replication synchronization requests on host:9182 have been failing for 15 minutes.'
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 23m
|
||||
eval_time: 28m
|
||||
exp_alerts: []
|
||||
|
||||
@@ -4,7 +4,15 @@ rule_files:
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Memory cache includes all lists, commit is a ratio and paging is a rate
|
||||
|
||||
@@ -111,3 +111,44 @@ tests:
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_bytes:rate{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
value: 30
|
||||
|
||||
- name: Processor queue per core averages the queue and packet rates keep the nic label
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '12x5 0x5'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="3", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="nocpu:9182", cluster="a"}'
|
||||
values: '12x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+600x10'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+1200x10'
|
||||
promql_expr_test:
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 3
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 0
|
||||
- expr: windows:net_received_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_received_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 10
|
||||
- expr: windows:net_sent_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 20
|
||||
|
||||
Reference in New Issue
Block a user