diff --git a/.github/workflows/mixin.yml b/.github/workflows/mixin.yml index 67c4064a..bfc9ee56 100644 --- a/.github/workflows/mixin.yml +++ b/.github/workflows/mixin.yml @@ -6,7 +6,8 @@ on: pull_request: paths: - contrib/mixin/** - - dashboard/*.json + - contrib/dashboard/*.json + - contrib/monitoring/*.yaml - docs/collector.*.md - docs/runbooks/** - docs/upgrade/** @@ -21,7 +22,8 @@ on: - main paths: - contrib/mixin/** - - dashboard/*.json + - contrib/dashboard/*.json + - contrib/monitoring/*.yaml - docs/collector.*.md - docs/runbooks/** - docs/upgrade/** @@ -64,6 +66,9 @@ jobs: - name: Build dashboard and check generated changes run: mise run check-dashboards + - name: Generate rules and check generated changes + run: mise run check-rules + - name: Lint dashboard and test rules run: mise run test diff --git a/AGENTS.md b/AGENTS.md index d95bb268..88b6882c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,9 +27,10 @@ their runtime behavior. and examples. [`docs/collector-template.md`](docs/collector-template.md) is the documentation template. - [`contrib/mixin`](contrib/mixin): Jsonnet recording rules, alerts and dashboard - sources. [`dashboard`](dashboard) contains the generated Grafana dashboards; - [`docs/runbooks`](docs/runbooks) documents the alerts. -- [`installer`](installer), [`kubernetes`](kubernetes) and [`tools`](tools): MSI + sources. [`contrib/dashboard`](contrib/dashboard) contains the generated Grafana + dashboards, [`contrib/monitoring`](contrib/monitoring) the rendered Prometheus + rules and alerts; [`docs/runbooks`](docs/runbooks) documents the alerts. +- [`installer`](installer), [`docs/kubernetes`](docs/kubernetes) and [`tools`](tools): MSI packaging, deployment examples, CI fixtures and end-to-end helpers. The CI package job validates installer changes with [`tools/test-installer.ps1`](tools/test-installer.ps1), which installs the @@ -356,9 +357,9 @@ dashes, followed by its age in hex. ## Dashboards, rules and runbooks Read [`contrib/mixin/README.md`](contrib/mixin/README.md) and -[`dashboard/README.md`](dashboard/README.md) before editing monitoring artifacts. +[`contrib/dashboard/README.md`](contrib/dashboard/README.md) before editing monitoring artifacts. Edit the Jsonnet/Grafonnet source under `contrib/mixin/dashboards`, including its -tab modules and shared builders, and regenerate `dashboard/windows-exporter-*.json`: +tab modules and shared builders, and regenerate `contrib/dashboard/windows-exporter-*.json`: the full dashboard `windows-exporter-dashboard.json` and one dashboard per tab. Do not hand-edit generated JSON. Tab dashboards keep only the variables their panels reference, so reference variables as `$name` or `${name}`. diff --git a/README.md b/README.md index 5b25dcf7..48cdf6bc 100644 --- a/README.md +++ b/README.md @@ -204,7 +204,7 @@ The Docker image is tagged with the version of the exporter. The `latest` tag is ## Kubernetes Implementation -See detailed steps to install on Windows Kubernetes [here](./kubernetes/kubernetes.md). +See detailed steps to install on Windows Kubernetes [here](./docs/kubernetes/kubernetes.md). ## Prometheus rules diff --git a/dashboard/README.md b/contrib/dashboard/README.md similarity index 95% rename from dashboard/README.md rename to contrib/dashboard/README.md index 438e18a9..20b4e2a0 100644 --- a/dashboard/README.md +++ b/contrib/dashboard/README.md @@ -10,7 +10,7 @@ and takes several ideas from Grafana's [windows-observ-lib](https://github.com/g - Grafana 13 or later with a Prometheus data source. The file uses the v2 dashboard schema, which Grafana needs for tabs. - windows_exporter with the default collectors (`cpu`, `logical_disk`, `memory`, `net`, `os`, `physical_disk`, `service`, `system`). The SMB, Processes, Scheduled tasks, Updates, GPU, Hyper-V, Failover Cluster and Time tabs, the MSSQL and Hyper-V Cluster dashboards, the TCP and UDP rows of the Network tab and the Drives and Storage Spaces rows of the Disk tab need the `smb` or `smbclient`, `process`, `scheduled_task`, `update`, `gpu`, `hyperv`, `mscluster`, `time`, `mssql`, `tcp`, `udp`, `diskdrive` and `storage_spaces` collectors, which are not enabled by default. - The sample shows every tab; build the dashboard from the [mixin](../contrib/mixin#collectors) with your collectors to hide the others. + The sample shows every tab; build the dashboard from the [mixin](../mixin#collectors) with your collectors to hide the others. The process names in the GPU tab need the `process` collector. GPU temperature, fan, power and clock panels require WDDM 2.4 or newer and driver support; unsupported sensors are absent. @@ -23,7 +23,7 @@ Import the JSON files through **Dashboards > New > Import** and pick the Prometh | Windows Exporter / MSSQL | [windows-exporter-mssql.json](windows-exporter-mssql.json) | `windows-exporter-mssql` | | Windows Exporter / Hyper-V Cluster | [windows-exporter-hyper-v-cluster.json](windows-exporter-hyper-v-cluster.json) | `windows-exporter-hyper-v-cluster` | -The MSSQL dashboard is not a tab of the full dashboard. The [mixin](../contrib/mixin#metadata) adds it as a tab with `separateDashboardTabs: []`. +The MSSQL dashboard is not a tab of the full dashboard. The [mixin](../mixin#metadata) adds it as a tab with `separateDashboardTabs: []`. The Hyper-V Cluster dashboard has tabs of its own, see [Hyper-V Cluster dashboard](#hyper-v-cluster-dashboard). The tab dashboards show only the variables their panels use. @@ -102,11 +102,11 @@ appear. I/O graphs combine read, write and other operations per process. ### Changing the dashboard -Don't edit the JSON files by hand. They are generated from Jsonnet with [grafonnet](https://github.com/grafana/grafonnet) as part of the [windows_exporter mixin](../contrib/mixin): +Don't edit the JSON files by hand. They are generated from Jsonnet with [grafonnet](https://github.com/grafana/grafonnet) as part of the [windows_exporter mixin](../mixin): -- [dashboards/windows-exporter.libsonnet](../contrib/mixin/dashboards/windows-exporter.libsonnet) composes the dashboards from one file per tab. -- [dashboards/builders.libsonnet](../contrib/mixin/dashboards/builders.libsonnet) holds the panel, query, and layout builders. -- [dashboards/styles.libsonnet](../contrib/mixin/dashboards/styles.libsonnet) shares the visualization defaults. +- [dashboards/windows-exporter.libsonnet](../mixin/dashboards/windows-exporter.libsonnet) composes the dashboards from one file per tab. +- [dashboards/builders.libsonnet](../mixin/dashboards/builders.libsonnet) holds the panel, query, and layout builders. +- [dashboards/styles.libsonnet](../mixin/dashboards/styles.libsonnet) shares the visualization defaults. The generator writes the v2 schema directly, so it needs neither Grafana nor Prometheus. Install [mise](https://mise.jdx.dev/) and Go, then install the pinned tools and diff --git a/dashboard/dashboard-cpu.png b/contrib/dashboard/dashboard-cpu.png similarity index 100% rename from dashboard/dashboard-cpu.png rename to contrib/dashboard/dashboard-cpu.png diff --git a/dashboard/dashboard-exporter.png b/contrib/dashboard/dashboard-exporter.png similarity index 100% rename from dashboard/dashboard-exporter.png rename to contrib/dashboard/dashboard-exporter.png diff --git a/dashboard/dashboard-fleet.png b/contrib/dashboard/dashboard-fleet.png similarity index 100% rename from dashboard/dashboard-fleet.png rename to contrib/dashboard/dashboard-fleet.png diff --git a/dashboard/dashboard-overview.png b/contrib/dashboard/dashboard-overview.png similarity index 100% rename from dashboard/dashboard-overview.png rename to contrib/dashboard/dashboard-overview.png diff --git a/dashboard/windows-exporter-cpu.json b/contrib/dashboard/windows-exporter-cpu.json similarity index 100% rename from dashboard/windows-exporter-cpu.json rename to contrib/dashboard/windows-exporter-cpu.json diff --git a/dashboard/windows-exporter-dashboard.json b/contrib/dashboard/windows-exporter-dashboard.json similarity index 100% rename from dashboard/windows-exporter-dashboard.json rename to contrib/dashboard/windows-exporter-dashboard.json diff --git a/dashboard/windows-exporter-disk.json b/contrib/dashboard/windows-exporter-disk.json similarity index 100% rename from dashboard/windows-exporter-disk.json rename to contrib/dashboard/windows-exporter-disk.json diff --git a/dashboard/windows-exporter-exporter.json b/contrib/dashboard/windows-exporter-exporter.json similarity index 100% rename from dashboard/windows-exporter-exporter.json rename to contrib/dashboard/windows-exporter-exporter.json diff --git a/dashboard/windows-exporter-failover-cluster.json b/contrib/dashboard/windows-exporter-failover-cluster.json similarity index 100% rename from dashboard/windows-exporter-failover-cluster.json rename to contrib/dashboard/windows-exporter-failover-cluster.json diff --git a/dashboard/windows-exporter-fleet.json b/contrib/dashboard/windows-exporter-fleet.json similarity index 100% rename from dashboard/windows-exporter-fleet.json rename to contrib/dashboard/windows-exporter-fleet.json diff --git a/dashboard/windows-exporter-gpu.json b/contrib/dashboard/windows-exporter-gpu.json similarity index 100% rename from dashboard/windows-exporter-gpu.json rename to contrib/dashboard/windows-exporter-gpu.json diff --git a/dashboard/windows-exporter-hyper-v-cluster.json b/contrib/dashboard/windows-exporter-hyper-v-cluster.json similarity index 100% rename from dashboard/windows-exporter-hyper-v-cluster.json rename to contrib/dashboard/windows-exporter-hyper-v-cluster.json diff --git a/dashboard/windows-exporter-hyper-v.json b/contrib/dashboard/windows-exporter-hyper-v.json similarity index 100% rename from dashboard/windows-exporter-hyper-v.json rename to contrib/dashboard/windows-exporter-hyper-v.json diff --git a/dashboard/windows-exporter-memory.json b/contrib/dashboard/windows-exporter-memory.json similarity index 100% rename from dashboard/windows-exporter-memory.json rename to contrib/dashboard/windows-exporter-memory.json diff --git a/dashboard/windows-exporter-mssql.json b/contrib/dashboard/windows-exporter-mssql.json similarity index 100% rename from dashboard/windows-exporter-mssql.json rename to contrib/dashboard/windows-exporter-mssql.json diff --git a/dashboard/windows-exporter-network.json b/contrib/dashboard/windows-exporter-network.json similarity index 100% rename from dashboard/windows-exporter-network.json rename to contrib/dashboard/windows-exporter-network.json diff --git a/dashboard/windows-exporter-overview.json b/contrib/dashboard/windows-exporter-overview.json similarity index 100% rename from dashboard/windows-exporter-overview.json rename to contrib/dashboard/windows-exporter-overview.json diff --git a/dashboard/windows-exporter-processes.json b/contrib/dashboard/windows-exporter-processes.json similarity index 100% rename from dashboard/windows-exporter-processes.json rename to contrib/dashboard/windows-exporter-processes.json diff --git a/dashboard/windows-exporter-scheduled-tasks.json b/contrib/dashboard/windows-exporter-scheduled-tasks.json similarity index 100% rename from dashboard/windows-exporter-scheduled-tasks.json rename to contrib/dashboard/windows-exporter-scheduled-tasks.json diff --git a/dashboard/windows-exporter-services.json b/contrib/dashboard/windows-exporter-services.json similarity index 100% rename from dashboard/windows-exporter-services.json rename to contrib/dashboard/windows-exporter-services.json diff --git a/dashboard/windows-exporter-smb.json b/contrib/dashboard/windows-exporter-smb.json similarity index 100% rename from dashboard/windows-exporter-smb.json rename to contrib/dashboard/windows-exporter-smb.json diff --git a/dashboard/windows-exporter-time.json b/contrib/dashboard/windows-exporter-time.json similarity index 100% rename from dashboard/windows-exporter-time.json rename to contrib/dashboard/windows-exporter-time.json diff --git a/dashboard/windows-exporter-updates.json b/contrib/dashboard/windows-exporter-updates.json similarity index 100% rename from dashboard/windows-exporter-updates.json rename to contrib/dashboard/windows-exporter-updates.json diff --git a/contrib/mixin/.gitignore b/contrib/mixin/.gitignore index bc8811b7..92adcbc9 100644 --- a/contrib/mixin/.gitignore +++ b/contrib/mixin/.gitignore @@ -1,5 +1,3 @@ /vendor/ -/windows_alerts.yaml -/windows_rules.yaml /tests/*_alerts.yaml /tests/*_rules.yaml diff --git a/contrib/mixin/AGENTS.md b/contrib/mixin/AGENTS.md new file mode 100644 index 00000000..3a290e45 --- /dev/null +++ b/contrib/mixin/AGENTS.md @@ -0,0 +1,88 @@ +# Mixin instructions + +Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers +`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources. +[`README.md`](README.md) documents the configuration and the generated rules; +keep it in sync with every change. + +## Alerts + +`alerts/` holds one file per collector, each rendering one rule group, and +`alerts/alerts.libsonnet` imports them in the order of the generated file. +A rule is a plain Prometheus alerting rule, like in kubernetes-mixin: + +```jsonnet +{ + alert: 'WindowsCPUHighUsage', + expr: ||| + windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { summary: '...', description: '...' }, +} +``` + +Write the PromQL out in the `expr` block with `%(name)s` placeholders for the +`_config` values; selectors go inside the braces as +`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet` +provides `a.group(name, collectors, rules)`, which keeps a rule only when every +collector in `collectors` is enabled in `_config.collectors` and the alert is +not switched off in `_config.alerts`, and adds `keep_firing_for` from +`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]` +when a rule reads other collectors than its group (the CPU saturation alert +reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`, +`== 1`) and one-shot alerts, which must resolve at once and get no +`keep_firing_for`. A one-shot alert has no `for`. + +A new alert touches six places, and CI checks all of them: + +1. The rule in `alerts/.libsonnet`, with `{{ $labels. }}` + templates that use the 0.32 label keys (`service`, `volume`, `nic`, + `task`, `device_id`, never `name`). Alerts for a new collector go in a new + file, imported from `alerts/alerts.libsonnet`, with the collector added to + `_config.collectors` in `config.libsonnet`; the group expectations in + `tests/alerts.test.jsonnet` list every group. +2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet` + fails when the map and the generated alerts differ. Thresholds are + `_config` knobs named `Threshold[Unit]`. +3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for + opt-in collectors, which `tests/config.libsonnet` enables) that checks the + first firing evaluation, the labels and annotations, and the recovery. With + `keep_firing_for` a threshold alert resolves `keepFiringFor` after its + condition clears. `rate()` extrapolates while a window has few samples, so + ratios of two rates activate earlier than a single rate; probe the + activation time with `ALERTS_FOR_STATE` instead of guessing it. +4. A runbook `docs/runbooks/.md` with the `Meaning`, + `Impact`, `Diagnosis` and `Mitigation` sections. + `go test ./internal/metriccontract/` checks every `windows_*` metric name + a runbook mentions against the metrics the exporter emits; the names in + the collector docs are not always current, so check the collector source. +5. A row in the alerts table of `README.md`. +6. The committed `contrib/monitoring/windows_rules.yaml` and + `windows_alerts.yaml`, regenerated + with `mise run rules`; `mise run check-rules` fails in CI when they are + stale. + +Metrics with a state label (`windows_service_state{state="running"} == 0`) +keep that label in the alert; drop it with `max without (state) (...)` so the +alert does not say `state="running"` for a stopped service. Join a state metric +with its info metric with `ignoring ()`, never +`on (job, instance, )`: `on` drops the other target labels, so two +targets with the same `instance` in different clusters match each other. +A target can be scraped again after a gap (a reboot, a restart of the +exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset +10m)`); a plain `m offset 10m` has no sample while the gap covers that point. + +## Recording rules + +New recordings go in `rules/rules.libsonnet` with the shared expressions in +`lib/queries.libsonnet`, a row in the `README.md` table and a test in +`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero +capacities produce no sample instead of `NaN` or `+Inf`. + +## Checks + +`mise run test` runs everything CI runs. Without mise, the tools from +`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of +the `lint`, `check-runbooks` and `test` tasks do the same. diff --git a/contrib/mixin/README.md b/contrib/mixin/README.md index 82437a52..7fccdded 100644 --- a/contrib/mixin/README.md +++ b/contrib/mixin/README.md @@ -23,16 +23,23 @@ Mise installs release binaries with checksum verification and builds Jsonnet Bundler using Go. `mise run` makes the pinned tools available automatically. Only the dashboard depends on Jsonnet libraries, so `jb install` is not required -to generate rules. Generated rule files are ignored by Git: +to generate rules. The rules rendered from the default configuration are +committed in [`contrib/monitoring`](../monitoring), so they can be used +without Jsonnet: - `windows_rules.yaml`: recording rules. - `windows_alerts.yaml`: alerting rules. +`mise run rules` regenerates them after a source change, and CI fails when the +committed files are out of date (`mise run check-rules`). + The default selector is `job="windows_exporter"`. Change it to match your scrape -job before generating rules. Resource rules use the default `cpu`, `memory`, -`logical_disk`, and `net` collectors. Enable `ad` on domain controllers when -opting into Active Directory alerts. Clock alerts require the `time` collector -with its `ntp` subcollector and are intended for hosts using NTP synchronization. +job before generating rules. The rules read the default `cpu`, `memory`, +`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors` +lists which collectors the targets run, and the alerts of a collector that is +turned off are not generated. Enable `ad` on domain controllers for the Active +Directory alerts and `time` for the clock alerts, which need the `ntp` and +`clock_source` subcollectors. Copy both generated files into your Prometheus rule directory and reference them in `prometheus.yml`: @@ -60,8 +67,9 @@ modifying its source. For example, create `custom.libsonnet` in this directory: nicSelector: 'nic!="Loopback"', rateInterval: '5m', cpuHighUsageThreshold: 95, - enableActiveDirectory: true, - enableTime: true, + collectors+: { ad: true, time: true, process: true }, + alerts+: { WindowsHostRebooted: false }, + keepFiringFor: '10m', runbookURLPattern: 'https://runbooks.example.com/windows/%s/', }, } @@ -77,9 +85,14 @@ jsonnet -S -e 'std.manifestYamlDoc((import "custom.libsonnet").prometheusAlerts) Selectors are comma-separated Prometheus label matchers without braces. They apply to both recording rules and alerts. All target labels, including `job`, `instance`, `cluster`, and custom selector labels, are preserved. CPU usage -averages idle rates across cores and removes only `core` and `mode`. Disk and -network recordings also retain `volume` and `nic` respectively. The CPU mode -recording retains `mode` and removes only `core`. +averages idle rates across cores and removes only `core` and `mode`. Disk, +network and service rules also retain `volume`, `nic` and `service` +respectively, filtered by `volumeSelector`, `nicSelector` and +`serviceSelector`. The scheduled task and cluster shared volume alerts +filter their `task` and `shared_volume` labels with `taskSelector` +(`task!=""`) and `sharedVolumeSelector` +(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only +`core`. `rateInterval` is the fixed window for recording rules and Active Directory failure rates; keep it at least four times your scrape interval. Grafana's @@ -93,6 +106,7 @@ rules. | `windows:cpu_usage:ratio` | Non-idle CPU time averaged across cores | Ratio, 0–1 | | `windows:cpu_cores:count` | Logical processors observed in idle CPU metrics | Count | | `windows:cpu_time:ratio_rate` | CPU time averaged across cores, retaining `mode` | Seconds/second | +| `windows:cpu_queue_per_core:ratio` | Threads waiting for a processor, averaged over `rateInterval`, per logical processor | Threads/processor | | `windows:memory_usage:ratio` | Physical memory in use, based on available bytes | Ratio, 0–1 | | `windows:memory_used:bytes` | Total physical memory minus available memory | Bytes | | `windows:memory_cached:bytes` | Cache, modified pages, and three standby lists | Bytes | @@ -109,6 +123,8 @@ rules. | `windows:logical_disk_write_latency:seconds` | Write-time rate divided by write-operation rate | Seconds/operation | | `windows:net_received_bytes:rate` | Received network traffic per interface | Bytes/second | | `windows:net_sent_bytes:rate` | Sent network traffic per interface | Bytes/second | +| `windows:net_received_packets:rate` | Received packets per interface | Packets/second | +| `windows:net_sent_packets:rate` | Sent packets per interface | Packets/second | | `windows:net_received_utilization:ratio` | Received throughput divided by interface bandwidth | Ratio | | `windows:net_sent_utilization:ratio` | Sent throughput divided by interface bandwidth | Ratio | | `windows:net_received_errors:rate` | Inbound packet errors | Packets/second | @@ -127,46 +143,137 @@ which can exceed wall time for concurrent I/O. Latency is an average and is omitted when there are no operations. Memory, disk space, commit, and network utilization ratios are omitted when their capacity is zero or missing. Network bandwidth is already in bytes/second; receive and send utilization are separate -for full-duplex links. +for full-duplex links. The processor queue length is an instantaneous sample, +so the queue recording averages it over `rateInterval` before dividing by the +core count. The cache recording includes modified pages and is not a measure of immediately reclaimable memory. Paging operations can include file-backed pages as well as pagefile traffic. ## Alerts -Threshold configuration values below are percentages except where noted. +The alerts come in one rule group per collector, from one file per collector +in `alerts/` (`alerts/cpu.libsonnet`, `alerts/service.libsonnet`, and so on). +Each group lists the collectors its rules read; a group is generated only with +those collectors enabled in `_config.collectors`, and a rule that needs two +collectors, such as the CPU saturation alert, only with both. Threshold configuration values below are +percentages except where noted. -| Alert | Default condition | Duration | Severity | -| --- | --- | --- | --- | -| `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical | -| `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning | -| `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning | -| `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning | -| `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning | -| `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning | -| `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning | -| `WindowsNetworkErrors` | Combined receive/outbound errors above `networkErrorRateThreshold` (1 packet/second) | 15m | Warning | -| `WindowsNetworkDiscards` | Combined receive/outbound discards above `networkDiscardRateThreshold` (1 packet/second) | 15m | Warning | -| `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning | -| `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical | -| `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning | -| `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning | -| `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning | -| `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning | -| `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning | +| Group | Collectors | Alert | Default condition | Duration | Severity | +| --- | --- | --- | --- | --- | --- | +| `windows-exporter` | | `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical | +| `windows-exporter` | | `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning | +| `windows-exporter` | | `WindowsCollectorTimeout` | `windows_exporter_collector_timeout == 1` | 10m | Warning | +| `windows-exporter` | | `WindowsScrapeQueueTimeout` | `windows_exporter_scrape_queue_timeout == 1` | 10m | Warning | +| `windows-cpu` | `cpu` | `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning | +| `windows-cpu` | `cpu`, `system` | `WindowsCPUSaturation` | Waiting threads per logical processor above `cpuQueuePerCoreThreshold` (2) | 15m | Warning | +| `windows-memory` | `memory` | `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning | +| `windows-memory` | `memory` | `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning | +| `windows-memory` | `memory` | `WindowsMemoryPagingHigh` | Pages read or written per second above `memoryPagingThreshold` (1000) | 15m | Warning | +| `windows-disk` | `logical_disk` | `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning | +| `windows-disk` | `logical_disk` | `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning | +| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning | +| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical | +| `windows-disk` | `logical_disk` | `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning | +| `windows-network` | `net` | `WindowsNetworkErrors` | Receive and outbound errors above `networkErrorRatioThreshold` (1%) of all packets | 15m | Warning | +| `windows-network` | `net` | `WindowsNetworkDiscards` | Receive and outbound discards above `networkDiscardRatioThreshold` (1%) of all packets | 15m | Warning | +| `windows-network` | `net` | `WindowsNetworkUtilizationHigh` | Received or sent throughput above `networkUtilizationThreshold` (90%) of the link bandwidth | 15m | Warning | +| `windows-network` | `net` | `WindowsNetworkInterfaceFlapping` | Operational status changed more than twice within `rateInterval` | 2m | Warning | +| `windows-service` | `service` | `WindowsServiceNotRunning` | Automatic service without triggers not running | 15m | Warning | +| `windows-service` | `service` | `WindowsServiceCrashLooping` | More than `serviceRestartThreshold` (2) restarts in 15 minutes | 0m | Warning | +| `windows-system` | `system` | `WindowsHostRebooted` | Boot time less than ten minutes old and changed | 0m | Info | +| `windows-active-directory` | `ad` | `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning | +| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning | +| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning | +| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning | +| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning | +| `windows-pagefile` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning | +| `windows-diskdrive` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical | +| `windows-license` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning | +| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning | +| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info | +| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning | +| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical | +| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning | +| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning | +| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning | +| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning | +| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical | +| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning | -Active Directory alerts are generated only when `enableActiveDirectory` is -`true`. `criticalSeverity` defaults to `critical` and can be overridden to match -your routing labels. Below the critical disk threshold, both disk severity -levels fire; configure Alertmanager inhibition if only the critical +`_config.alerts` lists every alert by name. Set one to `false` to leave it out, +for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the +build, and a name missing from a replaced object stays enabled. A group whose +alerts are all turned off is not generated. + +Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a +value that oscillates around a threshold does not flap. The state alerts +`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`, +`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`, +`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors +(clock source, drive status, license, task result, pending updates, Hyper-V, +cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`, +have no `keep_firing_for` and resolve as soon as their condition clears. +`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for +older servers. + +The network error and discard thresholds are percentages of all packets on the +interface. + +`criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override +them to match your routing labels. Below the critical disk threshold, both disk +severity levels fire; configure Alertmanager inhibition if only the critical notification should be sent. -Clock alerts are generated only when `enableTime` is `true`. A zero NTP source -count may be expected on hosts using another time provider. The offset compares -the host to its chosen source and does not independently verify that source. -Disk latency and network error/discard thresholds should be tuned for the +Active Directory alerts need `collectors.ad`, clock alerts `collectors.time`; +both are off by default. A zero NTP +source count may be expected on hosts using another time provider; +`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and +fires only for the local clock or no synchronization. The offset compares the +host to its chosen source and does not independently verify that source. + +The other opt-in alerts follow `_config.collectors`: enable `pagefile`, +`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`, +`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are +generated, each collector in a group of its own. +`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code +without a name, including the exit codes of the task's program, as a failure; +narrow the tasks with `taskSelector`. Disabling a task clears the alert. +`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and +virtual drives report permanently. `WindowsLicenseNotGenuine` reports the +`state` the host has; hosts without internet access can stay in `offline`, +so turn the alert off there. Every failover cluster node reports the state of +the whole cluster, and every Storage Spaces Direct node reports the clustered +pools and virtual disks, so the cluster and storage alerts fire once per node; +group them by their `node`, `resource`, `shared_volume`, `pool` or +`virtual_disk` labels in Alertmanager. The storage pool alert joins +`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP +scope alert only counts scopes in an enabled state, and a scope without +addresses produces no sample. `WindowsUpdatesPending` is an info alert that +waits `updatePendingDuration` for the normal maintenance window. Prometheus +restores the waiting time after a restart shorter than its +`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage +or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the storage tier, traffic volume, and application requirements. +`WindowsServiceNotRunning` watches services with the `auto` start mode and +skips trigger-start services, which stop by design, through the +`trigger_start` label of `windows_service_info`. Delayed-start services are +included; the 15 minute duration covers their later start after a reboot. +`WindowsServiceCrashLooping` counts the `windows_service_process` series of a +service within 15 minutes, one per process the service ran as. Exclude services +with `serviceSelector`, for example automatic services that exit by design +after their work, such as the software protection or updater services of some +products. `WindowsHostRebooted` fires once for about ten minutes after a +reboot and resolves by itself. + Windows disk size/free-space counters may lag by 10–15 minutes. Predictions depend on a sustained trend and are intentionally delayed for an hour. A target removed from service discovery has no `up` series and will not trigger @@ -204,10 +311,10 @@ Fork users should override `runbookURLPattern` to match their published URL. `grafanaDashboards` contains `windows-exporter.json`, the full sample dashboard with all tabs in -[dashboard/windows-exporter-dashboard.json](../../dashboard/windows-exporter-dashboard.json), +[contrib/dashboard/windows-exporter-dashboard.json](../dashboard/windows-exporter-dashboard.json), and a shorter dashboard per tab, such as `windows-exporter-cpu.json`, in -[dashboard](../../dashboard). They are Grafana v2 dashboards and need Grafana 13 -or later. See the [dashboard documentation](../../dashboard/README.md) for their +[contrib/dashboard](../dashboard). They are Grafana v2 dashboards and need Grafana 13 +or later. See the [dashboard documentation](../dashboard/README.md) for their content. The dashboard uses additive builders with [grafonnet](https://github.com/grafana/grafonnet), @@ -254,7 +361,7 @@ Regenerate the committed files after changing the dashboard source: mise run dashboard ``` -This runs `jb install` and writes `../../dashboard/windows-exporter-*.json`. +This runs `jb install` and writes `../dashboard/windows-exporter-*.json`. `mise run dashboards` is an alias. `mise run generate` builds both the rules and the dashboard, while `mise run rules` builds only the rules. The same commands work inside WSL; mise supplies the tool binaries on `PATH`. @@ -322,7 +429,7 @@ tab to the full dashboard: names, such as `cpu`, `physical_disk`, `diskdrive`, `smb` or `update`. The dashboard renders only the tabs, rows, panels and variables of enabled collectors, so a tab for a collector you do not run does not appear. The -defaults match the exporter's default collectors; `time` follows `enableTime`. +defaults match the exporter's default collectors. The `os` collector is required, because the variables select hosts through `windows_os_hostname`. An unknown collector name fails the build. Enable optional collectors on top of the defaults with `collectors+:`; replacing the @@ -366,14 +473,16 @@ mise run test This checks Jsonnet formatting, runs `jsonnet-lint` on the Jsonnet sources to find unused variables and references to missing fields, checks that the -committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with +committed dashboards and rule files are up to date, runs `dashboard-linter lint --strict`, validates generated rules with `promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic time series. -Tests cover all 25 recordings, alert timing and recovery, threshold -boundaries, counter resets, zero denominators, disk forecasts, target-label -preservation, and custom selectors/thresholds with Active Directory and clock -alerts enabled. Runbook checks include opt-in alerts and verify that their pages -and URLs exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as +Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`, +threshold boundaries, counter resets, zero denominators, disk forecasts, +target-label preservation, and custom selectors/thresholds with Active +Directory and clock alerts enabled. `tests/alerts.test.jsonnet` checks the +collector gating, the per-alert switches and the group layout. Runbook checks +render the alerts of every collector and verify that their pages and URLs +exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as `mise install`. To list the available tasks: @@ -405,12 +514,12 @@ prometheus "local" { } ``` -Then run `pint --config /path/to/local.pint.hcl lint windows_rules.yaml windows_alerts.yaml` +Then run `pint --config /path/to/local.pint.hcl lint ../monitoring/windows_rules.yaml ../monitoring/windows_alerts.yaml` without `--offline`. See the [pint documentation](https://cloudflare.github.io/pint/) for available checks. -Run `mise run fmt` after source changes and `mise run clean` to remove generated rule -files. For downstream composition with Jsonnet Bundler, import +Run `mise run fmt` and `mise run rules` after source changes, and `mise run clean` +to remove the rule files generated for the tests. For downstream composition with Jsonnet Bundler, import `github.com/prometheus-community/windows_exporter/contrib/mixin/mixin.libsonnet` and extend `_config`. See the [monitoring mixin documentation](https://github.com/monitoring-mixins/docs) for the standard composition workflow. diff --git a/contrib/mixin/alerts.jsonnet b/contrib/mixin/alerts.jsonnet index 75e7c1b2..b4434cf8 100644 --- a/contrib/mixin/alerts.jsonnet +++ b/contrib/mixin/alerts.jsonnet @@ -1 +1 @@ -std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts) +std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts, quote_keys=false) diff --git a/contrib/mixin/alerts/ad.libsonnet b/contrib/mixin/alerts/ad.libsonnet new file mode 100644 index 00000000..82bfce6c --- /dev/null +++ b/contrib/mixin/alerts/ad.libsonnet @@ -0,0 +1,32 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-active-directory', collectors=['ad'], rules=[ + { + alert: 'WindowsADPendingReplication', + expr: ||| + windows_ad_replication_pending_operations{%(windowsExporterSelector)s} > %(adPendingReplicationThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Active Directory replication queue is high', + description: 'Pending replication operations on {{ $labels.instance }} have exceeded %(adPendingReplicationThreshold)s for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsADReplicationFailures', + expr: ||| + (rate(windows_ad_replication_sync_requests_total{%(windowsExporterSelector)s}[%(rateInterval)s]) - rate(windows_ad_replication_sync_requests_success_total{%(windowsExporterSelector)s}[%(rateInterval)s])) > %(adReplicationFailureRateThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Active Directory replication requests are failing', + description: 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.', + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/alerts.libsonnet b/contrib/mixin/alerts/alerts.libsonnet index 0da169d2..0195ba78 100644 --- a/contrib/mixin/alerts/alerts.libsonnet +++ b/contrib/mixin/alerts/alerts.libsonnet @@ -1,158 +1,28 @@ +// One file per collector, each adding one rule group. To add alerts for a +// collector, copy one of these files, name it after the collector and import +// it here; the group order below is the order in the generated file. { local c = self._config, - local q = (import '../lib/queries.libsonnet')(c), - local alert(name, expr, duration, severity, summary, description) = { - alert: name, - expr: expr, - 'for': duration, - labels: { severity: severity }, - annotations: { summary: summary, description: description }, - }, - prometheusAlerts+:: { - groups: [ - { - name: 'windows-exporter-alerts', - rules: [ - alert( - 'WindowsExporterDown', - q.metric('up') + ' == 0', - '5m', - c.criticalSeverity, - 'Windows exporter is unreachable', - 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).' - ), - alert( - 'WindowsCollectorFailed', - q.metric('windows_exporter_collector_success') + ' == 0', - '5m', - 'warning', - 'Windows collector failed', - 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.' - ), - alert( - 'WindowsCPUHighUsage', - q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold, - '15m', - 'warning', - 'Windows CPU usage is high', - 'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold - ), - alert( - 'WindowsMemoryHighUsage', - q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold, - '15m', - 'warning', - 'Windows memory usage is high', - 'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold - ), - alert( - 'WindowsMemoryCommitHighUsage', - q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold, - '15m', - 'warning', - 'Windows memory commit usage is high', - 'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold - ), - ] + [ - alert( - 'WindowsDisk%sLatencyHigh' % direction[0], - q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds, - '15m', - 'warning', - 'Windows disk %s latency is high' % direction[1], - 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]] - ) - for direction in [['Read', 'read'], ['Write', 'write']] - ] + [ - alert( - 'WindowsNetwork%s' % outcome[0], - '(%s + %s) > %g' % [ - q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]), - q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]), - outcome[2], - ], - '15m', - 'warning', - 'Windows network packet %s are high' % outcome[3], - 'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g per second for 15 minutes.' % [outcome[3], outcome[2]] - ) - for outcome in [ - ['Errors', 'errors', c.networkErrorRateThreshold, 'errors'], - ['Discards', 'discarded', c.networkDiscardRateThreshold, 'discards'], - ] - ] + [ - alert( - 'WindowsDiskAlmostFull', - q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1], - '15m', - entry[0], - 'Windows disk space is low', - 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1] - ) - for entry in [ - ['warning', c.diskFreeWarningThreshold], - [c.criticalSeverity, c.diskFreeCriticalThreshold], - ] - ] + [ - alert( - 'WindowsDiskFillingUp', - q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [ - c.diskFreeWarningThreshold, - q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]), - c.diskPredictionWindow, - c.diskPredictionHours * 3600, - ], - '1h', - 'warning', - 'Windows disk is predicted to fill up', - 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours - ), - ], - }, - ] + (if c.enableActiveDirectory then [ - { - name: 'windows-active-directory', - rules: [ - alert( - 'WindowsADPendingReplication', - q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold, - '15m', - 'warning', - 'Active Directory replication queue is high', - 'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold - ), - alert( - 'WindowsADReplicationFailures', - '(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold, - '15m', - 'warning', - 'Active Directory replication requests are failing', - 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.' - ), - ], - }, - ] else []) + if c.enableTime then [ - { - name: 'windows-time', - rules: [ - alert( - 'WindowsClockNotSynchronising', - q.metric('windows_time_ntp_client_time_sources') + ' == 0', - '10m', - 'warning', - 'Windows clock has no NTP time source', - 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.' - ), - alert( - 'WindowsClockSkewDetected', - 'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds], - '10m', - 'warning', - 'Windows clock offset is high', - 'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds - ), - ], - }, - ] else [], - }, + local known = std.objectFields((import '../config.libsonnet')._config.alerts), + local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)), + assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown), } ++ (import 'exporter.libsonnet') ++ (import 'cpu.libsonnet') ++ (import 'memory.libsonnet') ++ (import 'pagefile.libsonnet') ++ (import 'logical_disk.libsonnet') ++ (import 'diskdrive.libsonnet') ++ (import 'net.libsonnet') ++ (import 'service.libsonnet') ++ (import 'system.libsonnet') ++ (import 'license.libsonnet') ++ (import 'ad.libsonnet') ++ (import 'time.libsonnet') ++ (import 'scheduled_task.libsonnet') ++ (import 'update.libsonnet') ++ (import 'hyperv.libsonnet') ++ (import 'mscluster.libsonnet') ++ (import 'storage_spaces.libsonnet') ++ (import 'dhcp.libsonnet') ++ (import 'dns.libsonnet') diff --git a/contrib/mixin/alerts/cpu.libsonnet b/contrib/mixin/alerts/cpu.libsonnet new file mode 100644 index 00000000..35c5887e --- /dev/null +++ b/contrib/mixin/alerts/cpu.libsonnet @@ -0,0 +1,33 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-cpu', collectors=['cpu'], rules=[ + { + alert: 'WindowsCPUHighUsage', + expr: ||| + windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows CPU usage is high', + description: 'CPU usage on {{ $labels.instance }} has exceeded %(cpuHighUsageThreshold)s%% for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsCPUSaturation', + expr: ||| + windows:cpu_queue_per_core:ratio{%(windowsExporterSelector)s} > %(cpuQueuePerCoreThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows processor queue is long', + description: 'More than %(cpuQueuePerCoreThreshold)s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c, + }, + collectors:: ['cpu', 'system'], + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/dhcp.libsonnet b/contrib/mixin/alerts/dhcp.libsonnet new file mode 100644 index 00000000..508534a0 --- /dev/null +++ b/contrib/mixin/alerts/dhcp.libsonnet @@ -0,0 +1,36 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-dhcp', collectors=['dhcp'], rules=[ + // Percent of free addresses in a scope; a scope without addresses has + // no sample, and only scopes in an enabled state count. + { + alert: 'WindowsDHCPScopeAlmostExhausted', + expr: ||| + ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeWarningThreshold)s + and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'DHCP scope is almost exhausted', + description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeWarningThreshold)s%% free addresses for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDHCPScopeAlmostExhausted', + expr: ||| + ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeCriticalThreshold)s + and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'DHCP scope is almost exhausted', + description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeCriticalThreshold)s%% free addresses for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/diskdrive.libsonnet b/contrib/mixin/alerts/diskdrive.libsonnet new file mode 100644 index 00000000..89f21179 --- /dev/null +++ b/contrib/mixin/alerts/diskdrive.libsonnet @@ -0,0 +1,24 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-diskdrive', collectors=['diskdrive'], rules=[ + { + alert: 'WindowsDiskDriveNotOK', + // One series per status with the current status set to 1, so the + // alert carries the status the drive reports. Some USB and virtual + // drives report Unknown for good. + expr: ||| + windows_diskdrive_status{%(windowsExporterSelector)s, status!~"OK|Unknown"} == 1 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows disk drive is not healthy', + description: 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/dns.libsonnet b/contrib/mixin/alerts/dns.libsonnet new file mode 100644 index 00000000..4c9abdf2 --- /dev/null +++ b/contrib/mixin/alerts/dns.libsonnet @@ -0,0 +1,20 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-dns', collectors=['dns'], rules=[ + { + alert: 'WindowsDNSZoneTransferFailures', + expr: ||| + rate(windows_dns_zone_transfer_failures_total{%(windowsExporterSelector)s}[%(rateInterval)s]) > %(dnsZoneTransferFailureRateThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'DNS zone transfers are failing', + description: 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.', + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/exporter.libsonnet b/contrib/mixin/alerts/exporter.libsonnet new file mode 100644 index 00000000..61924c74 --- /dev/null +++ b/contrib/mixin/alerts/exporter.libsonnet @@ -0,0 +1,60 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-exporter', collectors=[], rules=[ + { + alert: 'WindowsExporterDown', + expr: ||| + up{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows exporter is unreachable', + description: 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).', + }, + state:: true, + }, + { + alert: 'WindowsCollectorFailed', + expr: ||| + windows_exporter_collector_success{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '5m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows collector failed', + description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsCollectorTimeout', + expr: ||| + windows_exporter_collector_timeout{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows collector exceeds the scrape timeout', + description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsScrapeQueueTimeout', + expr: ||| + windows_exporter_scrape_queue_timeout{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows exporter scrapes time out in the queue', + description: 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/hyperv.libsonnet b/contrib/mixin/alerts/hyperv.libsonnet new file mode 100644 index 00000000..81559c00 --- /dev/null +++ b/contrib/mixin/alerts/hyperv.libsonnet @@ -0,0 +1,47 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-hyperv', collectors=['hyperv'], rules=[ + { + alert: 'WindowsHyperVWMIUnhealthy', + expr: ||| + windows_hyperv_wmi_health{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Hyper-V WMI namespace is not responding', + description: 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.', + }, + state:: true, + }, + { + alert: 'WindowsHyperVReplicationCritical', + expr: ||| + windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 3 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Hyper-V replication is critical', + description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsHyperVReplicationWarning', + expr: ||| + windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Hyper-V replication has a warning', + description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/license.libsonnet b/contrib/mixin/alerts/license.libsonnet new file mode 100644 index 00000000..6874dca6 --- /dev/null +++ b/contrib/mixin/alerts/license.libsonnet @@ -0,0 +1,21 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-license', collectors=['license'], rules=[ + { + alert: 'WindowsLicenseNotGenuine', + expr: ||| + windows_license_status{%(windowsExporterSelector)s, state!="genuine"} == 1 + ||| % c, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows license is not genuine', + description: 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/logical_disk.libsonnet b/contrib/mixin/alerts/logical_disk.libsonnet new file mode 100644 index 00000000..394108ee --- /dev/null +++ b/contrib/mixin/alerts/logical_disk.libsonnet @@ -0,0 +1,69 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-disk', collectors=['logical_disk'], rules=[ + { + alert: 'WindowsDiskReadLatencyHigh', + expr: ||| + windows:logical_disk_read_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk read latency is high', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per read operation for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskWriteLatencyHigh', + expr: ||| + windows:logical_disk_write_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk write latency is high', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per write operation for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskAlmostFull', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk space is low', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeWarningThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskAlmostFull', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeCriticalThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows disk space is low', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeCriticalThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskFillingUp', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s + and predict_linear(windows_logical_disk_free_bytes{%(windowsExporterSelector)s, %(volumeSelector)s}[%(diskPredictionWindow)s], %(diskPredictionSeconds)s) < 0 + ||| % c { diskPredictionSeconds: c.diskPredictionHours * 3600 }, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk is predicted to fill up', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %(diskPredictionHours)s hours.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/memory.libsonnet b/contrib/mixin/alerts/memory.libsonnet new file mode 100644 index 00000000..80512211 --- /dev/null +++ b/contrib/mixin/alerts/memory.libsonnet @@ -0,0 +1,44 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-memory', collectors=['memory'], rules=[ + { + alert: 'WindowsMemoryHighUsage', + expr: ||| + windows:memory_usage:ratio{%(windowsExporterSelector)s} * 100 > %(memoryHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory usage is high', + description: 'Physical memory usage on {{ $labels.instance }} has exceeded %(memoryHighUsageThreshold)s%% for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsMemoryCommitHighUsage', + expr: ||| + windows:memory_committed:ratio{%(windowsExporterSelector)s} * 100 > %(memoryCommitHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory commit usage is high', + description: 'Committed memory on {{ $labels.instance }} has exceeded %(memoryCommitHighUsageThreshold)s%% of the commit limit for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsMemoryPagingHigh', + expr: ||| + windows:memory_swap_page_operations:rate{%(windowsExporterSelector)s} > %(memoryPagingThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory paging is high', + description: 'Paging on {{ $labels.instance }} has exceeded %(memoryPagingThreshold)s pages per second for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/mscluster.libsonnet b/contrib/mixin/alerts/mscluster.libsonnet new file mode 100644 index 00000000..64b72f8c --- /dev/null +++ b/contrib/mixin/alerts/mscluster.libsonnet @@ -0,0 +1,73 @@ +// Every cluster node reports the state of the whole cluster, so these +// alerts fire once per node; group them by cluster labels in Alertmanager. +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-cluster', collectors=['mscluster'], rules=[ + { + alert: 'WindowsClusterNodeDown', + expr: ||| + windows_mscluster_node_state{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster node is down', + description: 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterResourceFailed', + expr: ||| + windows_mscluster_resource_state{%(windowsExporterSelector)s} == 4 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster resource has failed', + description: 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterResourceGroupFailed', + expr: ||| + windows_mscluster_resourcegroup_state{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster resource group has failed', + description: 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterSharedVolumeAlmostFull', + expr: ||| + (windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeWarningThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Cluster shared volume is almost full', + description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeWarningThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c, + }, + }, + { + alert: 'WindowsClusterSharedVolumeAlmostFull', + expr: ||| + (windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeCriticalThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Cluster shared volume is almost full', + description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeCriticalThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/net.libsonnet b/contrib/mixin/alerts/net.libsonnet new file mode 100644 index 00000000..798fe27f --- /dev/null +++ b/contrib/mixin/alerts/net.libsonnet @@ -0,0 +1,63 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-network', collectors=['net'], rules=[ + { + alert: 'WindowsNetworkErrors', + expr: ||| + ( + (windows:net_received_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s}) + / ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0) + ) * 100 > %(networkErrorRatioThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network packet errors are high', + description: 'Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkErrorRatioThreshold)g%% of all packets for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkDiscards', + expr: ||| + ( + (windows:net_received_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s}) + / ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0) + ) * 100 > %(networkDiscardRatioThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network packet discards are high', + description: 'Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkDiscardRatioThreshold)g%% of all packets for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkUtilizationHigh', + expr: ||| + (windows:net_received_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s) + or (windows:net_sent_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network interface utilization is high', + description: 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %(networkUtilizationThreshold)s%% of its bandwidth in at least one direction for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkInterfaceFlapping', + expr: ||| + max without (status) (changes(windows_net_nic_operation_status{%(windowsExporterSelector)s, %(nicSelector)s, status="up"}[%(rateInterval)s])) > 2 + ||| % c, + 'for': '2m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network interface is flapping', + description: 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %(rateInterval)s.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/pagefile.libsonnet b/contrib/mixin/alerts/pagefile.libsonnet new file mode 100644 index 00000000..b751a165 --- /dev/null +++ b/contrib/mixin/alerts/pagefile.libsonnet @@ -0,0 +1,20 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-pagefile', collectors=['pagefile'], rules=[ + { + alert: 'WindowsPagefileAlmostFull', + expr: ||| + (1 - (windows_pagefile_free_bytes{%(windowsExporterSelector)s} / (windows_pagefile_limit_bytes{%(windowsExporterSelector)s} > 0))) * 100 > %(pagefileUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows page file is almost full', + description: 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %(pagefileUsageThreshold)s%% in use for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/scheduled_task.libsonnet b/contrib/mixin/alerts/scheduled_task.libsonnet new file mode 100644 index 00000000..9699c941 --- /dev/null +++ b/contrib/mixin/alerts/scheduled_task.libsonnet @@ -0,0 +1,26 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-scheduled-task', collectors=['scheduled_task'], rules=[ + { + alert: 'WindowsScheduledTaskFailed', + // unknown is any result code without a named status, including the + // exit codes of the task's program. + // A disabled task keeps its last result, so disabling a broken task + // clears the alert. + expr: ||| + windows_scheduled_task_last_result_status{%(windowsExporterSelector)s, %(taskSelector)s, status=~"unknown|terminated"} == 1 + unless ignoring (status, state) (windows_scheduled_task_state{%(windowsExporterSelector)s, %(taskSelector)s, state="disabled"} == 1) + ||| % c, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows scheduled task did not succeed', + description: 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/service.libsonnet b/contrib/mixin/alerts/service.libsonnet new file mode 100644 index 00000000..49d99bbc --- /dev/null +++ b/contrib/mixin/alerts/service.libsonnet @@ -0,0 +1,38 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-service', collectors=['service'], rules=[ + { + alert: 'WindowsServiceNotRunning', + // The matches ignore only the labels of the right-hand metric, so + // every target label, including ones a selector adds, takes part. + expr: ||| + max without (state) (windows_service_state{%(windowsExporterSelector)s, %(serviceSelector)s, state="running"} == 0) + and ignoring (start_mode) (windows_service_start_mode{%(windowsExporterSelector)s, %(serviceSelector)s, start_mode="auto"} == 1) + unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{%(windowsExporterSelector)s, %(serviceSelector)s, trigger_start="true"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows automatic service is not running', + description: 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsServiceCrashLooping', + // Every service process has its own series, so the series seen in the + // window count the processes the service ran as. + expr: ||| + count without (process_id) (count_over_time(windows_service_process{%(windowsExporterSelector)s, %(serviceSelector)s}[15m])) - 1 > %(serviceRestartThreshold)s + ||| % c, + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows service is restarting repeatedly', + description: 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %(serviceRestartThreshold)s times in 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/storage_spaces.libsonnet b/contrib/mixin/alerts/storage_spaces.libsonnet new file mode 100644 index 00000000..b3d9dc01 --- /dev/null +++ b/contrib/mixin/alerts/storage_spaces.libsonnet @@ -0,0 +1,66 @@ +// On Storage Spaces Direct every node reports the clustered pools and +// virtual disks, so these alerts fire once per node as well. +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-storage-spaces', collectors=['storage_spaces'], rules=[ + { + alert: 'WindowsStoragePoolUnhealthy', + // The primordial pool holds the disks that are not in a concrete pool + // yet; only the info metric tells it apart. + expr: ||| + (windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 2) + and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"} + ||| % c, + 'for': '10m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Storage pool is unhealthy', + description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStoragePoolUnhealthy', + expr: ||| + (windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 1) + and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"} + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Storage pool has a health warning', + description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStorageVirtualDiskUnhealthy', + expr: ||| + windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '10m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Storage Spaces virtual disk is unhealthy', + description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStorageVirtualDiskUnhealthy', + expr: ||| + windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Storage Spaces virtual disk has a health warning', + description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/system.libsonnet b/contrib/mixin/alerts/system.libsonnet new file mode 100644 index 00000000..5c8843c1 --- /dev/null +++ b/contrib/mixin/alerts/system.libsonnet @@ -0,0 +1,25 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-system', collectors=['system'], rules=[ + { + alert: 'WindowsHostRebooted', + // Fires once per reboot: for the first ten minutes after the boot, + // and only on a host that was monitored before it. The exporter is + // down while the host reboots, so the second operand takes the last + // boot time of the hour before, not the sample of ten minutes ago. + expr: ||| + (time() - windows_system_boot_time_timestamp{%(windowsExporterSelector)s}) < 600 + and (time() - last_over_time(windows_system_boot_time_timestamp{%(windowsExporterSelector)s}[1h] offset 10m)) > 600 + ||| % c, + labels: { severity: c.infoSeverity }, + annotations: { + summary: 'Windows host has rebooted', + description: '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/time.libsonnet b/contrib/mixin/alerts/time.libsonnet new file mode 100644 index 00000000..8955f8b6 --- /dev/null +++ b/contrib/mixin/alerts/time.libsonnet @@ -0,0 +1,47 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-time', collectors=['time'], rules=[ + { + alert: 'WindowsClockNotSynchronising', + expr: ||| + windows_time_ntp_client_time_sources{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock has no NTP time source', + description: 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsClockSkewDetected', + expr: ||| + abs(windows_time_computed_time_offset_seconds{%(windowsExporterSelector)s}) > %(clockOffsetThresholdSeconds)g + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock offset is high', + description: 'Windows Time on {{ $labels.instance }} reports a clock offset above %(clockOffsetThresholdSeconds)g seconds for 10 minutes.' % c, + }, + }, + { + alert: 'WindowsClockSourceLocal', + // Covers domain members (NT5DS) too, which the NTP source count misses. + expr: ||| + windows_time_clock_sync_source{%(windowsExporterSelector)s, type=~"Local CMOS Clock|NoSync"} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock has no external time source', + description: 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/update.libsonnet b/contrib/mixin/alerts/update.libsonnet new file mode 100644 index 00000000..7eff2968 --- /dev/null +++ b/contrib/mixin/alerts/update.libsonnet @@ -0,0 +1,21 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-update', collectors=['update'], rules=[ + { + alert: 'WindowsUpdatesPending', + expr: ||| + count without (id, revision, category, severity, title) (windows_update_pending_info{%(windowsExporterSelector)s, deployment_action="installation"}) > 0 + ||| % c, + 'for': c.updatePendingDuration, + labels: { severity: c.infoSeverity }, + annotations: { + summary: 'Windows updates are waiting for installation', + description: '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %(updatePendingDuration)s.' % c, + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/config.libsonnet b/contrib/mixin/config.libsonnet index 079c49ba..248a1e7f 100644 --- a/contrib/mixin/config.libsonnet +++ b/contrib/mixin/config.libsonnet @@ -4,13 +4,12 @@ windowsExporterSelector: 'job="windows_exporter"', volumeSelector: 'volume!=""', nicSelector: 'nic!=""', + serviceSelector: 'service!=""', + taskSelector: 'task!=""', + // ClusterPerformanceHistory is the volume of the cluster's own health history. + sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"', rateInterval: '5m', - // Opt in only for targets with the ad collector enabled. - enableActiveDirectory: false, - // Requires the time collector with its ntp subcollector enabled. - enableTime: false, - // Grafana dashboard metadata. A UID is the v2 metadata.name; change the // UIDs to import the dashboards next to the sample ones. // Every tab also renders as a dashboard of its own with the UID @@ -37,18 +36,25 @@ // Collectors enabled on the windows_exporter targets, by collector name. // The dashboard only renders the tabs, rows and panels of enabled - // collectors. The defaults match the exporter's default collectors. + // collectors, and the alerts only the rules whose collectors are enabled. + // The defaults match the exporter's default collectors. collectors: { + // Only for targets with the ad collector enabled, such as domain controllers. + ad: false, cpu: true, + dhcp: false, diskdrive: false, + dns: false, gpu: false, hyperv: false, + license: false, logical_disk: true, memory: true, mscluster: false, mssql: false, net: true, os: true, + pagefile: false, physical_disk: true, process: false, scheduled_task: false, @@ -58,28 +64,101 @@ storage_spaces: false, system: true, tcp: false, - time: $._config.enableTime, + // Needs the time collector with its ntp and clock_source subcollectors. + time: false, udp: false, update: false, }, + // Alerts by name. Set one to false to leave it out of the generated + // rules, for example alerts+: { WindowsHostRebooted: false }. Names that + // are not listed here fail the build; names left out of a replaced object + // stay enabled. + alerts: { + WindowsExporterDown: true, + WindowsCollectorFailed: true, + WindowsCollectorTimeout: true, + WindowsScrapeQueueTimeout: true, + WindowsCPUHighUsage: true, + WindowsCPUSaturation: true, + WindowsMemoryHighUsage: true, + WindowsMemoryCommitHighUsage: true, + WindowsMemoryPagingHigh: true, + WindowsDiskReadLatencyHigh: true, + WindowsDiskWriteLatencyHigh: true, + WindowsDiskAlmostFull: true, + WindowsDiskFillingUp: true, + WindowsNetworkErrors: true, + WindowsNetworkDiscards: true, + WindowsNetworkUtilizationHigh: true, + WindowsNetworkInterfaceFlapping: true, + WindowsServiceNotRunning: true, + WindowsServiceCrashLooping: true, + WindowsHostRebooted: true, + WindowsADPendingReplication: true, + WindowsADReplicationFailures: true, + WindowsClockNotSynchronising: true, + WindowsClockSkewDetected: true, + WindowsClockSourceLocal: true, + WindowsDiskDriveNotOK: true, + WindowsPagefileAlmostFull: true, + WindowsLicenseNotGenuine: true, + WindowsScheduledTaskFailed: true, + WindowsUpdatesPending: true, + WindowsHyperVWMIUnhealthy: true, + WindowsHyperVReplicationCritical: true, + WindowsHyperVReplicationWarning: true, + WindowsClusterNodeDown: true, + WindowsClusterResourceFailed: true, + WindowsClusterResourceGroupFailed: true, + WindowsClusterSharedVolumeAlmostFull: true, + WindowsStoragePoolUnhealthy: true, + WindowsStorageVirtualDiskUnhealthy: true, + WindowsDHCPScopeAlmostExhausted: true, + WindowsDNSZoneTransferFailures: true, + }, + runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/', + // Threshold alerts keep firing this long after their condition clears, + // so a value oscillating around a threshold does not flap. Needs + // Prometheus 2.42 or later; set it to '' to leave keep_firing_for out. + keepFiringFor: '5m', + cpuHighUsageThreshold: 90, + // Threads waiting for a processor, per logical processor. + cpuQueuePerCoreThreshold: 2, memoryHighUsageThreshold: 90, memoryCommitHighUsageThreshold: 90, + // Pages read from or written to disk per second. + memoryPagingThreshold: 1000, diskLatencyThresholdSeconds: 0.05, // Average latency per I/O above which a Hyper-V virtual disk counts as slow. hypervDiskLatencyThresholdSeconds: 0.01, - networkErrorRateThreshold: 1, - networkDiscardRateThreshold: 1, + // Percent of all packets on the interface. + networkErrorRatioThreshold: 1, + networkDiscardRatioThreshold: 1, + // Percent of the interface bandwidth, in either direction. + networkUtilizationThreshold: 90, clockOffsetThresholdSeconds: 0.05, diskFreeWarningThreshold: 10, diskFreeCriticalThreshold: 5, diskPredictionWindow: '6h', diskPredictionHours: 24, + // Restarts of one service within 15 minutes. + serviceRestartThreshold: 2, criticalSeverity: 'critical', + infoSeverity: 'info', adPendingReplicationThreshold: 50, adReplicationFailureRateThreshold: 0, + // Percent of the page file in use. + pagefileUsageThreshold: 80, + // Percent of free addresses in an active DHCP scope. + dhcpScopeFreeWarningThreshold: 10, + dhcpScopeFreeCriticalThreshold: 2, + // How long updates may wait for installation before WindowsUpdatesPending fires. + updatePendingDuration: '7d', + // Failed zone transfers per second. + dnsZoneTransferFailureRateThreshold: 0, }, } diff --git a/contrib/mixin/dashboard.jsonnet b/contrib/mixin/dashboard.jsonnet index 27b1bad9..9d448e8f 100644 --- a/contrib/mixin/dashboard.jsonnet +++ b/contrib/mixin/dashboard.jsonnet @@ -1,4 +1,4 @@ -// Renders the sample dashboards into ../../dashboard. Run with +// Renders the sample dashboards into ../dashboard. Run with // jsonnet -J vendor -S -m . The sample dashboards show every // collector; mixin users select theirs with _config.collectors. local manifest = import 'lib/manifest.libsonnet'; diff --git a/contrib/mixin/lib/alerts.libsonnet b/contrib/mixin/lib/alerts.libsonnet new file mode 100644 index 00000000..8f0868ea --- /dev/null +++ b/contrib/mixin/lib/alerts.libsonnet @@ -0,0 +1,23 @@ +// Helpers shared by the alert files in alerts/. Takes the mixin _config. +function(c) + { + // A collector missing from a replaced collectors object counts as disabled; + // an alert missing from a replaced alerts object counts as enabled. + collectorEnabled(collector): std.get(c.collectors, collector, false), + alertEnabled(name): std.get(c.alerts, name, true), + // group renders the rules of one collector as a rule group, or nothing + // when none of them is enabled. Each rule is a plain Prometheus alerting + // rule plus two hidden fields: collectors:: lists the collectors the rule + // reads when it differs from the group's, and state:: true marks state + // alerts (== 0, == 1) and one-shot alerts, which must resolve at once. + // Every other alert is a threshold alert and keeps firing for + // keepFiringFor after its condition clears, so a value oscillating around + // the threshold does not flap. + group(name, collectors, rules): + local enabled = [ + rule + (if c.keepFiringFor != '' && !std.get(rule, 'state', false) then { keep_firing_for: c.keepFiringFor } else {}) + for rule in rules + if self.alertEnabled(rule.alert) && std.all([self.collectorEnabled(collector) for collector in std.get(rule, 'collectors', collectors)]) + ]; + if std.length(enabled) > 0 then [{ name: name, rules: enabled }] else [], + } diff --git a/contrib/mixin/lib/queries.libsonnet b/contrib/mixin/lib/queries.libsonnet index fcf655c0..98c0772a 100644 --- a/contrib/mixin/lib/queries.libsonnet +++ b/contrib/mixin/lib/queries.libsonnet @@ -7,6 +7,13 @@ function(c) ratio(numerator, denominator): '(%s / (%s > 0))' % [numerator, denominator], cpuUsage: '1 - avg without (core, mode) (%s)' % self.rate('windows_cpu_time_seconds_total', ['mode="idle"']), + // Threads waiting for a processor, averaged over the rate interval, per + // logical processor. The queue length is an instantaneous sample. + cpuQueuePerCore: + self.ratio( + 'avg_over_time(%s[%s])' % [self.metric('windows_system_processor_queue_length'), c.rateInterval], + self.metric('windows:cpu_cores:count') + ), memoryUsage: '1 - %s' % self.ratio( self.metric('windows_memory_available_bytes'), diff --git a/contrib/mixin/mise.toml b/contrib/mixin/mise.toml index a74da7ac..3c499787 100644 --- a/contrib/mixin/mise.toml +++ b/contrib/mixin/mise.toml @@ -8,7 +8,8 @@ [vars] sources = "*.libsonnet *.jsonnet lib/*.libsonnet rules/*.libsonnet alerts/*.libsonnet dashboards/*.libsonnet tests/*.libsonnet tests/*.jsonnet" fmt_flags = "-n 2 --max-blank-lines 2 --string-style s --comment-style s" -dashboards = "../../dashboard" +dashboards = "../dashboard" +rules = "../monitoring" [tasks.generate] description = "Generate Prometheus rules, alerts, and the dashboard" @@ -16,10 +17,10 @@ alias = "default" depends = ["rules", "dashboard"] [tasks.rules] -description = "Generate Prometheus rules and alerts" +description = "Generate the Prometheus rules and alerts in contrib/monitoring" run = [ - "jsonnet -S rules.jsonnet > windows_rules.yaml", - "jsonnet -S alerts.jsonnet > windows_alerts.yaml", + "jsonnet -S rules.jsonnet > {{vars.rules}}/windows_rules.yaml", + "jsonnet -S alerts.jsonnet > {{vars.rules}}/windows_alerts.yaml", ] [tasks.vendor] @@ -39,6 +40,15 @@ rm -f {{vars.dashboards}}/windows-exporter-*.json cp "$out"/*.json {{vars.dashboards}}/ ''' +[tasks.check-rules] +description = "Regenerate the rules and alerts in contrib/monitoring and fail on uncommitted changes" +depends = ["rules"] +run = ''' +set -eu +git add --intent-to-add -- {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml +git diff --exit-code -- {{vars.rules}} +''' + [tasks.check-dashboards] description = "Regenerate the dashboards and fail on uncommitted changes" depends = ["dashboard"] @@ -65,8 +75,8 @@ run = "jsonnetfmt {{vars.fmt_flags}} -i {{vars.sources}}" description = "Generate rules and alerts with the test configuration" hide = true run = [ - '''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules)' > tests/custom_rules.yaml''', - '''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts)' > tests/custom_alerts.yaml''', + '''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules, quote_keys=false)' > tests/custom_rules.yaml''', + '''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts, quote_keys=false)' > tests/custom_alerts.yaml''', ] [tasks.lint] @@ -88,25 +98,26 @@ for file in $(cat "$out/generated"); do diff -u {{vars.dashboards}}/"$file" "$out/$file" \ || { echo "{{vars.dashboards}}/$file is out of date; run mise run dashboard"; exit 1; } done -promtool check rules windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom -pint --offline lint windows_rules.yaml windows_alerts.yaml +promtool check rules {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom +pint --offline lint {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml pint --offline lint tests/custom_rules.yaml tests/custom_alerts.yaml ''' [tasks.check-runbooks] description = "Check runbook coverage for all alerts, including opt-in alerts" -run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { enableActiveDirectory: true, enableTime: true } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks''' +run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [name]: true for name in std.objectFields(super.collectors) } } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks''' [tasks.test] description = "Run all mixin lint, runbook, and rule tests" -depends = ["lint", "check-runbooks", "vendor"] +depends = ["lint", "check-rules", "check-runbooks", "vendor"] run = [ "jsonnet tests/runbooks.test.jsonnet", "jsonnet tests/hyperv.test.jsonnet > /dev/null", + "jsonnet tests/alerts.test.jsonnet", "jsonnet -J vendor tests/dashboard.test.jsonnet > /dev/null", "promtool test rules tests/*.test.yaml", ] [tasks.clean] -description = "Remove generated rule and alert files" -run = "rm -f windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml" +description = "Remove the generated test rule and alert files" +run = "rm -f tests/custom_rules.yaml tests/custom_alerts.yaml" diff --git a/contrib/mixin/rules.jsonnet b/contrib/mixin/rules.jsonnet index dbe13f41..2e8dc3bc 100644 --- a/contrib/mixin/rules.jsonnet +++ b/contrib/mixin/rules.jsonnet @@ -1 +1 @@ -std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules) +std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules, quote_keys=false) diff --git a/contrib/mixin/rules/rules.libsonnet b/contrib/mixin/rules/rules.libsonnet index b5016b98..e229b362 100644 --- a/contrib/mixin/rules/rules.libsonnet +++ b/contrib/mixin/rules/rules.libsonnet @@ -9,6 +9,7 @@ { record: 'windows:cpu_usage:ratio', expr: q.cpuUsage }, { record: 'windows:cpu_cores:count', expr: 'count without (core, mode) (%s)' % q.metric('windows_cpu_time_seconds_total', ['mode="idle"']) }, { record: 'windows:cpu_time:ratio_rate', expr: 'avg without (core) (%s)' % q.rate('windows_cpu_time_seconds_total') }, + { record: 'windows:cpu_queue_per_core:ratio', expr: q.cpuQueuePerCore }, { record: 'windows:memory_usage:ratio', expr: q.memoryUsage }, { record: 'windows:memory_used:bytes', @@ -56,6 +57,12 @@ expr: q.rate('windows_net_bytes_%s_total' % direction, [c.nicSelector]), } for direction in ['received', 'sent'] + ] + [ + { + record: 'windows:net_%s_packets:rate' % direction, + expr: q.rate('windows_net_packets_%s_total' % direction, [c.nicSelector]), + } + for direction in ['received', 'sent'] ] + [ { record: 'windows:net_%s_utilization:ratio' % direction, expr: q.networkUtilization(direction) } for direction in ['received', 'sent'] diff --git a/contrib/mixin/tests/alerts.test.jsonnet b/contrib/mixin/tests/alerts.test.jsonnet new file mode 100644 index 00000000..513f6f84 --- /dev/null +++ b/contrib/mixin/tests/alerts.test.jsonnet @@ -0,0 +1,81 @@ +// Checks the alert configuration: collector gating, per-alert switches, +// keep_firing_for and the group layout. +local mixin = import '../mixin.libsonnet'; +local config = (import '../config.libsonnet')._config; +local everything = mixin { _config+:: { collectors: { [name]: true for name in std.objectFields(config.collectors) } } }; +local optionalGroups = ['windows-pagefile', 'windows-diskdrive', 'windows-license', 'windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns']; +local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]); +local groupNames(alerts) = [group.name for group in alerts.groups]; + +// Every configured switch belongs to a generated alert, and every alert has a switch. +local all = std.set(names(everything.prometheusAlerts)); +assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs from generated alerts: %s' % std.manifestJson([std.setDiff(all, std.set(std.objectFields(config.alerts))), std.setDiff(std.set(std.objectFields(config.alerts)), all)]); + +// Groups follow the collector areas; opt-in groups appear only when enabled. +assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts)); +// One file per collector, one group per file: the groups of every enabled +// collector come in the order of the imports in alerts/alerts.libsonnet. +assert std.set(groupNames(everything.prometheusAlerts)) == std.set(groupNames(mixin.prometheusAlerts) + optionalGroups) : std.manifestJson(groupNames(everything.prometheusAlerts)); +assert groupNames(everything.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-pagefile', 'windows-disk', 'windows-diskdrive', 'windows-network', 'windows-service', 'windows-system', 'windows-license'] + optionalGroups[3:] : std.manifestJson(groupNames(everything.prometheusAlerts)); +local adOnly = mixin { _config+:: { collectors+: { ad: true } } }; +assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory']; +local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } }; +assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull']; + +// Disabling a collector removes its alerts and an empty group; the alerts of a +// rule that needs two collectors go with either. +local noNet = mixin { _config+:: { collectors+: { net: false } } }; +assert !std.member(groupNames(noNet.prometheusAlerts), 'windows-network'); +local noSystem = mixin { _config+:: { collectors+: { system: false } } }; +assert names(noSystem.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUSaturation', 'WindowsHostRebooted']); +assert std.member(names(noSystem.prometheusAlerts), 'WindowsCPUHighUsage'); +// A replaced collectors object counts missing collectors as disabled. +local onlyCPU = mixin { _config+:: { collectors: { cpu: true } } }; +assert names(onlyCPU.prometheusAlerts) == ['WindowsCPUHighUsage', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsExporterDown', 'WindowsScrapeQueueTimeout'] : std.manifestJson(names(onlyCPU.prometheusAlerts)); + +// Single alerts switch off by name, with alerts+: or a replaced object. +local noReboot = mixin { _config+:: { alerts+: { WindowsHostRebooted: false } } }; +assert names(noReboot.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsHostRebooted']); +assert !std.member(groupNames(noReboot.prometheusAlerts), 'windows-system'); +local replaced = mixin { _config+:: { alerts: { WindowsCPUHighUsage: false } } }; +assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUHighUsage']); + +// keep_firing_for applies to threshold alerts only and can be turned off. +local rules(alerts) = [rule for group in alerts.groups for rule in group.rules]; +local stateAlerts = [ + 'WindowsExporterDown', + 'WindowsCollectorFailed', + 'WindowsCollectorTimeout', + 'WindowsScrapeQueueTimeout', + 'WindowsServiceNotRunning', + 'WindowsHostRebooted', + 'WindowsClockNotSynchronising', + 'WindowsClockSourceLocal', + 'WindowsDiskDriveNotOK', + 'WindowsLicenseNotGenuine', + 'WindowsScheduledTaskFailed', + 'WindowsUpdatesPending', + 'WindowsHyperVWMIUnhealthy', + 'WindowsHyperVReplicationCritical', + 'WindowsHyperVReplicationWarning', + 'WindowsClusterNodeDown', + 'WindowsClusterResourceFailed', + 'WindowsClusterResourceGroupFailed', + 'WindowsStoragePoolUnhealthy', + 'WindowsStorageVirtualDiskUnhealthy', +]; +assert std.all([ + std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert) + for rule in rules(everything.prometheusAlerts) +]) : 'keep_firing_for set on a state alert or missing on a threshold alert'; +assert std.all([rule.keep_firing_for == '5m' for rule in rules(everything.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]); +local noKeep = mixin { _config+:: { keepFiringFor: '' } }; +assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]) == 0; + +// The info severity label is configurable. +local ticket = mixin { _config+:: { infoSeverity: 'ticket' } }; +assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket']; +local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } }; +assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']]; + +{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) } diff --git a/contrib/mixin/tests/alerts.test.yaml b/contrib/mixin/tests/alerts.test.yaml index ca449fc7..8af2d341 100644 --- a/contrib/mixin/tests/alerts.test.yaml +++ b/contrib/mixin/tests/alerts.test.yaml @@ -1,10 +1,18 @@ rule_files: - - ../windows_rules.yaml - - ../windows_alerts.yaml + - ../../monitoring/windows_rules.yaml + - ../../monitoring/windows_alerts.yaml evaluation_interval: 1m group_eval_order: - windows-exporter-recording - - windows-exporter-alerts + - windows-exporter + - windows-cpu + - windows-memory + - windows-disk + - windows-network + - windows-service + - windows-system + - windows-active-directory + - windows-time tests: - name: Commit pressure, disk latency and network failures wait and recover @@ -30,6 +38,10 @@ tests: values: '0x31' - series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' values: '0+120x20 2520x10' + - series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + values: '0+3000x31' + - series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + values: '0+3000x31' alert_rule_test: - {alertname: WindowsMemoryCommitHighUsage, eval_time: 14m, exp_alerts: []} - alertname: WindowsMemoryCommitHighUsage @@ -40,7 +52,8 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/ summary: Windows memory commit usage is high description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.' - - {alertname: WindowsMemoryCommitHighUsage, eval_time: 21m, exp_alerts: []} + - {alertname: WindowsMemoryCommitHighUsage, eval_time: 25m, exp_alerts: [{exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}, exp_annotations: {runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/, summary: Windows memory commit usage is high, description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'}}]} + - {alertname: WindowsMemoryCommitHighUsage, eval_time: 26m, exp_alerts: []} - {alertname: WindowsDiskReadLatencyHigh, eval_time: 15m, exp_alerts: []} - alertname: WindowsDiskReadLatencyHigh eval_time: 16m @@ -50,7 +63,7 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/ summary: Windows disk read latency is high description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per read operation for 15 minutes.' - - {alertname: WindowsDiskReadLatencyHigh, eval_time: 25m, exp_alerts: []} + - {alertname: WindowsDiskReadLatencyHigh, eval_time: 30m, exp_alerts: []} - {alertname: WindowsDiskWriteLatencyHigh, eval_time: 15m, exp_alerts: []} - alertname: WindowsDiskWriteLatencyHigh eval_time: 16m @@ -60,27 +73,27 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/ summary: Windows disk write latency is high description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per write operation for 15 minutes.' - - {alertname: WindowsDiskWriteLatencyHigh, eval_time: 25m, exp_alerts: []} - - {alertname: WindowsNetworkErrors, eval_time: 17m, exp_alerts: []} + - {alertname: WindowsDiskWriteLatencyHigh, eval_time: 30m, exp_alerts: []} + - {alertname: WindowsNetworkErrors, eval_time: 15m, exp_alerts: []} - alertname: WindowsNetworkErrors - eval_time: 18m + eval_time: 16m exp_alerts: - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning} exp_annotations: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/ summary: Windows network packet errors are high - description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.' - - {alertname: WindowsNetworkErrors, eval_time: 25m, exp_alerts: []} - - {alertname: WindowsNetworkDiscards, eval_time: 17m, exp_alerts: []} + description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.' + - {alertname: WindowsNetworkErrors, eval_time: 30m, exp_alerts: []} + - {alertname: WindowsNetworkDiscards, eval_time: 15m, exp_alerts: []} - alertname: WindowsNetworkDiscards - eval_time: 18m + eval_time: 16m exp_alerts: - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning} exp_annotations: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/ summary: Windows network packet discards are high - description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.' - - {alertname: WindowsNetworkDiscards, eval_time: 25m, exp_alerts: []} + description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.' + - {alertname: WindowsNetworkDiscards, eval_time: 30m, exp_alerts: []} - name: New resource alerts stay quiet at exact thresholds and clock alerts are disabled interval: 1m @@ -105,6 +118,10 @@ tests: values: '0+30x20' - series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}' values: '0+30x20' + - series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}' + values: '0+3600x20' + - series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}' + values: '0+3600x20' - series: 'windows_time_ntp_client_time_sources{job="windows_exporter", instance="host:9182"}' values: '0x20' - series: 'windows_time_computed_time_offset_seconds{job="windows_exporter", instance="host:9182"}' @@ -178,7 +195,7 @@ tests: summary: Windows CPU usage is high description: 'CPU usage on host:9182 has exceeded 90% for 15 minutes.' - alertname: WindowsCPUHighUsage - eval_time: 25m + eval_time: 30m exp_alerts: [] - alertname: WindowsMemoryHighUsage eval_time: 14m @@ -192,16 +209,16 @@ tests: summary: Windows memory usage is high description: 'Physical memory usage on host:9182 has exceeded 90% for 15 minutes.' - alertname: WindowsMemoryHighUsage - eval_time: 21m + eval_time: 26m exp_alerts: [] - name: Disk alerts retain volume labels and fire at both severity levels interval: 1m input_series: - series: 'windows_logical_disk_free_bytes{job="windows_exporter", instance="host:9182", volume="C:"}' - values: '3x15 30x5' + values: '3x15 30x10' - series: 'windows_logical_disk_size_bytes{job="windows_exporter", instance="host:9182", volume="C:"}' - values: '100x21' + values: '100x26' alert_rule_test: - alertname: WindowsDiskAlmostFull eval_time: 14m @@ -220,7 +237,7 @@ tests: summary: Windows disk space is low description: 'Volume C: on host:9182 has less than 5% free space. Windows may delay space metrics by 10–15 minutes.' - alertname: WindowsDiskAlmostFull - eval_time: 16m + eval_time: 21m exp_alerts: [] - name: Healthy resources at the exact thresholds do not fire @@ -291,3 +308,274 @@ tests: - {alertname: WindowsMemoryHighUsage, eval_time: 70m, exp_alerts: []} - {alertname: WindowsDiskAlmostFull, eval_time: 70m, exp_alerts: []} - {alertname: WindowsDiskFillingUp, eval_time: 70m, exp_alerts: []} + + - name: Collector and scrape queue timeouts wait ten minutes and recover without keep_firing_for + interval: 1m + input_series: + - series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="service"}' + values: '1x10 0x5' + - series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="cpu"}' + values: '0x15' + - series: 'windows_exporter_scrape_queue_timeout{job="windows_exporter", instance="host:9182"}' + values: '1x10 0x5' + alert_rule_test: + - {alertname: WindowsCollectorTimeout, eval_time: 9m, exp_alerts: []} + - alertname: WindowsCollectorTimeout + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', collector: service, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/ + summary: Windows collector exceeds the scrape timeout + description: 'Collector service on host:9182 has not finished within the scrape timeout for 10 minutes.' + - {alertname: WindowsCollectorTimeout, eval_time: 11m, exp_alerts: []} + - {alertname: WindowsScrapeQueueTimeout, eval_time: 9m, exp_alerts: []} + - alertname: WindowsScrapeQueueTimeout + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/ + summary: Windows exporter scrapes time out in the queue + description: 'Scrapes of host:9182 have timed out waiting for an earlier scrape to finish for 10 minutes.' + - {alertname: WindowsScrapeQueueTimeout, eval_time: 11m, exp_alerts: []} + + - name: CPU saturation and paging use recordings, wait fifteen minutes and keep firing five + interval: 1m + input_series: + - series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}' + values: '9x25 0x10' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}' + values: '0+60x35' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}' + values: '0+60x35' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}' + values: '0+60x35' + - series: 'windows_memory_swap_page_operations_total{job="windows_exporter", instance="host:9182", cluster="a"}' + values: '0+90000x20 1800000x15' + alert_rule_test: + - {alertname: WindowsCPUSaturation, eval_time: 14m, exp_alerts: []} + - alertname: WindowsCPUSaturation + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/ + summary: Windows processor queue is long + description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.' + - alertname: WindowsCPUSaturation + eval_time: 31m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/ + summary: Windows processor queue is long + description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.' + - {alertname: WindowsCPUSaturation, eval_time: 32m, exp_alerts: []} + - {alertname: WindowsMemoryPagingHigh, eval_time: 18m, exp_alerts: []} + - alertname: WindowsMemoryPagingHigh + eval_time: 19m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/ + summary: Windows memory paging is high + description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.' + - alertname: WindowsMemoryPagingHigh + eval_time: 26m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/ + summary: Windows memory paging is high + description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.' + - {alertname: WindowsMemoryPagingHigh, eval_time: 27m, exp_alerts: []} + + - name: Network utilization fires in one direction and interface flapping counts status changes + interval: 1m + input_series: + - series: 'windows_net_bytes_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}' + values: '0+6000x20 120000x15' + - series: 'windows_net_bytes_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}' + values: '0x36' + - series: 'windows_net_current_bandwidth_bytes{job="windows_exporter", instance="host:9182", nic="Ethernet"}' + values: '100x36' + - series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="up"}' + values: '1 0 1 0 1 0 1 0 1 0 1x10' + - series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="down"}' + values: '0 1 0 1 0 1 0 1 0 1 0x10' + - series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Wi-Fi", status="up"}' + values: '0x21' + alert_rule_test: + - {alertname: WindowsNetworkUtilizationHigh, eval_time: 19m, exp_alerts: []} + - alertname: WindowsNetworkUtilizationHigh + eval_time: 20m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/ + summary: Windows network interface utilization is high + description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.' + - alertname: WindowsNetworkUtilizationHigh + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/ + summary: Windows network interface utilization is high + description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.' + - {alertname: WindowsNetworkUtilizationHigh, eval_time: 26m, exp_alerts: []} + - {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []} + - alertname: WindowsNetworkInterfaceFlapping + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/ + summary: Windows network interface is flapping + description: 'Interface Ethernet on host:9182 has changed its operational status more than twice in 5m.' + - {alertname: WindowsNetworkInterfaceFlapping, eval_time: 17m, exp_alerts: []} + + - name: Service alerts skip trigger-start and manual services and count restarts by process + interval: 1m + input_series: + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}' + values: '0x15 1x8' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}' + values: '1x23' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}' + values: '1x23' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}' + values: '0x23' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}' + values: '1x23' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}' + values: '1x23' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}' + values: '0x23' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}' + values: '1x23' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}' + values: '0x23' + - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}' + values: '1000x4 _x19' + - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}' + values: '_x5 2000x4 _x14' + - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}' + values: '_x10 3000x4 _x9' + - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}' + values: '_x15 4000x9' + - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}' + values: '_x16 5000x8' + alert_rule_test: + - {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []} + - alertname: WindowsServiceNotRunning + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', service: Spooler, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/ + summary: Windows automatic service is not running + description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.' + - {alertname: WindowsServiceNotRunning, eval_time: 16m, exp_alerts: []} + - {alertname: WindowsServiceCrashLooping, eval_time: 14m, exp_alerts: []} + - alertname: WindowsServiceCrashLooping + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/ + summary: Windows service is restarting repeatedly + description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.' + - alertname: WindowsServiceCrashLooping + eval_time: 23m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/ + summary: Windows service is restarting repeatedly + description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.' + - {alertname: WindowsServiceCrashLooping, eval_time: 24m, exp_alerts: []} + + - name: Service alert matches within a cluster when instance names repeat across clusters + interval: 1m + input_series: + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", state="running"}' + values: '0x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", trigger_start="false", delayed_start="false"}' + values: '1x20' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", state="running"}' + values: '1x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", trigger_start="false", delayed_start="false"}' + values: '1x20' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", state="running"}' + values: '0x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", trigger_start="true", delayed_start="false"}' + values: '1x20' + alert_rule_test: + - alertname: WindowsServiceNotRunning + eval_time: 20m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, service: Spooler, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/ + summary: Windows automatic service is not running + description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.' + + - name: Host reboot fires once for the first ten minutes after a new boot time + interval: 1m + input_series: + - series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}' + values: '-100000x10 540x15' + - series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="steady:9182"}' + values: '-100000x25' + alert_rule_test: + - {alertname: WindowsHostRebooted, eval_time: 10m, exp_alerts: []} + - alertname: WindowsHostRebooted + eval_time: 11m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 2m 0s ago.' + - alertname: WindowsHostRebooted + eval_time: 18m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 9m 0s ago.' + - {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []} + + - name: Host reboot fires after a scrape gap longer than ten minutes + interval: 1m + input_series: + - series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}' + values: '-100000x5 stale _x11 960x20' + alert_rule_test: + - {alertname: WindowsHostRebooted, eval_time: 17m, exp_alerts: []} + - alertname: WindowsHostRebooted + eval_time: 18m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 2m 0s ago.' + - alertname: WindowsHostRebooted + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 9m 0s ago.' + - {alertname: WindowsHostRebooted, eval_time: 26m, exp_alerts: []} diff --git a/contrib/mixin/tests/config.libsonnet b/contrib/mixin/tests/config.libsonnet index fc9837e3..81ac211a 100644 --- a/contrib/mixin/tests/config.libsonnet +++ b/contrib/mixin/tests/config.libsonnet @@ -9,15 +9,33 @@ diskFreeWarningThreshold: 20, diskFreeCriticalThreshold: 10, criticalSeverity: 'page', - enableActiveDirectory: true, - enableTime: true, runbookURLPattern: 'https://runbooks.example.com/windows/%s/', memoryCommitHighUsageThreshold: 80, diskLatencyThresholdSeconds: 0.02, - networkErrorRateThreshold: 0.5, - networkDiscardRateThreshold: 0.5, + networkErrorRatioThreshold: 0.5, + networkDiscardRatioThreshold: 0.5, clockOffsetThresholdSeconds: 0.1, adPendingReplicationThreshold: 10, adReplicationFailureRateThreshold: 0.5, + collectors+: { + ad: true, + dhcp: true, + diskdrive: true, + dns: true, + hyperv: true, + license: true, + mscluster: true, + pagefile: true, + scheduled_task: true, + storage_spaces: true, + time: true, + update: true, + }, + pagefileUsageThreshold: 75, + dhcpScopeFreeWarningThreshold: 20, + dhcpScopeFreeCriticalThreshold: 5, + updatePendingDuration: '2d', + dnsZoneTransferFailureRateThreshold: 0.05, + taskSelector: 'task!~"/Ignored.*"', }, } diff --git a/contrib/mixin/tests/custom.test.yaml b/contrib/mixin/tests/custom.test.yaml index 1bee3fd8..8a47ce28 100644 --- a/contrib/mixin/tests/custom.test.yaml +++ b/contrib/mixin/tests/custom.test.yaml @@ -4,7 +4,13 @@ rule_files: evaluation_interval: 1m group_eval_order: - windows-exporter-recording - - windows-exporter-alerts + - windows-exporter + - windows-cpu + - windows-memory + - windows-disk + - windows-network + - windows-service + - windows-system - windows-active-directory - windows-time @@ -52,6 +58,14 @@ tests: values: '0+120x20' - series: 'windows_net_packets_outbound_discarded_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}' values: '0+120x20' + - series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}' + values: '0+3000x20' + - series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}' + values: '0+3000x20' + - series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}' + values: '0+120x20' + - series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}' + values: '0+120x20' alert_rule_test: - alertname: WindowsMemoryCommitHighUsage eval_time: 20m @@ -84,7 +98,7 @@ tests: exp_annotations: runbook_url: https://runbooks.example.com/windows/windowsnetworkerrors/ summary: Windows network packet errors are high - description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.' + description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.' - alertname: WindowsNetworkDiscards eval_time: 20m exp_alerts: @@ -92,7 +106,7 @@ tests: exp_annotations: runbook_url: https://runbooks.example.com/windows/windowsnetworkdiscards/ summary: Windows network packet discards are high - description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.' + description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.' - name: Clock alerts opt in, use absolute offset, custom URLs and recover interval: 1m @@ -129,7 +143,7 @@ tests: runbook_url: https://runbooks.example.com/windows/windowsclockskewdetected/ summary: Windows clock offset is high description: 'Windows Time on host:9182 reports a clock offset above 0.1 seconds for 10 minutes.' - - {alertname: WindowsClockSkewDetected, eval_time: 11m, exp_alerts: []} + - {alertname: WindowsClockSkewDetected, eval_time: 16m, exp_alerts: []} - name: Custom selectors, thresholds and critical severity propagate to alerts interval: 1m input_series: @@ -239,7 +253,7 @@ tests: summary: Active Directory replication queue is high description: 'Pending replication operations on host:9182 have exceeded 10 for 15 minutes.' - alertname: WindowsADPendingReplication - eval_time: 21m + eval_time: 26m exp_alerts: [] - alertname: WindowsADReplicationFailures eval_time: 16m @@ -253,5 +267,5 @@ tests: summary: Active Directory replication requests are failing description: 'Replication synchronization requests on host:9182 have been failing for 15 minutes.' - alertname: WindowsADReplicationFailures - eval_time: 23m + eval_time: 28m exp_alerts: [] diff --git a/contrib/mixin/tests/optional.test.yaml b/contrib/mixin/tests/optional.test.yaml new file mode 100644 index 00000000..912f71e9 --- /dev/null +++ b/contrib/mixin/tests/optional.test.yaml @@ -0,0 +1,508 @@ +# Alerts of the opt-in collectors, rendered with tests/config.libsonnet. +rule_files: + - custom_rules.yaml + - custom_alerts.yaml +evaluation_interval: 1m +group_eval_order: + - windows-exporter-recording + - windows-memory + - windows-disk + - windows-system + - windows-time + - windows-scheduled-task + - windows-update + - windows-hyperv + - windows-cluster + - windows-storage-spaces + - windows-dhcp + - windows-dns + +tests: + - name: Page file usage fires above the custom threshold and resolves after keep_firing_for + interval: 1m + input_series: + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}' + values: '20x19 50x10' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}' + values: '100x30' + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}' + values: '30x30' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}' + values: '100x30' + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}' + values: '0x30' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}' + values: '0x30' + alert_rule_test: + - {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []} + - alertname: WindowsPagefileAlmostFull + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/ + summary: Windows page file is almost full + description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.' + - alertname: WindowsPagefileAlmostFull + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/ + summary: Windows page file is almost full + description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.' + - {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []} + + - name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour + interval: 1m + input_series: + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}' + values: '0x9 1x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}' + values: '1x9 0x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}' + values: '0x20' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}' + values: '1x20' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}' + values: '1x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}' + values: '0x10' + - series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}' + values: '0x60' + - series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}' + values: '1x60' + - series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}' + values: '1x60' + - series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}' + values: '0x60' + alert_rule_test: + - {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []} + - alertname: WindowsDiskDriveNotOK + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/ + summary: Windows disk drive is not healthy + description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.' + - alertname: WindowsDiskDriveNotOK + eval_time: 9m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/ + summary: Windows disk drive is not healthy + description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.' + - {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []} + - {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []} + - alertname: WindowsLicenseNotGenuine + eval_time: 60m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/ + summary: Windows license is not genuine + description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.' + + - name: Clock source alert names the local source and ignores NTP and domain sources + interval: 1m + input_series: + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}' + values: '1x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}' + values: '0x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}' + values: '1x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}' + values: '0x10' + alert_rule_test: + - {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []} + - alertname: WindowsClockSourceLocal + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/ + summary: Windows clock has no external time source + description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.' + + - name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks + interval: 1m + input_series: + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}' + values: '0x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}' + values: '0x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}' + values: '1x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}' + values: '0x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}' + values: '0x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}' + values: '1x60' + alert_rule_test: + - {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []} + - alertname: WindowsScheduledTaskFailed + eval_time: 60m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/ + summary: Windows scheduled task did not succeed + description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/ + summary: Windows scheduled task did not succeed + description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.' + + - name: Pending updates fire as info after the configured duration and count installations only + interval: 1m + input_series: + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}' + values: '1x2880' + alert_rule_test: + - {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []} + - alertname: WindowsUpdatesPending + eval_time: 48h + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsupdatespending/ + summary: Windows updates are waiting for installation + description: 'host:9182 has had 2 updates waiting for installation for 2d.' + + - name: Hyper-V alerts cover the WMI namespace and both replication health levels + interval: 1m + input_series: + - series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}' + values: '0x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}' + values: '3x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}' + values: '2x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}' + values: '0x15' + alert_rule_test: + - {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []} + - alertname: WindowsHyperVWMIUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/ + summary: Hyper-V WMI namespace is not responding + description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.' + - {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []} + - alertname: WindowsHyperVReplicationCritical + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/ + summary: Hyper-V replication is critical + description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.' + - {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []} + - alertname: WindowsHyperVReplicationWarning + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/ + summary: Hyper-V replication has a warning + description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.' + + - name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes + interval: 1m + input_series: + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}' + values: '1x30' + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}' + values: '0x30' + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}' + values: '1x30' + - series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}' + values: '4x30' + - series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}' + values: '3x30' + - series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}' + values: '2x30' + - series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}' + values: '1x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}' + values: '15x19 50x10' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}' + values: '5x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}' + values: '1x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}' + values: '0x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}' + values: '0x30' + alert_rule_test: + - {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []} + - alertname: WindowsClusterNodeDown + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/ + summary: Failover cluster node is down + description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/ + summary: Failover cluster node is down + description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.' + - alertname: WindowsClusterResourceFailed + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/ + summary: Failover cluster resource has failed + description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.' + - alertname: WindowsClusterResourceGroupFailed + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/ + summary: Failover cluster resource group has failed + description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.' + - {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []} + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + + - name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy + interval: 1m + input_series: + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}' + values: '2x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}' + values: '2x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}' + values: '1x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}' + values: '2x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}' + values: '1x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}' + values: '0x15' + alert_rule_test: + - {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []} + - alertname: WindowsStoragePoolUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/ + summary: Storage pool is unhealthy + description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/ + summary: Storage pool has a health warning + description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.' + - alertname: WindowsStorageVirtualDiskUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/ + summary: Storage Spaces virtual disk is unhealthy + description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/ + summary: Storage Spaces virtual disk has a health warning + description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.' + + - name: DHCP scope exhaustion counts active scopes only and uses both thresholds + interval: 1m + input_series: + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}' + values: '10x19 50x10' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}' + values: '90x19 50x10' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}' + values: '1x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}' + values: '2x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}' + values: '98x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}' + values: '1x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}' + values: '100x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}' + values: '1x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}' + values: '1x30' + alert_rule_test: + - {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []} + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + + - name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for + interval: 1m + input_series: + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}' + values: '0+6x20 120x15' + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}' + values: '7x36' + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}' + values: '0+1x36' + alert_rule_test: + - {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []} + - alertname: WindowsDNSZoneTransferFailures + eval_time: 17m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/ + summary: DNS zone transfers are failing + description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.' + - alertname: WindowsDNSZoneTransferFailures + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/ + summary: DNS zone transfers are failing + description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.' + - {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []} diff --git a/contrib/mixin/tests/resources.test.yaml b/contrib/mixin/tests/resources.test.yaml index 27a8d1f0..c4b3bdc6 100644 --- a/contrib/mixin/tests/resources.test.yaml +++ b/contrib/mixin/tests/resources.test.yaml @@ -1,10 +1,18 @@ rule_files: - - ../windows_rules.yaml - - ../windows_alerts.yaml + - ../../monitoring/windows_rules.yaml + - ../../monitoring/windows_alerts.yaml evaluation_interval: 1m group_eval_order: - windows-exporter-recording - - windows-exporter-alerts + - windows-exporter + - windows-cpu + - windows-memory + - windows-disk + - windows-network + - windows-service + - windows-system + - windows-active-directory + - windows-time tests: - name: Memory cache includes all lists, commit is a ratio and paging is a rate diff --git a/contrib/mixin/tests/rules.test.yaml b/contrib/mixin/tests/rules.test.yaml index 95bc9096..12ec7e33 100644 --- a/contrib/mixin/tests/rules.test.yaml +++ b/contrib/mixin/tests/rules.test.yaml @@ -1,5 +1,5 @@ rule_files: - - ../windows_rules.yaml + - ../../monitoring/windows_rules.yaml evaluation_interval: 1m fuzzy_compare: true @@ -111,3 +111,44 @@ tests: exp_samples: - labels: 'windows:net_sent_bytes:rate{job="windows_exporter", instance="host:9182", nic="Ethernet"}' value: 30 + + - name: Processor queue per core averages the queue and packet rates keep the nic label + interval: 1m + input_series: + - series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}' + values: '12x5 0x5' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}' + values: '0+60x10' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}' + values: '0+60x10' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}' + values: '0+60x10' + - series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="3", mode="idle"}' + values: '0+60x10' + - series: 'windows_system_processor_queue_length{job="windows_exporter", instance="nocpu:9182", cluster="a"}' + values: '12x10' + - series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + values: '0+600x10' + - series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + values: '0+1200x10' + promql_expr_test: + - expr: windows:cpu_queue_per_core:ratio + eval_time: 5m + exp_samples: + - labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}' + value: 3 + - expr: windows:cpu_queue_per_core:ratio + eval_time: 10m + exp_samples: + - labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}' + value: 0 + - expr: windows:net_received_packets:rate + eval_time: 10m + exp_samples: + - labels: 'windows:net_received_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + value: 10 + - expr: windows:net_sent_packets:rate + eval_time: 10m + exp_samples: + - labels: 'windows:net_sent_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}' + value: 20 diff --git a/contrib/monitoring/README.md b/contrib/monitoring/README.md new file mode 100644 index 00000000..09837605 --- /dev/null +++ b/contrib/monitoring/README.md @@ -0,0 +1,25 @@ +# Prometheus rules for windows_exporter + +Ready-to-use Prometheus rule files, rendered from the +[mixin](../mixin/README.md) with its default configuration: + +- [`windows_rules.yaml`](windows_rules.yaml): recording rules. +- [`windows_alerts.yaml`](windows_alerts.yaml): alerting rules for the default + collectors. + +The alerts build on the recording rules, so load both files: + +```yaml +rule_files: + - /etc/prometheus/rules/windows_rules.yaml + - /etc/prometheus/rules/windows_alerts.yaml +``` + +The files expect the exporter under the job name `windows_exporter`. For other +job names, thresholds, selectors or the alerts of opt-in collectors, render +the mixin with your own configuration as the +[mixin documentation](../mixin/README.md#customize-or-compose) describes. Every alert links +to its [runbook](https://prometheus-community.github.io/windows_exporter/runbooks/). + +Do not edit these files by hand: `mise run rules` in `contrib/mixin` +regenerates them, and CI fails when they are out of date. diff --git a/contrib/monitoring/windows_alerts.yaml b/contrib/monitoring/windows_alerts.yaml new file mode 100644 index 00000000..e621e892 --- /dev/null +++ b/contrib/monitoring/windows_alerts.yaml @@ -0,0 +1,249 @@ +groups: +- name: "windows-exporter" + rules: + - alert: "WindowsExporterDown" + annotations: + description: "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/" + summary: "Windows exporter is unreachable" + expr: | + up{job="windows_exporter"} == 0 + for: "5m" + labels: + severity: "critical" + - alert: "WindowsCollectorFailed" + annotations: + description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/" + summary: "Windows collector failed" + expr: | + windows_exporter_collector_success{job="windows_exporter"} == 0 + for: "5m" + labels: + severity: "warning" + - alert: "WindowsCollectorTimeout" + annotations: + description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/" + summary: "Windows collector exceeds the scrape timeout" + expr: | + windows_exporter_collector_timeout{job="windows_exporter"} == 1 + for: "10m" + labels: + severity: "warning" + - alert: "WindowsScrapeQueueTimeout" + annotations: + description: "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/" + summary: "Windows exporter scrapes time out in the queue" + expr: | + windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1 + for: "10m" + labels: + severity: "warning" +- name: "windows-cpu" + rules: + - alert: "WindowsCPUHighUsage" + annotations: + description: "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/" + summary: "Windows CPU usage is high" + expr: | + windows:cpu_usage:ratio{job="windows_exporter"} * 100 > 90 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsCPUSaturation" + annotations: + description: "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/" + summary: "Windows processor queue is long" + expr: | + windows:cpu_queue_per_core:ratio{job="windows_exporter"} > 2 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" +- name: "windows-memory" + rules: + - alert: "WindowsMemoryHighUsage" + annotations: + description: "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/" + summary: "Windows memory usage is high" + expr: | + windows:memory_usage:ratio{job="windows_exporter"} * 100 > 90 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsMemoryCommitHighUsage" + annotations: + description: "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/" + summary: "Windows memory commit usage is high" + expr: | + windows:memory_committed:ratio{job="windows_exporter"} * 100 > 90 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsMemoryPagingHigh" + annotations: + description: "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/" + summary: "Windows memory paging is high" + expr: | + windows:memory_swap_page_operations:rate{job="windows_exporter"} > 1000 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" +- name: "windows-disk" + rules: + - alert: "WindowsDiskReadLatencyHigh" + annotations: + description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/" + summary: "Windows disk read latency is high" + expr: | + windows:logical_disk_read_latency:seconds{job="windows_exporter", volume!=""} > 0.05 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsDiskWriteLatencyHigh" + annotations: + description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/" + summary: "Windows disk write latency is high" + expr: | + windows:logical_disk_write_latency:seconds{job="windows_exporter", volume!=""} > 0.05 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsDiskAlmostFull" + annotations: + description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/" + summary: "Windows disk space is low" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsDiskAlmostFull" + annotations: + description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/" + summary: "Windows disk space is low" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 5 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "critical" + - alert: "WindowsDiskFillingUp" + annotations: + description: "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/" + summary: "Windows disk is predicted to fill up" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10 + and predict_linear(windows_logical_disk_free_bytes{job="windows_exporter", volume!=""}[6h], 86400) < 0 + for: "1h" + keep_firing_for: "5m" + labels: + severity: "warning" +- name: "windows-network" + rules: + - alert: "WindowsNetworkErrors" + annotations: + description: "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/" + summary: "Windows network packet errors are high" + expr: | + ( + (windows:net_received_errors:rate{job="windows_exporter", nic!=""} + windows:net_outbound_errors:rate{job="windows_exporter", nic!=""}) + / ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0) + ) * 100 > 1 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsNetworkDiscards" + annotations: + description: "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/" + summary: "Windows network packet discards are high" + expr: | + ( + (windows:net_received_discarded:rate{job="windows_exporter", nic!=""} + windows:net_outbound_discarded:rate{job="windows_exporter", nic!=""}) + / ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0) + ) * 100 > 1 + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsNetworkUtilizationHigh" + annotations: + description: "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/" + summary: "Windows network interface utilization is high" + expr: | + (windows:net_received_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90) + or (windows:net_sent_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90) + for: "15m" + keep_firing_for: "5m" + labels: + severity: "warning" + - alert: "WindowsNetworkInterfaceFlapping" + annotations: + description: "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/" + summary: "Windows network interface is flapping" + expr: | + max without (status) (changes(windows_net_nic_operation_status{job="windows_exporter", nic!="", status="up"}[5m])) > 2 + for: "2m" + keep_firing_for: "5m" + labels: + severity: "warning" +- name: "windows-service" + rules: + - alert: "WindowsServiceNotRunning" + annotations: + description: "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/" + summary: "Windows automatic service is not running" + expr: | + max without (state) (windows_service_state{job="windows_exporter", service!="", state="running"} == 0) + and ignoring (start_mode) (windows_service_start_mode{job="windows_exporter", service!="", start_mode="auto"} == 1) + unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job="windows_exporter", service!="", trigger_start="true"} == 1) + for: "15m" + labels: + severity: "warning" + - alert: "WindowsServiceCrashLooping" + annotations: + description: "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/" + summary: "Windows service is restarting repeatedly" + expr: | + count without (process_id) (count_over_time(windows_service_process{job="windows_exporter", service!=""}[15m])) - 1 > 2 + keep_firing_for: "5m" + labels: + severity: "warning" +- name: "windows-system" + rules: + - alert: "WindowsHostRebooted" + annotations: + description: "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago." + runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/" + summary: "Windows host has rebooted" + expr: | + (time() - windows_system_boot_time_timestamp{job="windows_exporter"}) < 600 + and (time() - last_over_time(windows_system_boot_time_timestamp{job="windows_exporter"}[1h] offset 10m)) > 600 + labels: + severity: "info" diff --git a/contrib/monitoring/windows_rules.yaml b/contrib/monitoring/windows_rules.yaml new file mode 100644 index 00000000..f83fc51f --- /dev/null +++ b/contrib/monitoring/windows_rules.yaml @@ -0,0 +1,59 @@ +groups: +- name: "windows-exporter-recording" + rules: + - expr: "1 - avg without (core, mode) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"}[5m]))" + record: "windows:cpu_usage:ratio" + - expr: "count without (core, mode) (windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"})" + record: "windows:cpu_cores:count" + - expr: "avg without (core) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\"}[5m]))" + record: "windows:cpu_time:ratio_rate" + - expr: "(avg_over_time(windows_system_processor_queue_length{job=\"windows_exporter\"}[5m]) / (windows:cpu_cores:count{job=\"windows_exporter\"} > 0))" + record: "windows:cpu_queue_per_core:ratio" + - expr: "1 - (windows_memory_available_bytes{job=\"windows_exporter\"} / (windows_memory_physical_total_bytes{job=\"windows_exporter\"} > 0))" + record: "windows:memory_usage:ratio" + - expr: "windows_memory_physical_total_bytes{job=\"windows_exporter\"} - windows_memory_available_bytes{job=\"windows_exporter\"}" + record: "windows:memory_used:bytes" + - expr: "windows_memory_cache_bytes{job=\"windows_exporter\"} + windows_memory_modified_page_list_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_core_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_normal_priority_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_reserve_bytes{job=\"windows_exporter\"}" + record: "windows:memory_cached:bytes" + - expr: "(windows_memory_committed_bytes{job=\"windows_exporter\"} / (windows_memory_commit_limit_bytes{job=\"windows_exporter\"} > 0))" + record: "windows:memory_committed:ratio" + - expr: "rate(windows_memory_swap_page_operations_total{job=\"windows_exporter\"}[5m])" + record: "windows:memory_swap_page_operations:rate" + - expr: "(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"} / (windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} > 0))" + record: "windows:logical_disk_free:ratio" + - expr: "windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} - windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}" + record: "windows:logical_disk_used:bytes" + - expr: "clamp(1 - rate(windows_logical_disk_idle_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]), 0, 1)" + record: "windows:logical_disk_busy:ratio" + - expr: "rate(windows_logical_disk_read_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])" + record: "windows:logical_disk_read_bytes:rate" + - expr: "rate(windows_logical_disk_write_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])" + record: "windows:logical_disk_write_bytes:rate" + - expr: "rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m])" + record: "windows:logical_disk_read_operations:rate" + - expr: "rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m])" + record: "windows:logical_disk_write_operations:rate" + - expr: "(rate(windows_logical_disk_read_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))" + record: "windows:logical_disk_read_latency:seconds" + - expr: "(rate(windows_logical_disk_write_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))" + record: "windows:logical_disk_write_latency:seconds" + - expr: "rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_received_bytes:rate" + - expr: "rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_sent_bytes:rate" + - expr: "rate(windows_net_packets_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_received_packets:rate" + - expr: "rate(windows_net_packets_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_sent_packets:rate" + - expr: "(rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))" + record: "windows:net_received_utilization:ratio" + - expr: "(rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))" + record: "windows:net_sent_utilization:ratio" + - expr: "rate(windows_net_packets_received_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_received_errors:rate" + - expr: "rate(windows_net_packets_received_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_received_discarded:rate" + - expr: "rate(windows_net_packets_outbound_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_outbound_errors:rate" + - expr: "rate(windows_net_packets_outbound_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])" + record: "windows:net_outbound_discarded:rate" diff --git a/kubernetes/kubernetes.md b/docs/kubernetes/kubernetes.md similarity index 97% rename from kubernetes/kubernetes.md rename to docs/kubernetes/kubernetes.md index 47640176..3c49a060 100644 --- a/kubernetes/kubernetes.md +++ b/docs/kubernetes/kubernetes.md @@ -30,7 +30,7 @@ Before beginning you need to deploy the [Prometheus Operator](https://github.com This creates a pod on every Windows node. A ConfigMap holds the configuration of the Windows exporter with [configuration file](../README.md#using-a-configuration-file). Adjust the configuration file for the collectors you are interested in. ```bash -kubectl apply -f kubernetes/windows-exporter-daemonset.yaml +kubectl apply -f docs/kubernetes/windows-exporter-daemonset.yaml ``` > Note: This example manifest deploys the latest bleeding edge image `ghcr.io/prometheus-community/windows-exporter:latest` built from the main branch. You should update this to use a released version which you can find at https://github.com/prometheus-community/windows_exporter/releases @@ -58,7 +58,7 @@ spec: Create the [Pod Monitor](https://prometheus-operator.dev/docs/operator/design/#podmonitor) to configure the scraping: ```bash -kubectl apply -f kubernetes/windows-exporter-podmonitor.yaml +kubectl apply -f docs/kubernetes/windows-exporter-podmonitor.yaml ``` ### View Metrics diff --git a/kubernetes/windows-exporter-daemonset.yaml b/docs/kubernetes/windows-exporter-daemonset.yaml similarity index 100% rename from kubernetes/windows-exporter-daemonset.yaml rename to docs/kubernetes/windows-exporter-daemonset.yaml diff --git a/kubernetes/windows-exporter-podmonitor.yaml b/docs/kubernetes/windows-exporter-podmonitor.yaml similarity index 100% rename from kubernetes/windows-exporter-podmonitor.yaml rename to docs/kubernetes/windows-exporter-podmonitor.yaml diff --git a/docs/runbooks/_index.md b/docs/runbooks/_index.md index 200967bc..15a741d2 100644 --- a/docs/runbooks/_index.md +++ b/docs/runbooks/_index.md @@ -8,5 +8,9 @@ the affected host. Disk alerts also identify a `volume`; network alerts identify a `nic`. Adjust example PromQL selectors to match your scrape configuration. The default thresholds are starting points. Check the generated alert rules for -your configured thresholds and durations. Active Directory and NTP clock alerts -are opt-in and require their collectors to be enabled. +your configured thresholds and durations. The rules are grouped by collector +area, and each alert is generated only when the collectors it reads are enabled +in the mixin configuration. Active Directory and NTP clock alerts need the `ad` +and `time` collectors, which are off by default, and the alerts for page files, +disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters, +Storage Spaces, DHCP and DNS need the collector of that role. diff --git a/docs/runbooks/windowsadpendingreplication.md b/docs/runbooks/windowsadpendingreplication.md index 856de2f1..61d2c849 100644 --- a/docs/runbooks/windowsadpendingreplication.md +++ b/docs/runbooks/windowsadpendingreplication.md @@ -6,7 +6,7 @@ title: WindowsADPendingReplication Pending Active Directory replication operations exceed the configured threshold for 15 minutes. The default is 50 operations. This alert is generated only with -`enableActiveDirectory: true` and requires the `ad` collector on domain controllers. +`collectors+: { ad: true }` and requires the `ad` collector on domain controllers. ## Impact diff --git a/docs/runbooks/windowsclocknotsynchronising.md b/docs/runbooks/windowsclocknotsynchronising.md index b7b42064..fbf5dc2d 100644 --- a/docs/runbooks/windowsclocknotsynchronising.md +++ b/docs/runbooks/windowsclocknotsynchronising.md @@ -5,7 +5,7 @@ title: WindowsClockNotSynchronising ## Meaning Windows Time reports no active NTP time source for ten minutes. This alert is -generated only with `enableTime: true` and requires the `time` collector and its +generated only with `collectors+: { time: true }` and requires the `time` collector and its `ntp` subcollector. Enable it on hosts expected to synchronize through NTP; hosts intentionally using another clock source may legitimately report zero. diff --git a/docs/runbooks/windowsclocksourcelocal.md b/docs/runbooks/windowsclocksourcelocal.md new file mode 100644 index 00000000..849c0c11 --- /dev/null +++ b/docs/runbooks/windowsclocksourcelocal.md @@ -0,0 +1,56 @@ +--- +title: WindowsClockSourceLocal +--- + +## Meaning + +Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system +clock for ten minutes. The alert carries the source in its `type` label. It +needs the `time` collector with its `clock_source` subcollector and covers +domain members too, whose source is `NT5DS` when they synchronize from the +domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does +not see those hosts. A host that is meant to be the free-running reference of +an isolated network reports the local clock by design; exclude it with +`alerts+: { WindowsClockSourceLocal: false }` on that configuration. + +## Impact + +The clock drifts without an external source. Clock errors break Kerberos +authentication beyond the five minute tolerance, TLS validation, scheduled +work, log correlation and distributed systems that depend on time order. + +## Diagnosis + +1. Inspect `windows_time_clock_sync_source` for the host, and + `windows_time_computed_time_offset_seconds` to see how far the clock has + already drifted. +2. On the host, check the configured and the effective source: + + ```powershell + w32tm /query /source + w32tm /query /status + w32tm /query /configuration + Get-Service W32Time + ``` + +3. `Local CMOS Clock` after a reboot often means the Windows Time service + started before the network was up or the configured peers were unreachable. + `NoSync` means the time provider is configured not to synchronize. +4. For domain members, check that a domain controller is reachable and that + the PDC emulator synchronizes from an external source. For virtual machines, + check whether hypervisor time integration overrides the configured source. + A Hyper-V guest outside a domain that is kept in time by the integration + service reports `NoSync` while `w32tm /query /source` shows the + `VM IC Time Synchronization Provider`; that is by design, so turn the alert + off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or + configure an NTP source in the guest. + +## Mitigation + +Configure the intended source and restart synchronization with +`w32tm /config /syncfromflags:domhier /update` on domain members or +`w32tm /config /manualpeerlist:"" /syncfromflags:manual /update` on +stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the +peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the +offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/) +if it does not. diff --git a/docs/runbooks/windowsclusternodedown.md b/docs/runbooks/windowsclusternodedown.md new file mode 100644 index 00000000..69534fa6 --- /dev/null +++ b/docs/runbooks/windowsclusternodedown.md @@ -0,0 +1,46 @@ +--- +title: WindowsClusterNodeDown +--- + +## Meaning + +The failover cluster reports the node `node` as down for five minutes. The +alert needs the `mscluster` collector. Every node that runs the exporter +reports the state of all nodes, so the alert fires once per reporting node; +the `instance` label names the reporter, and the `node` label the node that is +down. A paused node, for example during cluster-aware updating, has a +different state and does not raise this alert. + +## Impact + +The roles that the node hosted have failed over to the remaining nodes, with a +short outage during the move. The cluster has less capacity and less +redundancy; with too few nodes for quorum, the whole cluster stops. + +## Diagnosis + +1. Inspect `windows_mscluster_node_state` for all nodes and + `windows_mscluster_cluster_quorum_type_value` and the dynamic quorum metrics to + see how far the cluster is from losing quorum. Check whether the node's own + exporter target is up. +2. On a working node, read the cluster state: + + ```powershell + Get-ClusterNode + Get-ClusterGroup + Get-ClusterLog -Node '' -TimeSpan 60 -Destination . + Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50 + ``` + +3. Check whether the node is powered on and reachable, whether the Cluster + Service runs on it, and whether the cluster networks between the nodes are + working. Events 1135 and 1177 in the System log name the lost node and the + quorum state. + +## Mitigation + +Bring the node back: power it on, repair its network, or start the Cluster +Service with `Start-ClusterNode`. Evict a node that is removed for good with +`Remove-ClusterNode` so the quorum configuration matches the remaining nodes. +Move the roles back to their preferred owners once the node is up and confirm +the node state returns to up. diff --git a/docs/runbooks/windowsclusterresourcefailed.md b/docs/runbooks/windowsclusterresourcefailed.md new file mode 100644 index 00000000..0e31d1d3 --- /dev/null +++ b/docs/runbooks/windowsclusterresourcefailed.md @@ -0,0 +1,45 @@ +--- +title: WindowsClusterResourceFailed +--- + +## Meaning + +The failover cluster reports the resource `resource` of type `type` in the +group `resource_group` as failed for five minutes. The alert needs the +`mscluster` collector and fires once per reporting node, named by the +`instance` label. A failed resource has exhausted its restart attempts on the +current node, or its group has reached its failover threshold. + +## Impact + +The role that depends on the resource is down: a clustered disk, IP address, +network name, file share, SQL Server instance or virtual machine. Other +resources in the same group that depend on it are offline too. + +## Diagnosis + +1. Inspect `windows_mscluster_resource_state` for the group and + `windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node` + to see where the group is and whether the whole group failed. +2. On a cluster node, read the resource and the reason: + + ```powershell + Get-ClusterResource -Name '' | Format-List * + Get-ClusterResource -Name '' | Get-ClusterParameter + Get-ClusterLog -TimeSpan 30 -Destination . + Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20 + ``` + + Event 1069 names the failed resource and its error code. +3. Check the dependency of the resource: a disk that is not reachable from the + node, an IP address whose network is down, a network name whose computer + object in Active Directory is missing or disabled, or an application whose + service fails to start. + +## Mitigation + +Fix the dependency, then bring the resource online with +`Start-ClusterResource`, or move the group to another node with +`Move-ClusterGroup`. Review the restart and failover thresholds of the +resource if it failed because of a transient problem. Confirm the resource +state returns to online and the group is online. diff --git a/docs/runbooks/windowsclusterresourcegroupfailed.md b/docs/runbooks/windowsclusterresourcegroupfailed.md new file mode 100644 index 00000000..70fde0f6 --- /dev/null +++ b/docs/runbooks/windowsclusterresourcegroupfailed.md @@ -0,0 +1,41 @@ +--- +title: WindowsClusterResourceGroupFailed +--- + +## Meaning + +The failover cluster reports the resource group `resource_group` as failed for +five minutes. The alert needs the `mscluster` collector and fires once per +reporting node, named by the `instance` label. A group fails when a resource in +it fails and the group cannot fail over anymore: it reached its failover +threshold within the failover period, or no other node can host it. + +## Impact + +The whole clustered role is down and stays down until an operator intervenes. + +## Diagnosis + +1. Inspect `windows_mscluster_resource_state` for the resources of the group + to find the failed resource, and `windows_mscluster_resourcegroup_owner_node` + to see on which node the group stopped. +2. On a cluster node, read the group and its history: + + ```powershell + Get-ClusterGroup -Name '' | Format-List * + Get-ClusterGroup -Name '' | Get-ClusterResource + Get-ClusterLog -TimeSpan 60 -Destination . + Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20 + ``` + + Event 1254 says the group exceeded its failover threshold. +3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for + the failed resource, and check the possible owners of the group. + +## Mitigation + +Fix the failed resource, then bring the group online with `Start-ClusterGroup` +on a node that can host it. A group that failed over repeatedly may need a +higher failover threshold or a longer failover period, but a resource that +fails on every node needs its cause fixed first. Confirm the group state +returns to online. diff --git a/docs/runbooks/windowsclustersharedvolumealmostfull.md b/docs/runbooks/windowsclustersharedvolumealmostfull.md new file mode 100644 index 00000000..1172c72c --- /dev/null +++ b/docs/runbooks/windowsclustersharedvolumealmostfull.md @@ -0,0 +1,44 @@ +--- +title: WindowsClusterSharedVolumeAlmostFull +--- + +## Meaning + +The Cluster Shared Volume `shared_volume` has less free space than the +configured threshold for 15 minutes. The warning fires below +`diskFreeWarningThreshold` (10 percent by default) and the critical alert +below `diskFreeCriticalThreshold` (5 percent); both fire below the critical +threshold, so inhibit the warning in Alertmanager. The alert needs the +`mscluster` collector and fires once per reporting node, named by the +`instance` label. `sharedVolumeSelector` excludes the +`ClusterPerformanceHistory` volume of the cluster's own health history. + +## Impact + +Virtual machines and databases on the volume cannot grow their files. Dynamic +virtual hard disks pause the virtual machine when the volume is full, Hyper-V +checkpoints and backups fail, and SQL Server stops writing. + +## Diagnosis + +1. Compare `windows_mscluster_shared_volumes_free_bytes` and + `windows_mscluster_shared_volumes_total_bytes` over the last days to see + whether the volume fills steadily or filled suddenly. +2. On a cluster node, find the large files on the volume: + + ```powershell + Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}} + Get-ChildItem 'C:\ClusterStorage\' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}} + ``` + +3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard + disks, backup snapshots that were not cleaned up, and dynamic disks that + grew. + +## Mitigation + +Delete or merge checkpoints, remove orphaned files, move virtual machines to +another volume with storage migration, or extend the volume on the storage +array and then in the cluster with `Resize-Partition`. Confirm free space is +above the threshold. Raise the thresholds only for volumes whose growth is +known and planned. diff --git a/docs/runbooks/windowscollectortimeout.md b/docs/runbooks/windowscollectortimeout.md new file mode 100644 index 00000000..924a7323 --- /dev/null +++ b/docs/runbooks/windowscollectortimeout.md @@ -0,0 +1,46 @@ +--- +title: WindowsCollectorTimeout +--- + +## Meaning + +The collector identified by the `collector` label has reported +`windows_exporter_collector_timeout == 1` for ten minutes. The collector did +not finish within the scrape timeout, or its call from an earlier scrape is +still running and later scrapes skip it until it returns. + +## Impact + +The metrics of this collector are missing or stale on every scrape that times +out, so dashboards show gaps and alerts on those metrics stop evaluating. A +collector that never returns can hold a scrape slot and delay the other +collectors of the host. + +## Diagnosis + +1. Find the collector and compare how long it usually takes: + + ```promql + windows_exporter_collector_timeout{job="windows_exporter"} == 1 + windows_exporter_collector_duration_seconds{job="windows_exporter"} + ``` + +2. Compare the duration with the scrape timeout of the job. The exporter + subtracts `--scrape.timeout-margin` from the `scrape_timeout` that + Prometheus sends. +3. Check the exporter log for the collector. WMI and Management Infrastructure + queries, the Task Scheduler and Windows Update are slow on hosts with many + objects or a busy WMI provider; the collector's documentation lists the + expensive options. +4. Check whether the host was under CPU, disk or memory pressure at the same + time, and whether a Windows service the collector depends on hangs, for + example `Winmgmt` for WMI-based collectors. + +## Mitigation + +Raise the `scrape_timeout` of the job within its `scrape_interval`, narrow the +collector with its include and exclude flags, or move the collector to a +separate job with a longer timeout. Restart a hanging provider service when the +log identifies it. Disable the collector on hosts where its data is not needed. +Confirm `windows_exporter_collector_timeout` returns to `0` and the collector's +metrics are published again. diff --git a/docs/runbooks/windowscpusaturation.md b/docs/runbooks/windowscpusaturation.md new file mode 100644 index 00000000..640065dc --- /dev/null +++ b/docs/runbooks/windowscpusaturation.md @@ -0,0 +1,39 @@ +--- +title: WindowsCPUSaturation +--- + +## Meaning + +The number of threads waiting for a processor, averaged over the rate window +and divided by the number of logical processors, exceeds the configured +threshold for 15 minutes. The default is two waiting threads per logical +processor. The metric is `windows:cpu_queue_per_core:ratio`, built from +`windows_system_processor_queue_length` and `windows:cpu_cores:count`. + +## Impact + +Runnable threads wait for CPU time, so applications respond slowly even when +their own CPU usage looks moderate. A long queue with CPU usage below 100% +points at a few busy cores, a single-threaded bottleneck, or processor +scheduling problems such as parked cores or hypervisor contention. + +## Diagnosis + +1. Inspect `windows:cpu_queue_per_core:ratio` next to `windows:cpu_usage:ratio` + for the host. Saturation with high usage is a capacity problem; saturation + with low usage points at scheduling. +2. Compare the per-core rates of `windows_cpu_time_seconds_total`. One or two + cores at 100% while others idle indicate a thread that cannot spread. +3. Check `windows_cpu_parking_status`, `windows_cpu_core_frequency_hertz` and + the power plan. On virtual machines check the hypervisor for CPU ready time + or steal. +4. Use Task Manager or Performance Monitor (`System\Processor Queue Length`, + `Process\Thread Count`) to find the processes with many runnable threads. + +## Mitigation + +Reduce or redistribute the workload, add logical processors, or fix the power +plan and parking settings. Give the virtual machine more CPU share or move it to +a less loaded host. Tune the threshold for hosts whose workload keeps a long +queue by design, such as build servers. Confirm the queue length per core and +application latency return to normal. diff --git a/docs/runbooks/windowsdhcpscopealmostexhausted.md b/docs/runbooks/windowsdhcpscopealmostexhausted.md new file mode 100644 index 00000000..9500cf27 --- /dev/null +++ b/docs/runbooks/windowsdhcpscopealmostexhausted.md @@ -0,0 +1,48 @@ +--- +title: WindowsDHCPScopeAlmostExhausted +--- + +## Meaning + +The enabled DHCP scope `scope` has fewer free addresses than the configured +share of its address pool for 15 minutes. The warning fires below +`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert +below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the +critical threshold, so inhibit the warning in Alertmanager. The share divides +`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses, +and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled. +On a failover pair the free and in-use counts are the totals of both servers. +The alert needs the `dhcp` collector. + +## Impact + +Once the scope is exhausted, new clients receive no address and cannot use +the network. Existing clients keep their lease until it expires and then lose +their address too. + +## Diagnosis + +1. Compare the free and in-use addresses of the scope over the last days to + see whether the exhaustion is a trend or a sudden burst, and inspect + `windows_dhcp_scope_reserved_address` and + `windows_dhcp_scope_pending_offers` for reservations and offers in flight. +2. On the server, read the scope and its leases: + + ```powershell + Get-DhcpServerv4ScopeStatistics -ScopeId + Get-DhcpServerv4Scope -ScopeId | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State + Get-DhcpServerv4Lease -ScopeId | Group-Object AddressState + Get-DhcpServerv4ExclusionRange -ScopeId + ``` + +3. Look for a long lease duration on a network with many short-lived clients, + large exclusion ranges, a flood of leases from one device that changes its + MAC address, or a failover partner that holds the free addresses. + +## Mitigation + +Shorten the lease duration for networks with transient clients, remove +exclusion ranges that are not needed, add a range through a superscope or a +larger subnet, or reclaim bad leases. On a failover pair, check the split of +the addresses between the partners. Confirm the free share stays above the +threshold. diff --git a/docs/runbooks/windowsdiskdrivenotok.md b/docs/runbooks/windowsdiskdrivenotok.md new file mode 100644 index 00000000..62e3f2f2 --- /dev/null +++ b/docs/runbooks/windowsdiskdrivenotok.md @@ -0,0 +1,46 @@ +--- +title: WindowsDiskDriveNotOK +--- + +## Meaning + +The physical drive `device_id` reports an operational status other than `OK` +for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded` +or `Error`, in its `status` label. It needs the `diskdrive` collector, which +reads the status that the drive and its driver report through WMI. The +`Unknown` status does not raise the alert: some USB and virtual drives report +it permanently. Transient statuses such as `Starting` or `Stopping` raise it +only when they last longer than five minutes. + +## Impact + +`Pred Fail` is the drive's own prediction that it is about to fail. Any other +status means the drive is not fully working. Data on a single drive is at risk; +a drive in a mirror or parity set leaves the set without redundancy until it is +replaced. + +## Diagnosis + +1. Inspect `windows_diskdrive_status` for the host and check whether the + volumes on the drive still report normal latency with + `windows:logical_disk_read_latency:seconds` and + `windows:logical_disk_write_latency:seconds`. +2. On the host, read the drive and its reliability counters: + + ```powershell + Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status + Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear + ``` + +3. Check the System event log for disk, storahci, stornvme or RAID controller + events. On a virtual machine, check the hypervisor and the datastore of the + virtual disk. + +## Mitigation + +Back up the data on the drive first. Replace a drive that predicts its failure +or reports errors through the hardware or virtualization team. For a drive in +a storage pool or RAID set, follow the vendor's replacement and rebuild +procedure and confirm the set is redundant again. The alert resolves when the +status returns to `OK`. diff --git a/docs/runbooks/windowsdnszonetransferfailures.md b/docs/runbooks/windowsdnszonetransferfailures.md new file mode 100644 index 00000000..ef692e66 --- /dev/null +++ b/docs/runbooks/windowsdnszonetransferfailures.md @@ -0,0 +1,54 @@ +--- +title: WindowsDNSZoneTransferFailures +--- + +## Meaning + +The DNS server has failed zone transfers for 15 minutes, measured as the rate +of `windows_dns_zone_transfer_failures_total` over `rateInterval` above +`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts +transfers this server sends as the primary to its secondaries, including +requests it refuses, and it has no zone label. The alert needs the `dns` +collector and keeps firing for `keepFiringFor` after the failures stop. Raise +the threshold on servers that refuse transfer requests from scanners all the +time. + +## Impact + +The secondary servers of the zone serve stale records. When the zone on a +secondary expires, it stops answering for the zone altogether. Active +Directory integrated zones replicate through Active Directory and are not +affected unless they also have secondaries. + +## Diagnosis + +1. Inspect `windows_dns_zone_transfer_failures_total` next to + `windows_dns_zone_transfer_requests_received_total` and + `windows_dns_zone_transfer_success_sent_total` to see what share of the + transfers fail, and whether the failures started with a change. +2. On this primary, find the refused or failed requests and the zone + settings; the DNS Server log records a refused transfer request with the + address of the requester: + + ```powershell + Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer' + Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy + Get-DnsServerZone -Name '' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers + ``` + + On the secondary, events 6522 to 6527 of the DNS Server log say why its + transfer of the zone failed: refused by the primary, invalid response or + an expired zone. + +3. Check that the requesting secondary is in the zone transfer list, that TCP + port 53 is open between the servers, and that the zone is not paused or + broken. Failures from unknown addresses are refused transfers and may be + expected. + +## Mitigation + +Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries` +and the server list, fix the network path, or remove a decommissioned +secondary from the zone's notify list so it stops requesting transfers. +Confirm the failure counter stops rising and the secondaries show a current +serial number. diff --git a/docs/runbooks/windowshostrebooted.md b/docs/runbooks/windowshostrebooted.md new file mode 100644 index 00000000..07439706 --- /dev/null +++ b/docs/runbooks/windowshostrebooted.md @@ -0,0 +1,37 @@ +--- +title: WindowsHostRebooted +--- + +## Meaning + +The boot time of the host, `windows_system_boot_time_timestamp`, is less than +ten minutes old and differs from the boot time known ten minutes earlier. The +alert fires once for about ten minutes after a reboot and then resolves by +itself. It has the `info` severity and is meant for a dashboard, a ticket or a +chat notification, not for a page. + +## Impact + +Every service of the host restarted, connections to it were dropped and +caches are cold. A reboot outside a maintenance window can mean a crash, an +unattended Windows Update restart or a power event. + +## Diagnosis + +1. Compare the boot time with the maintenance calendar and the Windows Update + schedule. With the `update` collector enabled, `windows_update_pending_info` + shows updates that were pending before the reboot. +2. Check the System event log: event 1074 names the user or process that + requested the shutdown, event 6008 marks an unexpected shutdown and event 41 + a reset without a clean shutdown. Check for a memory dump when it was a + crash. +3. Confirm the host came back with the expected services; the + `WindowsServiceNotRunning` alert covers automatic services that did not + start. + +## Mitigation + +No action is needed for a planned reboot. For an unplanned one, find the cause +in the event log and fix the driver, update setting or power problem. Turn the +alert off with `alerts+: { WindowsHostRebooted: false }` when reboots are not +worth a notification. diff --git a/docs/runbooks/windowshypervreplicationcritical.md b/docs/runbooks/windowshypervreplicationcritical.md new file mode 100644 index 00000000..72132afc --- /dev/null +++ b/docs/runbooks/windowshypervreplicationcritical.md @@ -0,0 +1,47 @@ +--- +title: WindowsHyperVReplicationCritical +--- + +## Meaning + +Hyper-V Replica reports critical health for the virtual machine `vm` for five +minutes. The alert carries the `relationship` label, `primary` for the normal +replication and `extended` for an extended replica, and needs the `hyperv` +collector with its `replica_vm` subcollector. Critical health means that +replication has not succeeded for long enough that the replica misses the +recovery point objective, or that replication is paused because of an error. + +## Impact + +The replica of the virtual machine is stale. A failover to it loses the +changes since the last successful replication, and the recovery point +objective of the service is not met. If the primary host fails now, data is +lost. + +## Diagnosis + +1. Inspect `windows_hyperv_replica_vm_health` and + `windows_hyperv_replica_vm_state` for the virtual machine; the state tells + whether replication is suspended, resynchronizing or in error, and + `windows_hyperv_replica_vm_replication_latency_seconds` how long the last + cycle took. +2. On the primary host, read the replication details: + + ```powershell + Measure-VMReplication -VMName '' | Format-List * + Get-VMReplication -VMName '' | Format-List State, Health, LastReplicationTime, ReplicationMode + Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '**' + ``` + +3. Check the replica server: free space on the replica storage, the + Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication, + and the network path between the hosts. Check whether a snapshot merge or a + storage migration on the primary blocks replication. + +## Mitigation + +Fix the cause on the replica server or the network, then resume replication +with `Resume-VMReplication -VMName ''`; add `-Resynchronize` when the +state asks for a resynchronization. Confirm the health returns to normal and +the last replication time advances. If the replica cannot be recovered, remove +and re-enable replication, which sends a full initial copy. diff --git a/docs/runbooks/windowshypervreplicationwarning.md b/docs/runbooks/windowshypervreplicationwarning.md new file mode 100644 index 00000000..e8e36490 --- /dev/null +++ b/docs/runbooks/windowshypervreplicationwarning.md @@ -0,0 +1,43 @@ +--- +title: WindowsHyperVReplicationWarning +--- + +## Meaning + +Hyper-V Replica reports warning health for the virtual machine `vm` for 15 +minutes. Warning health means that some replication cycles missed their +interval, that more than a fifth of the cycles in the last hour failed, or +that the replication is waiting for an initial copy or a resynchronization. +The alert carries the `relationship` label and needs the `hyperv` collector +with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/) +follows when replication stops working. + +## Impact + +The replica lags behind the primary more than planned. The recovery point +objective is at risk, and a failover would lose more changes than expected. + +## Diagnosis + +1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state` + and `windows_hyperv_replica_vm_replication_latency_seconds` for the + virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes` + to see whether the change rate of the virtual machine has grown. +2. On the primary host, read the statistics of the last hour: + + ```powershell + Measure-VMReplication -VMName '' | Format-List * + Get-VMReplication -VMName '' | Format-List State, Health, LastReplicationTime, FrequencySec + ``` + +3. Check the bandwidth between the hosts, the load on the replica storage, + and whether the replication interval is too short for the change rate of + the virtual machine. + +## Mitigation + +Relieve the bottleneck: more bandwidth, less load on the replica storage, or a +longer replication interval through `Set-VMReplication -ReplicationFrequencySec`. +Resume a waiting replication with `Resume-VMReplication`. Reset the statistics +with `Reset-VMReplicationStatistics` after the fix; the health is computed +over the last hour and takes up to an hour to return to normal. diff --git a/docs/runbooks/windowshypervwmiunhealthy.md b/docs/runbooks/windowshypervwmiunhealthy.md new file mode 100644 index 00000000..d13b248d --- /dev/null +++ b/docs/runbooks/windowshypervwmiunhealthy.md @@ -0,0 +1,43 @@ +--- +title: WindowsHyperVWMIUnhealthy +--- + +## Meaning + +The Hyper-V WMI namespace `root/virtualization/v2` on the host has not +answered a minimal query for ten minutes. The alert needs the `hyperv` +collector with its `wmi_health` subcollector. While the namespace is broken, +the virtual machine, replication and virtual hard disk metrics that come from +WMI are missing, so the other Hyper-V alerts cannot fire. + +## Impact + +Management tools such as Hyper-V Manager, Failover Cluster Manager and System +Center cannot manage the virtual machines on the host. Backups that use the +Hyper-V VSS writer fail. The virtual machines themselves usually keep running. + +## Diagnosis + +1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}` + for the host. +2. On the host, test the namespace and the management service: + + ```powershell + Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1 + Get-Service vmms, vmcompute, winmgmt + winmgmt /verifyrepository + ``` + +3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of + the Virtual Machine Management service, and the WMI-Activity log for + provider failures. A stuck `vmms` process or a corrupt WMI repository are + the usual causes. + +## Mitigation + +Restart the Virtual Machine Management service with `Restart-Service vmms`; +running virtual machines are not affected. If the repository is inconsistent, +run `winmgmt /salvagerepository`, and register the Hyper-V provider again with +`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a +restart of the host if the service cannot be recovered. Confirm the health +metric returns to 1 and the Hyper-V metrics appear again. diff --git a/docs/runbooks/windowslicensenotgenuine.md b/docs/runbooks/windowslicensenotgenuine.md new file mode 100644 index 00000000..1fb33ff0 --- /dev/null +++ b/docs/runbooks/windowslicensenotgenuine.md @@ -0,0 +1,45 @@ +--- +title: WindowsLicenseNotGenuine +--- + +## Meaning + +Windows reports a license status other than `genuine` for one hour. The alert +carries the status in its `state` label and needs the `license` collector. The +status comes from the Software Licensing service and changes when an +activation expires, a KMS host is unreachable for more than the grace period, +the hardware changed, or the key was blocked. `offline` means the genuine +validation could not reach Microsoft; on hosts without internet access that +is the permanent state, so turn the alert off there with +`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter +selector. + +## Impact + +An unlicensed Windows shows activation notices, may limit personalization and, +on some editions, restarts on a schedule. A Windows Server that leaves its +activation grace period is out of compliance. + +## Diagnosis + +1. The `state` label of the alert names the status; `windows_license_status` + shows all states of the host. +2. On the host, read the license details: + + ```powershell + slmgr.vbs /dlv + slmgr.vbs /xpr + Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining + ``` + +3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp` + DNS record and answers on TCP port 1688. For Active Directory based + activation, check the activation object in the forest. For MAK or retail + keys, check whether the activation count is exhausted. + +## Mitigation + +Restore reachability to the KMS host or the activation object, then run +`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk ` if the +key was wrong, or reactivate after a hardware change. Confirm the status +returns to `genuine`. diff --git a/docs/runbooks/windowsmemorypaginghigh.md b/docs/runbooks/windowsmemorypaginghigh.md new file mode 100644 index 00000000..82b5342c --- /dev/null +++ b/docs/runbooks/windowsmemorypaginghigh.md @@ -0,0 +1,42 @@ +--- +title: WindowsMemoryPagingHigh +--- + +## Meaning + +The rate of pages read from or written to disk to resolve hard page faults, +`windows:memory_swap_page_operations:rate`, exceeds the configured threshold +for 15 minutes. The default is 1000 pages per second. The counter counts +pagefile traffic and memory-mapped file I/O alike. + +## Impact + +The host is short of physical memory and uses the disk as backing store, so +every affected process slows down to disk speed. Sustained paging also +competes with application I/O on the system disk. + +## Diagnosis + +1. Inspect the paging rate together with `windows:memory_usage:ratio`, + `windows_memory_available_bytes` and `windows:memory_committed:ratio`. Low + available memory confirms memory pressure; a high rate with ample memory + points at a process reading a large memory-mapped file, such as a backup or + an index rebuild. +2. Split the rate into reads and writes with + `windows_memory_swap_page_reads_total` and + `windows_memory_swap_page_writes_total`. Reads dominate when the working + sets of processes do not fit; writes dominate when the system trims and + writes modified pages. +3. Find the processes with the largest working sets and page fault rates in + Resource Monitor or, with the `process` collector enabled, with + `windows_process_working_set_private_bytes` and + `windows_process_page_faults_total`. +4. Check `windows_memory_standby_cache_lifetime_seconds`; a low lifetime means + cached pages are repurposed quickly, another sign of memory pressure. + +## Mitigation + +Stop or move the process that uses the memory, fix a memory leak, or add +physical memory. Move large sequential file jobs off peak hours when they cause +the paging. Confirm the paging rate drops and available memory recovers. Raise +the threshold on hosts whose workload maps large files by design. diff --git a/docs/runbooks/windowsnetworkdiscards.md b/docs/runbooks/windowsnetworkdiscards.md index d96cdaca..ffa562c8 100644 --- a/docs/runbooks/windowsnetworkdiscards.md +++ b/docs/runbooks/windowsnetworkdiscards.md @@ -4,9 +4,15 @@ title: WindowsNetworkDiscards ## Meaning -The sum of inbound and outbound discarded packet rates exceeds the configured -threshold for 15 minutes. The default is one discard per second per interface. -Discards count packets dropped without a packet error, such as buffer exhaustion. +Discarded inbound and outbound packets make up more than the configured share +of all packets on an interface for 15 minutes. The default +`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates +(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`) +by the packet rates (`windows:net_received_packets:rate` plus +`windows:net_sent_packets:rate`). Discards count packets dropped without a packet +error, such as buffer exhaustion. Interfaces that carry no packets produce no +sample and cannot alert. The alert keeps firing for `keepFiringFor` after the +ratio drops below the threshold. ## Impact @@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy. 1. Inspect `windows:net_received_discarded:rate` and `windows:net_outbound_discarded:rate` to identify the direction and affected - `nic`. + `nic`, and compare them with the packet rate recordings to see whether a low + packet rate inflates the ratio. 2. Compare traffic, directional utilization ratios, and `windows_net_output_queue_length_packets` with the normal baseline. 3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers, @@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy. Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver, or virtual switch problems when evidence identifies them. Review intentional -drop policies with the network owner before changing thresholds. Confirm discard -rates and application retransmissions return to the expected baseline. +drop policies with the network owner before raising +`networkDiscardRatioThreshold`. Confirm discard rates and application +retransmissions return to the expected baseline. diff --git a/docs/runbooks/windowsnetworkerrors.md b/docs/runbooks/windowsnetworkerrors.md index 882cb728..b5ba471c 100644 --- a/docs/runbooks/windowsnetworkerrors.md +++ b/docs/runbooks/windowsnetworkerrors.md @@ -4,9 +4,16 @@ title: WindowsNetworkErrors ## Meaning -The sum of inbound and outbound packet error rates exceeds the configured -threshold for 15 minutes. The default is one error per second per interface. -The alert identifies the interface using its `nic` label. +Packets with inbound or outbound errors make up more than the configured share +of all packets on an interface for 15 minutes. The default +`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates +(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by +the packet rates (`windows:net_received_packets:rate` plus +`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors +alerts while a busy interface with the same error count does not. Interfaces +that carry no packets produce no sample and cannot alert. The alert identifies +the interface using its `nic` label and keeps firing for `keepFiringFor` after +the ratio drops below the threshold. ## Impact @@ -16,7 +23,10 @@ connection failures. ## Diagnosis 1. Inspect `windows:net_received_errors:rate` and - `windows:net_outbound_errors:rate` separately to identify the direction. + `windows:net_outbound_errors:rate` separately to identify the direction, and + compare them with `windows:net_received_packets:rate` and + `windows:net_sent_packets:rate` to see whether a low packet rate inflates the + ratio. 2. Compare traffic and `windows:net_received_utilization:ratio` and `windows:net_sent_utilization:ratio` with the normal baseline. These ratios use bandwidth in bytes per second and keep each direction separate. @@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration problems. Apply driver and firmware fixes through the normal change process. Relieve congestion if it contributes. Verify errors stop increasing and affected application connections recover; cumulative counters do not need to return to -zero for the rate alert to resolve. +zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for +interfaces whose error share is known and accepted. diff --git a/docs/runbooks/windowsnetworkinterfaceflapping.md b/docs/runbooks/windowsnetworkinterfaceflapping.md new file mode 100644 index 00000000..6123e71f --- /dev/null +++ b/docs/runbooks/windowsnetworkinterfaceflapping.md @@ -0,0 +1,35 @@ +--- +title: WindowsNetworkInterfaceFlapping +--- + +## Meaning + +The operational status of the interface `nic` changed more than twice within +the rate window, and did so for two minutes. The alert counts the changes of +`windows_net_nic_operation_status{status="up"}` between `1` and `0`. + +## Impact + +Every transition drops the connections that use the interface and triggers +route, DHCP and DNS registration changes. A flapping uplink on a server causes +intermittent application failures that are hard to attribute. + +## Diagnosis + +1. Inspect `windows_net_nic_operation_status` for the interface to see the + status it flips to; `down` and `lower layer down` point at the link, + `dormant` at authentication such as 802.1X. +2. Check the System event log for NDIS, the adapter driver and + `Microsoft-Windows-NetworkProfile` events, and the switch port for link + state changes, errors and port security actions. +3. Check the cable, SFP, speed and duplex negotiation, and whether a teaming or + failover configuration switches between members. +4. On virtual machines, check the hypervisor for virtual switch changes, live + migrations, or a host adapter that flaps. + +## Mitigation + +Replace the cable or transceiver, fix the speed and duplex settings, update the +adapter driver, or correct the switch port and authentication configuration. +Exclude an interface that flaps by design, such as a Wi-Fi or VPN adapter, +with the `nicSelector`. Confirm the status stays `up`. diff --git a/docs/runbooks/windowsnetworkutilizationhigh.md b/docs/runbooks/windowsnetworkutilizationhigh.md new file mode 100644 index 00000000..70f838ac --- /dev/null +++ b/docs/runbooks/windowsnetworkutilizationhigh.md @@ -0,0 +1,40 @@ +--- +title: WindowsNetworkUtilizationHigh +--- + +## Meaning + +Received or sent throughput on the interface `nic` exceeds the configured share +of its link bandwidth for 15 minutes. The default is 90%. The recordings +`windows:net_received_utilization:ratio` and +`windows:net_sent_utilization:ratio` divide the byte rate of each direction by +`windows_net_current_bandwidth_bytes`. + +## Impact + +The link is close to saturation in at least one direction. Further traffic +queues at the adapter, which raises latency and causes discards and TCP +retransmissions for every application on the host. + +## Diagnosis + +1. Inspect both utilization ratios to find the direction, and + `windows:net_received_bytes:rate` and `windows:net_sent_bytes:rate` for the + absolute volume. +2. Check `windows_net_current_bandwidth_bytes`. A link that negotiated a lower + speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high + utilization at normal traffic; compare it with `Get-NetAdapter`. +3. Correlate with `windows:net_received_discarded:rate`, + `windows:net_outbound_discarded:rate` and + `windows_net_output_queue_length_packets` to see whether the saturation + already drops packets. +4. Identify the traffic with Resource Monitor, `Get-NetTCPConnection`, or + switch-port statistics: backups, replication, file copies and monitoring + bursts are common causes. + +## Mitigation + +Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that +negotiated the wrong speed. Confirm utilization returns below the threshold and +discards stop. Tune the threshold for interfaces that run near line rate by +design, such as storage replication links. diff --git a/docs/runbooks/windowspagefilealmostfull.md b/docs/runbooks/windowspagefilealmostfull.md new file mode 100644 index 00000000..62ebbea4 --- /dev/null +++ b/docs/runbooks/windowspagefilealmostfull.md @@ -0,0 +1,41 @@ +--- +title: WindowsPagefileAlmostFull +--- + +## Meaning + +The page file `file` has been more than the configured share in use for 15 +minutes. The default `pagefileUsageThreshold` is 80 percent of +`windows_pagefile_limit_bytes`, the current size of the file. The alert needs +the `pagefile` collector. A system-managed page file grows up to its maximum +when it fills, so a high usage of the current size on such a system shows that +the file has already grown. + +## Impact + +When the page file is full, Windows cannot commit more memory. Allocations +fail, processes crash with out-of-memory errors, and the system can become +unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/) +usually fires at the same time. + +## Diagnosis + +1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes` + with `windows:memory_committed:ratio` and `windows_memory_available_bytes` + to see whether physical memory is exhausted too. +2. Find the processes that hold the commit charge: + + ```powershell + Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}} + Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage + ``` + +3. Check for a recent change: a new workload, a memory leak that grows over + days, or a page file that was made smaller or fixed in size. + +## Mitigation + +Stop or restart the process that leaks memory, or move workload off the host. +Add physical memory or enlarge the page file through the normal change +process; a fixed page file needs enough room for the peak commit charge of the +host. Confirm usage falls below the threshold and stays there. diff --git a/docs/runbooks/windowsscheduledtaskfailed.md b/docs/runbooks/windowsscheduledtaskfailed.md new file mode 100644 index 00000000..366a0eea --- /dev/null +++ b/docs/runbooks/windowsscheduledtaskfailed.md @@ -0,0 +1,51 @@ +--- +title: WindowsScheduledTaskFailed +--- + +## Meaning + +The last run of the scheduled task `task` ended with the result `unknown` or +`terminated`, and the task has not completed successfully since for one hour. +The alert needs the `scheduled_task` collector. `terminated` means the run was +stopped by a user or by the scheduler's time limit. `unknown` is every result +code without a named status, which includes the exit code of the task's +program and the error codes of the Task Scheduler; the documentation of the +collector lists the named codes. A program that exits with a code other than +zero on success raises this alert on every run; exclude such tasks with +`taskSelector` or with the include and exclude flags of the collector. A +disabled task keeps its last result but does not raise the alert. + +## Impact + +The work of the task did not happen: backups, cleanups, reports, certificate +renewals or synchronizations that are scheduled on this host. The impact +depends on the task. + +## Diagnosis + +1. Inspect `windows_scheduled_task_last_result_status` for the task and + `windows_scheduled_task_state` to see whether the task is still enabled + and queued. +2. On the host, read the last result and the history: + + ```powershell + Get-ScheduledTaskInfo -TaskName '' -TaskPath '' + Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '**' + ``` + + The history log is off by default; enable it with + `wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`. +3. Translate the result code: `0x1` is a generic failure of the program, `0x2` + a missing file, `0x80070005` an access denied, and codes of the form + `0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other + codes between `0x41300` and `0x41325` are named statuses, such as `ready` + or `running`, and do not raise this alert. +4. Run the program by hand under the task's account to reproduce the failure. + +## Mitigation + +Fix the program, its arguments, its working directory or the permissions of +the account the task runs as. Start the task once with `Start-ScheduledTask` +and confirm the result turns to `success`. Disable or delete tasks that are +not needed anymore instead of leaving them failing; disabling a task clears +the alert. diff --git a/docs/runbooks/windowsscrapequeuetimeout.md b/docs/runbooks/windowsscrapequeuetimeout.md new file mode 100644 index 00000000..2732ee04 --- /dev/null +++ b/docs/runbooks/windowsscrapequeuetimeout.md @@ -0,0 +1,40 @@ +--- +title: WindowsScrapeQueueTimeout +--- + +## Meaning + +`windows_exporter_scrape_queue_timeout == 1` for ten minutes. Scrapes run one +at a time; a scrape whose timeout ends while it waits for an earlier scrape to +finish runs no collector and reports this metric as `1`. On such a scrape +`windows_exporter_collector_success` is `0` for every collector. + +## Impact + +The affected scrapes return no collector metrics, so the host has gaps in all +of its series and `WindowsCollectorFailed` can fire for every collector at +once. The exporter itself is reachable. + +## Diagnosis + +1. Confirm the queue timeout and find the scrape that holds the slot: + + ```promql + windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1 + windows_exporter_collector_timeout{job="windows_exporter"} == 1 + scrape_duration_seconds{job="windows_exporter"} + ``` + +2. Check whether more than one Prometheus server or job scrapes the same + exporter, or whether a job scrapes it more often than its scrape duration + allows. Compare `scrape_duration_seconds` with the `scrape_interval`. +3. Identify the slow collector with `windows_exporter_collector_duration_seconds` + and follow `WindowsCollectorTimeout` for it. + +## Mitigation + +Scrape each exporter from one job, or give the jobs a `scrape_interval` longer +than a full scrape takes. Shorten the slow collector or raise the +`scrape_timeout` so a scrape finishes before the next one starts. Confirm +`windows_exporter_scrape_queue_timeout` stays `0` and collector success returns +to `1`. diff --git a/docs/runbooks/windowsservicecrashlooping.md b/docs/runbooks/windowsservicecrashlooping.md new file mode 100644 index 00000000..5a050726 --- /dev/null +++ b/docs/runbooks/windowsservicecrashlooping.md @@ -0,0 +1,51 @@ +--- +title: WindowsServiceCrashLooping +--- + +## Meaning + +The service `service` has run as more processes in the last 15 minutes than +the configured number of restarts allows. The default is two restarts. Every +service process has its own `windows_service_process` series, identified by +`process_id`, so the series seen within 15 minutes count the restarts. + +Three limits apply. Windows reuses process IDs, so a restart that lands on the +same ID is not counted. Services that share a process, such as `svchost.exe` +groups, share a `process_id`: one crash of the shared process alerts for every +service in it. Services without a `windows_service_process` series, for example +stopped services or services the exporter cannot read the process of, are +invisible to this alert. + +## Impact + +The service crashes or is stopped and started repeatedly. It is unavailable +during every restart, and the recovery options of the Service Control Manager +give up after a configurable number of failures, after which +`WindowsServiceNotRunning` follows. + +## Diagnosis + +1. List the processes the service ran as: + + ```promql + count_over_time(windows_service_process{job="windows_exporter", service=""}[15m]) + ``` + + The value of each series is the start time of the process. If several + services report the same `process_id`, the shared host process crashed and + the faulty service is one of that group. +2. Check the Application log for crashes of the service executable (events + 1000 and 1026, .NET Runtime errors) and the System log for Service Control + Manager events 7031 and 7034, which also show the recovery action. +3. Check whether a deployment, configuration push, scheduled task or another + automation restarts the service on purpose. +4. Check for resources the service fails to acquire at start, such as a port + that is in use, a certificate, a database or a license. + +## Mitigation + +Fix the crash or the start failure, roll back the deployment that caused it, or +stop the automation that restarts the service. Keep the service stopped until +the cause is known if every restart has side effects. Confirm the service keeps +one process and stays running. Raise the threshold for services that restart by +design, for example worker services that recycle themselves. diff --git a/docs/runbooks/windowsservicenotrunning.md b/docs/runbooks/windowsservicenotrunning.md new file mode 100644 index 00000000..52b32be7 --- /dev/null +++ b/docs/runbooks/windowsservicenotrunning.md @@ -0,0 +1,45 @@ +--- +title: WindowsServiceNotRunning +--- + +## Meaning + +The service `service` has the start mode `auto` but has not been in the +`running` state for 15 minutes. Trigger-start services, which Windows stops by +design when their trigger condition ends, are left out through the +`trigger_start` label of `windows_service_info`. Delayed-start services are +included; the 15 minute duration covers their later start after a reboot. + +## Impact + +A service that should always run is stopped, so the function it provides is +unavailable, for example printing, time synchronization, a site or a +database instance. Windows restarts a crashed service only when its recovery +options say so. + +## Diagnosis + +1. Inspect the current state of the service and when it changed: + + ```promql + windows_service_state{job="windows_exporter", service=""} == 1 + ``` + +2. Check the System event log for Service Control Manager events 7000, 7009, + 7031 and 7034, and the Application log for crashes of the service process. +3. Check `windows_service_info` for the account in `run_as`; an expired + password or a removed right to log on as a service stops the service at the + next start. `windows_service_process` shows whether a process is running. +4. Check whether the service was stopped on purpose for maintenance or + disabled by policy, and whether it depends on another service that is also + stopped. + +## Mitigation + +Start the service after fixing the cause: the account, a missing dependency, a +configuration error or a crash. Set recovery options so Windows restarts the +service itself. Change the start mode to `manual` or `disabled` for services +that should not run, or exclude them with the `serviceSelector` when the mixin +should not watch them. For example +`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that +exit on their own by design. Confirm the service stays running. diff --git a/docs/runbooks/windowsstoragepoolunhealthy.md b/docs/runbooks/windowsstoragepoolunhealthy.md new file mode 100644 index 00000000..b8cdd118 --- /dev/null +++ b/docs/runbooks/windowsstoragepoolunhealthy.md @@ -0,0 +1,48 @@ +--- +title: WindowsStoragePoolUnhealthy +--- + +## Meaning + +The Storage Spaces pool `pool` reports a health status of warning or unhealthy +for ten minutes. The critical alert fires for unhealthy, the warning for +warning. The alert needs the `storage_spaces` collector and joins +`windows_storage_spaces_pool_info` so the primordial pool, which only groups +the disks that are not in a pool yet, does not raise it. The pool's +operational status, exposed as `windows_storage_spaces_pool_operational_status`, +tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every +node reports the clustered pool, so the alert fires once per node; the +`instance` label names the reporter. + +## Impact + +A pool in warning state has lost a physical disk or redundancy; the virtual +disks on it still work but have no tolerance left for another failure, or +cannot allocate more space. An unhealthy pool has lost more disks than its +resiliency tolerates, so virtual disks on it are detached or read-only. + +## Diagnosis + +1. Inspect `windows_storage_spaces_pool_health_status`, + `windows_storage_spaces_pool_operational_status` and + `windows_storage_spaces_virtual_disk_health_status` for the pool and its + virtual disks. +2. On the host, find the failed disk: + + ```powershell + Get-StoragePool -FriendlyName '' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly + Get-StoragePool -FriendlyName '' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage + Get-StoragePool -FriendlyName '' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-StorageJob + ``` + +3. Check the System log for disk and storage events, and whether a repair job + is already running. + +## Mitigation + +Replace the failed physical disk: retire it with +`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`, +repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk +with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the +health returns to healthy once the repair jobs finish. diff --git a/docs/runbooks/windowsstoragevirtualdiskunhealthy.md b/docs/runbooks/windowsstoragevirtualdiskunhealthy.md new file mode 100644 index 00000000..37a15904 --- /dev/null +++ b/docs/runbooks/windowsstoragevirtualdiskunhealthy.md @@ -0,0 +1,43 @@ +--- +title: WindowsStorageVirtualDiskUnhealthy +--- + +## Meaning + +The Storage Spaces virtual disk `virtual_disk` reports a health status of +warning or unhealthy for ten minutes. The critical alert fires for unhealthy, +the warning for warning. The alert needs the `storage_spaces` collector. A +virtual disk is in warning state when it is degraded or incomplete and still +accessible, and unhealthy when it is detached or has lost data. On Storage +Spaces Direct every node reports the clustered virtual disks, so the alert +fires once per node; the `instance` label names the reporter. + +## Impact + +A degraded virtual disk still serves data but has no redundancy left; a +further disk failure loses the data. A detached virtual disk takes its volumes +offline, and everything stored on them is unavailable. + +## Diagnosis + +1. Inspect `windows_storage_spaces_virtual_disk_health_status` and + `windows_storage_spaces_pool_health_status` to see whether the pool is + affected too; [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/) + then names the pool. +2. On the host, read the virtual disk, its footprint and the repair jobs: + + ```powershell + Get-VirtualDisk -FriendlyName '' | Select-Object FriendlyName, HealthStatus, OperationalStatus, ResiliencySettingName, DetachedReason + Get-VirtualDisk -FriendlyName '' | Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-StorageJob + ``` + +3. Check for a failed physical disk, a pool that ran out of space for + allocation, or a disk that was detached by a policy after an error. + +## Mitigation + +Replace the failed physical disk as in [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/) +and repair the virtual disk with `Repair-VirtualDisk`. Attach a detached disk +with `Connect-VirtualDisk` once the cause is fixed. Confirm the health returns +to healthy once the repair jobs finish. diff --git a/docs/runbooks/windowsupdatespending.md b/docs/runbooks/windowsupdatespending.md new file mode 100644 index 00000000..37d25411 --- /dev/null +++ b/docs/runbooks/windowsupdatespending.md @@ -0,0 +1,50 @@ +--- +title: WindowsUpdatesPending +--- + +## Meaning + +The host has updates waiting for installation for longer than +`updatePendingDuration`, seven days by default. The alert is an info alert, its +value is the number of pending updates, and it needs the `update` collector, +which queries the Windows Update agent in the background at the interval of +its scrape setting. Updates waiting for uninstallation do not count. Prometheus counts the +duration and restores it after a restart shorter than its +`--rules.alert.for-outage-tolerance`, one hour by default; a longer outage or +a new server starts the wait again. + +## Impact + +Security updates that wait longer than the patch window leave known +vulnerabilities open. Other updates carry fixes for reliability problems. The +host may also be waiting for a restart to finish an installation. + +## Diagnosis + +1. List the pending updates with their category and severity: + + ```promql + windows_update_pending_info{job="windows_exporter", instance="", deployment_action="installation"} + ``` + + `windows_update_pending_published_timestamp` shows how old each update is. +2. On the host, check the update client and the last installation attempts: + + ```powershell + Get-WindowsUpdateLog + Get-WinEvent -LogName Microsoft-Windows-WindowsUpdateClient/Operational -MaxEvents 50 + Get-CimInstance -Namespace root/cimv2 -ClassName Win32_QuickFixEngineering | Sort-Object InstalledOn -Descending | Select-Object -First 5 + ``` + +3. Check whether the host is in a maintenance window, a deferral period or a + WSUS or Configuration Manager deployment that has not approved the updates + yet, and whether a pending restart blocks the installation. + +## Mitigation + +Install the updates in the next maintenance window, or approve them in the +update management tool. Restart the host if an installation waits for it. +Raise `updatePendingDuration` to the patch window of the environment if seven +days is shorter than the policy, or turn the alert off with +`alerts+: { WindowsUpdatesPending: false }` where another system tracks patch +compliance. diff --git a/docs/upgrade/0.32.md b/docs/upgrade/0.32.md index aab18db2..f7bf82a3 100644 --- a/docs/upgrade/0.32.md +++ b/docs/upgrade/0.32.md @@ -116,7 +116,7 @@ grep -rnwFf old-names.txt --exclude=windows_exporter-0.32.prom path/to/dashboard This doesn't find the label renames and the metrics without a compatibility rule; check those against the tables. -If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/alerts.libsonnet`](../../contrib/mixin/alerts/alerts.libsonnet) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too. +If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/`](../../contrib/mixin/alerts) (one file per collector) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too. ### 5. Remove the compatibility rules @@ -339,7 +339,7 @@ The exporter's own metrics now also include Go scheduler metrics (`go_sched_*`), ### Dashboard and alerts -- The sample dashboard in [`dashboard/`](../../dashboard/README.md) was rebuilt as one dashboard with tabs. It uses the Grafana v2 dashboard schema and requires **Grafana 13 or later**. Import the new JSON file; the 0.31 dashboard can't be updated in place. +- The sample dashboard in [`contrib/dashboard/`](../../contrib/dashboard/README.md) was rebuilt as one dashboard with tabs. It uses the Grafana v2 dashboard schema and requires **Grafana 13 or later**. Import the new JSON file; the 0.31 dashboard can't be updated in place. - New: [`contrib/mixin`](../../contrib/mixin) provides Prometheus recording rules and alerts for windows_exporter, with a runbook for each alert in [`docs/runbooks`](../runbooks). ### Installer @@ -375,7 +375,7 @@ These changes affect Go programs that embed `github.com/prometheus-community/win ## Metric reference windows_exporter 0.32 follows the Prometheus naming conventions for every metric: values are in base units (seconds, bytes, hertz, ratios from 0 to 1), the unit is part of the name, and counters end in `_total`. -**The old names are no longer exported, there are no aliases.** The sample dashboards in [`dashboard/`](../../dashboard/README.md) and the rules in [`contrib/mixin`](../../contrib/mixin) already use the new names. +**The old names are no longer exported, there are no aliases.** The sample dashboards in [`contrib/dashboard/`](../../contrib/dashboard/README.md) and the rules in [`contrib/mixin`](../../contrib/mixin) already use the new names. The tables below are also the source of [`0.32.prom`](0.32.prom): a test generates the compatibility rules from the 0.31 and 0.32 columns and the notes. ### Unit changes diff --git a/internal/metriccontract/metrics_test.go b/internal/metriccontract/metrics_test.go index 1530706e..d8052a7a 100644 --- a/internal/metriccontract/metrics_test.go +++ b/internal/metriccontract/metrics_test.go @@ -189,7 +189,7 @@ type reference struct { } // metricReferences returns the windows_ metric names in the collector -// documentation, the mixin sources and the generated dashboards. +// documentation, the mixin sources, the rendered rules and the generated dashboards. func metricReferences(t *testing.T) []reference { t.Helper() @@ -209,11 +209,16 @@ func metricReferences(t *testing.T) []reference { files = append(files, runbooks...) - dashboards, err := filepath.Glob(filepath.Join(repoRoot, "dashboard", "*.json")) + dashboards, err := filepath.Glob(filepath.Join(repoRoot, "contrib", "dashboard", "*.json")) require.NoError(t, err) files = append(files, dashboards...) + rules, err := filepath.Glob(filepath.Join(repoRoot, "contrib", "monitoring", "*.yaml")) + require.NoError(t, err) + + files = append(files, rules...) + err = filepath.WalkDir(filepath.Join(repoRoot, "contrib", "mixin"), func(path string, entry fs.DirEntry, err error) error { if err != nil { return err @@ -239,6 +244,7 @@ func metricReferences(t *testing.T) []reference { require.NoError(t, err) require.NotEmpty(t, docs) require.NotEmpty(t, dashboards) + require.NotEmpty(t, rules) var refs []reference