mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-18 18:49:05 +02:00
mixin: group alerts by collector area and add alerts for default and opt-in collectors (#2777)
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
@@ -6,7 +6,8 @@ on:
|
||||
pull_request:
|
||||
paths:
|
||||
- contrib/mixin/**
|
||||
- dashboard/*.json
|
||||
- contrib/dashboard/*.json
|
||||
- contrib/monitoring/*.yaml
|
||||
- docs/collector.*.md
|
||||
- docs/runbooks/**
|
||||
- docs/upgrade/**
|
||||
@@ -21,7 +22,8 @@ on:
|
||||
- main
|
||||
paths:
|
||||
- contrib/mixin/**
|
||||
- dashboard/*.json
|
||||
- contrib/dashboard/*.json
|
||||
- contrib/monitoring/*.yaml
|
||||
- docs/collector.*.md
|
||||
- docs/runbooks/**
|
||||
- docs/upgrade/**
|
||||
@@ -64,6 +66,9 @@ jobs:
|
||||
- name: Build dashboard and check generated changes
|
||||
run: mise run check-dashboards
|
||||
|
||||
- name: Generate rules and check generated changes
|
||||
run: mise run check-rules
|
||||
|
||||
- name: Lint dashboard and test rules
|
||||
run: mise run test
|
||||
|
||||
|
||||
@@ -27,9 +27,10 @@ their runtime behavior.
|
||||
and examples. [`docs/collector-template.md`](docs/collector-template.md) is the
|
||||
documentation template.
|
||||
- [`contrib/mixin`](contrib/mixin): Jsonnet recording rules, alerts and dashboard
|
||||
sources. [`dashboard`](dashboard) contains the generated Grafana dashboards;
|
||||
[`docs/runbooks`](docs/runbooks) documents the alerts.
|
||||
- [`installer`](installer), [`kubernetes`](kubernetes) and [`tools`](tools): MSI
|
||||
sources. [`contrib/dashboard`](contrib/dashboard) contains the generated Grafana
|
||||
dashboards, [`contrib/monitoring`](contrib/monitoring) the rendered Prometheus
|
||||
rules and alerts; [`docs/runbooks`](docs/runbooks) documents the alerts.
|
||||
- [`installer`](installer), [`docs/kubernetes`](docs/kubernetes) and [`tools`](tools): MSI
|
||||
packaging, deployment examples, CI fixtures and end-to-end helpers. The CI
|
||||
package job validates installer changes with
|
||||
[`tools/test-installer.ps1`](tools/test-installer.ps1), which installs the
|
||||
@@ -356,9 +357,9 @@ dashes, followed by its age in hex.
|
||||
## Dashboards, rules and runbooks
|
||||
|
||||
Read [`contrib/mixin/README.md`](contrib/mixin/README.md) and
|
||||
[`dashboard/README.md`](dashboard/README.md) before editing monitoring artifacts.
|
||||
[`contrib/dashboard/README.md`](contrib/dashboard/README.md) before editing monitoring artifacts.
|
||||
Edit the Jsonnet/Grafonnet source under `contrib/mixin/dashboards`, including its
|
||||
tab modules and shared builders, and regenerate `dashboard/windows-exporter-*.json`:
|
||||
tab modules and shared builders, and regenerate `contrib/dashboard/windows-exporter-*.json`:
|
||||
the full dashboard `windows-exporter-dashboard.json` and one dashboard per tab.
|
||||
Do not hand-edit generated JSON. Tab dashboards keep only the variables their
|
||||
panels reference, so reference variables as `$name` or `${name}`.
|
||||
|
||||
@@ -204,7 +204,7 @@ The Docker image is tagged with the version of the exporter. The `latest` tag is
|
||||
|
||||
## Kubernetes Implementation
|
||||
|
||||
See detailed steps to install on Windows Kubernetes [here](./kubernetes/kubernetes.md).
|
||||
See detailed steps to install on Windows Kubernetes [here](./docs/kubernetes/kubernetes.md).
|
||||
|
||||
## Prometheus rules
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ and takes several ideas from Grafana's [windows-observ-lib](https://github.com/g
|
||||
- Grafana 13 or later with a Prometheus data source. The file uses the v2 dashboard schema, which Grafana needs for tabs.
|
||||
- windows_exporter with the default collectors (`cpu`, `logical_disk`, `memory`, `net`, `os`, `physical_disk`, `service`, `system`).
|
||||
The SMB, Processes, Scheduled tasks, Updates, GPU, Hyper-V, Failover Cluster and Time tabs, the MSSQL and Hyper-V Cluster dashboards, the TCP and UDP rows of the Network tab and the Drives and Storage Spaces rows of the Disk tab need the `smb` or `smbclient`, `process`, `scheduled_task`, `update`, `gpu`, `hyperv`, `mscluster`, `time`, `mssql`, `tcp`, `udp`, `diskdrive` and `storage_spaces` collectors, which are not enabled by default.
|
||||
The sample shows every tab; build the dashboard from the [mixin](../contrib/mixin#collectors) with your collectors to hide the others.
|
||||
The sample shows every tab; build the dashboard from the [mixin](../mixin#collectors) with your collectors to hide the others.
|
||||
The process names in the GPU tab need the `process` collector.
|
||||
GPU temperature, fan, power and clock panels require WDDM 2.4 or newer and driver support; unsupported sensors are absent.
|
||||
|
||||
@@ -23,7 +23,7 @@ Import the JSON files through **Dashboards > New > Import** and pick the Prometh
|
||||
| Windows Exporter / MSSQL | [windows-exporter-mssql.json](windows-exporter-mssql.json) | `windows-exporter-mssql` |
|
||||
| Windows Exporter / Hyper-V Cluster | [windows-exporter-hyper-v-cluster.json](windows-exporter-hyper-v-cluster.json) | `windows-exporter-hyper-v-cluster` |
|
||||
|
||||
The MSSQL dashboard is not a tab of the full dashboard. The [mixin](../contrib/mixin#metadata) adds it as a tab with `separateDashboardTabs: []`.
|
||||
The MSSQL dashboard is not a tab of the full dashboard. The [mixin](../mixin#metadata) adds it as a tab with `separateDashboardTabs: []`.
|
||||
The Hyper-V Cluster dashboard has tabs of its own, see [Hyper-V Cluster dashboard](#hyper-v-cluster-dashboard).
|
||||
|
||||
The tab dashboards show only the variables their panels use.
|
||||
@@ -102,11 +102,11 @@ appear. I/O graphs combine read, write and other operations per process.
|
||||
|
||||
### Changing the dashboard
|
||||
|
||||
Don't edit the JSON files by hand. They are generated from Jsonnet with [grafonnet](https://github.com/grafana/grafonnet) as part of the [windows_exporter mixin](../contrib/mixin):
|
||||
Don't edit the JSON files by hand. They are generated from Jsonnet with [grafonnet](https://github.com/grafana/grafonnet) as part of the [windows_exporter mixin](../mixin):
|
||||
|
||||
- [dashboards/windows-exporter.libsonnet](../contrib/mixin/dashboards/windows-exporter.libsonnet) composes the dashboards from one file per tab.
|
||||
- [dashboards/builders.libsonnet](../contrib/mixin/dashboards/builders.libsonnet) holds the panel, query, and layout builders.
|
||||
- [dashboards/styles.libsonnet](../contrib/mixin/dashboards/styles.libsonnet) shares the visualization defaults.
|
||||
- [dashboards/windows-exporter.libsonnet](../mixin/dashboards/windows-exporter.libsonnet) composes the dashboards from one file per tab.
|
||||
- [dashboards/builders.libsonnet](../mixin/dashboards/builders.libsonnet) holds the panel, query, and layout builders.
|
||||
- [dashboards/styles.libsonnet](../mixin/dashboards/styles.libsonnet) shares the visualization defaults.
|
||||
|
||||
The generator writes the v2 schema directly, so it needs neither Grafana nor Prometheus.
|
||||
Install [mise](https://mise.jdx.dev/) and Go, then install the pinned tools and
|
||||
|
Before Width: | Height: | Size: 299 KiB After Width: | Height: | Size: 299 KiB |
|
Before Width: | Height: | Size: 565 KiB After Width: | Height: | Size: 565 KiB |
|
Before Width: | Height: | Size: 162 KiB After Width: | Height: | Size: 162 KiB |
|
Before Width: | Height: | Size: 164 KiB After Width: | Height: | Size: 164 KiB |
@@ -1,5 +1,3 @@
|
||||
/vendor/
|
||||
/windows_alerts.yaml
|
||||
/windows_rules.yaml
|
||||
/tests/*_alerts.yaml
|
||||
/tests/*_rules.yaml
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Mixin instructions
|
||||
|
||||
Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers
|
||||
`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources.
|
||||
[`README.md`](README.md) documents the configuration and the generated rules;
|
||||
keep it in sync with every change.
|
||||
|
||||
## Alerts
|
||||
|
||||
`alerts/` holds one file per collector, each rendering one rule group, and
|
||||
`alerts/alerts.libsonnet` imports them in the order of the generated file.
|
||||
A rule is a plain Prometheus alerting rule, like in kubernetes-mixin:
|
||||
|
||||
```jsonnet
|
||||
{
|
||||
alert: 'WindowsCPUHighUsage',
|
||||
expr: |||
|
||||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: { summary: '...', description: '...' },
|
||||
}
|
||||
```
|
||||
|
||||
Write the PromQL out in the `expr` block with `%(name)s` placeholders for the
|
||||
`_config` values; selectors go inside the braces as
|
||||
`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet`
|
||||
provides `a.group(name, collectors, rules)`, which keeps a rule only when every
|
||||
collector in `collectors` is enabled in `_config.collectors` and the alert is
|
||||
not switched off in `_config.alerts`, and adds `keep_firing_for` from
|
||||
`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]`
|
||||
when a rule reads other collectors than its group (the CPU saturation alert
|
||||
reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`,
|
||||
`== 1`) and one-shot alerts, which must resolve at once and get no
|
||||
`keep_firing_for`. A one-shot alert has no `for`.
|
||||
|
||||
A new alert touches six places, and CI checks all of them:
|
||||
|
||||
1. The rule in `alerts/<collector>.libsonnet`, with `{{ $labels.<key> }}`
|
||||
templates that use the 0.32 label keys (`service`, `volume`, `nic`,
|
||||
`task`, `device_id`, never `name`). Alerts for a new collector go in a new
|
||||
file, imported from `alerts/alerts.libsonnet`, with the collector added to
|
||||
`_config.collectors` in `config.libsonnet`; the group expectations in
|
||||
`tests/alerts.test.jsonnet` list every group.
|
||||
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
|
||||
fails when the map and the generated alerts differ. Thresholds are
|
||||
`_config` knobs named `<area><Condition>Threshold[Unit]`.
|
||||
3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for
|
||||
opt-in collectors, which `tests/config.libsonnet` enables) that checks the
|
||||
first firing evaluation, the labels and annotations, and the recovery. With
|
||||
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
|
||||
condition clears. `rate()` extrapolates while a window has few samples, so
|
||||
ratios of two rates activate earlier than a single rate; probe the
|
||||
activation time with `ALERTS_FOR_STATE` instead of guessing it.
|
||||
4. A runbook `docs/runbooks/<lowercase-alert-name>.md` with the `Meaning`,
|
||||
`Impact`, `Diagnosis` and `Mitigation` sections.
|
||||
`go test ./internal/metriccontract/` checks every `windows_*` metric name
|
||||
a runbook mentions against the metrics the exporter emits; the names in
|
||||
the collector docs are not always current, so check the collector source.
|
||||
5. A row in the alerts table of `README.md`.
|
||||
6. The committed `contrib/monitoring/windows_rules.yaml` and
|
||||
`windows_alerts.yaml`, regenerated
|
||||
with `mise run rules`; `mise run check-rules` fails in CI when they are
|
||||
stale.
|
||||
|
||||
Metrics with a state label (`windows_service_state{state="running"} == 0`)
|
||||
keep that label in the alert; drop it with `max without (state) (...)` so the
|
||||
alert does not say `state="running"` for a stopped service. Join a state metric
|
||||
with its info metric with `ignoring (<labels of the info metric>)`, never
|
||||
`on (job, instance, <key>)`: `on` drops the other target labels, so two
|
||||
targets with the same `instance` in different clusters match each other.
|
||||
A target can be scraped again after a gap (a reboot, a restart of the
|
||||
exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset
|
||||
10m)`); a plain `m offset 10m` has no sample while the gap covers that point.
|
||||
|
||||
## Recording rules
|
||||
|
||||
New recordings go in `rules/rules.libsonnet` with the shared expressions in
|
||||
`lib/queries.libsonnet`, a row in the `README.md` table and a test in
|
||||
`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero
|
||||
capacities produce no sample instead of `NaN` or `+Inf`.
|
||||
|
||||
## Checks
|
||||
|
||||
`mise run test` runs everything CI runs. Without mise, the tools from
|
||||
`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
|
||||
the `lint`, `check-runbooks` and `test` tasks do the same.
|
||||
+161
-52
@@ -23,16 +23,23 @@ Mise installs release binaries with checksum verification and builds Jsonnet
|
||||
Bundler using Go. `mise run` makes the pinned tools available automatically.
|
||||
|
||||
Only the dashboard depends on Jsonnet libraries, so `jb install` is not required
|
||||
to generate rules. Generated rule files are ignored by Git:
|
||||
to generate rules. The rules rendered from the default configuration are
|
||||
committed in [`contrib/monitoring`](../monitoring), so they can be used
|
||||
without Jsonnet:
|
||||
|
||||
- `windows_rules.yaml`: recording rules.
|
||||
- `windows_alerts.yaml`: alerting rules.
|
||||
|
||||
`mise run rules` regenerates them after a source change, and CI fails when the
|
||||
committed files are out of date (`mise run check-rules`).
|
||||
|
||||
The default selector is `job="windows_exporter"`. Change it to match your scrape
|
||||
job before generating rules. Resource rules use the default `cpu`, `memory`,
|
||||
`logical_disk`, and `net` collectors. Enable `ad` on domain controllers when
|
||||
opting into Active Directory alerts. Clock alerts require the `time` collector
|
||||
with its `ntp` subcollector and are intended for hosts using NTP synchronization.
|
||||
job before generating rules. The rules read the default `cpu`, `memory`,
|
||||
`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors`
|
||||
lists which collectors the targets run, and the alerts of a collector that is
|
||||
turned off are not generated. Enable `ad` on domain controllers for the Active
|
||||
Directory alerts and `time` for the clock alerts, which need the `ntp` and
|
||||
`clock_source` subcollectors.
|
||||
|
||||
Copy both generated files into your Prometheus rule directory and reference
|
||||
them in `prometheus.yml`:
|
||||
@@ -60,8 +67,9 @@ modifying its source. For example, create `custom.libsonnet` in this directory:
|
||||
nicSelector: 'nic!="Loopback"',
|
||||
rateInterval: '5m',
|
||||
cpuHighUsageThreshold: 95,
|
||||
enableActiveDirectory: true,
|
||||
enableTime: true,
|
||||
collectors+: { ad: true, time: true, process: true },
|
||||
alerts+: { WindowsHostRebooted: false },
|
||||
keepFiringFor: '10m',
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
},
|
||||
}
|
||||
@@ -77,9 +85,14 @@ jsonnet -S -e 'std.manifestYamlDoc((import "custom.libsonnet").prometheusAlerts)
|
||||
Selectors are comma-separated Prometheus label matchers without braces. They
|
||||
apply to both recording rules and alerts. All target labels, including `job`,
|
||||
`instance`, `cluster`, and custom selector labels, are preserved. CPU usage
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk and
|
||||
network recordings also retain `volume` and `nic` respectively. The CPU mode
|
||||
recording retains `mode` and removes only `core`.
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk,
|
||||
network and service rules also retain `volume`, `nic` and `service`
|
||||
respectively, filtered by `volumeSelector`, `nicSelector` and
|
||||
`serviceSelector`. The scheduled task and cluster shared volume alerts
|
||||
filter their `task` and `shared_volume` labels with `taskSelector`
|
||||
(`task!=""`) and `sharedVolumeSelector`
|
||||
(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only
|
||||
`core`.
|
||||
|
||||
`rateInterval` is the fixed window for recording rules and Active Directory
|
||||
failure rates; keep it at least four times your scrape interval. Grafana's
|
||||
@@ -93,6 +106,7 @@ rules.
|
||||
| `windows:cpu_usage:ratio` | Non-idle CPU time averaged across cores | Ratio, 0–1 |
|
||||
| `windows:cpu_cores:count` | Logical processors observed in idle CPU metrics | Count |
|
||||
| `windows:cpu_time:ratio_rate` | CPU time averaged across cores, retaining `mode` | Seconds/second |
|
||||
| `windows:cpu_queue_per_core:ratio` | Threads waiting for a processor, averaged over `rateInterval`, per logical processor | Threads/processor |
|
||||
| `windows:memory_usage:ratio` | Physical memory in use, based on available bytes | Ratio, 0–1 |
|
||||
| `windows:memory_used:bytes` | Total physical memory minus available memory | Bytes |
|
||||
| `windows:memory_cached:bytes` | Cache, modified pages, and three standby lists | Bytes |
|
||||
@@ -109,6 +123,8 @@ rules.
|
||||
| `windows:logical_disk_write_latency:seconds` | Write-time rate divided by write-operation rate | Seconds/operation |
|
||||
| `windows:net_received_bytes:rate` | Received network traffic per interface | Bytes/second |
|
||||
| `windows:net_sent_bytes:rate` | Sent network traffic per interface | Bytes/second |
|
||||
| `windows:net_received_packets:rate` | Received packets per interface | Packets/second |
|
||||
| `windows:net_sent_packets:rate` | Sent packets per interface | Packets/second |
|
||||
| `windows:net_received_utilization:ratio` | Received throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_sent_utilization:ratio` | Sent throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_received_errors:rate` | Inbound packet errors | Packets/second |
|
||||
@@ -127,46 +143,137 @@ which can exceed wall time for concurrent I/O. Latency is an average and is
|
||||
omitted when there are no operations. Memory, disk space, commit, and network
|
||||
utilization ratios are omitted when their capacity is zero or missing. Network
|
||||
bandwidth is already in bytes/second; receive and send utilization are separate
|
||||
for full-duplex links.
|
||||
for full-duplex links. The processor queue length is an instantaneous sample,
|
||||
so the queue recording averages it over `rateInterval` before dividing by the
|
||||
core count.
|
||||
The cache recording includes modified pages and is not a measure of immediately
|
||||
reclaimable memory. Paging operations can include file-backed pages as well as
|
||||
pagefile traffic.
|
||||
|
||||
## Alerts
|
||||
|
||||
Threshold configuration values below are percentages except where noted.
|
||||
The alerts come in one rule group per collector, from one file per collector
|
||||
in `alerts/` (`alerts/cpu.libsonnet`, `alerts/service.libsonnet`, and so on).
|
||||
Each group lists the collectors its rules read; a group is generated only with
|
||||
those collectors enabled in `_config.collectors`, and a rule that needs two
|
||||
collectors, such as the CPU saturation alert, only with both. Threshold configuration values below are
|
||||
percentages except where noted.
|
||||
|
||||
| Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- |
|
||||
| `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsNetworkErrors` | Combined receive/outbound errors above `networkErrorRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsNetworkDiscards` | Combined receive/outbound discards above `networkDiscardRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
| Group | Collectors | Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `windows-exporter` | | `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `windows-exporter` | | `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `windows-exporter` | | `WindowsCollectorTimeout` | `windows_exporter_collector_timeout == 1` | 10m | Warning |
|
||||
| `windows-exporter` | | `WindowsScrapeQueueTimeout` | `windows_exporter_scrape_queue_timeout == 1` | 10m | Warning |
|
||||
| `windows-cpu` | `cpu` | `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-cpu` | `cpu`, `system` | `WindowsCPUSaturation` | Waiting threads per logical processor above `cpuQueuePerCoreThreshold` (2) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryPagingHigh` | Pages read or written per second above `memoryPagingThreshold` (1000) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkErrors` | Receive and outbound errors above `networkErrorRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkDiscards` | Receive and outbound discards above `networkDiscardRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkUtilizationHigh` | Received or sent throughput above `networkUtilizationThreshold` (90%) of the link bandwidth | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkInterfaceFlapping` | Operational status changed more than twice within `rateInterval` | 2m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceNotRunning` | Automatic service without triggers not running | 15m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceCrashLooping` | More than `serviceRestartThreshold` (2) restarts in 15 minutes | 0m | Warning |
|
||||
| `windows-system` | `system` | `WindowsHostRebooted` | Boot time less than ten minutes old and changed | 0m | Info |
|
||||
| `windows-active-directory` | `ad` | `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning |
|
||||
| `windows-pagefile` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
|
||||
| `windows-diskdrive` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
|
||||
| `windows-license` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
|
||||
| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning |
|
||||
| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning |
|
||||
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical |
|
||||
| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning |
|
||||
|
||||
Active Directory alerts are generated only when `enableActiveDirectory` is
|
||||
`true`. `criticalSeverity` defaults to `critical` and can be overridden to match
|
||||
your routing labels. Below the critical disk threshold, both disk severity
|
||||
levels fire; configure Alertmanager inhibition if only the critical
|
||||
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
|
||||
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
|
||||
build, and a name missing from a replaced object stays enabled. A group whose
|
||||
alerts are all turned off is not generated.
|
||||
|
||||
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
|
||||
value that oscillates around a threshold does not flap. The state alerts
|
||||
`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`,
|
||||
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`,
|
||||
`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors
|
||||
(clock source, drive status, license, task result, pending updates, Hyper-V,
|
||||
cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`,
|
||||
have no `keep_firing_for` and resolve as soon as their condition clears.
|
||||
`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for
|
||||
older servers.
|
||||
|
||||
The network error and discard thresholds are percentages of all packets on the
|
||||
interface.
|
||||
|
||||
`criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override
|
||||
them to match your routing labels. Below the critical disk threshold, both disk
|
||||
severity levels fire; configure Alertmanager inhibition if only the critical
|
||||
notification should be sent.
|
||||
|
||||
Clock alerts are generated only when `enableTime` is `true`. A zero NTP source
|
||||
count may be expected on hosts using another time provider. The offset compares
|
||||
the host to its chosen source and does not independently verify that source.
|
||||
Disk latency and network error/discard thresholds should be tuned for the
|
||||
Active Directory alerts need `collectors.ad`, clock alerts `collectors.time`;
|
||||
both are off by default. A zero NTP
|
||||
source count may be expected on hosts using another time provider;
|
||||
`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and
|
||||
fires only for the local clock or no synchronization. The offset compares the
|
||||
host to its chosen source and does not independently verify that source.
|
||||
|
||||
The other opt-in alerts follow `_config.collectors`: enable `pagefile`,
|
||||
`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`,
|
||||
`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are
|
||||
generated, each collector in a group of its own.
|
||||
`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code
|
||||
without a name, including the exit codes of the task's program, as a failure;
|
||||
narrow the tasks with `taskSelector`. Disabling a task clears the alert.
|
||||
`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and
|
||||
virtual drives report permanently. `WindowsLicenseNotGenuine` reports the
|
||||
`state` the host has; hosts without internet access can stay in `offline`,
|
||||
so turn the alert off there. Every failover cluster node reports the state of
|
||||
the whole cluster, and every Storage Spaces Direct node reports the clustered
|
||||
pools and virtual disks, so the cluster and storage alerts fire once per node;
|
||||
group them by their `node`, `resource`, `shared_volume`, `pool` or
|
||||
`virtual_disk` labels in Alertmanager. The storage pool alert joins
|
||||
`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP
|
||||
scope alert only counts scopes in an enabled state, and a scope without
|
||||
addresses produces no sample. `WindowsUpdatesPending` is an info alert that
|
||||
waits `updatePendingDuration` for the normal maintenance window. Prometheus
|
||||
restores the waiting time after a restart shorter than its
|
||||
`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage
|
||||
or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the
|
||||
storage tier, traffic volume, and application requirements.
|
||||
|
||||
`WindowsServiceNotRunning` watches services with the `auto` start mode and
|
||||
skips trigger-start services, which stop by design, through the
|
||||
`trigger_start` label of `windows_service_info`. Delayed-start services are
|
||||
included; the 15 minute duration covers their later start after a reboot.
|
||||
`WindowsServiceCrashLooping` counts the `windows_service_process` series of a
|
||||
service within 15 minutes, one per process the service ran as. Exclude services
|
||||
with `serviceSelector`, for example automatic services that exit by design
|
||||
after their work, such as the software protection or updater services of some
|
||||
products. `WindowsHostRebooted` fires once for about ten minutes after a
|
||||
reboot and resolves by itself.
|
||||
|
||||
Windows disk size/free-space counters may lag by 10–15 minutes. Predictions
|
||||
depend on a sustained trend and are intentionally delayed for an hour. A target
|
||||
removed from service discovery has no `up` series and will not trigger
|
||||
@@ -204,10 +311,10 @@ Fork users should override `runbookURLPattern` to match their published URL.
|
||||
|
||||
`grafanaDashboards` contains `windows-exporter.json`, the full sample dashboard
|
||||
with all tabs in
|
||||
[dashboard/windows-exporter-dashboard.json](../../dashboard/windows-exporter-dashboard.json),
|
||||
[contrib/dashboard/windows-exporter-dashboard.json](../dashboard/windows-exporter-dashboard.json),
|
||||
and a shorter dashboard per tab, such as `windows-exporter-cpu.json`, in
|
||||
[dashboard](../../dashboard). They are Grafana v2 dashboards and need Grafana 13
|
||||
or later. See the [dashboard documentation](../../dashboard/README.md) for their
|
||||
[contrib/dashboard](../dashboard). They are Grafana v2 dashboards and need Grafana 13
|
||||
or later. See the [dashboard documentation](../dashboard/README.md) for their
|
||||
content.
|
||||
|
||||
The dashboard uses additive builders with [grafonnet](https://github.com/grafana/grafonnet),
|
||||
@@ -254,7 +361,7 @@ Regenerate the committed files after changing the dashboard source:
|
||||
mise run dashboard
|
||||
```
|
||||
|
||||
This runs `jb install` and writes `../../dashboard/windows-exporter-*.json`.
|
||||
This runs `jb install` and writes `../dashboard/windows-exporter-*.json`.
|
||||
`mise run dashboards` is an alias. `mise run generate` builds both the rules and the
|
||||
dashboard, while `mise run rules` builds only the rules. The same commands work
|
||||
inside WSL; mise supplies the tool binaries on `PATH`.
|
||||
@@ -322,7 +429,7 @@ tab to the full dashboard:
|
||||
names, such as `cpu`, `physical_disk`, `diskdrive`, `smb` or `update`. The
|
||||
dashboard renders only the tabs, rows, panels and variables of enabled
|
||||
collectors, so a tab for a collector you do not run does not appear. The
|
||||
defaults match the exporter's default collectors; `time` follows `enableTime`.
|
||||
defaults match the exporter's default collectors.
|
||||
The `os` collector is required, because the variables select hosts through
|
||||
`windows_os_hostname`. An unknown collector name fails the build. Enable
|
||||
optional collectors on top of the defaults with `collectors+:`; replacing the
|
||||
@@ -366,14 +473,16 @@ mise run test
|
||||
|
||||
This checks Jsonnet formatting, runs `jsonnet-lint` on the Jsonnet sources to
|
||||
find unused variables and references to missing fields, checks that the
|
||||
committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
|
||||
committed dashboards and rule files are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
|
||||
`promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic
|
||||
time series.
|
||||
Tests cover all 25 recordings, alert timing and recovery, threshold
|
||||
boundaries, counter resets, zero denominators, disk forecasts, target-label
|
||||
preservation, and custom selectors/thresholds with Active Directory and clock
|
||||
alerts enabled. Runbook checks include opt-in alerts and verify that their pages
|
||||
and URLs exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`,
|
||||
threshold boundaries, counter resets, zero denominators, disk forecasts,
|
||||
target-label preservation, and custom selectors/thresholds with Active
|
||||
Directory and clock alerts enabled. `tests/alerts.test.jsonnet` checks the
|
||||
collector gating, the per-alert switches and the group layout. Runbook checks
|
||||
render the alerts of every collector and verify that their pages and URLs
|
||||
exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
`mise install`.
|
||||
|
||||
To list the available tasks:
|
||||
@@ -405,12 +514,12 @@ prometheus "local" {
|
||||
}
|
||||
```
|
||||
|
||||
Then run `pint --config /path/to/local.pint.hcl lint windows_rules.yaml windows_alerts.yaml`
|
||||
Then run `pint --config /path/to/local.pint.hcl lint ../monitoring/windows_rules.yaml ../monitoring/windows_alerts.yaml`
|
||||
without `--offline`. See the [pint documentation](https://cloudflare.github.io/pint/)
|
||||
for available checks.
|
||||
|
||||
Run `mise run fmt` after source changes and `mise run clean` to remove generated rule
|
||||
files. For downstream composition with Jsonnet Bundler, import
|
||||
Run `mise run fmt` and `mise run rules` after source changes, and `mise run clean`
|
||||
to remove the rule files generated for the tests. For downstream composition with Jsonnet Bundler, import
|
||||
`github.com/prometheus-community/windows_exporter/contrib/mixin/mixin.libsonnet`
|
||||
and extend `_config`. See the [monitoring mixin documentation](https://github.com/monitoring-mixins/docs)
|
||||
for the standard composition workflow.
|
||||
|
||||
@@ -1 +1 @@
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts)
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts, quote_keys=false)
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-active-directory', collectors=['ad'], rules=[
|
||||
{
|
||||
alert: 'WindowsADPendingReplication',
|
||||
expr: |||
|
||||
windows_ad_replication_pending_operations{%(windowsExporterSelector)s} > %(adPendingReplicationThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Active Directory replication queue is high',
|
||||
description: 'Pending replication operations on {{ $labels.instance }} have exceeded %(adPendingReplicationThreshold)s for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsADReplicationFailures',
|
||||
expr: |||
|
||||
(rate(windows_ad_replication_sync_requests_total{%(windowsExporterSelector)s}[%(rateInterval)s]) - rate(windows_ad_replication_sync_requests_success_total{%(windowsExporterSelector)s}[%(rateInterval)s])) > %(adReplicationFailureRateThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Active Directory replication requests are failing',
|
||||
description: 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -1,158 +1,28 @@
|
||||
// One file per collector, each adding one rule group. To add alerts for a
|
||||
// collector, copy one of these files, name it after the collector and import
|
||||
// it here; the group order below is the order in the generated file.
|
||||
{
|
||||
local c = self._config,
|
||||
local q = (import '../lib/queries.libsonnet')(c),
|
||||
local alert(name, expr, duration, severity, summary, description) = {
|
||||
alert: name,
|
||||
expr: expr,
|
||||
'for': duration,
|
||||
labels: { severity: severity },
|
||||
annotations: { summary: summary, description: description },
|
||||
},
|
||||
prometheusAlerts+:: {
|
||||
groups: [
|
||||
{
|
||||
name: 'windows-exporter-alerts',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsExporterDown',
|
||||
q.metric('up') + ' == 0',
|
||||
'5m',
|
||||
c.criticalSeverity,
|
||||
'Windows exporter is unreachable',
|
||||
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).'
|
||||
),
|
||||
alert(
|
||||
'WindowsCollectorFailed',
|
||||
q.metric('windows_exporter_collector_success') + ' == 0',
|
||||
'5m',
|
||||
'warning',
|
||||
'Windows collector failed',
|
||||
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsCPUHighUsage',
|
||||
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows CPU usage is high',
|
||||
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryHighUsage',
|
||||
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory usage is high',
|
||||
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryCommitHighUsage',
|
||||
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory commit usage is high',
|
||||
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold
|
||||
),
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDisk%sLatencyHigh' % direction[0],
|
||||
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows disk %s latency is high' % direction[1],
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]]
|
||||
)
|
||||
for direction in [['Read', 'read'], ['Write', 'write']]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsNetwork%s' % outcome[0],
|
||||
'(%s + %s) > %g' % [
|
||||
q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
outcome[2],
|
||||
],
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows network packet %s are high' % outcome[3],
|
||||
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g per second for 15 minutes.' % [outcome[3], outcome[2]]
|
||||
)
|
||||
for outcome in [
|
||||
['Errors', 'errors', c.networkErrorRateThreshold, 'errors'],
|
||||
['Discards', 'discarded', c.networkDiscardRateThreshold, 'discards'],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskAlmostFull',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
|
||||
'15m',
|
||||
entry[0],
|
||||
'Windows disk space is low',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1]
|
||||
)
|
||||
for entry in [
|
||||
['warning', c.diskFreeWarningThreshold],
|
||||
[c.criticalSeverity, c.diskFreeCriticalThreshold],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskFillingUp',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
|
||||
c.diskFreeWarningThreshold,
|
||||
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
|
||||
c.diskPredictionWindow,
|
||||
c.diskPredictionHours * 3600,
|
||||
],
|
||||
'1h',
|
||||
'warning',
|
||||
'Windows disk is predicted to fill up',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours
|
||||
),
|
||||
],
|
||||
},
|
||||
] + (if c.enableActiveDirectory then [
|
||||
{
|
||||
name: 'windows-active-directory',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsADPendingReplication',
|
||||
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication queue is high',
|
||||
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsADReplicationFailures',
|
||||
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication requests are failing',
|
||||
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.'
|
||||
),
|
||||
],
|
||||
},
|
||||
] else []) + if c.enableTime then [
|
||||
{
|
||||
name: 'windows-time',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsClockNotSynchronising',
|
||||
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock has no NTP time source',
|
||||
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsClockSkewDetected',
|
||||
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock offset is high',
|
||||
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds
|
||||
),
|
||||
],
|
||||
},
|
||||
] else [],
|
||||
},
|
||||
local known = std.objectFields((import '../config.libsonnet')._config.alerts),
|
||||
local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
|
||||
assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
|
||||
}
|
||||
+ (import 'exporter.libsonnet')
|
||||
+ (import 'cpu.libsonnet')
|
||||
+ (import 'memory.libsonnet')
|
||||
+ (import 'pagefile.libsonnet')
|
||||
+ (import 'logical_disk.libsonnet')
|
||||
+ (import 'diskdrive.libsonnet')
|
||||
+ (import 'net.libsonnet')
|
||||
+ (import 'service.libsonnet')
|
||||
+ (import 'system.libsonnet')
|
||||
+ (import 'license.libsonnet')
|
||||
+ (import 'ad.libsonnet')
|
||||
+ (import 'time.libsonnet')
|
||||
+ (import 'scheduled_task.libsonnet')
|
||||
+ (import 'update.libsonnet')
|
||||
+ (import 'hyperv.libsonnet')
|
||||
+ (import 'mscluster.libsonnet')
|
||||
+ (import 'storage_spaces.libsonnet')
|
||||
+ (import 'dhcp.libsonnet')
|
||||
+ (import 'dns.libsonnet')
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-cpu', collectors=['cpu'], rules=[
|
||||
{
|
||||
alert: 'WindowsCPUHighUsage',
|
||||
expr: |||
|
||||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows CPU usage is high',
|
||||
description: 'CPU usage on {{ $labels.instance }} has exceeded %(cpuHighUsageThreshold)s%% for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCPUSaturation',
|
||||
expr: |||
|
||||
windows:cpu_queue_per_core:ratio{%(windowsExporterSelector)s} > %(cpuQueuePerCoreThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows processor queue is long',
|
||||
description: 'More than %(cpuQueuePerCoreThreshold)s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c,
|
||||
},
|
||||
collectors:: ['cpu', 'system'],
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-dhcp', collectors=['dhcp'], rules=[
|
||||
// Percent of free addresses in a scope; a scope without addresses has
|
||||
// no sample, and only scopes in an enabled state count.
|
||||
{
|
||||
alert: 'WindowsDHCPScopeAlmostExhausted',
|
||||
expr: |||
|
||||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeWarningThreshold)s
|
||||
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'DHCP scope is almost exhausted',
|
||||
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeWarningThreshold)s%% free addresses for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDHCPScopeAlmostExhausted',
|
||||
expr: |||
|
||||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeCriticalThreshold)s
|
||||
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'DHCP scope is almost exhausted',
|
||||
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeCriticalThreshold)s%% free addresses for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-diskdrive', collectors=['diskdrive'], rules=[
|
||||
{
|
||||
alert: 'WindowsDiskDriveNotOK',
|
||||
// One series per status with the current status set to 1, so the
|
||||
// alert carries the status the drive reports. Some USB and virtual
|
||||
// drives report Unknown for good.
|
||||
expr: |||
|
||||
windows_diskdrive_status{%(windowsExporterSelector)s, status!~"OK|Unknown"} == 1
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows disk drive is not healthy',
|
||||
description: 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-dns', collectors=['dns'], rules=[
|
||||
{
|
||||
alert: 'WindowsDNSZoneTransferFailures',
|
||||
expr: |||
|
||||
rate(windows_dns_zone_transfer_failures_total{%(windowsExporterSelector)s}[%(rateInterval)s]) > %(dnsZoneTransferFailureRateThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'DNS zone transfers are failing',
|
||||
description: 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-exporter', collectors=[], rules=[
|
||||
{
|
||||
alert: 'WindowsExporterDown',
|
||||
expr: |||
|
||||
up{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows exporter is unreachable',
|
||||
description: 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCollectorFailed',
|
||||
expr: |||
|
||||
windows_exporter_collector_success{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows collector failed',
|
||||
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCollectorTimeout',
|
||||
expr: |||
|
||||
windows_exporter_collector_timeout{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows collector exceeds the scrape timeout',
|
||||
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsScrapeQueueTimeout',
|
||||
expr: |||
|
||||
windows_exporter_scrape_queue_timeout{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows exporter scrapes time out in the queue',
|
||||
description: 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-hyperv', collectors=['hyperv'], rules=[
|
||||
{
|
||||
alert: 'WindowsHyperVWMIUnhealthy',
|
||||
expr: |||
|
||||
windows_hyperv_wmi_health{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Hyper-V WMI namespace is not responding',
|
||||
description: 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsHyperVReplicationCritical',
|
||||
expr: |||
|
||||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 3
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Hyper-V replication is critical',
|
||||
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsHyperVReplicationWarning',
|
||||
expr: |||
|
||||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Hyper-V replication has a warning',
|
||||
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-license', collectors=['license'], rules=[
|
||||
{
|
||||
alert: 'WindowsLicenseNotGenuine',
|
||||
expr: |||
|
||||
windows_license_status{%(windowsExporterSelector)s, state!="genuine"} == 1
|
||||
||| % c,
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows license is not genuine',
|
||||
description: 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-disk', collectors=['logical_disk'], rules=[
|
||||
{
|
||||
alert: 'WindowsDiskReadLatencyHigh',
|
||||
expr: |||
|
||||
windows:logical_disk_read_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk read latency is high',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per read operation for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskWriteLatencyHigh',
|
||||
expr: |||
|
||||
windows:logical_disk_write_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk write latency is high',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per write operation for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskAlmostFull',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk space is low',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeWarningThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskAlmostFull',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeCriticalThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows disk space is low',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeCriticalThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskFillingUp',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
|
||||
and predict_linear(windows_logical_disk_free_bytes{%(windowsExporterSelector)s, %(volumeSelector)s}[%(diskPredictionWindow)s], %(diskPredictionSeconds)s) < 0
|
||||
||| % c { diskPredictionSeconds: c.diskPredictionHours * 3600 },
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk is predicted to fill up',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %(diskPredictionHours)s hours.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-memory', collectors=['memory'], rules=[
|
||||
{
|
||||
alert: 'WindowsMemoryHighUsage',
|
||||
expr: |||
|
||||
windows:memory_usage:ratio{%(windowsExporterSelector)s} * 100 > %(memoryHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory usage is high',
|
||||
description: 'Physical memory usage on {{ $labels.instance }} has exceeded %(memoryHighUsageThreshold)s%% for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsMemoryCommitHighUsage',
|
||||
expr: |||
|
||||
windows:memory_committed:ratio{%(windowsExporterSelector)s} * 100 > %(memoryCommitHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory commit usage is high',
|
||||
description: 'Committed memory on {{ $labels.instance }} has exceeded %(memoryCommitHighUsageThreshold)s%% of the commit limit for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsMemoryPagingHigh',
|
||||
expr: |||
|
||||
windows:memory_swap_page_operations:rate{%(windowsExporterSelector)s} > %(memoryPagingThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory paging is high',
|
||||
description: 'Paging on {{ $labels.instance }} has exceeded %(memoryPagingThreshold)s pages per second for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
// Every cluster node reports the state of the whole cluster, so these
|
||||
// alerts fire once per node; group them by cluster labels in Alertmanager.
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-cluster', collectors=['mscluster'], rules=[
|
||||
{
|
||||
alert: 'WindowsClusterNodeDown',
|
||||
expr: |||
|
||||
windows_mscluster_node_state{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster node is down',
|
||||
description: 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterResourceFailed',
|
||||
expr: |||
|
||||
windows_mscluster_resource_state{%(windowsExporterSelector)s} == 4
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster resource has failed',
|
||||
description: 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterResourceGroupFailed',
|
||||
expr: |||
|
||||
windows_mscluster_resourcegroup_state{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster resource group has failed',
|
||||
description: 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterSharedVolumeAlmostFull',
|
||||
expr: |||
|
||||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeWarningThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Cluster shared volume is almost full',
|
||||
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeWarningThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterSharedVolumeAlmostFull',
|
||||
expr: |||
|
||||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeCriticalThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Cluster shared volume is almost full',
|
||||
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeCriticalThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-network', collectors=['net'], rules=[
|
||||
{
|
||||
alert: 'WindowsNetworkErrors',
|
||||
expr: |||
|
||||
(
|
||||
(windows:net_received_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s})
|
||||
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
|
||||
) * 100 > %(networkErrorRatioThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network packet errors are high',
|
||||
description: 'Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkErrorRatioThreshold)g%% of all packets for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkDiscards',
|
||||
expr: |||
|
||||
(
|
||||
(windows:net_received_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s})
|
||||
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
|
||||
) * 100 > %(networkDiscardRatioThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network packet discards are high',
|
||||
description: 'Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkDiscardRatioThreshold)g%% of all packets for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkUtilizationHigh',
|
||||
expr: |||
|
||||
(windows:net_received_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
|
||||
or (windows:net_sent_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network interface utilization is high',
|
||||
description: 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %(networkUtilizationThreshold)s%% of its bandwidth in at least one direction for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkInterfaceFlapping',
|
||||
expr: |||
|
||||
max without (status) (changes(windows_net_nic_operation_status{%(windowsExporterSelector)s, %(nicSelector)s, status="up"}[%(rateInterval)s])) > 2
|
||||
||| % c,
|
||||
'for': '2m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network interface is flapping',
|
||||
description: 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %(rateInterval)s.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-pagefile', collectors=['pagefile'], rules=[
|
||||
{
|
||||
alert: 'WindowsPagefileAlmostFull',
|
||||
expr: |||
|
||||
(1 - (windows_pagefile_free_bytes{%(windowsExporterSelector)s} / (windows_pagefile_limit_bytes{%(windowsExporterSelector)s} > 0))) * 100 > %(pagefileUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows page file is almost full',
|
||||
description: 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %(pagefileUsageThreshold)s%% in use for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-scheduled-task', collectors=['scheduled_task'], rules=[
|
||||
{
|
||||
alert: 'WindowsScheduledTaskFailed',
|
||||
// unknown is any result code without a named status, including the
|
||||
// exit codes of the task's program.
|
||||
// A disabled task keeps its last result, so disabling a broken task
|
||||
// clears the alert.
|
||||
expr: |||
|
||||
windows_scheduled_task_last_result_status{%(windowsExporterSelector)s, %(taskSelector)s, status=~"unknown|terminated"} == 1
|
||||
unless ignoring (status, state) (windows_scheduled_task_state{%(windowsExporterSelector)s, %(taskSelector)s, state="disabled"} == 1)
|
||||
||| % c,
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows scheduled task did not succeed',
|
||||
description: 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-service', collectors=['service'], rules=[
|
||||
{
|
||||
alert: 'WindowsServiceNotRunning',
|
||||
// The matches ignore only the labels of the right-hand metric, so
|
||||
// every target label, including ones a selector adds, takes part.
|
||||
expr: |||
|
||||
max without (state) (windows_service_state{%(windowsExporterSelector)s, %(serviceSelector)s, state="running"} == 0)
|
||||
and ignoring (start_mode) (windows_service_start_mode{%(windowsExporterSelector)s, %(serviceSelector)s, start_mode="auto"} == 1)
|
||||
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{%(windowsExporterSelector)s, %(serviceSelector)s, trigger_start="true"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows automatic service is not running',
|
||||
description: 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsServiceCrashLooping',
|
||||
// Every service process has its own series, so the series seen in the
|
||||
// window count the processes the service ran as.
|
||||
expr: |||
|
||||
count without (process_id) (count_over_time(windows_service_process{%(windowsExporterSelector)s, %(serviceSelector)s}[15m])) - 1 > %(serviceRestartThreshold)s
|
||||
||| % c,
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows service is restarting repeatedly',
|
||||
description: 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %(serviceRestartThreshold)s times in 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// On Storage Spaces Direct every node reports the clustered pools and
|
||||
// virtual disks, so these alerts fire once per node as well.
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-storage-spaces', collectors=['storage_spaces'], rules=[
|
||||
{
|
||||
alert: 'WindowsStoragePoolUnhealthy',
|
||||
// The primordial pool holds the disks that are not in a concrete pool
|
||||
// yet; only the info metric tells it apart.
|
||||
expr: |||
|
||||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 2)
|
||||
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Storage pool is unhealthy',
|
||||
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStoragePoolUnhealthy',
|
||||
expr: |||
|
||||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 1)
|
||||
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Storage pool has a health warning',
|
||||
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStorageVirtualDiskUnhealthy',
|
||||
expr: |||
|
||||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Storage Spaces virtual disk is unhealthy',
|
||||
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStorageVirtualDiskUnhealthy',
|
||||
expr: |||
|
||||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Storage Spaces virtual disk has a health warning',
|
||||
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-system', collectors=['system'], rules=[
|
||||
{
|
||||
alert: 'WindowsHostRebooted',
|
||||
// Fires once per reboot: for the first ten minutes after the boot,
|
||||
// and only on a host that was monitored before it. The exporter is
|
||||
// down while the host reboots, so the second operand takes the last
|
||||
// boot time of the hour before, not the sample of ten minutes ago.
|
||||
expr: |||
|
||||
(time() - windows_system_boot_time_timestamp{%(windowsExporterSelector)s}) < 600
|
||||
and (time() - last_over_time(windows_system_boot_time_timestamp{%(windowsExporterSelector)s}[1h] offset 10m)) > 600
|
||||
||| % c,
|
||||
labels: { severity: c.infoSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows host has rebooted',
|
||||
description: '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-time', collectors=['time'], rules=[
|
||||
{
|
||||
alert: 'WindowsClockNotSynchronising',
|
||||
expr: |||
|
||||
windows_time_ntp_client_time_sources{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock has no NTP time source',
|
||||
description: 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClockSkewDetected',
|
||||
expr: |||
|
||||
abs(windows_time_computed_time_offset_seconds{%(windowsExporterSelector)s}) > %(clockOffsetThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock offset is high',
|
||||
description: 'Windows Time on {{ $labels.instance }} reports a clock offset above %(clockOffsetThresholdSeconds)g seconds for 10 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClockSourceLocal',
|
||||
// Covers domain members (NT5DS) too, which the NTP source count misses.
|
||||
expr: |||
|
||||
windows_time_clock_sync_source{%(windowsExporterSelector)s, type=~"Local CMOS Clock|NoSync"} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock has no external time source',
|
||||
description: 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-update', collectors=['update'], rules=[
|
||||
{
|
||||
alert: 'WindowsUpdatesPending',
|
||||
expr: |||
|
||||
count without (id, revision, category, severity, title) (windows_update_pending_info{%(windowsExporterSelector)s, deployment_action="installation"}) > 0
|
||||
||| % c,
|
||||
'for': c.updatePendingDuration,
|
||||
labels: { severity: c.infoSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows updates are waiting for installation',
|
||||
description: '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %(updatePendingDuration)s.' % c,
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -4,13 +4,12 @@
|
||||
windowsExporterSelector: 'job="windows_exporter"',
|
||||
volumeSelector: 'volume!=""',
|
||||
nicSelector: 'nic!=""',
|
||||
serviceSelector: 'service!=""',
|
||||
taskSelector: 'task!=""',
|
||||
// ClusterPerformanceHistory is the volume of the cluster's own health history.
|
||||
sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"',
|
||||
rateInterval: '5m',
|
||||
|
||||
// Opt in only for targets with the ad collector enabled.
|
||||
enableActiveDirectory: false,
|
||||
// Requires the time collector with its ntp subcollector enabled.
|
||||
enableTime: false,
|
||||
|
||||
// Grafana dashboard metadata. A UID is the v2 metadata.name; change the
|
||||
// UIDs to import the dashboards next to the sample ones.
|
||||
// Every tab also renders as a dashboard of its own with the UID
|
||||
@@ -37,18 +36,25 @@
|
||||
|
||||
// Collectors enabled on the windows_exporter targets, by collector name.
|
||||
// The dashboard only renders the tabs, rows and panels of enabled
|
||||
// collectors. The defaults match the exporter's default collectors.
|
||||
// collectors, and the alerts only the rules whose collectors are enabled.
|
||||
// The defaults match the exporter's default collectors.
|
||||
collectors: {
|
||||
// Only for targets with the ad collector enabled, such as domain controllers.
|
||||
ad: false,
|
||||
cpu: true,
|
||||
dhcp: false,
|
||||
diskdrive: false,
|
||||
dns: false,
|
||||
gpu: false,
|
||||
hyperv: false,
|
||||
license: false,
|
||||
logical_disk: true,
|
||||
memory: true,
|
||||
mscluster: false,
|
||||
mssql: false,
|
||||
net: true,
|
||||
os: true,
|
||||
pagefile: false,
|
||||
physical_disk: true,
|
||||
process: false,
|
||||
scheduled_task: false,
|
||||
@@ -58,28 +64,101 @@
|
||||
storage_spaces: false,
|
||||
system: true,
|
||||
tcp: false,
|
||||
time: $._config.enableTime,
|
||||
// Needs the time collector with its ntp and clock_source subcollectors.
|
||||
time: false,
|
||||
udp: false,
|
||||
update: false,
|
||||
},
|
||||
|
||||
// Alerts by name. Set one to false to leave it out of the generated
|
||||
// rules, for example alerts+: { WindowsHostRebooted: false }. Names that
|
||||
// are not listed here fail the build; names left out of a replaced object
|
||||
// stay enabled.
|
||||
alerts: {
|
||||
WindowsExporterDown: true,
|
||||
WindowsCollectorFailed: true,
|
||||
WindowsCollectorTimeout: true,
|
||||
WindowsScrapeQueueTimeout: true,
|
||||
WindowsCPUHighUsage: true,
|
||||
WindowsCPUSaturation: true,
|
||||
WindowsMemoryHighUsage: true,
|
||||
WindowsMemoryCommitHighUsage: true,
|
||||
WindowsMemoryPagingHigh: true,
|
||||
WindowsDiskReadLatencyHigh: true,
|
||||
WindowsDiskWriteLatencyHigh: true,
|
||||
WindowsDiskAlmostFull: true,
|
||||
WindowsDiskFillingUp: true,
|
||||
WindowsNetworkErrors: true,
|
||||
WindowsNetworkDiscards: true,
|
||||
WindowsNetworkUtilizationHigh: true,
|
||||
WindowsNetworkInterfaceFlapping: true,
|
||||
WindowsServiceNotRunning: true,
|
||||
WindowsServiceCrashLooping: true,
|
||||
WindowsHostRebooted: true,
|
||||
WindowsADPendingReplication: true,
|
||||
WindowsADReplicationFailures: true,
|
||||
WindowsClockNotSynchronising: true,
|
||||
WindowsClockSkewDetected: true,
|
||||
WindowsClockSourceLocal: true,
|
||||
WindowsDiskDriveNotOK: true,
|
||||
WindowsPagefileAlmostFull: true,
|
||||
WindowsLicenseNotGenuine: true,
|
||||
WindowsScheduledTaskFailed: true,
|
||||
WindowsUpdatesPending: true,
|
||||
WindowsHyperVWMIUnhealthy: true,
|
||||
WindowsHyperVReplicationCritical: true,
|
||||
WindowsHyperVReplicationWarning: true,
|
||||
WindowsClusterNodeDown: true,
|
||||
WindowsClusterResourceFailed: true,
|
||||
WindowsClusterResourceGroupFailed: true,
|
||||
WindowsClusterSharedVolumeAlmostFull: true,
|
||||
WindowsStoragePoolUnhealthy: true,
|
||||
WindowsStorageVirtualDiskUnhealthy: true,
|
||||
WindowsDHCPScopeAlmostExhausted: true,
|
||||
WindowsDNSZoneTransferFailures: true,
|
||||
},
|
||||
|
||||
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
|
||||
|
||||
// Threshold alerts keep firing this long after their condition clears,
|
||||
// so a value oscillating around a threshold does not flap. Needs
|
||||
// Prometheus 2.42 or later; set it to '' to leave keep_firing_for out.
|
||||
keepFiringFor: '5m',
|
||||
|
||||
cpuHighUsageThreshold: 90,
|
||||
// Threads waiting for a processor, per logical processor.
|
||||
cpuQueuePerCoreThreshold: 2,
|
||||
memoryHighUsageThreshold: 90,
|
||||
memoryCommitHighUsageThreshold: 90,
|
||||
// Pages read from or written to disk per second.
|
||||
memoryPagingThreshold: 1000,
|
||||
diskLatencyThresholdSeconds: 0.05,
|
||||
// Average latency per I/O above which a Hyper-V virtual disk counts as slow.
|
||||
hypervDiskLatencyThresholdSeconds: 0.01,
|
||||
networkErrorRateThreshold: 1,
|
||||
networkDiscardRateThreshold: 1,
|
||||
// Percent of all packets on the interface.
|
||||
networkErrorRatioThreshold: 1,
|
||||
networkDiscardRatioThreshold: 1,
|
||||
// Percent of the interface bandwidth, in either direction.
|
||||
networkUtilizationThreshold: 90,
|
||||
clockOffsetThresholdSeconds: 0.05,
|
||||
diskFreeWarningThreshold: 10,
|
||||
diskFreeCriticalThreshold: 5,
|
||||
diskPredictionWindow: '6h',
|
||||
diskPredictionHours: 24,
|
||||
// Restarts of one service within 15 minutes.
|
||||
serviceRestartThreshold: 2,
|
||||
criticalSeverity: 'critical',
|
||||
infoSeverity: 'info',
|
||||
adPendingReplicationThreshold: 50,
|
||||
adReplicationFailureRateThreshold: 0,
|
||||
// Percent of the page file in use.
|
||||
pagefileUsageThreshold: 80,
|
||||
// Percent of free addresses in an active DHCP scope.
|
||||
dhcpScopeFreeWarningThreshold: 10,
|
||||
dhcpScopeFreeCriticalThreshold: 2,
|
||||
// How long updates may wait for installation before WindowsUpdatesPending fires.
|
||||
updatePendingDuration: '7d',
|
||||
// Failed zone transfers per second.
|
||||
dnsZoneTransferFailureRateThreshold: 0,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// Renders the sample dashboards into ../../dashboard. Run with
|
||||
// Renders the sample dashboards into ../dashboard. Run with
|
||||
// jsonnet -J vendor -S -m <directory>. The sample dashboards show every
|
||||
// collector; mixin users select theirs with _config.collectors.
|
||||
local manifest = import 'lib/manifest.libsonnet';
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
// Helpers shared by the alert files in alerts/. Takes the mixin _config.
|
||||
function(c)
|
||||
{
|
||||
// A collector missing from a replaced collectors object counts as disabled;
|
||||
// an alert missing from a replaced alerts object counts as enabled.
|
||||
collectorEnabled(collector): std.get(c.collectors, collector, false),
|
||||
alertEnabled(name): std.get(c.alerts, name, true),
|
||||
// group renders the rules of one collector as a rule group, or nothing
|
||||
// when none of them is enabled. Each rule is a plain Prometheus alerting
|
||||
// rule plus two hidden fields: collectors:: lists the collectors the rule
|
||||
// reads when it differs from the group's, and state:: true marks state
|
||||
// alerts (== 0, == 1) and one-shot alerts, which must resolve at once.
|
||||
// Every other alert is a threshold alert and keeps firing for
|
||||
// keepFiringFor after its condition clears, so a value oscillating around
|
||||
// the threshold does not flap.
|
||||
group(name, collectors, rules):
|
||||
local enabled = [
|
||||
rule + (if c.keepFiringFor != '' && !std.get(rule, 'state', false) then { keep_firing_for: c.keepFiringFor } else {})
|
||||
for rule in rules
|
||||
if self.alertEnabled(rule.alert) && std.all([self.collectorEnabled(collector) for collector in std.get(rule, 'collectors', collectors)])
|
||||
];
|
||||
if std.length(enabled) > 0 then [{ name: name, rules: enabled }] else [],
|
||||
}
|
||||
@@ -7,6 +7,13 @@ function(c)
|
||||
ratio(numerator, denominator): '(%s / (%s > 0))' % [numerator, denominator],
|
||||
cpuUsage:
|
||||
'1 - avg without (core, mode) (%s)' % self.rate('windows_cpu_time_seconds_total', ['mode="idle"']),
|
||||
// Threads waiting for a processor, averaged over the rate interval, per
|
||||
// logical processor. The queue length is an instantaneous sample.
|
||||
cpuQueuePerCore:
|
||||
self.ratio(
|
||||
'avg_over_time(%s[%s])' % [self.metric('windows_system_processor_queue_length'), c.rateInterval],
|
||||
self.metric('windows:cpu_cores:count')
|
||||
),
|
||||
memoryUsage:
|
||||
'1 - %s' % self.ratio(
|
||||
self.metric('windows_memory_available_bytes'),
|
||||
|
||||
+23
-12
@@ -8,7 +8,8 @@
|
||||
[vars]
|
||||
sources = "*.libsonnet *.jsonnet lib/*.libsonnet rules/*.libsonnet alerts/*.libsonnet dashboards/*.libsonnet tests/*.libsonnet tests/*.jsonnet"
|
||||
fmt_flags = "-n 2 --max-blank-lines 2 --string-style s --comment-style s"
|
||||
dashboards = "../../dashboard"
|
||||
dashboards = "../dashboard"
|
||||
rules = "../monitoring"
|
||||
|
||||
[tasks.generate]
|
||||
description = "Generate Prometheus rules, alerts, and the dashboard"
|
||||
@@ -16,10 +17,10 @@ alias = "default"
|
||||
depends = ["rules", "dashboard"]
|
||||
|
||||
[tasks.rules]
|
||||
description = "Generate Prometheus rules and alerts"
|
||||
description = "Generate the Prometheus rules and alerts in contrib/monitoring"
|
||||
run = [
|
||||
"jsonnet -S rules.jsonnet > windows_rules.yaml",
|
||||
"jsonnet -S alerts.jsonnet > windows_alerts.yaml",
|
||||
"jsonnet -S rules.jsonnet > {{vars.rules}}/windows_rules.yaml",
|
||||
"jsonnet -S alerts.jsonnet > {{vars.rules}}/windows_alerts.yaml",
|
||||
]
|
||||
|
||||
[tasks.vendor]
|
||||
@@ -39,6 +40,15 @@ rm -f {{vars.dashboards}}/windows-exporter-*.json
|
||||
cp "$out"/*.json {{vars.dashboards}}/
|
||||
'''
|
||||
|
||||
[tasks.check-rules]
|
||||
description = "Regenerate the rules and alerts in contrib/monitoring and fail on uncommitted changes"
|
||||
depends = ["rules"]
|
||||
run = '''
|
||||
set -eu
|
||||
git add --intent-to-add -- {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml
|
||||
git diff --exit-code -- {{vars.rules}}
|
||||
'''
|
||||
|
||||
[tasks.check-dashboards]
|
||||
description = "Regenerate the dashboards and fail on uncommitted changes"
|
||||
depends = ["dashboard"]
|
||||
@@ -65,8 +75,8 @@ run = "jsonnetfmt {{vars.fmt_flags}} -i {{vars.sources}}"
|
||||
description = "Generate rules and alerts with the test configuration"
|
||||
hide = true
|
||||
run = [
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules)' > tests/custom_rules.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts)' > tests/custom_alerts.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules, quote_keys=false)' > tests/custom_rules.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts, quote_keys=false)' > tests/custom_alerts.yaml''',
|
||||
]
|
||||
|
||||
[tasks.lint]
|
||||
@@ -88,25 +98,26 @@ for file in $(cat "$out/generated"); do
|
||||
diff -u {{vars.dashboards}}/"$file" "$out/$file" \
|
||||
|| { echo "{{vars.dashboards}}/$file is out of date; run mise run dashboard"; exit 1; }
|
||||
done
|
||||
promtool check rules windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom
|
||||
pint --offline lint windows_rules.yaml windows_alerts.yaml
|
||||
promtool check rules {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom
|
||||
pint --offline lint {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml
|
||||
pint --offline lint tests/custom_rules.yaml tests/custom_alerts.yaml
|
||||
'''
|
||||
|
||||
[tasks.check-runbooks]
|
||||
description = "Check runbook coverage for all alerts, including opt-in alerts"
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { enableActiveDirectory: true, enableTime: true } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [name]: true for name in std.objectFields(super.collectors) } } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
|
||||
[tasks.test]
|
||||
description = "Run all mixin lint, runbook, and rule tests"
|
||||
depends = ["lint", "check-runbooks", "vendor"]
|
||||
depends = ["lint", "check-rules", "check-runbooks", "vendor"]
|
||||
run = [
|
||||
"jsonnet tests/runbooks.test.jsonnet",
|
||||
"jsonnet tests/hyperv.test.jsonnet > /dev/null",
|
||||
"jsonnet tests/alerts.test.jsonnet",
|
||||
"jsonnet -J vendor tests/dashboard.test.jsonnet > /dev/null",
|
||||
"promtool test rules tests/*.test.yaml",
|
||||
]
|
||||
|
||||
[tasks.clean]
|
||||
description = "Remove generated rule and alert files"
|
||||
run = "rm -f windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml"
|
||||
description = "Remove the generated test rule and alert files"
|
||||
run = "rm -f tests/custom_rules.yaml tests/custom_alerts.yaml"
|
||||
|
||||
@@ -1 +1 @@
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules)
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules, quote_keys=false)
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
{ record: 'windows:cpu_usage:ratio', expr: q.cpuUsage },
|
||||
{ record: 'windows:cpu_cores:count', expr: 'count without (core, mode) (%s)' % q.metric('windows_cpu_time_seconds_total', ['mode="idle"']) },
|
||||
{ record: 'windows:cpu_time:ratio_rate', expr: 'avg without (core) (%s)' % q.rate('windows_cpu_time_seconds_total') },
|
||||
{ record: 'windows:cpu_queue_per_core:ratio', expr: q.cpuQueuePerCore },
|
||||
{ record: 'windows:memory_usage:ratio', expr: q.memoryUsage },
|
||||
{
|
||||
record: 'windows:memory_used:bytes',
|
||||
@@ -56,6 +57,12 @@
|
||||
expr: q.rate('windows_net_bytes_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{
|
||||
record: 'windows:net_%s_packets:rate' % direction,
|
||||
expr: q.rate('windows_net_packets_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{ record: 'windows:net_%s_utilization:ratio' % direction, expr: q.networkUtilization(direction) }
|
||||
for direction in ['received', 'sent']
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
// Checks the alert configuration: collector gating, per-alert switches,
|
||||
// keep_firing_for and the group layout.
|
||||
local mixin = import '../mixin.libsonnet';
|
||||
local config = (import '../config.libsonnet')._config;
|
||||
local everything = mixin { _config+:: { collectors: { [name]: true for name in std.objectFields(config.collectors) } } };
|
||||
local optionalGroups = ['windows-pagefile', 'windows-diskdrive', 'windows-license', 'windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
|
||||
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
|
||||
local groupNames(alerts) = [group.name for group in alerts.groups];
|
||||
|
||||
// Every configured switch belongs to a generated alert, and every alert has a switch.
|
||||
local all = std.set(names(everything.prometheusAlerts));
|
||||
assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs from generated alerts: %s' % std.manifestJson([std.setDiff(all, std.set(std.objectFields(config.alerts))), std.setDiff(std.set(std.objectFields(config.alerts)), all)]);
|
||||
|
||||
// Groups follow the collector areas; opt-in groups appear only when enabled.
|
||||
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
|
||||
// One file per collector, one group per file: the groups of every enabled
|
||||
// collector come in the order of the imports in alerts/alerts.libsonnet.
|
||||
assert std.set(groupNames(everything.prometheusAlerts)) == std.set(groupNames(mixin.prometheusAlerts) + optionalGroups) : std.manifestJson(groupNames(everything.prometheusAlerts));
|
||||
assert groupNames(everything.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-pagefile', 'windows-disk', 'windows-diskdrive', 'windows-network', 'windows-service', 'windows-system', 'windows-license'] + optionalGroups[3:] : std.manifestJson(groupNames(everything.prometheusAlerts));
|
||||
local adOnly = mixin { _config+:: { collectors+: { ad: true } } };
|
||||
assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory'];
|
||||
local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } };
|
||||
assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull'];
|
||||
|
||||
// Disabling a collector removes its alerts and an empty group; the alerts of a
|
||||
// rule that needs two collectors go with either.
|
||||
local noNet = mixin { _config+:: { collectors+: { net: false } } };
|
||||
assert !std.member(groupNames(noNet.prometheusAlerts), 'windows-network');
|
||||
local noSystem = mixin { _config+:: { collectors+: { system: false } } };
|
||||
assert names(noSystem.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUSaturation', 'WindowsHostRebooted']);
|
||||
assert std.member(names(noSystem.prometheusAlerts), 'WindowsCPUHighUsage');
|
||||
// A replaced collectors object counts missing collectors as disabled.
|
||||
local onlyCPU = mixin { _config+:: { collectors: { cpu: true } } };
|
||||
assert names(onlyCPU.prometheusAlerts) == ['WindowsCPUHighUsage', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsExporterDown', 'WindowsScrapeQueueTimeout'] : std.manifestJson(names(onlyCPU.prometheusAlerts));
|
||||
|
||||
// Single alerts switch off by name, with alerts+: or a replaced object.
|
||||
local noReboot = mixin { _config+:: { alerts+: { WindowsHostRebooted: false } } };
|
||||
assert names(noReboot.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsHostRebooted']);
|
||||
assert !std.member(groupNames(noReboot.prometheusAlerts), 'windows-system');
|
||||
local replaced = mixin { _config+:: { alerts: { WindowsCPUHighUsage: false } } };
|
||||
assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUHighUsage']);
|
||||
|
||||
// keep_firing_for applies to threshold alerts only and can be turned off.
|
||||
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
|
||||
local stateAlerts = [
|
||||
'WindowsExporterDown',
|
||||
'WindowsCollectorFailed',
|
||||
'WindowsCollectorTimeout',
|
||||
'WindowsScrapeQueueTimeout',
|
||||
'WindowsServiceNotRunning',
|
||||
'WindowsHostRebooted',
|
||||
'WindowsClockNotSynchronising',
|
||||
'WindowsClockSourceLocal',
|
||||
'WindowsDiskDriveNotOK',
|
||||
'WindowsLicenseNotGenuine',
|
||||
'WindowsScheduledTaskFailed',
|
||||
'WindowsUpdatesPending',
|
||||
'WindowsHyperVWMIUnhealthy',
|
||||
'WindowsHyperVReplicationCritical',
|
||||
'WindowsHyperVReplicationWarning',
|
||||
'WindowsClusterNodeDown',
|
||||
'WindowsClusterResourceFailed',
|
||||
'WindowsClusterResourceGroupFailed',
|
||||
'WindowsStoragePoolUnhealthy',
|
||||
'WindowsStorageVirtualDiskUnhealthy',
|
||||
];
|
||||
assert std.all([
|
||||
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
|
||||
for rule in rules(everything.prometheusAlerts)
|
||||
]) : 'keep_firing_for set on a state alert or missing on a threshold alert';
|
||||
assert std.all([rule.keep_firing_for == '5m' for rule in rules(everything.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]);
|
||||
local noKeep = mixin { _config+:: { keepFiringFor: '' } };
|
||||
assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]) == 0;
|
||||
|
||||
// The info severity label is configurable.
|
||||
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
|
||||
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
|
||||
local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } };
|
||||
assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']];
|
||||
|
||||
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
|
||||
@@ -1,10 +1,18 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../windows_alerts.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
- ../../monitoring/windows_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Commit pressure, disk latency and network failures wait and recover
|
||||
@@ -30,6 +38,10 @@ tests:
|
||||
values: '0x31'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+120x20 2520x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
@@ -40,7 +52,8 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/
|
||||
summary: Windows memory commit usage is high
|
||||
description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 21m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 25m, exp_alerts: [{exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}, exp_annotations: {runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/, summary: Windows memory commit usage is high, description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'}}]}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 26m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskReadLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -50,7 +63,7 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/
|
||||
summary: Windows disk read latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per read operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskWriteLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -60,27 +73,27 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/
|
||||
summary: Windows disk write latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per write operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 17m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkErrors
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 17m, exp_alerts: []}
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 25m, exp_alerts: []}
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 30m, exp_alerts: []}
|
||||
|
||||
- name: New resource alerts stay quiet at exact thresholds and clock alerts are disabled
|
||||
interval: 1m
|
||||
@@ -105,6 +118,10 @@ tests:
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_time_ntp_client_time_sources{job="windows_exporter", instance="host:9182"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_time_computed_time_offset_seconds{job="windows_exporter", instance="host:9182"}'
|
||||
@@ -178,7 +195,7 @@ tests:
|
||||
summary: Windows CPU usage is high
|
||||
description: 'CPU usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsCPUHighUsage
|
||||
eval_time: 25m
|
||||
eval_time: 30m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 14m
|
||||
@@ -192,16 +209,16 @@ tests:
|
||||
summary: Windows memory usage is high
|
||||
description: 'Physical memory usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Disk alerts retain volume labels and fire at both severity levels
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_logical_disk_free_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '3x15 30x5'
|
||||
values: '3x15 30x10'
|
||||
- series: 'windows_logical_disk_size_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '100x21'
|
||||
values: '100x26'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 14m
|
||||
@@ -220,7 +237,7 @@ tests:
|
||||
summary: Windows disk space is low
|
||||
description: 'Volume C: on host:9182 has less than 5% free space. Windows may delay space metrics by 10–15 minutes.'
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 16m
|
||||
eval_time: 21m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Healthy resources at the exact thresholds do not fire
|
||||
@@ -291,3 +308,274 @@ tests:
|
||||
- {alertname: WindowsMemoryHighUsage, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskAlmostFull, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskFillingUp, eval_time: 70m, exp_alerts: []}
|
||||
|
||||
- name: Collector and scrape queue timeouts wait ten minutes and recover without keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="service"}'
|
||||
values: '1x10 0x5'
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="cpu"}'
|
||||
values: '0x15'
|
||||
- series: 'windows_exporter_scrape_queue_timeout{job="windows_exporter", instance="host:9182"}'
|
||||
values: '1x10 0x5'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsCollectorTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', collector: service, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/
|
||||
summary: Windows collector exceeds the scrape timeout
|
||||
description: 'Collector service on host:9182 has not finished within the scrape timeout for 10 minutes.'
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsScrapeQueueTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/
|
||||
summary: Windows exporter scrapes time out in the queue
|
||||
description: 'Scrapes of host:9182 have timed out waiting for an earlier scrape to finish for 10 minutes.'
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 11m, exp_alerts: []}
|
||||
|
||||
- name: CPU saturation and paging use recordings, wait fifteen minutes and keep firing five
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '9x25 0x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_memory_swap_page_operations_total{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '0+90000x20 1800000x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 31m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 32m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 18m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryPagingHigh
|
||||
eval_time: 19m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
|
||||
summary: Windows memory paging is high
|
||||
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
|
||||
- alertname: WindowsMemoryPagingHigh
|
||||
eval_time: 26m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
|
||||
summary: Windows memory paging is high
|
||||
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 27m, exp_alerts: []}
|
||||
|
||||
- name: Network utilization fires in one direction and interface flapping counts status changes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_net_bytes_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+6000x20 120000x15'
|
||||
- series: 'windows_net_bytes_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0x36'
|
||||
- series: 'windows_net_current_bandwidth_bytes{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '100x36'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="up"}'
|
||||
values: '1 0 1 0 1 0 1 0 1 0 1x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="down"}'
|
||||
values: '0 1 0 1 0 1 0 1 0 1 0x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Wi-Fi", status="up"}'
|
||||
values: '0x21'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 19m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkUtilizationHigh
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
|
||||
summary: Windows network interface utilization is high
|
||||
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
|
||||
- alertname: WindowsNetworkUtilizationHigh
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
|
||||
summary: Windows network interface utilization is high
|
||||
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 26m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkInterfaceFlapping
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/
|
||||
summary: Windows network interface is flapping
|
||||
description: 'Interface Ethernet on host:9182 has changed its operational status more than twice in 5m.'
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 17m, exp_alerts: []}
|
||||
|
||||
- name: Service alerts skip trigger-start and manual services and count restarts by process
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}'
|
||||
values: '0x15 1x8'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}'
|
||||
values: '1000x4 _x19'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}'
|
||||
values: '_x5 2000x4 _x14'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}'
|
||||
values: '_x10 3000x4 _x9'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}'
|
||||
values: '_x15 4000x9'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}'
|
||||
values: '_x16 5000x8'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceNotRunning
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: Spooler, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
|
||||
summary: Windows automatic service is not running
|
||||
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 16m, exp_alerts: []}
|
||||
- {alertname: WindowsServiceCrashLooping, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceCrashLooping
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
|
||||
summary: Windows service is restarting repeatedly
|
||||
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
|
||||
- alertname: WindowsServiceCrashLooping
|
||||
eval_time: 23m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
|
||||
summary: Windows service is restarting repeatedly
|
||||
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
|
||||
- {alertname: WindowsServiceCrashLooping, eval_time: 24m, exp_alerts: []}
|
||||
|
||||
- name: Service alert matches within a cluster when instance names repeat across clusters
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", state="running"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", trigger_start="true", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsServiceNotRunning
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, service: Spooler, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
|
||||
summary: Windows automatic service is not running
|
||||
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
|
||||
|
||||
- name: Host reboot fires once for the first ten minutes after a new boot time
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
|
||||
values: '-100000x10 540x15'
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="steady:9182"}'
|
||||
values: '-100000x25'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHostRebooted, eval_time: 10m, exp_alerts: []}
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 11m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 2m 0s ago.'
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 18m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 9m 0s ago.'
|
||||
- {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []}
|
||||
|
||||
- name: Host reboot fires after a scrape gap longer than ten minutes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
|
||||
values: '-100000x5 stale _x11 960x20'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHostRebooted, eval_time: 17m, exp_alerts: []}
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 18m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 2m 0s ago.'
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 9m 0s ago.'
|
||||
- {alertname: WindowsHostRebooted, eval_time: 26m, exp_alerts: []}
|
||||
|
||||
@@ -9,15 +9,33 @@
|
||||
diskFreeWarningThreshold: 20,
|
||||
diskFreeCriticalThreshold: 10,
|
||||
criticalSeverity: 'page',
|
||||
enableActiveDirectory: true,
|
||||
enableTime: true,
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
memoryCommitHighUsageThreshold: 80,
|
||||
diskLatencyThresholdSeconds: 0.02,
|
||||
networkErrorRateThreshold: 0.5,
|
||||
networkDiscardRateThreshold: 0.5,
|
||||
networkErrorRatioThreshold: 0.5,
|
||||
networkDiscardRatioThreshold: 0.5,
|
||||
clockOffsetThresholdSeconds: 0.1,
|
||||
adPendingReplicationThreshold: 10,
|
||||
adReplicationFailureRateThreshold: 0.5,
|
||||
collectors+: {
|
||||
ad: true,
|
||||
dhcp: true,
|
||||
diskdrive: true,
|
||||
dns: true,
|
||||
hyperv: true,
|
||||
license: true,
|
||||
mscluster: true,
|
||||
pagefile: true,
|
||||
scheduled_task: true,
|
||||
storage_spaces: true,
|
||||
time: true,
|
||||
update: true,
|
||||
},
|
||||
pagefileUsageThreshold: 75,
|
||||
dhcpScopeFreeWarningThreshold: 20,
|
||||
dhcpScopeFreeCriticalThreshold: 5,
|
||||
updatePendingDuration: '2d',
|
||||
dnsZoneTransferFailureRateThreshold: 0.05,
|
||||
taskSelector: 'task!~"/Ignored.*"',
|
||||
},
|
||||
}
|
||||
|
||||
@@ -4,7 +4,13 @@ rule_files:
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
@@ -52,6 +58,14 @@ tests:
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
eval_time: 20m
|
||||
@@ -84,7 +98,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
@@ -92,7 +106,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
|
||||
- name: Clock alerts opt in, use absolute offset, custom URLs and recover
|
||||
interval: 1m
|
||||
@@ -129,7 +143,7 @@ tests:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclockskewdetected/
|
||||
summary: Windows clock offset is high
|
||||
description: 'Windows Time on host:9182 reports a clock offset above 0.1 seconds for 10 minutes.'
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 16m, exp_alerts: []}
|
||||
- name: Custom selectors, thresholds and critical severity propagate to alerts
|
||||
interval: 1m
|
||||
input_series:
|
||||
@@ -239,7 +253,7 @@ tests:
|
||||
summary: Active Directory replication queue is high
|
||||
description: 'Pending replication operations on host:9182 have exceeded 10 for 15 minutes.'
|
||||
- alertname: WindowsADPendingReplication
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 16m
|
||||
@@ -253,5 +267,5 @@ tests:
|
||||
summary: Active Directory replication requests are failing
|
||||
description: 'Replication synchronization requests on host:9182 have been failing for 15 minutes.'
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 23m
|
||||
eval_time: 28m
|
||||
exp_alerts: []
|
||||
|
||||
@@ -0,0 +1,508 @@
|
||||
# Alerts of the opt-in collectors, rendered with tests/config.libsonnet.
|
||||
rule_files:
|
||||
- custom_rules.yaml
|
||||
- custom_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-system
|
||||
- windows-time
|
||||
- windows-scheduled-task
|
||||
- windows-update
|
||||
- windows-hyperv
|
||||
- windows-cluster
|
||||
- windows-storage-spaces
|
||||
- windows-dhcp
|
||||
- windows-dns
|
||||
|
||||
tests:
|
||||
- name: Page file usage fires above the custom threshold and resolves after keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
|
||||
values: '20x19 50x10'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
|
||||
values: '30x30'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
|
||||
values: '0x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsPagefileAlmostFull
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
|
||||
summary: Windows page file is almost full
|
||||
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
|
||||
- alertname: WindowsPagefileAlmostFull
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
|
||||
summary: Windows page file is almost full
|
||||
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
|
||||
- {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []}
|
||||
|
||||
- name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}'
|
||||
values: '0x9 1x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}'
|
||||
values: '1x9 0x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}'
|
||||
values: '0x10'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}'
|
||||
values: '0x60'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsDiskDriveNotOK
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
|
||||
summary: Windows disk drive is not healthy
|
||||
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
|
||||
- alertname: WindowsDiskDriveNotOK
|
||||
eval_time: 9m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
|
||||
summary: Windows disk drive is not healthy
|
||||
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
|
||||
- {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []}
|
||||
- {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []}
|
||||
- alertname: WindowsLicenseNotGenuine
|
||||
eval_time: 60m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/
|
||||
summary: Windows license is not genuine
|
||||
description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.'
|
||||
|
||||
- name: Clock source alert names the local source and ignores NTP and domain sources
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}'
|
||||
values: '0x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}'
|
||||
values: '0x10'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsClockSourceLocal
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/
|
||||
summary: Windows clock has no external time source
|
||||
description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.'
|
||||
|
||||
- name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}'
|
||||
values: '1x60'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []}
|
||||
- alertname: WindowsScheduledTaskFailed
|
||||
eval_time: 60m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
|
||||
summary: Windows scheduled task did not succeed
|
||||
description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
|
||||
summary: Windows scheduled task did not succeed
|
||||
description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.'
|
||||
|
||||
- name: Pending updates fire as info after the configured duration and count installations only
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}'
|
||||
values: '1x2880'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []}
|
||||
- alertname: WindowsUpdatesPending
|
||||
eval_time: 48h
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsupdatespending/
|
||||
summary: Windows updates are waiting for installation
|
||||
description: 'host:9182 has had 2 updates waiting for installation for 2d.'
|
||||
|
||||
- name: Hyper-V alerts cover the WMI namespace and both replication health levels
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}'
|
||||
values: '0x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}'
|
||||
values: '3x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}'
|
||||
values: '0x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVWMIUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/
|
||||
summary: Hyper-V WMI namespace is not responding
|
||||
description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.'
|
||||
- {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVReplicationCritical
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/
|
||||
summary: Hyper-V replication is critical
|
||||
description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.'
|
||||
- {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVReplicationWarning
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/
|
||||
summary: Hyper-V replication has a warning
|
||||
description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.'
|
||||
|
||||
- name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}'
|
||||
values: '4x30'
|
||||
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}'
|
||||
values: '3x30'
|
||||
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}'
|
||||
values: '2x30'
|
||||
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
|
||||
values: '15x19 50x10'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
|
||||
values: '5x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
|
||||
values: '0x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsClusterNodeDown
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
|
||||
summary: Failover cluster node is down
|
||||
description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
|
||||
summary: Failover cluster node is down
|
||||
description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.'
|
||||
- alertname: WindowsClusterResourceFailed
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/
|
||||
summary: Failover cluster resource has failed
|
||||
description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterResourceGroupFailed
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/
|
||||
summary: Failover cluster resource group has failed
|
||||
description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
|
||||
- {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
|
||||
- name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}'
|
||||
values: '0x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsStoragePoolUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
|
||||
summary: Storage pool is unhealthy
|
||||
description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
|
||||
summary: Storage pool has a health warning
|
||||
description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.'
|
||||
- alertname: WindowsStorageVirtualDiskUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
|
||||
summary: Storage Spaces virtual disk is unhealthy
|
||||
description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
|
||||
summary: Storage Spaces virtual disk has a health warning
|
||||
description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.'
|
||||
|
||||
- name: DHCP scope exhaustion counts active scopes only and uses both thresholds
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
|
||||
values: '10x19 50x10'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
|
||||
values: '90x19 50x10'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
|
||||
values: '2x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
|
||||
values: '98x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
|
||||
- name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}'
|
||||
values: '0+6x20 120x15'
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}'
|
||||
values: '7x36'
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}'
|
||||
values: '0+1x36'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []}
|
||||
- alertname: WindowsDNSZoneTransferFailures
|
||||
eval_time: 17m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
|
||||
summary: DNS zone transfers are failing
|
||||
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
|
||||
- alertname: WindowsDNSZoneTransferFailures
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
|
||||
summary: DNS zone transfers are failing
|
||||
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
|
||||
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []}
|
||||
@@ -1,10 +1,18 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../windows_alerts.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
- ../../monitoring/windows_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Memory cache includes all lists, commit is a ratio and paging is a rate
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
evaluation_interval: 1m
|
||||
fuzzy_compare: true
|
||||
|
||||
@@ -111,3 +111,44 @@ tests:
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_bytes:rate{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
value: 30
|
||||
|
||||
- name: Processor queue per core averages the queue and packet rates keep the nic label
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '12x5 0x5'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="3", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="nocpu:9182", cluster="a"}'
|
||||
values: '12x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+600x10'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+1200x10'
|
||||
promql_expr_test:
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 3
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 0
|
||||
- expr: windows:net_received_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_received_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 10
|
||||
- expr: windows:net_sent_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 20
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Prometheus rules for windows_exporter
|
||||
|
||||
Ready-to-use Prometheus rule files, rendered from the
|
||||
[mixin](../mixin/README.md) with its default configuration:
|
||||
|
||||
- [`windows_rules.yaml`](windows_rules.yaml): recording rules.
|
||||
- [`windows_alerts.yaml`](windows_alerts.yaml): alerting rules for the default
|
||||
collectors.
|
||||
|
||||
The alerts build on the recording rules, so load both files:
|
||||
|
||||
```yaml
|
||||
rule_files:
|
||||
- /etc/prometheus/rules/windows_rules.yaml
|
||||
- /etc/prometheus/rules/windows_alerts.yaml
|
||||
```
|
||||
|
||||
The files expect the exporter under the job name `windows_exporter`. For other
|
||||
job names, thresholds, selectors or the alerts of opt-in collectors, render
|
||||
the mixin with your own configuration as the
|
||||
[mixin documentation](../mixin/README.md#customize-or-compose) describes. Every alert links
|
||||
to its [runbook](https://prometheus-community.github.io/windows_exporter/runbooks/).
|
||||
|
||||
Do not edit these files by hand: `mise run rules` in `contrib/mixin`
|
||||
regenerates them, and CI fails when they are out of date.
|
||||
@@ -0,0 +1,249 @@
|
||||
groups:
|
||||
- name: "windows-exporter"
|
||||
rules:
|
||||
- alert: "WindowsExporterDown"
|
||||
annotations:
|
||||
description: "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/"
|
||||
summary: "Windows exporter is unreachable"
|
||||
expr: |
|
||||
up{job="windows_exporter"} == 0
|
||||
for: "5m"
|
||||
labels:
|
||||
severity: "critical"
|
||||
- alert: "WindowsCollectorFailed"
|
||||
annotations:
|
||||
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/"
|
||||
summary: "Windows collector failed"
|
||||
expr: |
|
||||
windows_exporter_collector_success{job="windows_exporter"} == 0
|
||||
for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsCollectorTimeout"
|
||||
annotations:
|
||||
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/"
|
||||
summary: "Windows collector exceeds the scrape timeout"
|
||||
expr: |
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
for: "10m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsScrapeQueueTimeout"
|
||||
annotations:
|
||||
description: "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/"
|
||||
summary: "Windows exporter scrapes time out in the queue"
|
||||
expr: |
|
||||
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
|
||||
for: "10m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-cpu"
|
||||
rules:
|
||||
- alert: "WindowsCPUHighUsage"
|
||||
annotations:
|
||||
description: "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/"
|
||||
summary: "Windows CPU usage is high"
|
||||
expr: |
|
||||
windows:cpu_usage:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsCPUSaturation"
|
||||
annotations:
|
||||
description: "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/"
|
||||
summary: "Windows processor queue is long"
|
||||
expr: |
|
||||
windows:cpu_queue_per_core:ratio{job="windows_exporter"} > 2
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-memory"
|
||||
rules:
|
||||
- alert: "WindowsMemoryHighUsage"
|
||||
annotations:
|
||||
description: "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/"
|
||||
summary: "Windows memory usage is high"
|
||||
expr: |
|
||||
windows:memory_usage:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsMemoryCommitHighUsage"
|
||||
annotations:
|
||||
description: "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/"
|
||||
summary: "Windows memory commit usage is high"
|
||||
expr: |
|
||||
windows:memory_committed:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsMemoryPagingHigh"
|
||||
annotations:
|
||||
description: "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/"
|
||||
summary: "Windows memory paging is high"
|
||||
expr: |
|
||||
windows:memory_swap_page_operations:rate{job="windows_exporter"} > 1000
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-disk"
|
||||
rules:
|
||||
- alert: "WindowsDiskReadLatencyHigh"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/"
|
||||
summary: "Windows disk read latency is high"
|
||||
expr: |
|
||||
windows:logical_disk_read_latency:seconds{job="windows_exporter", volume!=""} > 0.05
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskWriteLatencyHigh"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/"
|
||||
summary: "Windows disk write latency is high"
|
||||
expr: |
|
||||
windows:logical_disk_write_latency:seconds{job="windows_exporter", volume!=""} > 0.05
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskAlmostFull"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
|
||||
summary: "Windows disk space is low"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskAlmostFull"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
|
||||
summary: "Windows disk space is low"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 5
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "critical"
|
||||
- alert: "WindowsDiskFillingUp"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/"
|
||||
summary: "Windows disk is predicted to fill up"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
|
||||
and predict_linear(windows_logical_disk_free_bytes{job="windows_exporter", volume!=""}[6h], 86400) < 0
|
||||
for: "1h"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-network"
|
||||
rules:
|
||||
- alert: "WindowsNetworkErrors"
|
||||
annotations:
|
||||
description: "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/"
|
||||
summary: "Windows network packet errors are high"
|
||||
expr: |
|
||||
(
|
||||
(windows:net_received_errors:rate{job="windows_exporter", nic!=""} + windows:net_outbound_errors:rate{job="windows_exporter", nic!=""})
|
||||
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
|
||||
) * 100 > 1
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkDiscards"
|
||||
annotations:
|
||||
description: "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/"
|
||||
summary: "Windows network packet discards are high"
|
||||
expr: |
|
||||
(
|
||||
(windows:net_received_discarded:rate{job="windows_exporter", nic!=""} + windows:net_outbound_discarded:rate{job="windows_exporter", nic!=""})
|
||||
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
|
||||
) * 100 > 1
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkUtilizationHigh"
|
||||
annotations:
|
||||
description: "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/"
|
||||
summary: "Windows network interface utilization is high"
|
||||
expr: |
|
||||
(windows:net_received_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
|
||||
or (windows:net_sent_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkInterfaceFlapping"
|
||||
annotations:
|
||||
description: "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/"
|
||||
summary: "Windows network interface is flapping"
|
||||
expr: |
|
||||
max without (status) (changes(windows_net_nic_operation_status{job="windows_exporter", nic!="", status="up"}[5m])) > 2
|
||||
for: "2m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-service"
|
||||
rules:
|
||||
- alert: "WindowsServiceNotRunning"
|
||||
annotations:
|
||||
description: "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/"
|
||||
summary: "Windows automatic service is not running"
|
||||
expr: |
|
||||
max without (state) (windows_service_state{job="windows_exporter", service!="", state="running"} == 0)
|
||||
and ignoring (start_mode) (windows_service_start_mode{job="windows_exporter", service!="", start_mode="auto"} == 1)
|
||||
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job="windows_exporter", service!="", trigger_start="true"} == 1)
|
||||
for: "15m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsServiceCrashLooping"
|
||||
annotations:
|
||||
description: "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/"
|
||||
summary: "Windows service is restarting repeatedly"
|
||||
expr: |
|
||||
count without (process_id) (count_over_time(windows_service_process{job="windows_exporter", service!=""}[15m])) - 1 > 2
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-system"
|
||||
rules:
|
||||
- alert: "WindowsHostRebooted"
|
||||
annotations:
|
||||
description: "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/"
|
||||
summary: "Windows host has rebooted"
|
||||
expr: |
|
||||
(time() - windows_system_boot_time_timestamp{job="windows_exporter"}) < 600
|
||||
and (time() - last_over_time(windows_system_boot_time_timestamp{job="windows_exporter"}[1h] offset 10m)) > 600
|
||||
labels:
|
||||
severity: "info"
|
||||
@@ -0,0 +1,59 @@
|
||||
groups:
|
||||
- name: "windows-exporter-recording"
|
||||
rules:
|
||||
- expr: "1 - avg without (core, mode) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"}[5m]))"
|
||||
record: "windows:cpu_usage:ratio"
|
||||
- expr: "count without (core, mode) (windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"})"
|
||||
record: "windows:cpu_cores:count"
|
||||
- expr: "avg without (core) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\"}[5m]))"
|
||||
record: "windows:cpu_time:ratio_rate"
|
||||
- expr: "(avg_over_time(windows_system_processor_queue_length{job=\"windows_exporter\"}[5m]) / (windows:cpu_cores:count{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:cpu_queue_per_core:ratio"
|
||||
- expr: "1 - (windows_memory_available_bytes{job=\"windows_exporter\"} / (windows_memory_physical_total_bytes{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:memory_usage:ratio"
|
||||
- expr: "windows_memory_physical_total_bytes{job=\"windows_exporter\"} - windows_memory_available_bytes{job=\"windows_exporter\"}"
|
||||
record: "windows:memory_used:bytes"
|
||||
- expr: "windows_memory_cache_bytes{job=\"windows_exporter\"} + windows_memory_modified_page_list_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_core_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_normal_priority_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_reserve_bytes{job=\"windows_exporter\"}"
|
||||
record: "windows:memory_cached:bytes"
|
||||
- expr: "(windows_memory_committed_bytes{job=\"windows_exporter\"} / (windows_memory_commit_limit_bytes{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:memory_committed:ratio"
|
||||
- expr: "rate(windows_memory_swap_page_operations_total{job=\"windows_exporter\"}[5m])"
|
||||
record: "windows:memory_swap_page_operations:rate"
|
||||
- expr: "(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"} / (windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} > 0))"
|
||||
record: "windows:logical_disk_free:ratio"
|
||||
- expr: "windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} - windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}"
|
||||
record: "windows:logical_disk_used:bytes"
|
||||
- expr: "clamp(1 - rate(windows_logical_disk_idle_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]), 0, 1)"
|
||||
record: "windows:logical_disk_busy:ratio"
|
||||
- expr: "rate(windows_logical_disk_read_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_read_bytes:rate"
|
||||
- expr: "rate(windows_logical_disk_write_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_write_bytes:rate"
|
||||
- expr: "rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_read_operations:rate"
|
||||
- expr: "rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_write_operations:rate"
|
||||
- expr: "(rate(windows_logical_disk_read_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
|
||||
record: "windows:logical_disk_read_latency:seconds"
|
||||
- expr: "(rate(windows_logical_disk_write_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
|
||||
record: "windows:logical_disk_write_latency:seconds"
|
||||
- expr: "rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_bytes:rate"
|
||||
- expr: "rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_sent_bytes:rate"
|
||||
- expr: "rate(windows_net_packets_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_packets:rate"
|
||||
- expr: "rate(windows_net_packets_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_sent_packets:rate"
|
||||
- expr: "(rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
|
||||
record: "windows:net_received_utilization:ratio"
|
||||
- expr: "(rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
|
||||
record: "windows:net_sent_utilization:ratio"
|
||||
- expr: "rate(windows_net_packets_received_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_errors:rate"
|
||||
- expr: "rate(windows_net_packets_received_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_discarded:rate"
|
||||
- expr: "rate(windows_net_packets_outbound_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_outbound_errors:rate"
|
||||
- expr: "rate(windows_net_packets_outbound_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_outbound_discarded:rate"
|
||||
@@ -30,7 +30,7 @@ Before beginning you need to deploy the [Prometheus Operator](https://github.com
|
||||
This creates a pod on every Windows node. A ConfigMap holds the configuration of the Windows exporter with [configuration file](../README.md#using-a-configuration-file). Adjust the configuration file for the collectors you are interested in.
|
||||
|
||||
```bash
|
||||
kubectl apply -f kubernetes/windows-exporter-daemonset.yaml
|
||||
kubectl apply -f docs/kubernetes/windows-exporter-daemonset.yaml
|
||||
```
|
||||
|
||||
> Note: This example manifest deploys the latest bleeding edge image `ghcr.io/prometheus-community/windows-exporter:latest` built from the main branch. You should update this to use a released version which you can find at https://github.com/prometheus-community/windows_exporter/releases
|
||||
@@ -58,7 +58,7 @@ spec:
|
||||
Create the [Pod Monitor](https://prometheus-operator.dev/docs/operator/design/#podmonitor) to configure the scraping:
|
||||
|
||||
```bash
|
||||
kubectl apply -f kubernetes/windows-exporter-podmonitor.yaml
|
||||
kubectl apply -f docs/kubernetes/windows-exporter-podmonitor.yaml
|
||||
```
|
||||
|
||||
### View Metrics
|
||||
@@ -8,5 +8,9 @@ the affected host. Disk alerts also identify a `volume`; network alerts identify
|
||||
a `nic`. Adjust example PromQL selectors to match your scrape configuration.
|
||||
|
||||
The default thresholds are starting points. Check the generated alert rules for
|
||||
your configured thresholds and durations. Active Directory and NTP clock alerts
|
||||
are opt-in and require their collectors to be enabled.
|
||||
your configured thresholds and durations. The rules are grouped by collector
|
||||
area, and each alert is generated only when the collectors it reads are enabled
|
||||
in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
|
||||
and `time` collectors, which are off by default, and the alerts for page files,
|
||||
disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters,
|
||||
Storage Spaces, DHCP and DNS need the collector of that role.
|
||||
|
||||
@@ -6,7 +6,7 @@ title: WindowsADPendingReplication
|
||||
|
||||
Pending Active Directory replication operations exceed the configured threshold
|
||||
for 15 minutes. The default is 50 operations. This alert is generated only with
|
||||
`enableActiveDirectory: true` and requires the `ad` collector on domain controllers.
|
||||
`collectors+: { ad: true }` and requires the `ad` collector on domain controllers.
|
||||
|
||||
## Impact
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ title: WindowsClockNotSynchronising
|
||||
## Meaning
|
||||
|
||||
Windows Time reports no active NTP time source for ten minutes. This alert is
|
||||
generated only with `enableTime: true` and requires the `time` collector and its
|
||||
generated only with `collectors+: { time: true }` and requires the `time` collector and its
|
||||
`ntp` subcollector. Enable it on hosts expected to synchronize through NTP;
|
||||
hosts intentionally using another clock source may legitimately report zero.
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
title: WindowsClockSourceLocal
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system
|
||||
clock for ten minutes. The alert carries the source in its `type` label. It
|
||||
needs the `time` collector with its `clock_source` subcollector and covers
|
||||
domain members too, whose source is `NT5DS` when they synchronize from the
|
||||
domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does
|
||||
not see those hosts. A host that is meant to be the free-running reference of
|
||||
an isolated network reports the local clock by design; exclude it with
|
||||
`alerts+: { WindowsClockSourceLocal: false }` on that configuration.
|
||||
|
||||
## Impact
|
||||
|
||||
The clock drifts without an external source. Clock errors break Kerberos
|
||||
authentication beyond the five minute tolerance, TLS validation, scheduled
|
||||
work, log correlation and distributed systems that depend on time order.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_time_clock_sync_source` for the host, and
|
||||
`windows_time_computed_time_offset_seconds` to see how far the clock has
|
||||
already drifted.
|
||||
2. On the host, check the configured and the effective source:
|
||||
|
||||
```powershell
|
||||
w32tm /query /source
|
||||
w32tm /query /status
|
||||
w32tm /query /configuration
|
||||
Get-Service W32Time
|
||||
```
|
||||
|
||||
3. `Local CMOS Clock` after a reboot often means the Windows Time service
|
||||
started before the network was up or the configured peers were unreachable.
|
||||
`NoSync` means the time provider is configured not to synchronize.
|
||||
4. For domain members, check that a domain controller is reachable and that
|
||||
the PDC emulator synchronizes from an external source. For virtual machines,
|
||||
check whether hypervisor time integration overrides the configured source.
|
||||
A Hyper-V guest outside a domain that is kept in time by the integration
|
||||
service reports `NoSync` while `w32tm /query /source` shows the
|
||||
`VM IC Time Synchronization Provider`; that is by design, so turn the alert
|
||||
off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or
|
||||
configure an NTP source in the guest.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Configure the intended source and restart synchronization with
|
||||
`w32tm /config /syncfromflags:domhier /update` on domain members or
|
||||
`w32tm /config /manualpeerlist:"<servers>" /syncfromflags:manual /update` on
|
||||
stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the
|
||||
peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the
|
||||
offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/)
|
||||
if it does not.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: WindowsClusterNodeDown
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The failover cluster reports the node `node` as down for five minutes. The
|
||||
alert needs the `mscluster` collector. Every node that runs the exporter
|
||||
reports the state of all nodes, so the alert fires once per reporting node;
|
||||
the `instance` label names the reporter, and the `node` label the node that is
|
||||
down. A paused node, for example during cluster-aware updating, has a
|
||||
different state and does not raise this alert.
|
||||
|
||||
## Impact
|
||||
|
||||
The roles that the node hosted have failed over to the remaining nodes, with a
|
||||
short outage during the move. The cluster has less capacity and less
|
||||
redundancy; with too few nodes for quorum, the whole cluster stops.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_mscluster_node_state` for all nodes and
|
||||
`windows_mscluster_cluster_quorum_type_value` and the dynamic quorum metrics to
|
||||
see how far the cluster is from losing quorum. Check whether the node's own
|
||||
exporter target is up.
|
||||
2. On a working node, read the cluster state:
|
||||
|
||||
```powershell
|
||||
Get-ClusterNode
|
||||
Get-ClusterGroup
|
||||
Get-ClusterLog -Node '<node>' -TimeSpan 60 -Destination .
|
||||
Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50
|
||||
```
|
||||
|
||||
3. Check whether the node is powered on and reachable, whether the Cluster
|
||||
Service runs on it, and whether the cluster networks between the nodes are
|
||||
working. Events 1135 and 1177 in the System log name the lost node and the
|
||||
quorum state.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Bring the node back: power it on, repair its network, or start the Cluster
|
||||
Service with `Start-ClusterNode`. Evict a node that is removed for good with
|
||||
`Remove-ClusterNode` so the quorum configuration matches the remaining nodes.
|
||||
Move the roles back to their preferred owners once the node is up and confirm
|
||||
the node state returns to up.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: WindowsClusterResourceFailed
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The failover cluster reports the resource `resource` of type `type` in the
|
||||
group `resource_group` as failed for five minutes. The alert needs the
|
||||
`mscluster` collector and fires once per reporting node, named by the
|
||||
`instance` label. A failed resource has exhausted its restart attempts on the
|
||||
current node, or its group has reached its failover threshold.
|
||||
|
||||
## Impact
|
||||
|
||||
The role that depends on the resource is down: a clustered disk, IP address,
|
||||
network name, file share, SQL Server instance or virtual machine. Other
|
||||
resources in the same group that depend on it are offline too.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_mscluster_resource_state` for the group and
|
||||
`windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node`
|
||||
to see where the group is and whether the whole group failed.
|
||||
2. On a cluster node, read the resource and the reason:
|
||||
|
||||
```powershell
|
||||
Get-ClusterResource -Name '<resource>' | Format-List *
|
||||
Get-ClusterResource -Name '<resource>' | Get-ClusterParameter
|
||||
Get-ClusterLog -TimeSpan 30 -Destination .
|
||||
Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20
|
||||
```
|
||||
|
||||
Event 1069 names the failed resource and its error code.
|
||||
3. Check the dependency of the resource: a disk that is not reachable from the
|
||||
node, an IP address whose network is down, a network name whose computer
|
||||
object in Active Directory is missing or disabled, or an application whose
|
||||
service fails to start.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the dependency, then bring the resource online with
|
||||
`Start-ClusterResource`, or move the group to another node with
|
||||
`Move-ClusterGroup`. Review the restart and failover thresholds of the
|
||||
resource if it failed because of a transient problem. Confirm the resource
|
||||
state returns to online and the group is online.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
title: WindowsClusterResourceGroupFailed
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The failover cluster reports the resource group `resource_group` as failed for
|
||||
five minutes. The alert needs the `mscluster` collector and fires once per
|
||||
reporting node, named by the `instance` label. A group fails when a resource in
|
||||
it fails and the group cannot fail over anymore: it reached its failover
|
||||
threshold within the failover period, or no other node can host it.
|
||||
|
||||
## Impact
|
||||
|
||||
The whole clustered role is down and stays down until an operator intervenes.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_mscluster_resource_state` for the resources of the group
|
||||
to find the failed resource, and `windows_mscluster_resourcegroup_owner_node`
|
||||
to see on which node the group stopped.
|
||||
2. On a cluster node, read the group and its history:
|
||||
|
||||
```powershell
|
||||
Get-ClusterGroup -Name '<group>' | Format-List *
|
||||
Get-ClusterGroup -Name '<group>' | Get-ClusterResource
|
||||
Get-ClusterLog -TimeSpan 60 -Destination .
|
||||
Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20
|
||||
```
|
||||
|
||||
Event 1254 says the group exceeded its failover threshold.
|
||||
3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for
|
||||
the failed resource, and check the possible owners of the group.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the failed resource, then bring the group online with `Start-ClusterGroup`
|
||||
on a node that can host it. A group that failed over repeatedly may need a
|
||||
higher failover threshold or a longer failover period, but a resource that
|
||||
fails on every node needs its cause fixed first. Confirm the group state
|
||||
returns to online.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
title: WindowsClusterSharedVolumeAlmostFull
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The Cluster Shared Volume `shared_volume` has less free space than the
|
||||
configured threshold for 15 minutes. The warning fires below
|
||||
`diskFreeWarningThreshold` (10 percent by default) and the critical alert
|
||||
below `diskFreeCriticalThreshold` (5 percent); both fire below the critical
|
||||
threshold, so inhibit the warning in Alertmanager. The alert needs the
|
||||
`mscluster` collector and fires once per reporting node, named by the
|
||||
`instance` label. `sharedVolumeSelector` excludes the
|
||||
`ClusterPerformanceHistory` volume of the cluster's own health history.
|
||||
|
||||
## Impact
|
||||
|
||||
Virtual machines and databases on the volume cannot grow their files. Dynamic
|
||||
virtual hard disks pause the virtual machine when the volume is full, Hyper-V
|
||||
checkpoints and backups fail, and SQL Server stops writing.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Compare `windows_mscluster_shared_volumes_free_bytes` and
|
||||
`windows_mscluster_shared_volumes_total_bytes` over the last days to see
|
||||
whether the volume fills steadily or filled suddenly.
|
||||
2. On a cluster node, find the large files on the volume:
|
||||
|
||||
```powershell
|
||||
Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}}
|
||||
Get-ChildItem 'C:\ClusterStorage\<volume>' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}}
|
||||
```
|
||||
|
||||
3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard
|
||||
disks, backup snapshots that were not cleaned up, and dynamic disks that
|
||||
grew.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Delete or merge checkpoints, remove orphaned files, move virtual machines to
|
||||
another volume with storage migration, or extend the volume on the storage
|
||||
array and then in the cluster with `Resize-Partition`. Confirm free space is
|
||||
above the threshold. Raise the thresholds only for volumes whose growth is
|
||||
known and planned.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: WindowsCollectorTimeout
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The collector identified by the `collector` label has reported
|
||||
`windows_exporter_collector_timeout == 1` for ten minutes. The collector did
|
||||
not finish within the scrape timeout, or its call from an earlier scrape is
|
||||
still running and later scrapes skip it until it returns.
|
||||
|
||||
## Impact
|
||||
|
||||
The metrics of this collector are missing or stale on every scrape that times
|
||||
out, so dashboards show gaps and alerts on those metrics stop evaluating. A
|
||||
collector that never returns can hold a scrape slot and delay the other
|
||||
collectors of the host.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Find the collector and compare how long it usually takes:
|
||||
|
||||
```promql
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
windows_exporter_collector_duration_seconds{job="windows_exporter"}
|
||||
```
|
||||
|
||||
2. Compare the duration with the scrape timeout of the job. The exporter
|
||||
subtracts `--scrape.timeout-margin` from the `scrape_timeout` that
|
||||
Prometheus sends.
|
||||
3. Check the exporter log for the collector. WMI and Management Infrastructure
|
||||
queries, the Task Scheduler and Windows Update are slow on hosts with many
|
||||
objects or a busy WMI provider; the collector's documentation lists the
|
||||
expensive options.
|
||||
4. Check whether the host was under CPU, disk or memory pressure at the same
|
||||
time, and whether a Windows service the collector depends on hangs, for
|
||||
example `Winmgmt` for WMI-based collectors.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Raise the `scrape_timeout` of the job within its `scrape_interval`, narrow the
|
||||
collector with its include and exclude flags, or move the collector to a
|
||||
separate job with a longer timeout. Restart a hanging provider service when the
|
||||
log identifies it. Disable the collector on hosts where its data is not needed.
|
||||
Confirm `windows_exporter_collector_timeout` returns to `0` and the collector's
|
||||
metrics are published again.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: WindowsCPUSaturation
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The number of threads waiting for a processor, averaged over the rate window
|
||||
and divided by the number of logical processors, exceeds the configured
|
||||
threshold for 15 minutes. The default is two waiting threads per logical
|
||||
processor. The metric is `windows:cpu_queue_per_core:ratio`, built from
|
||||
`windows_system_processor_queue_length` and `windows:cpu_cores:count`.
|
||||
|
||||
## Impact
|
||||
|
||||
Runnable threads wait for CPU time, so applications respond slowly even when
|
||||
their own CPU usage looks moderate. A long queue with CPU usage below 100%
|
||||
points at a few busy cores, a single-threaded bottleneck, or processor
|
||||
scheduling problems such as parked cores or hypervisor contention.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows:cpu_queue_per_core:ratio` next to `windows:cpu_usage:ratio`
|
||||
for the host. Saturation with high usage is a capacity problem; saturation
|
||||
with low usage points at scheduling.
|
||||
2. Compare the per-core rates of `windows_cpu_time_seconds_total`. One or two
|
||||
cores at 100% while others idle indicate a thread that cannot spread.
|
||||
3. Check `windows_cpu_parking_status`, `windows_cpu_core_frequency_hertz` and
|
||||
the power plan. On virtual machines check the hypervisor for CPU ready time
|
||||
or steal.
|
||||
4. Use Task Manager or Performance Monitor (`System\Processor Queue Length`,
|
||||
`Process\Thread Count`) to find the processes with many runnable threads.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Reduce or redistribute the workload, add logical processors, or fix the power
|
||||
plan and parking settings. Give the virtual machine more CPU share or move it to
|
||||
a less loaded host. Tune the threshold for hosts whose workload keeps a long
|
||||
queue by design, such as build servers. Confirm the queue length per core and
|
||||
application latency return to normal.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: WindowsDHCPScopeAlmostExhausted
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The enabled DHCP scope `scope` has fewer free addresses than the configured
|
||||
share of its address pool for 15 minutes. The warning fires below
|
||||
`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert
|
||||
below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the
|
||||
critical threshold, so inhibit the warning in Alertmanager. The share divides
|
||||
`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses,
|
||||
and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled.
|
||||
On a failover pair the free and in-use counts are the totals of both servers.
|
||||
The alert needs the `dhcp` collector.
|
||||
|
||||
## Impact
|
||||
|
||||
Once the scope is exhausted, new clients receive no address and cannot use
|
||||
the network. Existing clients keep their lease until it expires and then lose
|
||||
their address too.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Compare the free and in-use addresses of the scope over the last days to
|
||||
see whether the exhaustion is a trend or a sudden burst, and inspect
|
||||
`windows_dhcp_scope_reserved_address` and
|
||||
`windows_dhcp_scope_pending_offers` for reservations and offers in flight.
|
||||
2. On the server, read the scope and its leases:
|
||||
|
||||
```powershell
|
||||
Get-DhcpServerv4ScopeStatistics -ScopeId <scope>
|
||||
Get-DhcpServerv4Scope -ScopeId <scope> | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State
|
||||
Get-DhcpServerv4Lease -ScopeId <scope> | Group-Object AddressState
|
||||
Get-DhcpServerv4ExclusionRange -ScopeId <scope>
|
||||
```
|
||||
|
||||
3. Look for a long lease duration on a network with many short-lived clients,
|
||||
large exclusion ranges, a flood of leases from one device that changes its
|
||||
MAC address, or a failover partner that holds the free addresses.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Shorten the lease duration for networks with transient clients, remove
|
||||
exclusion ranges that are not needed, add a range through a superscope or a
|
||||
larger subnet, or reclaim bad leases. On a failover pair, check the split of
|
||||
the addresses between the partners. Confirm the free share stays above the
|
||||
threshold.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: WindowsDiskDriveNotOK
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The physical drive `device_id` reports an operational status other than `OK`
|
||||
for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded`
|
||||
or `Error`, in its `status` label. It needs the `diskdrive` collector, which
|
||||
reads the status that the drive and its driver report through WMI. The
|
||||
`Unknown` status does not raise the alert: some USB and virtual drives report
|
||||
it permanently. Transient statuses such as `Starting` or `Stopping` raise it
|
||||
only when they last longer than five minutes.
|
||||
|
||||
## Impact
|
||||
|
||||
`Pred Fail` is the drive's own prediction that it is about to fail. Any other
|
||||
status means the drive is not fully working. Data on a single drive is at risk;
|
||||
a drive in a mirror or parity set leaves the set without redundancy until it is
|
||||
replaced.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_diskdrive_status` for the host and check whether the
|
||||
volumes on the drive still report normal latency with
|
||||
`windows:logical_disk_read_latency:seconds` and
|
||||
`windows:logical_disk_write_latency:seconds`.
|
||||
2. On the host, read the drive and its reliability counters:
|
||||
|
||||
```powershell
|
||||
Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status
|
||||
Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
|
||||
Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear
|
||||
```
|
||||
|
||||
3. Check the System event log for disk, storahci, stornvme or RAID controller
|
||||
events. On a virtual machine, check the hypervisor and the datastore of the
|
||||
virtual disk.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Back up the data on the drive first. Replace a drive that predicts its failure
|
||||
or reports errors through the hardware or virtualization team. For a drive in
|
||||
a storage pool or RAID set, follow the vendor's replacement and rebuild
|
||||
procedure and confirm the set is redundant again. The alert resolves when the
|
||||
status returns to `OK`.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: WindowsDNSZoneTransferFailures
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The DNS server has failed zone transfers for 15 minutes, measured as the rate
|
||||
of `windows_dns_zone_transfer_failures_total` over `rateInterval` above
|
||||
`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts
|
||||
transfers this server sends as the primary to its secondaries, including
|
||||
requests it refuses, and it has no zone label. The alert needs the `dns`
|
||||
collector and keeps firing for `keepFiringFor` after the failures stop. Raise
|
||||
the threshold on servers that refuse transfer requests from scanners all the
|
||||
time.
|
||||
|
||||
## Impact
|
||||
|
||||
The secondary servers of the zone serve stale records. When the zone on a
|
||||
secondary expires, it stops answering for the zone altogether. Active
|
||||
Directory integrated zones replicate through Active Directory and are not
|
||||
affected unless they also have secondaries.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_dns_zone_transfer_failures_total` next to
|
||||
`windows_dns_zone_transfer_requests_received_total` and
|
||||
`windows_dns_zone_transfer_success_sent_total` to see what share of the
|
||||
transfers fail, and whether the failures started with a change.
|
||||
2. On this primary, find the refused or failed requests and the zone
|
||||
settings; the DNS Server log records a refused transfer request with the
|
||||
address of the requester:
|
||||
|
||||
```powershell
|
||||
Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer'
|
||||
Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy
|
||||
Get-DnsServerZone -Name '<zone>' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers
|
||||
```
|
||||
|
||||
On the secondary, events 6522 to 6527 of the DNS Server log say why its
|
||||
transfer of the zone failed: refused by the primary, invalid response or
|
||||
an expired zone.
|
||||
|
||||
3. Check that the requesting secondary is in the zone transfer list, that TCP
|
||||
port 53 is open between the servers, and that the zone is not paused or
|
||||
broken. Failures from unknown addresses are refused transfers and may be
|
||||
expected.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries`
|
||||
and the server list, fix the network path, or remove a decommissioned
|
||||
secondary from the zone's notify list so it stops requesting transfers.
|
||||
Confirm the failure counter stops rising and the secondaries show a current
|
||||
serial number.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: WindowsHostRebooted
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The boot time of the host, `windows_system_boot_time_timestamp`, is less than
|
||||
ten minutes old and differs from the boot time known ten minutes earlier. The
|
||||
alert fires once for about ten minutes after a reboot and then resolves by
|
||||
itself. It has the `info` severity and is meant for a dashboard, a ticket or a
|
||||
chat notification, not for a page.
|
||||
|
||||
## Impact
|
||||
|
||||
Every service of the host restarted, connections to it were dropped and
|
||||
caches are cold. A reboot outside a maintenance window can mean a crash, an
|
||||
unattended Windows Update restart or a power event.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Compare the boot time with the maintenance calendar and the Windows Update
|
||||
schedule. With the `update` collector enabled, `windows_update_pending_info`
|
||||
shows updates that were pending before the reboot.
|
||||
2. Check the System event log: event 1074 names the user or process that
|
||||
requested the shutdown, event 6008 marks an unexpected shutdown and event 41
|
||||
a reset without a clean shutdown. Check for a memory dump when it was a
|
||||
crash.
|
||||
3. Confirm the host came back with the expected services; the
|
||||
`WindowsServiceNotRunning` alert covers automatic services that did not
|
||||
start.
|
||||
|
||||
## Mitigation
|
||||
|
||||
No action is needed for a planned reboot. For an unplanned one, find the cause
|
||||
in the event log and fix the driver, update setting or power problem. Turn the
|
||||
alert off with `alerts+: { WindowsHostRebooted: false }` when reboots are not
|
||||
worth a notification.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: WindowsHyperVReplicationCritical
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Hyper-V Replica reports critical health for the virtual machine `vm` for five
|
||||
minutes. The alert carries the `relationship` label, `primary` for the normal
|
||||
replication and `extended` for an extended replica, and needs the `hyperv`
|
||||
collector with its `replica_vm` subcollector. Critical health means that
|
||||
replication has not succeeded for long enough that the replica misses the
|
||||
recovery point objective, or that replication is paused because of an error.
|
||||
|
||||
## Impact
|
||||
|
||||
The replica of the virtual machine is stale. A failover to it loses the
|
||||
changes since the last successful replication, and the recovery point
|
||||
objective of the service is not met. If the primary host fails now, data is
|
||||
lost.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_hyperv_replica_vm_health` and
|
||||
`windows_hyperv_replica_vm_state` for the virtual machine; the state tells
|
||||
whether replication is suspended, resynchronizing or in error, and
|
||||
`windows_hyperv_replica_vm_replication_latency_seconds` how long the last
|
||||
cycle took.
|
||||
2. On the primary host, read the replication details:
|
||||
|
||||
```powershell
|
||||
Measure-VMReplication -VMName '<vm>' | Format-List *
|
||||
Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, ReplicationMode
|
||||
Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '*<vm>*'
|
||||
```
|
||||
|
||||
3. Check the replica server: free space on the replica storage, the
|
||||
Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication,
|
||||
and the network path between the hosts. Check whether a snapshot merge or a
|
||||
storage migration on the primary blocks replication.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the cause on the replica server or the network, then resume replication
|
||||
with `Resume-VMReplication -VMName '<vm>'`; add `-Resynchronize` when the
|
||||
state asks for a resynchronization. Confirm the health returns to normal and
|
||||
the last replication time advances. If the replica cannot be recovered, remove
|
||||
and re-enable replication, which sends a full initial copy.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: WindowsHyperVReplicationWarning
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Hyper-V Replica reports warning health for the virtual machine `vm` for 15
|
||||
minutes. Warning health means that some replication cycles missed their
|
||||
interval, that more than a fifth of the cycles in the last hour failed, or
|
||||
that the replication is waiting for an initial copy or a resynchronization.
|
||||
The alert carries the `relationship` label and needs the `hyperv` collector
|
||||
with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/)
|
||||
follows when replication stops working.
|
||||
|
||||
## Impact
|
||||
|
||||
The replica lags behind the primary more than planned. The recovery point
|
||||
objective is at risk, and a failover would lose more changes than expected.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state`
|
||||
and `windows_hyperv_replica_vm_replication_latency_seconds` for the
|
||||
virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes`
|
||||
to see whether the change rate of the virtual machine has grown.
|
||||
2. On the primary host, read the statistics of the last hour:
|
||||
|
||||
```powershell
|
||||
Measure-VMReplication -VMName '<vm>' | Format-List *
|
||||
Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, FrequencySec
|
||||
```
|
||||
|
||||
3. Check the bandwidth between the hosts, the load on the replica storage,
|
||||
and whether the replication interval is too short for the change rate of
|
||||
the virtual machine.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Relieve the bottleneck: more bandwidth, less load on the replica storage, or a
|
||||
longer replication interval through `Set-VMReplication -ReplicationFrequencySec`.
|
||||
Resume a waiting replication with `Resume-VMReplication`. Reset the statistics
|
||||
with `Reset-VMReplicationStatistics` after the fix; the health is computed
|
||||
over the last hour and takes up to an hour to return to normal.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: WindowsHyperVWMIUnhealthy
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The Hyper-V WMI namespace `root/virtualization/v2` on the host has not
|
||||
answered a minimal query for ten minutes. The alert needs the `hyperv`
|
||||
collector with its `wmi_health` subcollector. While the namespace is broken,
|
||||
the virtual machine, replication and virtual hard disk metrics that come from
|
||||
WMI are missing, so the other Hyper-V alerts cannot fire.
|
||||
|
||||
## Impact
|
||||
|
||||
Management tools such as Hyper-V Manager, Failover Cluster Manager and System
|
||||
Center cannot manage the virtual machines on the host. Backups that use the
|
||||
Hyper-V VSS writer fail. The virtual machines themselves usually keep running.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}`
|
||||
for the host.
|
||||
2. On the host, test the namespace and the management service:
|
||||
|
||||
```powershell
|
||||
Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1
|
||||
Get-Service vmms, vmcompute, winmgmt
|
||||
winmgmt /verifyrepository
|
||||
```
|
||||
|
||||
3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of
|
||||
the Virtual Machine Management service, and the WMI-Activity log for
|
||||
provider failures. A stuck `vmms` process or a corrupt WMI repository are
|
||||
the usual causes.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Restart the Virtual Machine Management service with `Restart-Service vmms`;
|
||||
running virtual machines are not affected. If the repository is inconsistent,
|
||||
run `winmgmt /salvagerepository`, and register the Hyper-V provider again with
|
||||
`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a
|
||||
restart of the host if the service cannot be recovered. Confirm the health
|
||||
metric returns to 1 and the Hyper-V metrics appear again.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: WindowsLicenseNotGenuine
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Windows reports a license status other than `genuine` for one hour. The alert
|
||||
carries the status in its `state` label and needs the `license` collector. The
|
||||
status comes from the Software Licensing service and changes when an
|
||||
activation expires, a KMS host is unreachable for more than the grace period,
|
||||
the hardware changed, or the key was blocked. `offline` means the genuine
|
||||
validation could not reach Microsoft; on hosts without internet access that
|
||||
is the permanent state, so turn the alert off there with
|
||||
`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter
|
||||
selector.
|
||||
|
||||
## Impact
|
||||
|
||||
An unlicensed Windows shows activation notices, may limit personalization and,
|
||||
on some editions, restarts on a schedule. A Windows Server that leaves its
|
||||
activation grace period is out of compliance.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. The `state` label of the alert names the status; `windows_license_status`
|
||||
shows all states of the host.
|
||||
2. On the host, read the license details:
|
||||
|
||||
```powershell
|
||||
slmgr.vbs /dlv
|
||||
slmgr.vbs /xpr
|
||||
Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining
|
||||
```
|
||||
|
||||
3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp`
|
||||
DNS record and answers on TCP port 1688. For Active Directory based
|
||||
activation, check the activation object in the forest. For MAK or retail
|
||||
keys, check whether the activation count is exhausted.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Restore reachability to the KMS host or the activation object, then run
|
||||
`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk <key>` if the
|
||||
key was wrong, or reactivate after a hardware change. Confirm the status
|
||||
returns to `genuine`.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: WindowsMemoryPagingHigh
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The rate of pages read from or written to disk to resolve hard page faults,
|
||||
`windows:memory_swap_page_operations:rate`, exceeds the configured threshold
|
||||
for 15 minutes. The default is 1000 pages per second. The counter counts
|
||||
pagefile traffic and memory-mapped file I/O alike.
|
||||
|
||||
## Impact
|
||||
|
||||
The host is short of physical memory and uses the disk as backing store, so
|
||||
every affected process slows down to disk speed. Sustained paging also
|
||||
competes with application I/O on the system disk.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect the paging rate together with `windows:memory_usage:ratio`,
|
||||
`windows_memory_available_bytes` and `windows:memory_committed:ratio`. Low
|
||||
available memory confirms memory pressure; a high rate with ample memory
|
||||
points at a process reading a large memory-mapped file, such as a backup or
|
||||
an index rebuild.
|
||||
2. Split the rate into reads and writes with
|
||||
`windows_memory_swap_page_reads_total` and
|
||||
`windows_memory_swap_page_writes_total`. Reads dominate when the working
|
||||
sets of processes do not fit; writes dominate when the system trims and
|
||||
writes modified pages.
|
||||
3. Find the processes with the largest working sets and page fault rates in
|
||||
Resource Monitor or, with the `process` collector enabled, with
|
||||
`windows_process_working_set_private_bytes` and
|
||||
`windows_process_page_faults_total`.
|
||||
4. Check `windows_memory_standby_cache_lifetime_seconds`; a low lifetime means
|
||||
cached pages are repurposed quickly, another sign of memory pressure.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Stop or move the process that uses the memory, fix a memory leak, or add
|
||||
physical memory. Move large sequential file jobs off peak hours when they cause
|
||||
the paging. Confirm the paging rate drops and available memory recovers. Raise
|
||||
the threshold on hosts whose workload maps large files by design.
|
||||
@@ -4,9 +4,15 @@ title: WindowsNetworkDiscards
|
||||
|
||||
## Meaning
|
||||
|
||||
The sum of inbound and outbound discarded packet rates exceeds the configured
|
||||
threshold for 15 minutes. The default is one discard per second per interface.
|
||||
Discards count packets dropped without a packet error, such as buffer exhaustion.
|
||||
Discarded inbound and outbound packets make up more than the configured share
|
||||
of all packets on an interface for 15 minutes. The default
|
||||
`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates
|
||||
(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`)
|
||||
by the packet rates (`windows:net_received_packets:rate` plus
|
||||
`windows:net_sent_packets:rate`). Discards count packets dropped without a packet
|
||||
error, such as buffer exhaustion. Interfaces that carry no packets produce no
|
||||
sample and cannot alert. The alert keeps firing for `keepFiringFor` after the
|
||||
ratio drops below the threshold.
|
||||
|
||||
## Impact
|
||||
|
||||
@@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
|
||||
|
||||
1. Inspect `windows:net_received_discarded:rate` and
|
||||
`windows:net_outbound_discarded:rate` to identify the direction and affected
|
||||
`nic`.
|
||||
`nic`, and compare them with the packet rate recordings to see whether a low
|
||||
packet rate inflates the ratio.
|
||||
2. Compare traffic, directional utilization ratios, and
|
||||
`windows_net_output_queue_length_packets` with the normal baseline.
|
||||
3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers,
|
||||
@@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
|
||||
|
||||
Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver,
|
||||
or virtual switch problems when evidence identifies them. Review intentional
|
||||
drop policies with the network owner before changing thresholds. Confirm discard
|
||||
rates and application retransmissions return to the expected baseline.
|
||||
drop policies with the network owner before raising
|
||||
`networkDiscardRatioThreshold`. Confirm discard rates and application
|
||||
retransmissions return to the expected baseline.
|
||||
|
||||
@@ -4,9 +4,16 @@ title: WindowsNetworkErrors
|
||||
|
||||
## Meaning
|
||||
|
||||
The sum of inbound and outbound packet error rates exceeds the configured
|
||||
threshold for 15 minutes. The default is one error per second per interface.
|
||||
The alert identifies the interface using its `nic` label.
|
||||
Packets with inbound or outbound errors make up more than the configured share
|
||||
of all packets on an interface for 15 minutes. The default
|
||||
`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates
|
||||
(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by
|
||||
the packet rates (`windows:net_received_packets:rate` plus
|
||||
`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors
|
||||
alerts while a busy interface with the same error count does not. Interfaces
|
||||
that carry no packets produce no sample and cannot alert. The alert identifies
|
||||
the interface using its `nic` label and keeps firing for `keepFiringFor` after
|
||||
the ratio drops below the threshold.
|
||||
|
||||
## Impact
|
||||
|
||||
@@ -16,7 +23,10 @@ connection failures.
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows:net_received_errors:rate` and
|
||||
`windows:net_outbound_errors:rate` separately to identify the direction.
|
||||
`windows:net_outbound_errors:rate` separately to identify the direction, and
|
||||
compare them with `windows:net_received_packets:rate` and
|
||||
`windows:net_sent_packets:rate` to see whether a low packet rate inflates the
|
||||
ratio.
|
||||
2. Compare traffic and `windows:net_received_utilization:ratio` and
|
||||
`windows:net_sent_utilization:ratio` with the normal baseline. These ratios
|
||||
use bandwidth in bytes per second and keep each direction separate.
|
||||
@@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration
|
||||
problems. Apply driver and firmware fixes through the normal change process.
|
||||
Relieve congestion if it contributes. Verify errors stop increasing and affected
|
||||
application connections recover; cumulative counters do not need to return to
|
||||
zero for the rate alert to resolve.
|
||||
zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for
|
||||
interfaces whose error share is known and accepted.
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: WindowsNetworkInterfaceFlapping
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The operational status of the interface `nic` changed more than twice within
|
||||
the rate window, and did so for two minutes. The alert counts the changes of
|
||||
`windows_net_nic_operation_status{status="up"}` between `1` and `0`.
|
||||
|
||||
## Impact
|
||||
|
||||
Every transition drops the connections that use the interface and triggers
|
||||
route, DHCP and DNS registration changes. A flapping uplink on a server causes
|
||||
intermittent application failures that are hard to attribute.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_net_nic_operation_status` for the interface to see the
|
||||
status it flips to; `down` and `lower layer down` point at the link,
|
||||
`dormant` at authentication such as 802.1X.
|
||||
2. Check the System event log for NDIS, the adapter driver and
|
||||
`Microsoft-Windows-NetworkProfile` events, and the switch port for link
|
||||
state changes, errors and port security actions.
|
||||
3. Check the cable, SFP, speed and duplex negotiation, and whether a teaming or
|
||||
failover configuration switches between members.
|
||||
4. On virtual machines, check the hypervisor for virtual switch changes, live
|
||||
migrations, or a host adapter that flaps.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Replace the cable or transceiver, fix the speed and duplex settings, update the
|
||||
adapter driver, or correct the switch port and authentication configuration.
|
||||
Exclude an interface that flaps by design, such as a Wi-Fi or VPN adapter,
|
||||
with the `nicSelector`. Confirm the status stays `up`.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: WindowsNetworkUtilizationHigh
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
Received or sent throughput on the interface `nic` exceeds the configured share
|
||||
of its link bandwidth for 15 minutes. The default is 90%. The recordings
|
||||
`windows:net_received_utilization:ratio` and
|
||||
`windows:net_sent_utilization:ratio` divide the byte rate of each direction by
|
||||
`windows_net_current_bandwidth_bytes`.
|
||||
|
||||
## Impact
|
||||
|
||||
The link is close to saturation in at least one direction. Further traffic
|
||||
queues at the adapter, which raises latency and causes discards and TCP
|
||||
retransmissions for every application on the host.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect both utilization ratios to find the direction, and
|
||||
`windows:net_received_bytes:rate` and `windows:net_sent_bytes:rate` for the
|
||||
absolute volume.
|
||||
2. Check `windows_net_current_bandwidth_bytes`. A link that negotiated a lower
|
||||
speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high
|
||||
utilization at normal traffic; compare it with `Get-NetAdapter`.
|
||||
3. Correlate with `windows:net_received_discarded:rate`,
|
||||
`windows:net_outbound_discarded:rate` and
|
||||
`windows_net_output_queue_length_packets` to see whether the saturation
|
||||
already drops packets.
|
||||
4. Identify the traffic with Resource Monitor, `Get-NetTCPConnection`, or
|
||||
switch-port statistics: backups, replication, file copies and monitoring
|
||||
bursts are common causes.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that
|
||||
negotiated the wrong speed. Confirm utilization returns below the threshold and
|
||||
discards stop. Tune the threshold for interfaces that run near line rate by
|
||||
design, such as storage replication links.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
title: WindowsPagefileAlmostFull
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The page file `file` has been more than the configured share in use for 15
|
||||
minutes. The default `pagefileUsageThreshold` is 80 percent of
|
||||
`windows_pagefile_limit_bytes`, the current size of the file. The alert needs
|
||||
the `pagefile` collector. A system-managed page file grows up to its maximum
|
||||
when it fills, so a high usage of the current size on such a system shows that
|
||||
the file has already grown.
|
||||
|
||||
## Impact
|
||||
|
||||
When the page file is full, Windows cannot commit more memory. Allocations
|
||||
fail, processes crash with out-of-memory errors, and the system can become
|
||||
unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/)
|
||||
usually fires at the same time.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes`
|
||||
with `windows:memory_committed:ratio` and `windows_memory_available_bytes`
|
||||
to see whether physical memory is exhausted too.
|
||||
2. Find the processes that hold the commit charge:
|
||||
|
||||
```powershell
|
||||
Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}}
|
||||
Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
|
||||
```
|
||||
|
||||
3. Check for a recent change: a new workload, a memory leak that grows over
|
||||
days, or a page file that was made smaller or fixed in size.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Stop or restart the process that leaks memory, or move workload off the host.
|
||||
Add physical memory or enlarge the page file through the normal change
|
||||
process; a fixed page file needs enough room for the peak commit charge of the
|
||||
host. Confirm usage falls below the threshold and stays there.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
title: WindowsScheduledTaskFailed
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The last run of the scheduled task `task` ended with the result `unknown` or
|
||||
`terminated`, and the task has not completed successfully since for one hour.
|
||||
The alert needs the `scheduled_task` collector. `terminated` means the run was
|
||||
stopped by a user or by the scheduler's time limit. `unknown` is every result
|
||||
code without a named status, which includes the exit code of the task's
|
||||
program and the error codes of the Task Scheduler; the documentation of the
|
||||
collector lists the named codes. A program that exits with a code other than
|
||||
zero on success raises this alert on every run; exclude such tasks with
|
||||
`taskSelector` or with the include and exclude flags of the collector. A
|
||||
disabled task keeps its last result but does not raise the alert.
|
||||
|
||||
## Impact
|
||||
|
||||
The work of the task did not happen: backups, cleanups, reports, certificate
|
||||
renewals or synchronizations that are scheduled on this host. The impact
|
||||
depends on the task.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_scheduled_task_last_result_status` for the task and
|
||||
`windows_scheduled_task_state` to see whether the task is still enabled
|
||||
and queued.
|
||||
2. On the host, read the last result and the history:
|
||||
|
||||
```powershell
|
||||
Get-ScheduledTaskInfo -TaskName '<name>' -TaskPath '<folder>'
|
||||
Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '*<name>*'
|
||||
```
|
||||
|
||||
The history log is off by default; enable it with
|
||||
`wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`.
|
||||
3. Translate the result code: `0x1` is a generic failure of the program, `0x2`
|
||||
a missing file, `0x80070005` an access denied, and codes of the form
|
||||
`0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other
|
||||
codes between `0x41300` and `0x41325` are named statuses, such as `ready`
|
||||
or `running`, and do not raise this alert.
|
||||
4. Run the program by hand under the task's account to reproduce the failure.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the program, its arguments, its working directory or the permissions of
|
||||
the account the task runs as. Start the task once with `Start-ScheduledTask`
|
||||
and confirm the result turns to `success`. Disable or delete tasks that are
|
||||
not needed anymore instead of leaving them failing; disabling a task clears
|
||||
the alert.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: WindowsScrapeQueueTimeout
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
`windows_exporter_scrape_queue_timeout == 1` for ten minutes. Scrapes run one
|
||||
at a time; a scrape whose timeout ends while it waits for an earlier scrape to
|
||||
finish runs no collector and reports this metric as `1`. On such a scrape
|
||||
`windows_exporter_collector_success` is `0` for every collector.
|
||||
|
||||
## Impact
|
||||
|
||||
The affected scrapes return no collector metrics, so the host has gaps in all
|
||||
of its series and `WindowsCollectorFailed` can fire for every collector at
|
||||
once. The exporter itself is reachable.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Confirm the queue timeout and find the scrape that holds the slot:
|
||||
|
||||
```promql
|
||||
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
scrape_duration_seconds{job="windows_exporter"}
|
||||
```
|
||||
|
||||
2. Check whether more than one Prometheus server or job scrapes the same
|
||||
exporter, or whether a job scrapes it more often than its scrape duration
|
||||
allows. Compare `scrape_duration_seconds` with the `scrape_interval`.
|
||||
3. Identify the slow collector with `windows_exporter_collector_duration_seconds`
|
||||
and follow `WindowsCollectorTimeout` for it.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Scrape each exporter from one job, or give the jobs a `scrape_interval` longer
|
||||
than a full scrape takes. Shorten the slow collector or raise the
|
||||
`scrape_timeout` so a scrape finishes before the next one starts. Confirm
|
||||
`windows_exporter_scrape_queue_timeout` stays `0` and collector success returns
|
||||
to `1`.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
title: WindowsServiceCrashLooping
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The service `service` has run as more processes in the last 15 minutes than
|
||||
the configured number of restarts allows. The default is two restarts. Every
|
||||
service process has its own `windows_service_process` series, identified by
|
||||
`process_id`, so the series seen within 15 minutes count the restarts.
|
||||
|
||||
Three limits apply. Windows reuses process IDs, so a restart that lands on the
|
||||
same ID is not counted. Services that share a process, such as `svchost.exe`
|
||||
groups, share a `process_id`: one crash of the shared process alerts for every
|
||||
service in it. Services without a `windows_service_process` series, for example
|
||||
stopped services or services the exporter cannot read the process of, are
|
||||
invisible to this alert.
|
||||
|
||||
## Impact
|
||||
|
||||
The service crashes or is stopped and started repeatedly. It is unavailable
|
||||
during every restart, and the recovery options of the Service Control Manager
|
||||
give up after a configurable number of failures, after which
|
||||
`WindowsServiceNotRunning` follows.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. List the processes the service ran as:
|
||||
|
||||
```promql
|
||||
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
|
||||
```
|
||||
|
||||
The value of each series is the start time of the process. If several
|
||||
services report the same `process_id`, the shared host process crashed and
|
||||
the faulty service is one of that group.
|
||||
2. Check the Application log for crashes of the service executable (events
|
||||
1000 and 1026, .NET Runtime errors) and the System log for Service Control
|
||||
Manager events 7031 and 7034, which also show the recovery action.
|
||||
3. Check whether a deployment, configuration push, scheduled task or another
|
||||
automation restarts the service on purpose.
|
||||
4. Check for resources the service fails to acquire at start, such as a port
|
||||
that is in use, a certificate, a database or a license.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Fix the crash or the start failure, roll back the deployment that caused it, or
|
||||
stop the automation that restarts the service. Keep the service stopped until
|
||||
the cause is known if every restart has side effects. Confirm the service keeps
|
||||
one process and stays running. Raise the threshold for services that restart by
|
||||
design, for example worker services that recycle themselves.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: WindowsServiceNotRunning
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The service `service` has the start mode `auto` but has not been in the
|
||||
`running` state for 15 minutes. Trigger-start services, which Windows stops by
|
||||
design when their trigger condition ends, are left out through the
|
||||
`trigger_start` label of `windows_service_info`. Delayed-start services are
|
||||
included; the 15 minute duration covers their later start after a reboot.
|
||||
|
||||
## Impact
|
||||
|
||||
A service that should always run is stopped, so the function it provides is
|
||||
unavailable, for example printing, time synchronization, a site or a
|
||||
database instance. Windows restarts a crashed service only when its recovery
|
||||
options say so.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect the current state of the service and when it changed:
|
||||
|
||||
```promql
|
||||
windows_service_state{job="windows_exporter", service="<service>"} == 1
|
||||
```
|
||||
|
||||
2. Check the System event log for Service Control Manager events 7000, 7009,
|
||||
7031 and 7034, and the Application log for crashes of the service process.
|
||||
3. Check `windows_service_info` for the account in `run_as`; an expired
|
||||
password or a removed right to log on as a service stops the service at the
|
||||
next start. `windows_service_process` shows whether a process is running.
|
||||
4. Check whether the service was stopped on purpose for maintenance or
|
||||
disabled by policy, and whether it depends on another service that is also
|
||||
stopped.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Start the service after fixing the cause: the account, a missing dependency, a
|
||||
configuration error or a crash. Set recovery options so Windows restarts the
|
||||
service itself. Change the start mode to `manual` or `disabled` for services
|
||||
that should not run, or exclude them with the `serviceSelector` when the mixin
|
||||
should not watch them. For example
|
||||
`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that
|
||||
exit on their own by design. Confirm the service stays running.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: WindowsStoragePoolUnhealthy
|
||||
---
|
||||
|
||||
## Meaning
|
||||
|
||||
The Storage Spaces pool `pool` reports a health status of warning or unhealthy
|
||||
for ten minutes. The critical alert fires for unhealthy, the warning for
|
||||
warning. The alert needs the `storage_spaces` collector and joins
|
||||
`windows_storage_spaces_pool_info` so the primordial pool, which only groups
|
||||
the disks that are not in a pool yet, does not raise it. The pool's
|
||||
operational status, exposed as `windows_storage_spaces_pool_operational_status`,
|
||||
tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every
|
||||
node reports the clustered pool, so the alert fires once per node; the
|
||||
`instance` label names the reporter.
|
||||
|
||||
## Impact
|
||||
|
||||
A pool in warning state has lost a physical disk or redundancy; the virtual
|
||||
disks on it still work but have no tolerance left for another failure, or
|
||||
cannot allocate more space. An unhealthy pool has lost more disks than its
|
||||
resiliency tolerates, so virtual disks on it are detached or read-only.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
1. Inspect `windows_storage_spaces_pool_health_status`,
|
||||
`windows_storage_spaces_pool_operational_status` and
|
||||
`windows_storage_spaces_virtual_disk_health_status` for the pool and its
|
||||
virtual disks.
|
||||
2. On the host, find the failed disk:
|
||||
|
||||
```powershell
|
||||
Get-StoragePool -FriendlyName '<pool>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly
|
||||
Get-StoragePool -FriendlyName '<pool>' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage
|
||||
Get-StoragePool -FriendlyName '<pool>' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
|
||||
Get-StorageJob
|
||||
```
|
||||
|
||||
3. Check the System log for disk and storage events, and whether a repair job
|
||||
is already running.
|
||||
|
||||
## Mitigation
|
||||
|
||||
Replace the failed physical disk: retire it with
|
||||
`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`,
|
||||
repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk
|
||||
with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the
|
||||
health returns to healthy once the repair jobs finish.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user