sendnrw/windows_exporter
2
0
Fork 0
mirror of https://github.com/prometheus-community/windows_exporter.git synced 2026-10-17 18:19:04 +02:00
Code Issues Packages Projects Releases Wiki Activity

mixin: one alert file per collector with plain PromQL

Browse Source 
Split alerts/alerts.libsonnet into one file per collector under alerts/,
like kubernetes-mixin. Each rule is a plain Prometheus alerting rule with
the PromQL written out in an expr block and %(name)s placeholders for the
_config values; lib/alerts.libsonnet keeps the collector and per-alert
gating and keep_firing_for. The pagefile, diskdrive and license alerts
get groups of their own so that one file is one group. The rendered
default rules are unchanged apart from expr block scalars.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-17 14:53:14 +00:00
parent 220dbf52b7
commit 51a579e031
26 changed files with 915 additions and 584 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/mixin/AGENTS.md
+37 -12
View File
@@ -7,23 +7,48 @@ keep it in sync with every change.
## Alerts
`alerts/alerts.libsonnet` renders one rule group per collector area. Every
alert names the collectors it reads (`collectors=[...]`) and is generated only
with those enabled in `_config.collectors`; `_config.alerts` switches single
alerts off by name. Threshold alerts get `keep_firing_for` from
`keepFiringFor`; pass `threshold=false` for state alerts (`== 0`, `== 1`) and
one-shot alerts, which must resolve at once.
`alerts/` holds one file per collector, each rendering one rule group, and
`alerts/alerts.libsonnet` imports them in the order of the generated file.
A rule is a plain Prometheus alerting rule, like in kubernetes-mixin:
A new alert touches five places, and CI checks all of them:
```jsonnet
{
alert: 'WindowsCPUHighUsage',
expr: |||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: { summary: '...', description: '...' },
}
```
1. The rule in `alerts/alerts.libsonnet`, in the group of its area, with
`{{ $labels.<key> }}` templates that use the 0.32 label keys (`service`,
`volume`, `nic`, `task`, `device_id`, never `name`).
Write the PromQL out in the `expr` block with `%(name)s` placeholders for the
`_config` values; selectors go inside the braces as
`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet`
provides `a.group(name, collectors, rules)`, which keeps a rule only when every
collector in `collectors` is enabled in `_config.collectors` and the alert is
not switched off in `_config.alerts`, and adds `keep_firing_for` from
`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]`
when a rule reads other collectors than its group (the CPU saturation alert
reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`,
`== 1`) and one-shot alerts, which must resolve at once and get no
`keep_firing_for`. A one-shot alert has no `for`.
A new alert touches six places, and CI checks all of them:
1. The rule in `alerts/<collector>.libsonnet`, with `{{ $labels.<key> }}`
templates that use the 0.32 label keys (`service`, `volume`, `nic`,
`task`, `device_id`, never `name`). Alerts for a new collector go in a new
file, imported from `alerts/alerts.libsonnet`, with the collector added to
`_config.collectors` in `config.libsonnet`; the group expectations in
`tests/alerts.test.jsonnet` list every group.
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
fails when the map and the generated alerts differ. Thresholds are
`_config` knobs named `<area><Condition>Threshold[Unit]`.
3. A unit test in `tests/alerts.test.yaml` that checks the first firing
evaluation, the labels and annotations, and the recovery. With
3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for
opt-in collectors, which `tests/config.libsonnet` enables) that checks the
first firing evaluation, the labels and annotations, and the recovery. With
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
condition clears. `rate()` extrapolates while a window has few samples, so
ratios of two rates activate earlier than a single rate; probe the
contrib/mixin/README.md
+9 -8
View File
@@ -151,10 +151,11 @@ pagefile traffic.
## Alerts
The alerts are grouped by collector area. Each group lists the collectors its
rules read; a group is generated only with those collectors enabled in
`_config.collectors`, and a rule that needs two collectors, such as the CPU
saturation alert, only with both. Threshold configuration values below are
The alerts come in one rule group per collector, from one file per collector
in `alerts/` (`alerts/cpu.libsonnet`, `alerts/service.libsonnet`, and so on).
Each group lists the collectors its rules read; a group is generated only with
those collectors enabled in `_config.collectors`, and a rule that needs two
collectors, such as the CPU saturation alert, only with both. Threshold configuration values below are
percentages except where noted.
| Group | Collectors | Alert | Default condition | Duration | Severity |
@@ -185,9 +186,9 @@ percentages except where noted.
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning |
| `windows-memory` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
| `windows-disk` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
| `windows-system` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
| `windows-pagefile` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
| `windows-diskdrive` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
| `windows-license` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning |
| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info |
| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning |
@@ -244,7 +245,7 @@ host to its chosen source and does not independently verify that source.
The other opt-in alerts follow `_config.collectors`: enable `pagefile`,
`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`,
`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are
generated, in the group of their area or in a group of their own.
generated, each collector in a group of its own.
`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code
without a name, including the exit codes of the task's program, as a failure;
narrow the tasks with `taskSelector`. Disabling a task clears the alert.
contrib/mixin/alerts/ad.libsonnet
+32
View File
@@ -0,0 +1,32 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-active-directory', collectors=['ad'], rules=[
{
alert: 'WindowsADPendingReplication',
expr: |||
windows_ad_replication_pending_operations{%(windowsExporterSelector)s} > %(adPendingReplicationThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Active Directory replication queue is high',
description: 'Pending replication operations on {{ $labels.instance }} have exceeded %(adPendingReplicationThreshold)s for 15 minutes.' % c,
},
},
{
alert: 'WindowsADReplicationFailures',
expr: |||
(rate(windows_ad_replication_sync_requests_total{%(windowsExporterSelector)s}[%(rateInterval)s]) - rate(windows_ad_replication_sync_requests_success_total{%(windowsExporterSelector)s}[%(rateInterval)s])) > %(adReplicationFailureRateThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Active Directory replication requests are failing',
description: 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
},
},
]),
},
}
contrib/mixin/alerts/alerts.libsonnet
+22 -538
View File
@@ -1,6 +1,8 @@
// One file per collector, each adding one rule group. To add alerts for a
// collector, copy one of these files, name it after the collector and import
// it here; the group order below is the order in the generated file.
{
local c = self._config,
local q = (import '../lib/queries.libsonnet')(c),
local known = std.objectFields((import '../config.libsonnet')._config.alerts),
local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
@@ -8,541 +10,23 @@
'networkErrorRateThreshold and networkDiscardRateThreshold (packets per second) were replaced by networkErrorRatioThreshold and networkDiscardRatioThreshold (percent of all packets)',
assert !std.objectHas(c, 'enableActiveDirectory') && !std.objectHas(c, 'enableTime') :
'enableActiveDirectory and enableTime were replaced by collectors+: { ad: true } and collectors+: { time: true }',
// A collector missing from a replaced collectors object counts as disabled;
// an alert missing from a replaced alerts object counts as enabled.
local collectorEnabled(collector) = std.get(c.collectors, collector, false),
local alertEnabled(name) = std.get(c.alerts, name, true),
local keepFiring = if c.keepFiringFor != '' then { keep_firing_for: c.keepFiringFor } else {},
// alert renders a rule when every collector it needs is enabled and the
// alert is not turned off. Threshold alerts set keepFiringFor so a value
// oscillating around the threshold does not flap; state alerts and
// one-shot alerts leave it out.
local alert(name, expr, duration, severity, summary, description, collectors=[], threshold=true) =
if std.all([collectorEnabled(collector) for collector in collectors]) && alertEnabled(name) then [
{
alert: name,
expr: expr,
labels: { severity: severity },
annotations: { summary: summary, description: description },
} + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
] else [],
local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
// Percent of free addresses in a scope; a scope without addresses has no sample.
local dhcpScopeFree = '(%s * 100)' % q.ratio(
q.metric('windows_dhcp_scope_addresses_free'),
'(%s + %s)' % [q.metric('windows_dhcp_scope_addresses_free'), q.metric('windows_dhcp_scope_addresses_in_use')]
),
local dhcpScopeActive = q.metric('windows_dhcp_scope_state', ['state=~"Enabled|EnabledSwitched"']),
prometheusAlerts+:: {
groups:
group('windows-exporter', std.flattenArrays([
alert(
'WindowsExporterDown',
q.metric('up') + ' == 0',
'5m',
c.criticalSeverity,
'Windows exporter is unreachable',
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
threshold=false
),
alert(
'WindowsCollectorFailed',
q.metric('windows_exporter_collector_success') + ' == 0',
'5m',
'warning',
'Windows collector failed',
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
threshold=false
),
alert(
'WindowsCollectorTimeout',
q.metric('windows_exporter_collector_timeout') + ' == 1',
'10m',
'warning',
'Windows collector exceeds the scrape timeout',
'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
threshold=false
),
alert(
'WindowsScrapeQueueTimeout',
q.metric('windows_exporter_scrape_queue_timeout') + ' == 1',
'10m',
'warning',
'Windows exporter scrapes time out in the queue',
'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
threshold=false
),
]))
+ group('windows-cpu', std.flattenArrays([
alert(
'WindowsCPUHighUsage',
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
'15m',
'warning',
'Windows CPU usage is high',
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold,
collectors=['cpu']
),
alert(
'WindowsCPUSaturation',
q.metric('windows:cpu_queue_per_core:ratio') + ' > %s' % c.cpuQueuePerCoreThreshold,
'15m',
'warning',
'Windows processor queue is long',
'More than %s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c.cpuQueuePerCoreThreshold,
collectors=['cpu', 'system']
),
]))
+ group('windows-memory', std.flattenArrays([
alert(
'WindowsMemoryHighUsage',
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
'15m',
'warning',
'Windows memory usage is high',
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold,
collectors=['memory']
),
alert(
'WindowsMemoryCommitHighUsage',
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
'15m',
'warning',
'Windows memory commit usage is high',
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold,
collectors=['memory']
),
alert(
'WindowsMemoryPagingHigh',
q.metric('windows:memory_swap_page_operations:rate') + ' > %s' % c.memoryPagingThreshold,
'15m',
'warning',
'Windows memory paging is high',
'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold,
collectors=['memory']
),
alert(
'WindowsPagefileAlmostFull',
'(1 - %s) * 100 > %s' % [
q.ratio(q.metric('windows_pagefile_free_bytes'), q.metric('windows_pagefile_limit_bytes')),
c.pagefileUsageThreshold,
],
'15m',
'warning',
'Windows page file is almost full',
'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %s%% in use for 15 minutes.' % c.pagefileUsageThreshold,
collectors=['pagefile']
),
]))
+ group('windows-disk', std.flattenArrays([
alert(
'WindowsDisk%sLatencyHigh' % direction[0],
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
'15m',
'warning',
'Windows disk %s latency is high' % direction[1],
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]],
collectors=['logical_disk']
)
for direction in [['Read', 'read'], ['Write', 'write']]
] + [
alert(
'WindowsDiskAlmostFull',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
'15m',
entry[0],
'Windows disk space is low',
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1],
collectors=['logical_disk']
)
for entry in [
['warning', c.diskFreeWarningThreshold],
[c.criticalSeverity, c.diskFreeCriticalThreshold],
]
] + [
alert(
'WindowsDiskFillingUp',
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
c.diskFreeWarningThreshold,
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
c.diskPredictionWindow,
c.diskPredictionHours * 3600,
],
'1h',
'warning',
'Windows disk is predicted to fill up',
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours,
collectors=['logical_disk']
),
alert(
'WindowsDiskDriveNotOK',
// One series per status with the current status set to 1, so the
// alert carries the status the drive reports. Some USB and virtual
// drives report Unknown for good.
q.metric('windows_diskdrive_status', ['status!~"OK|Unknown"']) + ' == 1',
'5m',
c.criticalSeverity,
'Windows disk drive is not healthy',
'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
collectors=['diskdrive'],
threshold=false
),
]))
+ group('windows-network', std.flattenArrays([
alert(
'WindowsNetwork%s' % outcome[0],
q.ratio(
'(%s + %s)' % [
q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]),
q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]),
],
'(%s + %s)' % [
q.metric('windows:net_received_packets:rate', [c.nicSelector]),
q.metric('windows:net_sent_packets:rate', [c.nicSelector]),
]
) + ' * 100 > %g' % outcome[2],
'15m',
'warning',
'Windows network packet %s are high' % outcome[3],
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g%% of all packets for 15 minutes.' % [outcome[3], outcome[2]],
collectors=['net']
)
for outcome in [
['Errors', 'errors', c.networkErrorRatioThreshold, 'errors'],
['Discards', 'discarded', c.networkDiscardRatioThreshold, 'discards'],
]
] + [
alert(
'WindowsNetworkUtilizationHigh',
'(%s * 100 > %s) or (%s * 100 > %s)' % [
q.metric('windows:net_received_utilization:ratio', [c.nicSelector]),
c.networkUtilizationThreshold,
q.metric('windows:net_sent_utilization:ratio', [c.nicSelector]),
c.networkUtilizationThreshold,
],
'15m',
'warning',
'Windows network interface utilization is high',
'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %s%% of its bandwidth in at least one direction for 15 minutes.' % c.networkUtilizationThreshold,
collectors=['net']
),
alert(
'WindowsNetworkInterfaceFlapping',
'max without (status) (changes(%s[%s])) > 2' % [
q.metric('windows_net_nic_operation_status', [c.nicSelector, 'status="up"']),
c.rateInterval,
],
'2m',
'warning',
'Windows network interface is flapping',
'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %s.' % c.rateInterval,
collectors=['net']
),
]))
+ group('windows-service', std.flattenArrays([
alert(
'WindowsServiceNotRunning',
// The matches ignore only the labels of the right-hand metric, so
// every target label, including ones a selector adds, takes part.
std.join(' ', [
'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']),
'and ignoring (start_mode) (%s == 1)' % q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"']),
'unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (%s == 1)' % q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"']),
]),
'15m',
'warning',
'Windows automatic service is not running',
'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
collectors=['service'],
threshold=false
),
alert(
'WindowsServiceCrashLooping',
// Every service process has its own series, so the series seen in the
// window count the processes the service ran as.
'count without (process_id) (count_over_time(%s[15m])) - 1 > %s' % [
q.metric('windows_service_process', [c.serviceSelector]),
c.serviceRestartThreshold,
],
'0m',
'warning',
'Windows service is restarting repeatedly',
'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %s times in 15 minutes.' % c.serviceRestartThreshold,
collectors=['service']
),
]))
+ group('windows-system', std.flattenArrays([
alert(
'WindowsHostRebooted',
// Fires once per reboot: for the first ten minutes after the boot,
// and only on a host that was monitored before it. The exporter is
// down while the host reboots, so the second operand takes the last
// boot time of the hour before, not the sample of ten minutes ago.
'(time() - %s) < 600 and (time() - last_over_time(%s[1h] offset 10m)) > 600' % [
q.metric('windows_system_boot_time_timestamp'),
q.metric('windows_system_boot_time_timestamp'),
],
'0m',
c.infoSeverity,
'Windows host has rebooted',
'{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
collectors=['system'],
threshold=false
),
alert(
'WindowsLicenseNotGenuine',
q.metric('windows_license_status', ['state!="genuine"']) + ' == 1',
'1h',
'warning',
'Windows license is not genuine',
'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
collectors=['license'],
threshold=false
),
]))
+ group('windows-active-directory', std.flattenArrays([
alert(
'WindowsADPendingReplication',
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
'15m',
'warning',
'Active Directory replication queue is high',
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold,
collectors=['ad']
),
alert(
'WindowsADReplicationFailures',
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
'15m',
'warning',
'Active Directory replication requests are failing',
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
collectors=['ad']
),
]))
+ group('windows-time', std.flattenArrays([
alert(
'WindowsClockNotSynchronising',
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
'10m',
'warning',
'Windows clock has no NTP time source',
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
collectors=['time'],
threshold=false
),
alert(
'WindowsClockSkewDetected',
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
'10m',
'warning',
'Windows clock offset is high',
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds,
collectors=['time']
),
alert(
'WindowsClockSourceLocal',
// Covers domain members (NT5DS) too, which the NTP source count misses.
q.metric('windows_time_clock_sync_source', ['type=~"Local CMOS Clock|NoSync"']) + ' == 1',
'10m',
'warning',
'Windows clock has no external time source',
'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
collectors=['time'],
threshold=false
),
]))
+ group('windows-scheduled-task', std.flattenArrays([
alert(
'WindowsScheduledTaskFailed',
// unknown is any result code without a named status, including the
// exit codes of the task's program.
// A disabled task keeps its last result, so disabling a broken task
// clears the alert.
'%s == 1 unless ignoring (status, state) (%s == 1)' % [
q.metric('windows_scheduled_task_last_result_status', [c.taskSelector, 'status=~"unknown|terminated"']),
q.metric('windows_scheduled_task_state', [c.taskSelector, 'state="disabled"']),
],
'1h',
'warning',
'Windows scheduled task did not succeed',
'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
collectors=['scheduled_task'],
threshold=false
),
]))
+ group('windows-update', std.flattenArrays([
alert(
'WindowsUpdatesPending',
'count without (id, revision, category, severity, title) (%s) > 0' % q.metric('windows_update_pending_info', ['deployment_action="installation"']),
c.updatePendingDuration,
c.infoSeverity,
'Windows updates are waiting for installation',
'{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %s.' % c.updatePendingDuration,
collectors=['update'],
threshold=false
),
]))
+ group('windows-hyperv', std.flattenArrays([
alert(
'WindowsHyperVWMIUnhealthy',
q.metric('windows_hyperv_wmi_health') + ' == 0',
'10m',
'warning',
'Hyper-V WMI namespace is not responding',
'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationCritical',
q.metric('windows_hyperv_replica_vm_health') + ' == 3',
'5m',
c.criticalSeverity,
'Hyper-V replication is critical',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationWarning',
q.metric('windows_hyperv_replica_vm_health') + ' == 2',
'15m',
'warning',
'Hyper-V replication has a warning',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
collectors=['hyperv'],
threshold=false
),
]))
// Every cluster node reports the state of the whole cluster, so these
// alerts fire once per node; group them by cluster labels in Alertmanager.
+ group('windows-cluster', std.flattenArrays([
alert(
'WindowsClusterNodeDown',
q.metric('windows_mscluster_node_state') + ' == 1',
'5m',
c.criticalSeverity,
'Failover cluster node is down',
'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceFailed',
q.metric('windows_mscluster_resource_state') + ' == 4',
'5m',
c.criticalSeverity,
'Failover cluster resource has failed',
'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceGroupFailed',
q.metric('windows_mscluster_resourcegroup_state') + ' == 2',
'5m',
c.criticalSeverity,
'Failover cluster resource group has failed',
'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeWarningThreshold,
'15m',
'warning',
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeWarningThreshold,
collectors=['mscluster']
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeCriticalThreshold,
'15m',
c.criticalSeverity,
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeCriticalThreshold,
collectors=['mscluster']
),
]))
// On Storage Spaces Direct every node reports the clustered pools and
// virtual disks, so these alerts fire once per node as well.
+ group('windows-storage-spaces', std.flattenArrays([
// The primordial pool holds the disks that are not in a concrete pool
// yet; only the info metric tells it apart.
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 2) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
c.criticalSeverity,
'Storage pool is unhealthy',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 1) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
'warning',
'Storage pool has a health warning',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 2',
'10m',
c.criticalSeverity,
'Storage Spaces virtual disk is unhealthy',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 1',
'10m',
'warning',
'Storage Spaces virtual disk has a health warning',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
]))
+ group('windows-dhcp', std.flattenArrays([
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeWarningThreshold, dhcpScopeActive],
'15m',
'warning',
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeWarningThreshold,
collectors=['dhcp']
),
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeCriticalThreshold, dhcpScopeActive],
'15m',
c.criticalSeverity,
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeCriticalThreshold,
collectors=['dhcp']
),
]))
+ group('windows-dns', std.flattenArrays([
alert(
'WindowsDNSZoneTransferFailures',
q.rate('windows_dns_zone_transfer_failures_total') + ' > %g' % c.dnsZoneTransferFailureRateThreshold,
'15m',
'warning',
'DNS zone transfers are failing',
'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
collectors=['dns']
),
])),
},
}
+ (import 'exporter.libsonnet')
+ (import 'cpu.libsonnet')
+ (import 'memory.libsonnet')
+ (import 'pagefile.libsonnet')
+ (import 'logical_disk.libsonnet')
+ (import 'diskdrive.libsonnet')
+ (import 'net.libsonnet')
+ (import 'service.libsonnet')
+ (import 'system.libsonnet')
+ (import 'license.libsonnet')
+ (import 'ad.libsonnet')
+ (import 'time.libsonnet')
+ (import 'scheduled_task.libsonnet')
+ (import 'update.libsonnet')
+ (import 'hyperv.libsonnet')
+ (import 'mscluster.libsonnet')
+ (import 'storage_spaces.libsonnet')
+ (import 'dhcp.libsonnet')
+ (import 'dns.libsonnet')
contrib/mixin/alerts/cpu.libsonnet
+33
View File
@@ -0,0 +1,33 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-cpu', collectors=['cpu'], rules=[
{
alert: 'WindowsCPUHighUsage',
expr: |||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows CPU usage is high',
description: 'CPU usage on {{ $labels.instance }} has exceeded %(cpuHighUsageThreshold)s%% for 15 minutes.' % c,
},
},
{
alert: 'WindowsCPUSaturation',
expr: |||
windows:cpu_queue_per_core:ratio{%(windowsExporterSelector)s} > %(cpuQueuePerCoreThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows processor queue is long',
description: 'More than %(cpuQueuePerCoreThreshold)s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c,
},
collectors:: ['cpu', 'system'],
},
]),
},
}
contrib/mixin/alerts/dhcp.libsonnet
+36
View File
@@ -0,0 +1,36 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-dhcp', collectors=['dhcp'], rules=[
// Percent of free addresses in a scope; a scope without addresses has
// no sample, and only scopes in an enabled state count.
{
alert: 'WindowsDHCPScopeAlmostExhausted',
expr: |||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeWarningThreshold)s
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'DHCP scope is almost exhausted',
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeWarningThreshold)s%% free addresses for 15 minutes.' % c,
},
},
{
alert: 'WindowsDHCPScopeAlmostExhausted',
expr: |||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeCriticalThreshold)s
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
||| % c,
'for': '15m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'DHCP scope is almost exhausted',
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeCriticalThreshold)s%% free addresses for 15 minutes.' % c,
},
},
]),
},
}
contrib/mixin/alerts/diskdrive.libsonnet
+24
View File
@@ -0,0 +1,24 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-diskdrive', collectors=['diskdrive'], rules=[
{
alert: 'WindowsDiskDriveNotOK',
// One series per status with the current status set to 1, so the
// alert carries the status the drive reports. Some USB and virtual
// drives report Unknown for good.
expr: |||
windows_diskdrive_status{%(windowsExporterSelector)s, status!~"OK|Unknown"} == 1
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Windows disk drive is not healthy',
description: 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/dns.libsonnet
+20
View File
@@ -0,0 +1,20 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-dns', collectors=['dns'], rules=[
{
alert: 'WindowsDNSZoneTransferFailures',
expr: |||
rate(windows_dns_zone_transfer_failures_total{%(windowsExporterSelector)s}[%(rateInterval)s]) > %(dnsZoneTransferFailureRateThreshold)g
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'DNS zone transfers are failing',
description: 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
},
},
]),
},
}
contrib/mixin/alerts/exporter.libsonnet
+60
View File
@@ -0,0 +1,60 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-exporter', collectors=[], rules=[
{
alert: 'WindowsExporterDown',
expr: |||
up{%(windowsExporterSelector)s} == 0
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Windows exporter is unreachable',
description: 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
},
state:: true,
},
{
alert: 'WindowsCollectorFailed',
expr: |||
windows_exporter_collector_success{%(windowsExporterSelector)s} == 0
||| % c,
'for': '5m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows collector failed',
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
},
state:: true,
},
{
alert: 'WindowsCollectorTimeout',
expr: |||
windows_exporter_collector_timeout{%(windowsExporterSelector)s} == 1
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows collector exceeds the scrape timeout',
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
},
state:: true,
},
{
alert: 'WindowsScrapeQueueTimeout',
expr: |||
windows_exporter_scrape_queue_timeout{%(windowsExporterSelector)s} == 1
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows exporter scrapes time out in the queue',
description: 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/hyperv.libsonnet
+47
View File
@@ -0,0 +1,47 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-hyperv', collectors=['hyperv'], rules=[
{
alert: 'WindowsHyperVWMIUnhealthy',
expr: |||
windows_hyperv_wmi_health{%(windowsExporterSelector)s} == 0
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Hyper-V WMI namespace is not responding',
description: 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
},
state:: true,
},
{
alert: 'WindowsHyperVReplicationCritical',
expr: |||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 3
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Hyper-V replication is critical',
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
},
state:: true,
},
{
alert: 'WindowsHyperVReplicationWarning',
expr: |||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 2
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Hyper-V replication has a warning',
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/license.libsonnet
+21
View File
@@ -0,0 +1,21 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-license', collectors=['license'], rules=[
{
alert: 'WindowsLicenseNotGenuine',
expr: |||
windows_license_status{%(windowsExporterSelector)s, state!="genuine"} == 1
||| % c,
'for': '1h',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows license is not genuine',
description: 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/logical_disk.libsonnet
+69
View File
@@ -0,0 +1,69 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-disk', collectors=['logical_disk'], rules=[
{
alert: 'WindowsDiskReadLatencyHigh',
expr: |||
windows:logical_disk_read_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows disk read latency is high',
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per read operation for 15 minutes.' % c,
},
},
{
alert: 'WindowsDiskWriteLatencyHigh',
expr: |||
windows:logical_disk_write_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows disk write latency is high',
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per write operation for 15 minutes.' % c,
},
},
{
alert: 'WindowsDiskAlmostFull',
expr: |||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows disk space is low',
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeWarningThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
},
},
{
alert: 'WindowsDiskAlmostFull',
expr: |||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeCriticalThreshold)s
||| % c,
'for': '15m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Windows disk space is low',
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeCriticalThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
},
},
{
alert: 'WindowsDiskFillingUp',
expr: |||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
and predict_linear(windows_logical_disk_free_bytes{%(windowsExporterSelector)s, %(volumeSelector)s}[%(diskPredictionWindow)s], %(diskPredictionSeconds)s) < 0
||| % c { diskPredictionSeconds: c.diskPredictionHours * 3600 },
'for': '1h',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows disk is predicted to fill up',
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %(diskPredictionHours)s hours.' % c,
},
},
]),
},
}
contrib/mixin/alerts/memory.libsonnet
+44
View File
@@ -0,0 +1,44 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-memory', collectors=['memory'], rules=[
{
alert: 'WindowsMemoryHighUsage',
expr: |||
windows:memory_usage:ratio{%(windowsExporterSelector)s} * 100 > %(memoryHighUsageThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows memory usage is high',
description: 'Physical memory usage on {{ $labels.instance }} has exceeded %(memoryHighUsageThreshold)s%% for 15 minutes.' % c,
},
},
{
alert: 'WindowsMemoryCommitHighUsage',
expr: |||
windows:memory_committed:ratio{%(windowsExporterSelector)s} * 100 > %(memoryCommitHighUsageThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows memory commit usage is high',
description: 'Committed memory on {{ $labels.instance }} has exceeded %(memoryCommitHighUsageThreshold)s%% of the commit limit for 15 minutes.' % c,
},
},
{
alert: 'WindowsMemoryPagingHigh',
expr: |||
windows:memory_swap_page_operations:rate{%(windowsExporterSelector)s} > %(memoryPagingThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows memory paging is high',
description: 'Paging on {{ $labels.instance }} has exceeded %(memoryPagingThreshold)s pages per second for 15 minutes.' % c,
},
},
]),
},
}
contrib/mixin/alerts/mscluster.libsonnet
+73
View File
@@ -0,0 +1,73 @@
// Every cluster node reports the state of the whole cluster, so these
// alerts fire once per node; group them by cluster labels in Alertmanager.
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-cluster', collectors=['mscluster'], rules=[
{
alert: 'WindowsClusterNodeDown',
expr: |||
windows_mscluster_node_state{%(windowsExporterSelector)s} == 1
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Failover cluster node is down',
description: 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
},
state:: true,
},
{
alert: 'WindowsClusterResourceFailed',
expr: |||
windows_mscluster_resource_state{%(windowsExporterSelector)s} == 4
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Failover cluster resource has failed',
description: 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
},
state:: true,
},
{
alert: 'WindowsClusterResourceGroupFailed',
expr: |||
windows_mscluster_resourcegroup_state{%(windowsExporterSelector)s} == 2
||| % c,
'for': '5m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Failover cluster resource group has failed',
description: 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
},
state:: true,
},
{
alert: 'WindowsClusterSharedVolumeAlmostFull',
expr: |||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeWarningThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Cluster shared volume is almost full',
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeWarningThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
},
},
{
alert: 'WindowsClusterSharedVolumeAlmostFull',
expr: |||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeCriticalThreshold)s
||| % c,
'for': '15m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Cluster shared volume is almost full',
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeCriticalThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
},
},
]),
},
}
contrib/mixin/alerts/net.libsonnet
+63
View File
@@ -0,0 +1,63 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-network', collectors=['net'], rules=[
{
alert: 'WindowsNetworkErrors',
expr: |||
(
(windows:net_received_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s})
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
) * 100 > %(networkErrorRatioThreshold)g
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows network packet errors are high',
description: 'Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkErrorRatioThreshold)g%% of all packets for 15 minutes.' % c,
},
},
{
alert: 'WindowsNetworkDiscards',
expr: |||
(
(windows:net_received_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s})
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
) * 100 > %(networkDiscardRatioThreshold)g
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows network packet discards are high',
description: 'Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkDiscardRatioThreshold)g%% of all packets for 15 minutes.' % c,
},
},
{
alert: 'WindowsNetworkUtilizationHigh',
expr: |||
(windows:net_received_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
or (windows:net_sent_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows network interface utilization is high',
description: 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %(networkUtilizationThreshold)s%% of its bandwidth in at least one direction for 15 minutes.' % c,
},
},
{
alert: 'WindowsNetworkInterfaceFlapping',
expr: |||
max without (status) (changes(windows_net_nic_operation_status{%(windowsExporterSelector)s, %(nicSelector)s, status="up"}[%(rateInterval)s])) > 2
||| % c,
'for': '2m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows network interface is flapping',
description: 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %(rateInterval)s.' % c,
},
},
]),
},
}
contrib/mixin/alerts/pagefile.libsonnet
+20
View File
@@ -0,0 +1,20 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-pagefile', collectors=['pagefile'], rules=[
{
alert: 'WindowsPagefileAlmostFull',
expr: |||
(1 - (windows_pagefile_free_bytes{%(windowsExporterSelector)s} / (windows_pagefile_limit_bytes{%(windowsExporterSelector)s} > 0))) * 100 > %(pagefileUsageThreshold)s
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows page file is almost full',
description: 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %(pagefileUsageThreshold)s%% in use for 15 minutes.' % c,
},
},
]),
},
}
contrib/mixin/alerts/scheduled_task.libsonnet
+26
View File
@@ -0,0 +1,26 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-scheduled-task', collectors=['scheduled_task'], rules=[
{
alert: 'WindowsScheduledTaskFailed',
// unknown is any result code without a named status, including the
// exit codes of the task's program.
// A disabled task keeps its last result, so disabling a broken task
// clears the alert.
expr: |||
windows_scheduled_task_last_result_status{%(windowsExporterSelector)s, %(taskSelector)s, status=~"unknown|terminated"} == 1
unless ignoring (status, state) (windows_scheduled_task_state{%(windowsExporterSelector)s, %(taskSelector)s, state="disabled"} == 1)
||| % c,
'for': '1h',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows scheduled task did not succeed',
description: 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/service.libsonnet
+38
View File
@@ -0,0 +1,38 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-service', collectors=['service'], rules=[
{
alert: 'WindowsServiceNotRunning',
// The matches ignore only the labels of the right-hand metric, so
// every target label, including ones a selector adds, takes part.
expr: |||
max without (state) (windows_service_state{%(windowsExporterSelector)s, %(serviceSelector)s, state="running"} == 0)
and ignoring (start_mode) (windows_service_start_mode{%(windowsExporterSelector)s, %(serviceSelector)s, start_mode="auto"} == 1)
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{%(windowsExporterSelector)s, %(serviceSelector)s, trigger_start="true"} == 1)
||| % c,
'for': '15m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows automatic service is not running',
description: 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
},
state:: true,
},
{
alert: 'WindowsServiceCrashLooping',
// Every service process has its own series, so the series seen in the
// window count the processes the service ran as.
expr: |||
count without (process_id) (count_over_time(windows_service_process{%(windowsExporterSelector)s, %(serviceSelector)s}[15m])) - 1 > %(serviceRestartThreshold)s
||| % c,
labels: { severity: 'warning' },
annotations: {
summary: 'Windows service is restarting repeatedly',
description: 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %(serviceRestartThreshold)s times in 15 minutes.' % c,
},
},
]),
},
}
contrib/mixin/alerts/storage_spaces.libsonnet
+66
View File
@@ -0,0 +1,66 @@
// On Storage Spaces Direct every node reports the clustered pools and
// virtual disks, so these alerts fire once per node as well.
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-storage-spaces', collectors=['storage_spaces'], rules=[
{
alert: 'WindowsStoragePoolUnhealthy',
// The primordial pool holds the disks that are not in a concrete pool
// yet; only the info metric tells it apart.
expr: |||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 2)
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
||| % c,
'for': '10m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Storage pool is unhealthy',
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
},
state:: true,
},
{
alert: 'WindowsStoragePoolUnhealthy',
expr: |||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 1)
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Storage pool has a health warning',
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
},
state:: true,
},
{
alert: 'WindowsStorageVirtualDiskUnhealthy',
expr: |||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 2
||| % c,
'for': '10m',
labels: { severity: c.criticalSeverity },
annotations: {
summary: 'Storage Spaces virtual disk is unhealthy',
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
},
state:: true,
},
{
alert: 'WindowsStorageVirtualDiskUnhealthy',
expr: |||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 1
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Storage Spaces virtual disk has a health warning',
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/system.libsonnet
+25
View File
@@ -0,0 +1,25 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-system', collectors=['system'], rules=[
{
alert: 'WindowsHostRebooted',
// Fires once per reboot: for the first ten minutes after the boot,
// and only on a host that was monitored before it. The exporter is
// down while the host reboots, so the second operand takes the last
// boot time of the hour before, not the sample of ten minutes ago.
expr: |||
(time() - windows_system_boot_time_timestamp{%(windowsExporterSelector)s}) < 600
and (time() - last_over_time(windows_system_boot_time_timestamp{%(windowsExporterSelector)s}[1h] offset 10m)) > 600
||| % c,
labels: { severity: c.infoSeverity },
annotations: {
summary: 'Windows host has rebooted',
description: '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/time.libsonnet
+47
View File
@@ -0,0 +1,47 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-time', collectors=['time'], rules=[
{
alert: 'WindowsClockNotSynchronising',
expr: |||
windows_time_ntp_client_time_sources{%(windowsExporterSelector)s} == 0
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows clock has no NTP time source',
description: 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
},
state:: true,
},
{
alert: 'WindowsClockSkewDetected',
expr: |||
abs(windows_time_computed_time_offset_seconds{%(windowsExporterSelector)s}) > %(clockOffsetThresholdSeconds)g
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows clock offset is high',
description: 'Windows Time on {{ $labels.instance }} reports a clock offset above %(clockOffsetThresholdSeconds)g seconds for 10 minutes.' % c,
},
},
{
alert: 'WindowsClockSourceLocal',
// Covers domain members (NT5DS) too, which the NTP source count misses.
expr: |||
windows_time_clock_sync_source{%(windowsExporterSelector)s, type=~"Local CMOS Clock|NoSync"} == 1
||| % c,
'for': '10m',
labels: { severity: 'warning' },
annotations: {
summary: 'Windows clock has no external time source',
description: 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
},
state:: true,
},
]),
},
}
contrib/mixin/alerts/update.libsonnet
+21
View File
@@ -0,0 +1,21 @@
{
local c = self._config,
local a = (import '../lib/alerts.libsonnet')(c),
prometheusAlerts+:: {
groups+: a.group('windows-update', collectors=['update'], rules=[
{
alert: 'WindowsUpdatesPending',
expr: |||
count without (id, revision, category, severity, title) (windows_update_pending_info{%(windowsExporterSelector)s, deployment_action="installation"}) > 0
||| % c,
'for': c.updatePendingDuration,
labels: { severity: c.infoSeverity },
annotations: {
summary: 'Windows updates are waiting for installation',
description: '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %(updatePendingDuration)s.' % c,
},
state:: true,
},
]),
},
}
contrib/mixin/lib/alerts.libsonnet
+23
View File
@@ -0,0 +1,23 @@
// Helpers shared by the alert files in alerts/. Takes the mixin _config.
function(c)
{
// A collector missing from a replaced collectors object counts as disabled;
// an alert missing from a replaced alerts object counts as enabled.
collectorEnabled(collector): std.get(c.collectors, collector, false),
alertEnabled(name): std.get(c.alerts, name, true),
// group renders the rules of one collector as a rule group, or nothing
// when none of them is enabled. Each rule is a plain Prometheus alerting
// rule plus two hidden fields: collectors:: lists the collectors the rule
// reads when it differs from the group's, and state:: true marks state
// alerts (== 0, == 1) and one-shot alerts, which must resolve at once.
// Every other alert is a threshold alert and keeps firing for
// keepFiringFor after its condition clears, so a value oscillating around
// the threshold does not flap.
group(name, collectors, rules):
local enabled = [
rule + (if c.keepFiringFor != '' && !std.get(rule, 'state', false) then { keep_firing_for: c.keepFiringFor } else {})
for rule in rules
if self.alertEnabled(rule.alert) && std.all([self.collectorEnabled(collector) for collector in std.get(rule, 'collectors', collectors)])
];
if std.length(enabled) > 0 then [{ name: name, rules: enabled }] else [],
}
contrib/mixin/tests/alerts.test.jsonnet
+5 -4
View File
@@ -3,7 +3,7 @@
local mixin = import '../mixin.libsonnet';
local config = (import '../config.libsonnet')._config;
local everything = mixin { _config+:: { collectors: { [name]: true for name in std.objectFields(config.collectors) } } };
local optionalGroups = ['windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
local optionalGroups = ['windows-pagefile', 'windows-diskdrive', 'windows-license', 'windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
local groupNames(alerts) = [group.name for group in alerts.groups];
@@ -13,12 +13,13 @@ assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs fro
// Groups follow the collector areas; opt-in groups appear only when enabled.
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + optionalGroups : std.manifestJson(groupNames(everything.prometheusAlerts));
// Opt-in collectors add their alerts to the default groups of their area too.
// One file per collector, one group per file: the groups of every enabled
// collector come in the order of the imports in alerts/alerts.libsonnet.
assert std.set(groupNames(everything.prometheusAlerts)) == std.set(groupNames(mixin.prometheusAlerts) + optionalGroups) : std.manifestJson(groupNames(everything.prometheusAlerts));
assert groupNames(everything.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-pagefile', 'windows-disk', 'windows-diskdrive', 'windows-network', 'windows-service', 'windows-system', 'windows-license'] + optionalGroups[3:] : std.manifestJson(groupNames(everything.prometheusAlerts));
local adOnly = mixin { _config+:: { collectors+: { ad: true } } };
assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory'];
local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } };
assert groupNames(pagefile.prometheusAlerts) == groupNames(mixin.prometheusAlerts);
assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull'];
// Disabling a collector removes its alerts and an empty group; the alerts of a
contrib/mixin/windows_alerts.yaml
+53 -21
View File
@@ -6,7 +6,8 @@ groups:
description: "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/"
summary: "Windows exporter is unreachable"
expr: "up{job=\"windows_exporter\"} == 0"
expr: |
up{job="windows_exporter"} == 0
for: "5m"
labels:
severity: "critical"
@@ -15,7 +16,8 @@ groups:
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/"
summary: "Windows collector failed"
expr: "windows_exporter_collector_success{job=\"windows_exporter\"} == 0"
expr: |
windows_exporter_collector_success{job="windows_exporter"} == 0
for: "5m"
labels:
severity: "warning"
@@ -24,7 +26,8 @@ groups:
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/"
summary: "Windows collector exceeds the scrape timeout"
expr: "windows_exporter_collector_timeout{job=\"windows_exporter\"} == 1"
expr: |
windows_exporter_collector_timeout{job="windows_exporter"} == 1
for: "10m"
labels:
severity: "warning"
@@ -33,7 +36,8 @@ groups:
description: "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/"
summary: "Windows exporter scrapes time out in the queue"
expr: "windows_exporter_scrape_queue_timeout{job=\"windows_exporter\"} == 1"
expr: |
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
for: "10m"
labels:
severity: "warning"
@@ -44,7 +48,8 @@ groups:
description: "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/"
summary: "Windows CPU usage is high"
expr: "windows:cpu_usage:ratio{job=\"windows_exporter\"} * 100 > 90"
expr: |
windows:cpu_usage:ratio{job="windows_exporter"} * 100 > 90
for: "15m"
keep_firing_for: "5m"
labels:
@@ -54,7 +59,8 @@ groups:
description: "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/"
summary: "Windows processor queue is long"
expr: "windows:cpu_queue_per_core:ratio{job=\"windows_exporter\"} > 2"
expr: |
windows:cpu_queue_per_core:ratio{job="windows_exporter"} > 2
for: "15m"
keep_firing_for: "5m"
labels:
@@ -66,7 +72,8 @@ groups:
description: "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/"
summary: "Windows memory usage is high"
expr: "windows:memory_usage:ratio{job=\"windows_exporter\"} * 100 > 90"
expr: |
windows:memory_usage:ratio{job="windows_exporter"} * 100 > 90
for: "15m"
keep_firing_for: "5m"
labels:
@@ -76,7 +83,8 @@ groups:
description: "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/"
summary: "Windows memory commit usage is high"
expr: "windows:memory_committed:ratio{job=\"windows_exporter\"} * 100 > 90"
expr: |
windows:memory_committed:ratio{job="windows_exporter"} * 100 > 90
for: "15m"
keep_firing_for: "5m"
labels:
@@ -86,7 +94,8 @@ groups:
description: "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/"
summary: "Windows memory paging is high"
expr: "windows:memory_swap_page_operations:rate{job=\"windows_exporter\"} > 1000"
expr: |
windows:memory_swap_page_operations:rate{job="windows_exporter"} > 1000
for: "15m"
keep_firing_for: "5m"
labels:
@@ -98,7 +107,8 @@ groups:
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/"
summary: "Windows disk read latency is high"
expr: "windows:logical_disk_read_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05"
expr: |
windows:logical_disk_read_latency:seconds{job="windows_exporter", volume!=""} > 0.05
for: "15m"
keep_firing_for: "5m"
labels:
@@ -108,7 +118,8 @@ groups:
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/"
summary: "Windows disk write latency is high"
expr: "windows:logical_disk_write_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05"
expr: |
windows:logical_disk_write_latency:seconds{job="windows_exporter", volume!=""} > 0.05
for: "15m"
keep_firing_for: "5m"
labels:
@@ -118,7 +129,8 @@ groups:
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
summary: "Windows disk space is low"
expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10"
expr: |
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
for: "15m"
keep_firing_for: "5m"
labels:
@@ -128,7 +140,8 @@ groups:
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
summary: "Windows disk space is low"
expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 5"
expr: |
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 5
for: "15m"
keep_firing_for: "5m"
labels:
@@ -138,7 +151,9 @@ groups:
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/"
summary: "Windows disk is predicted to fill up"
expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10 and predict_linear(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}[6h], 86400) < 0"
expr: |
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
and predict_linear(windows_logical_disk_free_bytes{job="windows_exporter", volume!=""}[6h], 86400) < 0
for: "1h"
keep_firing_for: "5m"
labels:
@@ -150,7 +165,11 @@ groups:
description: "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/"
summary: "Windows network packet errors are high"
expr: "((windows:net_received_errors:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_errors:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1"
expr: |
(
(windows:net_received_errors:rate{job="windows_exporter", nic!=""} + windows:net_outbound_errors:rate{job="windows_exporter", nic!=""})
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
) * 100 > 1
for: "15m"
keep_firing_for: "5m"
labels:
@@ -160,7 +179,11 @@ groups:
description: "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/"
summary: "Windows network packet discards are high"
expr: "((windows:net_received_discarded:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_discarded:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1"
expr: |
(
(windows:net_received_discarded:rate{job="windows_exporter", nic!=""} + windows:net_outbound_discarded:rate{job="windows_exporter", nic!=""})
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
) * 100 > 1
for: "15m"
keep_firing_for: "5m"
labels:
@@ -170,7 +193,9 @@ groups:
description: "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/"
summary: "Windows network interface utilization is high"
expr: "(windows:net_received_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90) or (windows:net_sent_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90)"
expr: |
(windows:net_received_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
or (windows:net_sent_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
for: "15m"
keep_firing_for: "5m"
labels:
@@ -180,7 +205,8 @@ groups:
description: "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/"
summary: "Windows network interface is flapping"
expr: "max without (status) (changes(windows_net_nic_operation_status{job=\"windows_exporter\", nic!=\"\", status=\"up\"}[5m])) > 2"
expr: |
max without (status) (changes(windows_net_nic_operation_status{job="windows_exporter", nic!="", status="up"}[5m])) > 2
for: "2m"
keep_firing_for: "5m"
labels:
@@ -192,7 +218,10 @@ groups:
description: "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/"
summary: "Windows automatic service is not running"
expr: "max without (state) (windows_service_state{job=\"windows_exporter\", service!=\"\", state=\"running\"} == 0) and ignoring (start_mode) (windows_service_start_mode{job=\"windows_exporter\", service!=\"\", start_mode=\"auto\"} == 1) unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job=\"windows_exporter\", service!=\"\", trigger_start=\"true\"} == 1)"
expr: |
max without (state) (windows_service_state{job="windows_exporter", service!="", state="running"} == 0)
and ignoring (start_mode) (windows_service_start_mode{job="windows_exporter", service!="", start_mode="auto"} == 1)
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job="windows_exporter", service!="", trigger_start="true"} == 1)
for: "15m"
labels:
severity: "warning"
@@ -201,7 +230,8 @@ groups:
description: "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/"
summary: "Windows service is restarting repeatedly"
expr: "count without (process_id) (count_over_time(windows_service_process{job=\"windows_exporter\", service!=\"\"}[15m])) - 1 > 2"
expr: |
count without (process_id) (count_over_time(windows_service_process{job="windows_exporter", service!=""}[15m])) - 1 > 2
keep_firing_for: "5m"
labels:
severity: "warning"
@@ -212,6 +242,8 @@ groups:
description: "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago."
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/"
summary: "Windows host has rebooted"
expr: "(time() - windows_system_boot_time_timestamp{job=\"windows_exporter\"}) < 600 and (time() - last_over_time(windows_system_boot_time_timestamp{job=\"windows_exporter\"}[1h] offset 10m)) > 600"
expr: |
(time() - windows_system_boot_time_timestamp{job="windows_exporter"}) < 600
and (time() - last_over_time(windows_system_boot_time_timestamp{job="windows_exporter"}[1h] offset 10m)) > 600
labels:
severity: "info"
docs/upgrade/0.32.md
+1 -1
View File
@@ -116,7 +116,7 @@ grep -rnwFf old-names.txt --exclude=windows_exporter-0.32.prom path/to/dashboard
This doesn't find the label renames and the metrics without a compatibility rule; check those against the tables.
If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/alerts.libsonnet`](../../contrib/mixin/alerts/alerts.libsonnet) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too.
If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/`](../../contrib/mixin/alerts) (one file per collector) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too.
### 5. Remove the compatibility rules