sendnrw/windows_exporter
2
0
Fork 0
mirror of https://github.com/prometheus-community/windows_exporter.git synced 2026-10-17 18:19:04 +02:00
Code Issues Packages Projects Releases Wiki Activity

mixin: add alerts for the opt-in collectors

Browse Source 
Every optional collector of the mixin now has alerts, generated only with
that collector enabled in _config.collectors: clock source, disk drive
status, page file usage, license status, scheduled task results, pending
updates, Hyper-V WMI and replication health, failover cluster nodes,
resources, groups and shared volumes, Storage Spaces pool and virtual disk
health, DHCP scope exhaustion and DNS zone transfer failures. The alerts
of the pagefile, diskdrive and license collectors join the memory, disk
and system groups; the other collectors get a group of their own.

New knobs: taskSelector, sharedVolumeSelector, pagefileUsageThreshold,
dhcpScopeFreeWarningThreshold, dhcpScopeFreeCriticalThreshold and
updatePendingDuration. The test configuration enables every optional
collector and tests/optional.test.yaml covers the new alerts.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-17 02:28:35 +00:00
committed by Claude
parent d6afd48819
commit d819362150
24 changed files with 1688 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/mixin/README.md
+57 -7
View File
@@ -85,7 +85,10 @@ apply to both recording rules and alerts. All target labels, including `job`,
averages idle rates across cores and removes only `core` and `mode`. Disk,
network and service rules also retain `volume`, `nic` and `service`
respectively, filtered by `volumeSelector`, `nicSelector` and
`serviceSelector`. The CPU mode recording retains `mode` and removes only
`serviceSelector`. The scheduled task and cluster shared volume alerts
filter their `task` and `shared_volume` labels with `taskSelector`
(`task!=""`) and `sharedVolumeSelector`
(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only
`core`.
`rateInterval` is the fixed window for recording rules and Active Directory
@@ -179,6 +182,27 @@ percentages except where noted.
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning |
| `windows-memory` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
| `windows-disk` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
| `windows-system` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning |
| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info |
| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning |
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical |
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning |
| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning |
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning |
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning |
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical |
| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning |
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
@@ -188,9 +212,11 @@ alerts are all turned off is not generated.
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
value that oscillates around a threshold does not flap. The state alerts
`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`,
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning` and
`WindowsClockNotSynchronising`, and the one-shot `WindowsHostRebooted`, have no
`keep_firing_for` and resolve as soon as their condition clears.
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`,
`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors
(clock source, drive status, license, task result, pending updates, Hyper-V,
cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`,
have no `keep_firing_for` and resolve as soon as their condition clears.
`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for
older servers.
@@ -206,9 +232,33 @@ notification should be sent.
Active Directory alerts need `collectors.ad`, which `enableActiveDirectory`
sets. Clock alerts need `collectors.time`, which `enableTime` sets. A zero NTP
source count may be expected on hosts using another time provider. The offset
compares the host to its chosen source and does not independently verify that
source. Disk latency, paging and network thresholds should be tuned for the
source count may be expected on hosts using another time provider;
`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and
fires only for the local clock or no synchronization. The offset compares the
host to its chosen source and does not independently verify that source.
The other opt-in alerts follow `_config.collectors`: enable `pagefile`,
`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`,
`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are
generated, in the group of their area or in a group of their own.
`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code
without a name, including the exit codes of the task's program, as a failure;
narrow the tasks with `taskSelector`. Disabling a task clears the alert.
`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and
virtual drives report permanently. `WindowsLicenseNotGenuine` reports the
`state` the host has; hosts without internet access can stay in `offline`,
so turn the alert off there. Every failover cluster node reports the state of
the whole cluster, and every Storage Spaces Direct node reports the clustered
pools and virtual disks, so the cluster and storage alerts fire once per node;
group them by their `node`, `resource`, `shared_volume`, `pool` or
`virtual_disk` labels in Alertmanager. The storage pool alert joins
`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP
scope alert only counts scopes in an enabled state, and a scope without
addresses produces no sample. `WindowsUpdatesPending` is an info alert that
waits `updatePendingDuration` for the normal maintenance window. Prometheus
restores the waiting time after a restart shorter than its
`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage
or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the
storage tier, traffic volume, and application requirements.
`WindowsServiceNotRunning` watches services with the `auto` start mode and
contrib/mixin/alerts/alerts.libsonnet
+244
View File
@@ -25,6 +25,12 @@
} + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
] else [],
local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
// Percent of free addresses in a scope; a scope without addresses has no sample.
local dhcpScopeFree = '(%s * 100)' % q.ratio(
q.metric('windows_dhcp_scope_addresses_free'),
'(%s + %s)' % [q.metric('windows_dhcp_scope_addresses_free'), q.metric('windows_dhcp_scope_addresses_in_use')]
),
local dhcpScopeActive = q.metric('windows_dhcp_scope_state', ['state=~"Enabled|EnabledSwitched"']),
prometheusAlerts+:: {
groups:
group('windows-exporter', std.flattenArrays([
@@ -113,6 +119,18 @@
'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold,
collectors=['memory']
),
alert(
'WindowsPagefileAlmostFull',
'(1 - %s) * 100 > %s' % [
q.ratio(q.metric('windows_pagefile_free_bytes'), q.metric('windows_pagefile_limit_bytes')),
c.pagefileUsageThreshold,
],
'15m',
'warning',
'Windows page file is almost full',
'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %s%% in use for 15 minutes.' % c.pagefileUsageThreshold,
collectors=['pagefile']
),
]))
+ group('windows-disk', std.flattenArrays([
alert(
@@ -154,6 +172,19 @@
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours,
collectors=['logical_disk']
),
alert(
'WindowsDiskDriveNotOK',
// One series per status with the current status set to 1, so the
// alert carries the status the drive reports. Some USB and virtual
// drives report Unknown for good.
q.metric('windows_diskdrive_status', ['status!~"OK|Unknown"']) + ' == 1',
'5m',
c.criticalSeverity,
'Windows disk drive is not healthy',
'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
collectors=['diskdrive'],
threshold=false
),
]))
+ group('windows-network', std.flattenArrays([
alert(
@@ -256,6 +287,16 @@
collectors=['system'],
threshold=false
),
alert(
'WindowsLicenseNotGenuine',
q.metric('windows_license_status', ['state!="genuine"']) + ' == 1',
'1h',
'warning',
'Windows license is not genuine',
'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
collectors=['license'],
threshold=false
),
]))
+ group('windows-active-directory', std.flattenArrays([
alert(
@@ -297,6 +338,209 @@
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds,
collectors=['time']
),
alert(
'WindowsClockSourceLocal',
// Covers domain members (NT5DS) too, which the NTP source count misses.
q.metric('windows_time_clock_sync_source', ['type=~"Local CMOS Clock|NoSync"']) + ' == 1',
'10m',
'warning',
'Windows clock has no external time source',
'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
collectors=['time'],
threshold=false
),
]))
+ group('windows-scheduled-task', std.flattenArrays([
alert(
'WindowsScheduledTaskFailed',
// unknown is any result code without a named status, including the
// exit codes of the task's program.
// A disabled task keeps its last result, so disabling a broken task
// clears the alert.
'%s == 1 unless ignoring (status, state) (%s == 1)' % [
q.metric('windows_scheduled_task_last_result_status', [c.taskSelector, 'status=~"unknown|terminated"']),
q.metric('windows_scheduled_task_state', [c.taskSelector, 'state="disabled"']),
],
'1h',
'warning',
'Windows scheduled task did not succeed',
'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
collectors=['scheduled_task'],
threshold=false
),
]))
+ group('windows-update', std.flattenArrays([
alert(
'WindowsUpdatesPending',
'count without (id, revision, category, severity, title) (%s) > 0' % q.metric('windows_update_pending_info', ['deployment_action="installation"']),
c.updatePendingDuration,
c.infoSeverity,
'Windows updates are waiting for installation',
'{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %s.' % c.updatePendingDuration,
collectors=['update'],
threshold=false
),
]))
+ group('windows-hyperv', std.flattenArrays([
alert(
'WindowsHyperVWMIUnhealthy',
q.metric('windows_hyperv_wmi_health') + ' == 0',
'10m',
'warning',
'Hyper-V WMI namespace is not responding',
'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationCritical',
q.metric('windows_hyperv_replica_vm_health') + ' == 3',
'5m',
c.criticalSeverity,
'Hyper-V replication is critical',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationWarning',
q.metric('windows_hyperv_replica_vm_health') + ' == 2',
'15m',
'warning',
'Hyper-V replication has a warning',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
collectors=['hyperv'],
threshold=false
),
]))
// Every cluster node reports the state of the whole cluster, so these
// alerts fire once per node; group them by cluster labels in Alertmanager.
+ group('windows-cluster', std.flattenArrays([
alert(
'WindowsClusterNodeDown',
q.metric('windows_mscluster_node_state') + ' == 1',
'5m',
c.criticalSeverity,
'Failover cluster node is down',
'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceFailed',
q.metric('windows_mscluster_resource_state') + ' == 4',
'5m',
c.criticalSeverity,
'Failover cluster resource has failed',
'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceGroupFailed',
q.metric('windows_mscluster_resourcegroup_state') + ' == 2',
'5m',
c.criticalSeverity,
'Failover cluster resource group has failed',
'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeWarningThreshold,
'15m',
'warning',
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeWarningThreshold,
collectors=['mscluster']
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeCriticalThreshold,
'15m',
c.criticalSeverity,
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeCriticalThreshold,
collectors=['mscluster']
),
]))
// On Storage Spaces Direct every node reports the clustered pools and
// virtual disks, so these alerts fire once per node as well.
+ group('windows-storage-spaces', std.flattenArrays([
// The primordial pool holds the disks that are not in a concrete pool
// yet; only the info metric tells it apart.
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 2) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
c.criticalSeverity,
'Storage pool is unhealthy',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 1) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
'warning',
'Storage pool has a health warning',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 2',
'10m',
c.criticalSeverity,
'Storage Spaces virtual disk is unhealthy',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 1',
'10m',
'warning',
'Storage Spaces virtual disk has a health warning',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
]))
+ group('windows-dhcp', std.flattenArrays([
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeWarningThreshold, dhcpScopeActive],
'15m',
'warning',
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeWarningThreshold,
collectors=['dhcp']
),
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeCriticalThreshold, dhcpScopeActive],
'15m',
c.criticalSeverity,
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeCriticalThreshold,
collectors=['dhcp']
),
]))
+ group('windows-dns', std.flattenArrays([
alert(
'WindowsDNSZoneTransferFailures',
q.rate('windows_dns_zone_transfer_failures_total') + ' > %g' % c.dnsZoneTransferFailureRateThreshold,
'15m',
'warning',
'DNS zone transfers are failing',
'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
collectors=['dns']
),
])),
},
}
contrib/mixin/config.libsonnet
+34
View File
@@ -5,6 +5,9 @@
volumeSelector: 'volume!=""',
nicSelector: 'nic!=""',
serviceSelector: 'service!=""',
taskSelector: 'task!=""',
// ClusterPerformanceHistory is the volume of the cluster's own health history.
sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"',
rateInterval: '5m',
// Opt in only for targets with the ad collector enabled. Sets collectors.ad.
@@ -43,14 +46,19 @@
collectors: {
ad: $._config.enableActiveDirectory,
cpu: true,
dhcp: false,
diskdrive: false,
dns: false,
gpu: false,
hyperv: false,
license: false,
logical_disk: true,
memory: true,
mscluster: false,
mssql: false,
net: true,
os: true,
pagefile: false,
physical_disk: true,
process: false,
scheduled_task: false,
@@ -94,6 +102,23 @@
WindowsADReplicationFailures: true,
WindowsClockNotSynchronising: true,
WindowsClockSkewDetected: true,
WindowsClockSourceLocal: true,
WindowsDiskDriveNotOK: true,
WindowsPagefileAlmostFull: true,
WindowsLicenseNotGenuine: true,
WindowsScheduledTaskFailed: true,
WindowsUpdatesPending: true,
WindowsHyperVWMIUnhealthy: true,
WindowsHyperVReplicationCritical: true,
WindowsHyperVReplicationWarning: true,
WindowsClusterNodeDown: true,
WindowsClusterResourceFailed: true,
WindowsClusterResourceGroupFailed: true,
WindowsClusterSharedVolumeAlmostFull: true,
WindowsStoragePoolUnhealthy: true,
WindowsStorageVirtualDiskUnhealthy: true,
WindowsDHCPScopeAlmostExhausted: true,
WindowsDNSZoneTransferFailures: true,
},
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
@@ -127,5 +152,14 @@
infoSeverity: 'info',
adPendingReplicationThreshold: 50,
adReplicationFailureRateThreshold: 0,
// Percent of the page file in use.
pagefileUsageThreshold: 80,
// Percent of free addresses in an active DHCP scope.
dhcpScopeFreeWarningThreshold: 10,
dhcpScopeFreeCriticalThreshold: 2,
// How long updates may wait for installation before WindowsUpdatesPending fires.
updatePendingDuration: '7d',
// Failed zone transfers per second.
dnsZoneTransferFailureRateThreshold: 0,
},
}
contrib/mixin/tests/alerts.test.jsonnet
+33 -3
View File
@@ -2,7 +2,8 @@
// keep_firing_for and the group layout.
local mixin = import '../mixin.libsonnet';
local config = (import '../config.libsonnet')._config;
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true } };
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true, collectors: { [name]: true for name in std.objectFields(config.collectors) } } };
local optionalGroups = ['windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
local groupNames(alerts) = [group.name for group in alerts.groups];
@@ -12,7 +13,13 @@ assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs fro
// Groups follow the collector areas; opt-in groups appear only when enabled.
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory', 'windows-time'];
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + optionalGroups : std.manifestJson(groupNames(everything.prometheusAlerts));
// Opt-in collectors add their alerts to the default groups of their area too.
local adOnly = mixin { _config+:: { enableActiveDirectory: true } };
assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory'];
local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } };
assert groupNames(pagefile.prometheusAlerts) == groupNames(mixin.prometheusAlerts);
assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull'];
// Disabling a collector removes its alerts and an empty group; the alerts of a
// rule that needs two collectors go with either.
@@ -34,7 +41,28 @@ assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAle
// keep_firing_for applies to threshold alerts only and can be turned off.
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
local stateAlerts = ['WindowsExporterDown', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsScrapeQueueTimeout', 'WindowsServiceNotRunning', 'WindowsHostRebooted', 'WindowsClockNotSynchronising'];
local stateAlerts = [
'WindowsExporterDown',
'WindowsCollectorFailed',
'WindowsCollectorTimeout',
'WindowsScrapeQueueTimeout',
'WindowsServiceNotRunning',
'WindowsHostRebooted',
'WindowsClockNotSynchronising',
'WindowsClockSourceLocal',
'WindowsDiskDriveNotOK',
'WindowsLicenseNotGenuine',
'WindowsScheduledTaskFailed',
'WindowsUpdatesPending',
'WindowsHyperVWMIUnhealthy',
'WindowsHyperVReplicationCritical',
'WindowsHyperVReplicationWarning',
'WindowsClusterNodeDown',
'WindowsClusterResourceFailed',
'WindowsClusterResourceGroupFailed',
'WindowsStoragePoolUnhealthy',
'WindowsStorageVirtualDiskUnhealthy',
];
assert std.all([
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
for rule in rules(everything.prometheusAlerts)
@@ -46,5 +74,7 @@ assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.object
// The info severity label is configurable.
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } };
assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']];
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
contrib/mixin/tests/config.libsonnet
+18
View File
@@ -19,5 +19,23 @@
clockOffsetThresholdSeconds: 0.1,
adPendingReplicationThreshold: 10,
adReplicationFailureRateThreshold: 0.5,
collectors+: {
dhcp: true,
diskdrive: true,
dns: true,
hyperv: true,
license: true,
mscluster: true,
pagefile: true,
scheduled_task: true,
storage_spaces: true,
update: true,
},
pagefileUsageThreshold: 75,
dhcpScopeFreeWarningThreshold: 20,
dhcpScopeFreeCriticalThreshold: 5,
updatePendingDuration: '2d',
dnsZoneTransferFailureRateThreshold: 0.05,
taskSelector: 'task!~"/Ignored.*"',
},
}
contrib/mixin/tests/optional.test.yaml
+508
View File
@@ -0,0 +1,508 @@
# Alerts of the opt-in collectors, rendered with tests/config.libsonnet.
rule_files:
- custom_rules.yaml
- custom_alerts.yaml
evaluation_interval: 1m
group_eval_order:
- windows-exporter-recording
- windows-memory
- windows-disk
- windows-system
- windows-time
- windows-scheduled-task
- windows-update
- windows-hyperv
- windows-cluster
- windows-storage-spaces
- windows-dhcp
- windows-dns
tests:
- name: Page file usage fires above the custom threshold and resolves after keep_firing_for
interval: 1m
input_series:
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
values: '20x19 50x10'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
values: '100x30'
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
values: '30x30'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
values: '100x30'
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
values: '0x30'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
values: '0x30'
alert_rule_test:
- {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []}
- alertname: WindowsPagefileAlmostFull
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
summary: Windows page file is almost full
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
- alertname: WindowsPagefileAlmostFull
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
summary: Windows page file is almost full
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
- {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []}
- name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour
interval: 1m
input_series:
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}'
values: '0x9 1x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}'
values: '1x9 0x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}'
values: '0x20'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}'
values: '1x20'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}'
values: '1x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}'
values: '0x10'
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}'
values: '0x60'
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}'
values: '1x60'
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}'
values: '1x60'
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}'
values: '0x60'
alert_rule_test:
- {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []}
- alertname: WindowsDiskDriveNotOK
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
summary: Windows disk drive is not healthy
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
- alertname: WindowsDiskDriveNotOK
eval_time: 9m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
summary: Windows disk drive is not healthy
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
- {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []}
- {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []}
- alertname: WindowsLicenseNotGenuine
eval_time: 60m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/
summary: Windows license is not genuine
description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.'
- name: Clock source alert names the local source and ignores NTP and domain sources
interval: 1m
input_series:
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}'
values: '1x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}'
values: '0x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}'
values: '1x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}'
values: '0x10'
alert_rule_test:
- {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []}
- alertname: WindowsClockSourceLocal
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/
summary: Windows clock has no external time source
description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.'
- name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks
interval: 1m
input_series:
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}'
values: '0x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}'
values: '0x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}'
values: '1x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}'
values: '0x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}'
values: '0x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}'
values: '1x60'
alert_rule_test:
- {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []}
- alertname: WindowsScheduledTaskFailed
eval_time: 60m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
summary: Windows scheduled task did not succeed
description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
summary: Windows scheduled task did not succeed
description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.'
- name: Pending updates fire as info after the configured duration and count installations only
interval: 1m
input_series:
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}'
values: '1x2880'
alert_rule_test:
- {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []}
- alertname: WindowsUpdatesPending
eval_time: 48h
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsupdatespending/
summary: Windows updates are waiting for installation
description: 'host:9182 has had 2 updates waiting for installation for 2d.'
- name: Hyper-V alerts cover the WMI namespace and both replication health levels
interval: 1m
input_series:
- series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}'
values: '0x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}'
values: '3x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}'
values: '2x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}'
values: '0x15'
alert_rule_test:
- {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []}
- alertname: WindowsHyperVWMIUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/
summary: Hyper-V WMI namespace is not responding
description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.'
- {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []}
- alertname: WindowsHyperVReplicationCritical
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/
summary: Hyper-V replication is critical
description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.'
- {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []}
- alertname: WindowsHyperVReplicationWarning
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/
summary: Hyper-V replication has a warning
description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.'
- name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes
interval: 1m
input_series:
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}'
values: '1x30'
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}'
values: '0x30'
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}'
values: '1x30'
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}'
values: '4x30'
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}'
values: '3x30'
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}'
values: '2x30'
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}'
values: '1x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
values: '15x19 50x10'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
values: '5x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
values: '1x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
values: '0x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
values: '0x30'
alert_rule_test:
- {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []}
- alertname: WindowsClusterNodeDown
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
summary: Failover cluster node is down
description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
summary: Failover cluster node is down
description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.'
- alertname: WindowsClusterResourceFailed
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/
summary: Failover cluster resource has failed
description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
- alertname: WindowsClusterResourceGroupFailed
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/
summary: Failover cluster resource group has failed
description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
- {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []}
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy
interval: 1m
input_series:
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}'
values: '2x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}'
values: '2x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}'
values: '1x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}'
values: '2x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}'
values: '1x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}'
values: '0x15'
alert_rule_test:
- {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []}
- alertname: WindowsStoragePoolUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
summary: Storage pool is unhealthy
description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
summary: Storage pool has a health warning
description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.'
- alertname: WindowsStorageVirtualDiskUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
summary: Storage Spaces virtual disk is unhealthy
description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
summary: Storage Spaces virtual disk has a health warning
description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.'
- name: DHCP scope exhaustion counts active scopes only and uses both thresholds
interval: 1m
input_series:
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
values: '10x19 50x10'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
values: '90x19 50x10'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}'
values: '1x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
values: '2x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
values: '98x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}'
values: '1x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
values: '100x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}'
values: '1x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}'
values: '1x30'
alert_rule_test:
- {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []}
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for
interval: 1m
input_series:
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}'
values: '0+6x20 120x15'
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}'
values: '7x36'
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}'
values: '0+1x36'
alert_rule_test:
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []}
- alertname: WindowsDNSZoneTransferFailures
eval_time: 17m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
summary: DNS zone transfers are failing
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
- alertname: WindowsDNSZoneTransferFailures
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
summary: DNS zone transfers are failing
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []}
docs/runbooks/_index.md
+3 -1
View File
@@ -11,4 +11,6 @@ The default thresholds are starting points. Check the generated alert rules for
your configured thresholds and durations. The rules are grouped by collector
area, and each alert is generated only when the collectors it reads are enabled
in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
and `time` collectors, which are off by default.
and `time` collectors, which are off by default, and the alerts for page files,
disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters,
Storage Spaces, DHCP and DNS need the collector of that role.
docs/runbooks/windowsclocksourcelocal.md
+56
View File
@@ -0,0 +1,56 @@
---
title: WindowsClockSourceLocal
---
## Meaning
Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system
clock for ten minutes. The alert carries the source in its `type` label. It
needs the `time` collector with its `clock_source` subcollector and covers
domain members too, whose source is `NT5DS` when they synchronize from the
domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does
not see those hosts. A host that is meant to be the free-running reference of
an isolated network reports the local clock by design; exclude it with
`alerts+: { WindowsClockSourceLocal: false }` on that configuration.
## Impact
The clock drifts without an external source. Clock errors break Kerberos
authentication beyond the five minute tolerance, TLS validation, scheduled
work, log correlation and distributed systems that depend on time order.
## Diagnosis
1. Inspect `windows_time_clock_sync_source` for the host, and
`windows_time_computed_time_offset_seconds` to see how far the clock has
already drifted.
2. On the host, check the configured and the effective source:
```powershell
w32tm /query /source
w32tm /query /status
w32tm /query /configuration
Get-Service W32Time
```
3. `Local CMOS Clock` after a reboot often means the Windows Time service
started before the network was up or the configured peers were unreachable.
`NoSync` means the time provider is configured not to synchronize.
4. For domain members, check that a domain controller is reachable and that
the PDC emulator synchronizes from an external source. For virtual machines,
check whether hypervisor time integration overrides the configured source.
A Hyper-V guest outside a domain that is kept in time by the integration
service reports `NoSync` while `w32tm /query /source` shows the
`VM IC Time Synchronization Provider`; that is by design, so turn the alert
off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or
configure an NTP source in the guest.
## Mitigation
Configure the intended source and restart synchronization with
`w32tm /config /syncfromflags:domhier /update` on domain members or
`w32tm /config /manualpeerlist:"<servers>" /syncfromflags:manual /update` on
stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the
peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the
offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/)
if it does not.
docs/runbooks/windowsclusternodedown.md
+46
View File
@@ -0,0 +1,46 @@
---
title: WindowsClusterNodeDown
---
## Meaning
The failover cluster reports the node `node` as down for five minutes. The
alert needs the `mscluster` collector. Every node that runs the exporter
reports the state of all nodes, so the alert fires once per reporting node;
the `instance` label names the reporter, and the `node` label the node that is
down. A paused node, for example during cluster-aware updating, has a
different state and does not raise this alert.
## Impact
The roles that the node hosted have failed over to the remaining nodes, with a
short outage during the move. The cluster has less capacity and less
redundancy; with too few nodes for quorum, the whole cluster stops.
## Diagnosis
1. Inspect `windows_mscluster_node_state` for all nodes and
`windows_mscluster_cluster_quorum_type` and the dynamic quorum metrics to
see how far the cluster is from losing quorum. Check whether the node's own
exporter target is up.
2. On a working node, read the cluster state:
```powershell
Get-ClusterNode
Get-ClusterGroup
Get-ClusterLog -Node '<node>' -TimeSpan 60 -Destination .
Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50
```
3. Check whether the node is powered on and reachable, whether the Cluster
Service runs on it, and whether the cluster networks between the nodes are
working. Events 1135 and 1177 in the System log name the lost node and the
quorum state.
## Mitigation
Bring the node back: power it on, repair its network, or start the Cluster
Service with `Start-ClusterNode`. Evict a node that is removed for good with
`Remove-ClusterNode` so the quorum configuration matches the remaining nodes.
Move the roles back to their preferred owners once the node is up and confirm
the node state returns to up.
docs/runbooks/windowsclusterresourcefailed.md
+45
View File
@@ -0,0 +1,45 @@
---
title: WindowsClusterResourceFailed
---
## Meaning
The failover cluster reports the resource `resource` of type `type` in the
group `resource_group` as failed for five minutes. The alert needs the
`mscluster` collector and fires once per reporting node, named by the
`instance` label. A failed resource has exhausted its restart attempts on the
current node, or its group has reached its failover threshold.
## Impact
The role that depends on the resource is down: a clustered disk, IP address,
network name, file share, SQL Server instance or virtual machine. Other
resources in the same group that depend on it are offline too.
## Diagnosis
1. Inspect `windows_mscluster_resource_state` for the group and
`windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node`
to see where the group is and whether the whole group failed.
2. On a cluster node, read the resource and the reason:
```powershell
Get-ClusterResource -Name '<resource>' | Format-List *
Get-ClusterResource -Name '<resource>' | Get-ClusterParameter
Get-ClusterLog -TimeSpan 30 -Destination .
Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20
```
Event 1069 names the failed resource and its error code.
3. Check the dependency of the resource: a disk that is not reachable from the
node, an IP address whose network is down, a network name whose computer
object in Active Directory is missing or disabled, or an application whose
service fails to start.
## Mitigation
Fix the dependency, then bring the resource online with
`Start-ClusterResource`, or move the group to another node with
`Move-ClusterGroup`. Review the restart and failover thresholds of the
resource if it failed because of a transient problem. Confirm the resource
state returns to online and the group is online.
docs/runbooks/windowsclusterresourcegroupfailed.md
+41
View File
@@ -0,0 +1,41 @@
---
title: WindowsClusterResourceGroupFailed
---
## Meaning
The failover cluster reports the resource group `resource_group` as failed for
five minutes. The alert needs the `mscluster` collector and fires once per
reporting node, named by the `instance` label. A group fails when a resource in
it fails and the group cannot fail over anymore: it reached its failover
threshold within the failover period, or no other node can host it.
## Impact
The whole clustered role is down and stays down until an operator intervenes.
## Diagnosis
1. Inspect `windows_mscluster_resource_state` for the resources of the group
to find the failed resource, and `windows_mscluster_resourcegroup_owner_node`
to see on which node the group stopped.
2. On a cluster node, read the group and its history:
```powershell
Get-ClusterGroup -Name '<group>' | Format-List *
Get-ClusterGroup -Name '<group>' | Get-ClusterResource
Get-ClusterLog -TimeSpan 60 -Destination .
Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20
```
Event 1254 says the group exceeded its failover threshold.
3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for
the failed resource, and check the possible owners of the group.
## Mitigation
Fix the failed resource, then bring the group online with `Start-ClusterGroup`
on a node that can host it. A group that failed over repeatedly may need a
higher failover threshold or a longer failover period, but a resource that
fails on every node needs its cause fixed first. Confirm the group state
returns to online.
docs/runbooks/windowsclustersharedvolumealmostfull.md
+44
View File
@@ -0,0 +1,44 @@
---
title: WindowsClusterSharedVolumeAlmostFull
---
## Meaning
The Cluster Shared Volume `shared_volume` has less free space than the
configured threshold for 15 minutes. The warning fires below
`diskFreeWarningThreshold` (10 percent by default) and the critical alert
below `diskFreeCriticalThreshold` (5 percent); both fire below the critical
threshold, so inhibit the warning in Alertmanager. The alert needs the
`mscluster` collector and fires once per reporting node, named by the
`instance` label. `sharedVolumeSelector` excludes the
`ClusterPerformanceHistory` volume of the cluster's own health history.
## Impact
Virtual machines and databases on the volume cannot grow their files. Dynamic
virtual hard disks pause the virtual machine when the volume is full, Hyper-V
checkpoints and backups fail, and SQL Server stops writing.
## Diagnosis
1. Compare `windows_mscluster_shared_volumes_free_bytes` and
`windows_mscluster_shared_volumes_total_bytes` over the last days to see
whether the volume fills steadily or filled suddenly.
2. On a cluster node, find the large files on the volume:
```powershell
Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}}
Get-ChildItem 'C:\ClusterStorage\<volume>' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}}
```
3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard
disks, backup snapshots that were not cleaned up, and dynamic disks that
grew.
## Mitigation
Delete or merge checkpoints, remove orphaned files, move virtual machines to
another volume with storage migration, or extend the volume on the storage
array and then in the cluster with `Resize-Partition`. Confirm free space is
above the threshold. Raise the thresholds only for volumes whose growth is
known and planned.
docs/runbooks/windowsdhcpscopealmostexhausted.md
+48
View File
@@ -0,0 +1,48 @@
---
title: WindowsDHCPScopeAlmostExhausted
---
## Meaning
The enabled DHCP scope `scope` has fewer free addresses than the configured
share of its address pool for 15 minutes. The warning fires below
`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert
below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the
critical threshold, so inhibit the warning in Alertmanager. The share divides
`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses,
and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled.
On a failover pair the free and in-use counts are the totals of both servers.
The alert needs the `dhcp` collector.
## Impact
Once the scope is exhausted, new clients receive no address and cannot use
the network. Existing clients keep their lease until it expires and then lose
their address too.
## Diagnosis
1. Compare the free and in-use addresses of the scope over the last days to
see whether the exhaustion is a trend or a sudden burst, and inspect
`windows_dhcp_scope_reserved_address` and
`windows_dhcp_scope_pending_offers` for reservations and offers in flight.
2. On the server, read the scope and its leases:
```powershell
Get-DhcpServerv4ScopeStatistics -ScopeId <scope>
Get-DhcpServerv4Scope -ScopeId <scope> | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State
Get-DhcpServerv4Lease -ScopeId <scope> | Group-Object AddressState
Get-DhcpServerv4ExclusionRange -ScopeId <scope>
```
3. Look for a long lease duration on a network with many short-lived clients,
large exclusion ranges, a flood of leases from one device that changes its
MAC address, or a failover partner that holds the free addresses.
## Mitigation
Shorten the lease duration for networks with transient clients, remove
exclusion ranges that are not needed, add a range through a superscope or a
larger subnet, or reclaim bad leases. On a failover pair, check the split of
the addresses between the partners. Confirm the free share stays above the
threshold.
docs/runbooks/windowsdiskdrivenotok.md
+46
View File
@@ -0,0 +1,46 @@
---
title: WindowsDiskDriveNotOK
---
## Meaning
The physical drive `device_id` reports an operational status other than `OK`
for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded`
or `Error`, in its `status` label. It needs the `diskdrive` collector, which
reads the status that the drive and its driver report through WMI. The
`Unknown` status does not raise the alert: some USB and virtual drives report
it permanently. Transient statuses such as `Starting` or `Stopping` raise it
only when they last longer than five minutes.
## Impact
`Pred Fail` is the drive's own prediction that it is about to fail. Any other
status means the drive is not fully working. Data on a single drive is at risk;
a drive in a mirror or parity set leaves the set without redundancy until it is
replaced.
## Diagnosis
1. Inspect `windows_diskdrive_status` for the host and check whether the
volumes on the drive still report normal latency with
`windows:logical_disk_read_latency:seconds` and
`windows:logical_disk_write_latency:seconds`.
2. On the host, read the drive and its reliability counters:
```powershell
Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status
Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear
```
3. Check the System event log for disk, storahci, stornvme or RAID controller
events. On a virtual machine, check the hypervisor and the datastore of the
virtual disk.
## Mitigation
Back up the data on the drive first. Replace a drive that predicts its failure
or reports errors through the hardware or virtualization team. For a drive in
a storage pool or RAID set, follow the vendor's replacement and rebuild
procedure and confirm the set is redundant again. The alert resolves when the
status returns to `OK`.
docs/runbooks/windowsdnszonetransferfailures.md
+54
View File
@@ -0,0 +1,54 @@
---
title: WindowsDNSZoneTransferFailures
---
## Meaning
The DNS server has failed zone transfers for 15 minutes, measured as the rate
of `windows_dns_zone_transfer_failures_total` over `rateInterval` above
`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts
transfers this server sends as the primary to its secondaries, including
requests it refuses, and it has no zone label. The alert needs the `dns`
collector and keeps firing for `keepFiringFor` after the failures stop. Raise
the threshold on servers that refuse transfer requests from scanners all the
time.
## Impact
The secondary servers of the zone serve stale records. When the zone on a
secondary expires, it stops answering for the zone altogether. Active
Directory integrated zones replicate through Active Directory and are not
affected unless they also have secondaries.
## Diagnosis
1. Inspect `windows_dns_zone_transfer_failures_total` next to
`windows_dns_zone_transfer_requests_received_total` and
`windows_dns_zone_transfer_success_sent_total` to see what share of the
transfers fail, and whether the failures started with a change.
2. On this primary, find the refused or failed requests and the zone
settings; the DNS Server log records a refused transfer request with the
address of the requester:
```powershell
Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer'
Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy
Get-DnsServerZone -Name '<zone>' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers
```
On the secondary, events 6522 to 6527 of the DNS Server log say why its
transfer of the zone failed: refused by the primary, invalid response or
an expired zone.
3. Check that the requesting secondary is in the zone transfer list, that TCP
port 53 is open between the servers, and that the zone is not paused or
broken. Failures from unknown addresses are refused transfers and may be
expected.
## Mitigation
Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries`
and the server list, fix the network path, or remove a decommissioned
secondary from the zone's notify list so it stops requesting transfers.
Confirm the failure counter stops rising and the secondaries show a current
serial number.
docs/runbooks/windowshypervreplicationcritical.md
+47
View File
@@ -0,0 +1,47 @@
---
title: WindowsHyperVReplicationCritical
---
## Meaning
Hyper-V Replica reports critical health for the virtual machine `vm` for five
minutes. The alert carries the `relationship` label, `primary` for the normal
replication and `extended` for an extended replica, and needs the `hyperv`
collector with its `replica_vm` subcollector. Critical health means that
replication has not succeeded for long enough that the replica misses the
recovery point objective, or that replication is paused because of an error.
## Impact
The replica of the virtual machine is stale. A failover to it loses the
changes since the last successful replication, and the recovery point
objective of the service is not met. If the primary host fails now, data is
lost.
## Diagnosis
1. Inspect `windows_hyperv_replica_vm_health` and
`windows_hyperv_replica_vm_state` for the virtual machine; the state tells
whether replication is suspended, resynchronizing or in error, and
`windows_hyperv_replica_vm_replication_latency_seconds` how long the last
cycle took.
2. On the primary host, read the replication details:
```powershell
Measure-VMReplication -VMName '<vm>' | Format-List *
Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, ReplicationMode
Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '*<vm>*'
```
3. Check the replica server: free space on the replica storage, the
Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication,
and the network path between the hosts. Check whether a snapshot merge or a
storage migration on the primary blocks replication.
## Mitigation
Fix the cause on the replica server or the network, then resume replication
with `Resume-VMReplication -VMName '<vm>'`; add `-Resynchronize` when the
state asks for a resynchronization. Confirm the health returns to normal and
the last replication time advances. If the replica cannot be recovered, remove
and re-enable replication, which sends a full initial copy.
docs/runbooks/windowshypervreplicationwarning.md
+43
View File
@@ -0,0 +1,43 @@
---
title: WindowsHyperVReplicationWarning
---
## Meaning
Hyper-V Replica reports warning health for the virtual machine `vm` for 15
minutes. Warning health means that some replication cycles missed their
interval, that more than a fifth of the cycles in the last hour failed, or
that the replication is waiting for an initial copy or a resynchronization.
The alert carries the `relationship` label and needs the `hyperv` collector
with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/)
follows when replication stops working.
## Impact
The replica lags behind the primary more than planned. The recovery point
objective is at risk, and a failover would lose more changes than expected.
## Diagnosis
1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state`
and `windows_hyperv_replica_vm_replication_latency_seconds` for the
virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes`
to see whether the change rate of the virtual machine has grown.
2. On the primary host, read the statistics of the last hour:
```powershell
Measure-VMReplication -VMName '<vm>' | Format-List *
Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, FrequencySec
```
3. Check the bandwidth between the hosts, the load on the replica storage,
and whether the replication interval is too short for the change rate of
the virtual machine.
## Mitigation
Relieve the bottleneck: more bandwidth, less load on the replica storage, or a
longer replication interval through `Set-VMReplication -ReplicationFrequencySec`.
Resume a waiting replication with `Resume-VMReplication`. Reset the statistics
with `Reset-VMReplicationStatistics` after the fix; the health is computed
over the last hour and takes up to an hour to return to normal.
docs/runbooks/windowshypervwmiunhealthy.md
+43
View File
@@ -0,0 +1,43 @@
---
title: WindowsHyperVWMIUnhealthy
---
## Meaning
The Hyper-V WMI namespace `root/virtualization/v2` on the host has not
answered a minimal query for ten minutes. The alert needs the `hyperv`
collector with its `wmi_health` subcollector. While the namespace is broken,
the virtual machine, replication and virtual hard disk metrics that come from
WMI are missing, so the other Hyper-V alerts cannot fire.
## Impact
Management tools such as Hyper-V Manager, Failover Cluster Manager and System
Center cannot manage the virtual machines on the host. Backups that use the
Hyper-V VSS writer fail. The virtual machines themselves usually keep running.
## Diagnosis
1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}`
for the host.
2. On the host, test the namespace and the management service:
```powershell
Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1
Get-Service vmms, vmcompute, winmgmt
winmgmt /verifyrepository
```
3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of
the Virtual Machine Management service, and the WMI-Activity log for
provider failures. A stuck `vmms` process or a corrupt WMI repository are
the usual causes.
## Mitigation
Restart the Virtual Machine Management service with `Restart-Service vmms`;
running virtual machines are not affected. If the repository is inconsistent,
run `winmgmt /salvagerepository`, and register the Hyper-V provider again with
`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a
restart of the host if the service cannot be recovered. Confirm the health
metric returns to 1 and the Hyper-V metrics appear again.
docs/runbooks/windowslicensenotgenuine.md
+45
View File
@@ -0,0 +1,45 @@
---
title: WindowsLicenseNotGenuine
---
## Meaning
Windows reports a license status other than `genuine` for one hour. The alert
carries the status in its `state` label and needs the `license` collector. The
status comes from the Software Licensing service and changes when an
activation expires, a KMS host is unreachable for more than the grace period,
the hardware changed, or the key was blocked. `offline` means the genuine
validation could not reach Microsoft; on hosts without internet access that
is the permanent state, so turn the alert off there with
`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter
selector.
## Impact
An unlicensed Windows shows activation notices, may limit personalization and,
on some editions, restarts on a schedule. A Windows Server that leaves its
activation grace period is out of compliance.
## Diagnosis
1. The `state` label of the alert names the status; `windows_license_status`
shows all states of the host.
2. On the host, read the license details:
```powershell
slmgr.vbs /dlv
slmgr.vbs /xpr
Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining
```
3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp`
DNS record and answers on TCP port 1688. For Active Directory based
activation, check the activation object in the forest. For MAK or retail
keys, check whether the activation count is exhausted.
## Mitigation
Restore reachability to the KMS host or the activation object, then run
`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk <key>` if the
key was wrong, or reactivate after a hardware change. Confirm the status
returns to `genuine`.
docs/runbooks/windowspagefilealmostfull.md
+41
View File
@@ -0,0 +1,41 @@
---
title: WindowsPagefileAlmostFull
---
## Meaning
The page file `file` has been more than the configured share in use for 15
minutes. The default `pagefileUsageThreshold` is 80 percent of
`windows_pagefile_limit_bytes`, the current size of the file. The alert needs
the `pagefile` collector. A system-managed page file grows up to its maximum
when it fills, so a high usage of the current size on such a system shows that
the file has already grown.
## Impact
When the page file is full, Windows cannot commit more memory. Allocations
fail, processes crash with out-of-memory errors, and the system can become
unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/)
usually fires at the same time.
## Diagnosis
1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes`
with `windows:memory_committed:ratio` and `windows_memory_available_bytes`
to see whether physical memory is exhausted too.
2. Find the processes that hold the commit charge:
```powershell
Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}}
Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
```
3. Check for a recent change: a new workload, a memory leak that grows over
days, or a page file that was made smaller or fixed in size.
## Mitigation
Stop or restart the process that leaks memory, or move workload off the host.
Add physical memory or enlarge the page file through the normal change
process; a fixed page file needs enough room for the peak commit charge of the
host. Confirm usage falls below the threshold and stays there.
docs/runbooks/windowsscheduledtaskfailed.md
+51
View File
@@ -0,0 +1,51 @@
---
title: WindowsScheduledTaskFailed
---
## Meaning
The last run of the scheduled task `task` ended with the result `unknown` or
`terminated`, and the task has not completed successfully since for one hour.
The alert needs the `scheduled_task` collector. `terminated` means the run was
stopped by a user or by the scheduler's time limit. `unknown` is every result
code without a named status, which includes the exit code of the task's
program and the error codes of the Task Scheduler; the documentation of the
collector lists the named codes. A program that exits with a code other than
zero on success raises this alert on every run; exclude such tasks with
`taskSelector` or with the include and exclude flags of the collector. A
disabled task keeps its last result but does not raise the alert.
## Impact
The work of the task did not happen: backups, cleanups, reports, certificate
renewals or synchronizations that are scheduled on this host. The impact
depends on the task.
## Diagnosis
1. Inspect `windows_scheduled_task_last_result_status` for the task and
`windows_scheduled_task_state` to see whether the task is still enabled
and queued.
2. On the host, read the last result and the history:
```powershell
Get-ScheduledTaskInfo -TaskName '<name>' -TaskPath '<folder>'
Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '*<name>*'
```
The history log is off by default; enable it with
`wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`.
3. Translate the result code: `0x1` is a generic failure of the program, `0x2`
a missing file, `0x80070005` an access denied, and codes of the form
`0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other
codes between `0x41300` and `0x41325` are named statuses, such as `ready`
or `running`, and do not raise this alert.
4. Run the program by hand under the task's account to reproduce the failure.
## Mitigation
Fix the program, its arguments, its working directory or the permissions of
the account the task runs as. Start the task once with `Start-ScheduledTask`
and confirm the result turns to `success`. Disable or delete tasks that are
not needed anymore instead of leaving them failing; disabling a task clears
the alert.
docs/runbooks/windowsstoragepoolunhealthy.md
+48
View File
@@ -0,0 +1,48 @@
---
title: WindowsStoragePoolUnhealthy
---
## Meaning
The Storage Spaces pool `pool` reports a health status of warning or unhealthy
for ten minutes. The critical alert fires for unhealthy, the warning for
warning. The alert needs the `storage_spaces` collector and joins
`windows_storage_spaces_pool_info` so the primordial pool, which only groups
the disks that are not in a pool yet, does not raise it. The pool's
operational status, exposed as `windows_storage_spaces_pool_operational_status`,
tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every
node reports the clustered pool, so the alert fires once per node; the
`instance` label names the reporter.
## Impact
A pool in warning state has lost a physical disk or redundancy; the virtual
disks on it still work but have no tolerance left for another failure, or
cannot allocate more space. An unhealthy pool has lost more disks than its
resiliency tolerates, so virtual disks on it are detached or read-only.
## Diagnosis
1. Inspect `windows_storage_spaces_pool_health_status`,
`windows_storage_spaces_pool_operational_status` and
`windows_storage_spaces_virtual_disk_health_status` for the pool and its
virtual disks.
2. On the host, find the failed disk:
```powershell
Get-StoragePool -FriendlyName '<pool>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly
Get-StoragePool -FriendlyName '<pool>' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage
Get-StoragePool -FriendlyName '<pool>' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
Get-StorageJob
```
3. Check the System log for disk and storage events, and whether a repair job
is already running.
## Mitigation
Replace the failed physical disk: retire it with
`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`,
repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk
with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the
health returns to healthy once the repair jobs finish.
docs/runbooks/windowsstoragevirtualdiskunhealthy.md
+43
View File
@@ -0,0 +1,43 @@
---
title: WindowsStorageVirtualDiskUnhealthy
---
## Meaning
The Storage Spaces virtual disk `virtual_disk` reports a health status of
warning or unhealthy for ten minutes. The critical alert fires for unhealthy,
the warning for warning. The alert needs the `storage_spaces` collector. A
virtual disk is in warning state when it is degraded or incomplete and still
accessible, and unhealthy when it is detached or has lost data. On Storage
Spaces Direct every node reports the clustered virtual disks, so the alert
fires once per node; the `instance` label names the reporter.
## Impact
A degraded virtual disk still serves data but has no redundancy left; a
further disk failure loses the data. A detached virtual disk takes its volumes
offline, and everything stored on them is unavailable.
## Diagnosis
1. Inspect `windows_storage_spaces_virtual_disk_health_status` and
`windows_storage_spaces_pool_health_status` to see whether the pool is
affected too; [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
then names the pool.
2. On the host, read the virtual disk, its footprint and the repair jobs:
```powershell
Get-VirtualDisk -FriendlyName '<disk>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, ResiliencySettingName, DetachedReason
Get-VirtualDisk -FriendlyName '<disk>' | Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
Get-StorageJob
```
3. Check for a failed physical disk, a pool that ran out of space for
allocation, or a disk that was detached by a policy after an error.
## Mitigation
Replace the failed physical disk as in [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
and repair the virtual disk with `Repair-VirtualDisk`. Attach a detached disk
with `Connect-VirtualDisk` once the cause is fixed. Confirm the health returns
to healthy once the repair jobs finish.
docs/runbooks/windowsupdatespending.md
+50
View File
@@ -0,0 +1,50 @@
---
title: WindowsUpdatesPending
---
## Meaning
The host has updates waiting for installation for longer than
`updatePendingDuration`, seven days by default. The alert is an info alert, its
value is the number of pending updates, and it needs the `update` collector,
which queries the Windows Update agent in the background at the interval of
its scrape setting. Updates waiting for uninstallation do not count. Prometheus counts the
duration and restores it after a restart shorter than its
`--rules.alert.for-outage-tolerance`, one hour by default; a longer outage or
a new server starts the wait again.
## Impact
Security updates that wait longer than the patch window leave known
vulnerabilities open. Other updates carry fixes for reliability problems. The
host may also be waiting for a restart to finish an installation.
## Diagnosis
1. List the pending updates with their category and severity:
```promql
windows_update_pending_info{job="windows_exporter", instance="<instance>", deployment_action="installation"}
```
`windows_update_pending_published_timestamp` shows how old each update is.
2. On the host, check the update client and the last installation attempts:
```powershell
Get-WindowsUpdateLog
Get-WinEvent -LogName Microsoft-Windows-WindowsUpdateClient/Operational -MaxEvents 50
Get-CimInstance -Namespace root/cimv2 -ClassName Win32_QuickFixEngineering | Sort-Object InstalledOn -Descending | Select-Object -First 5
```
3. Check whether the host is in a maintenance window, a deferral period or a
WSUS or Configuration Manager deployment that has not approved the updates
yet, and whether a pending restart blocks the installation.
## Mitigation
Install the updates in the next maintenance window, or approve them in the
update management tool. Restart the host if an installation waits for it.
Raise `updatePendingDuration` to the patch window of the environment if seven
days is shorter than the policy, or turn the alert off with
`alerts+: { WindowsUpdatesPending: false }` where another system tracks patch
compliance.