diff --git a/contrib/mixin/README.md b/contrib/mixin/README.md index d0258f42..2edc9b42 100644 --- a/contrib/mixin/README.md +++ b/contrib/mixin/README.md @@ -85,7 +85,10 @@ apply to both recording rules and alerts. All target labels, including `job`, averages idle rates across cores and removes only `core` and `mode`. Disk, network and service rules also retain `volume`, `nic` and `service` respectively, filtered by `volumeSelector`, `nicSelector` and -`serviceSelector`. The CPU mode recording retains `mode` and removes only +`serviceSelector`. The scheduled task and cluster shared volume alerts +filter their `task` and `shared_volume` labels with `taskSelector` +(`task!=""`) and `sharedVolumeSelector` +(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only `core`. `rateInterval` is the fixed window for recording rules and Active Directory @@ -179,6 +182,27 @@ percentages except where noted. | `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning | | `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning | | `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning | +| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning | +| `windows-memory` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning | +| `windows-disk` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical | +| `windows-system` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning | +| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning | +| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info | +| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning | +| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical | +| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning | +| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical | +| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning | +| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical | +| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning | +| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning | +| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical | +| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning | `_config.alerts` lists every alert by name. Set one to `false` to leave it out, for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the @@ -188,9 +212,11 @@ alerts are all turned off is not generated. Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a value that oscillates around a threshold does not flap. The state alerts `WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`, -`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning` and -`WindowsClockNotSynchronising`, and the one-shot `WindowsHostRebooted`, have no -`keep_firing_for` and resolve as soon as their condition clears. +`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`, +`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors +(clock source, drive status, license, task result, pending updates, Hyper-V, +cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`, +have no `keep_firing_for` and resolve as soon as their condition clears. `keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for older servers. @@ -206,9 +232,33 @@ notification should be sent. Active Directory alerts need `collectors.ad`, which `enableActiveDirectory` sets. Clock alerts need `collectors.time`, which `enableTime` sets. A zero NTP -source count may be expected on hosts using another time provider. The offset -compares the host to its chosen source and does not independently verify that -source. Disk latency, paging and network thresholds should be tuned for the +source count may be expected on hosts using another time provider; +`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and +fires only for the local clock or no synchronization. The offset compares the +host to its chosen source and does not independently verify that source. + +The other opt-in alerts follow `_config.collectors`: enable `pagefile`, +`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`, +`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are +generated, in the group of their area or in a group of their own. +`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code +without a name, including the exit codes of the task's program, as a failure; +narrow the tasks with `taskSelector`. Disabling a task clears the alert. +`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and +virtual drives report permanently. `WindowsLicenseNotGenuine` reports the +`state` the host has; hosts without internet access can stay in `offline`, +so turn the alert off there. Every failover cluster node reports the state of +the whole cluster, and every Storage Spaces Direct node reports the clustered +pools and virtual disks, so the cluster and storage alerts fire once per node; +group them by their `node`, `resource`, `shared_volume`, `pool` or +`virtual_disk` labels in Alertmanager. The storage pool alert joins +`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP +scope alert only counts scopes in an enabled state, and a scope without +addresses produces no sample. `WindowsUpdatesPending` is an info alert that +waits `updatePendingDuration` for the normal maintenance window. Prometheus +restores the waiting time after a restart shorter than its +`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage +or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the storage tier, traffic volume, and application requirements. `WindowsServiceNotRunning` watches services with the `auto` start mode and diff --git a/contrib/mixin/alerts/alerts.libsonnet b/contrib/mixin/alerts/alerts.libsonnet index c70bb63c..15e3f50f 100644 --- a/contrib/mixin/alerts/alerts.libsonnet +++ b/contrib/mixin/alerts/alerts.libsonnet @@ -25,6 +25,12 @@ } + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}), ] else [], local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [], + // Percent of free addresses in a scope; a scope without addresses has no sample. + local dhcpScopeFree = '(%s * 100)' % q.ratio( + q.metric('windows_dhcp_scope_addresses_free'), + '(%s + %s)' % [q.metric('windows_dhcp_scope_addresses_free'), q.metric('windows_dhcp_scope_addresses_in_use')] + ), + local dhcpScopeActive = q.metric('windows_dhcp_scope_state', ['state=~"Enabled|EnabledSwitched"']), prometheusAlerts+:: { groups: group('windows-exporter', std.flattenArrays([ @@ -113,6 +119,18 @@ 'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold, collectors=['memory'] ), + alert( + 'WindowsPagefileAlmostFull', + '(1 - %s) * 100 > %s' % [ + q.ratio(q.metric('windows_pagefile_free_bytes'), q.metric('windows_pagefile_limit_bytes')), + c.pagefileUsageThreshold, + ], + '15m', + 'warning', + 'Windows page file is almost full', + 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %s%% in use for 15 minutes.' % c.pagefileUsageThreshold, + collectors=['pagefile'] + ), ])) + group('windows-disk', std.flattenArrays([ alert( @@ -154,6 +172,19 @@ 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours, collectors=['logical_disk'] ), + alert( + 'WindowsDiskDriveNotOK', + // One series per status with the current status set to 1, so the + // alert carries the status the drive reports. Some USB and virtual + // drives report Unknown for good. + q.metric('windows_diskdrive_status', ['status!~"OK|Unknown"']) + ' == 1', + '5m', + c.criticalSeverity, + 'Windows disk drive is not healthy', + 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.', + collectors=['diskdrive'], + threshold=false + ), ])) + group('windows-network', std.flattenArrays([ alert( @@ -256,6 +287,16 @@ collectors=['system'], threshold=false ), + alert( + 'WindowsLicenseNotGenuine', + q.metric('windows_license_status', ['state!="genuine"']) + ' == 1', + '1h', + 'warning', + 'Windows license is not genuine', + 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.', + collectors=['license'], + threshold=false + ), ])) + group('windows-active-directory', std.flattenArrays([ alert( @@ -297,6 +338,209 @@ 'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds, collectors=['time'] ), + alert( + 'WindowsClockSourceLocal', + // Covers domain members (NT5DS) too, which the NTP source count misses. + q.metric('windows_time_clock_sync_source', ['type=~"Local CMOS Clock|NoSync"']) + ' == 1', + '10m', + 'warning', + 'Windows clock has no external time source', + 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.', + collectors=['time'], + threshold=false + ), + ])) + + group('windows-scheduled-task', std.flattenArrays([ + alert( + 'WindowsScheduledTaskFailed', + // unknown is any result code without a named status, including the + // exit codes of the task's program. + // A disabled task keeps its last result, so disabling a broken task + // clears the alert. + '%s == 1 unless ignoring (status, state) (%s == 1)' % [ + q.metric('windows_scheduled_task_last_result_status', [c.taskSelector, 'status=~"unknown|terminated"']), + q.metric('windows_scheduled_task_state', [c.taskSelector, 'state="disabled"']), + ], + '1h', + 'warning', + 'Windows scheduled task did not succeed', + 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.', + collectors=['scheduled_task'], + threshold=false + ), + ])) + + group('windows-update', std.flattenArrays([ + alert( + 'WindowsUpdatesPending', + 'count without (id, revision, category, severity, title) (%s) > 0' % q.metric('windows_update_pending_info', ['deployment_action="installation"']), + c.updatePendingDuration, + c.infoSeverity, + 'Windows updates are waiting for installation', + '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %s.' % c.updatePendingDuration, + collectors=['update'], + threshold=false + ), + ])) + + group('windows-hyperv', std.flattenArrays([ + alert( + 'WindowsHyperVWMIUnhealthy', + q.metric('windows_hyperv_wmi_health') + ' == 0', + '10m', + 'warning', + 'Hyper-V WMI namespace is not responding', + 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.', + collectors=['hyperv'], + threshold=false + ), + alert( + 'WindowsHyperVReplicationCritical', + q.metric('windows_hyperv_replica_vm_health') + ' == 3', + '5m', + c.criticalSeverity, + 'Hyper-V replication is critical', + 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.', + collectors=['hyperv'], + threshold=false + ), + alert( + 'WindowsHyperVReplicationWarning', + q.metric('windows_hyperv_replica_vm_health') + ' == 2', + '15m', + 'warning', + 'Hyper-V replication has a warning', + 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.', + collectors=['hyperv'], + threshold=false + ), + ])) + // Every cluster node reports the state of the whole cluster, so these + // alerts fire once per node; group them by cluster labels in Alertmanager. + + group('windows-cluster', std.flattenArrays([ + alert( + 'WindowsClusterNodeDown', + q.metric('windows_mscluster_node_state') + ' == 1', + '5m', + c.criticalSeverity, + 'Failover cluster node is down', + 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.', + collectors=['mscluster'], + threshold=false + ), + alert( + 'WindowsClusterResourceFailed', + q.metric('windows_mscluster_resource_state') + ' == 4', + '5m', + c.criticalSeverity, + 'Failover cluster resource has failed', + 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + collectors=['mscluster'], + threshold=false + ), + alert( + 'WindowsClusterResourceGroupFailed', + q.metric('windows_mscluster_resourcegroup_state') + ' == 2', + '5m', + c.criticalSeverity, + 'Failover cluster resource group has failed', + 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + collectors=['mscluster'], + threshold=false + ), + alert( + 'WindowsClusterSharedVolumeAlmostFull', + q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeWarningThreshold, + '15m', + 'warning', + 'Cluster shared volume is almost full', + 'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeWarningThreshold, + collectors=['mscluster'] + ), + alert( + 'WindowsClusterSharedVolumeAlmostFull', + q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeCriticalThreshold, + '15m', + c.criticalSeverity, + 'Cluster shared volume is almost full', + 'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeCriticalThreshold, + collectors=['mscluster'] + ), + ])) + // On Storage Spaces Direct every node reports the clustered pools and + // virtual disks, so these alerts fire once per node as well. + + group('windows-storage-spaces', std.flattenArrays([ + // The primordial pool holds the disks that are not in a concrete pool + // yet; only the info metric tells it apart. + alert( + 'WindowsStoragePoolUnhealthy', + '(%s == 2) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])], + '10m', + c.criticalSeverity, + 'Storage pool is unhealthy', + 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + collectors=['storage_spaces'], + threshold=false + ), + alert( + 'WindowsStoragePoolUnhealthy', + '(%s == 1) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])], + '10m', + 'warning', + 'Storage pool has a health warning', + 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + collectors=['storage_spaces'], + threshold=false + ), + alert( + 'WindowsStorageVirtualDiskUnhealthy', + q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 2', + '10m', + c.criticalSeverity, + 'Storage Spaces virtual disk is unhealthy', + 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + collectors=['storage_spaces'], + threshold=false + ), + alert( + 'WindowsStorageVirtualDiskUnhealthy', + q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 1', + '10m', + 'warning', + 'Storage Spaces virtual disk has a health warning', + 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + collectors=['storage_spaces'], + threshold=false + ), + ])) + + group('windows-dhcp', std.flattenArrays([ + alert( + 'WindowsDHCPScopeAlmostExhausted', + dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeWarningThreshold, dhcpScopeActive], + '15m', + 'warning', + 'DHCP scope is almost exhausted', + 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeWarningThreshold, + collectors=['dhcp'] + ), + alert( + 'WindowsDHCPScopeAlmostExhausted', + dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeCriticalThreshold, dhcpScopeActive], + '15m', + c.criticalSeverity, + 'DHCP scope is almost exhausted', + 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeCriticalThreshold, + collectors=['dhcp'] + ), + ])) + + group('windows-dns', std.flattenArrays([ + alert( + 'WindowsDNSZoneTransferFailures', + q.rate('windows_dns_zone_transfer_failures_total') + ' > %g' % c.dnsZoneTransferFailureRateThreshold, + '15m', + 'warning', + 'DNS zone transfers are failing', + 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.', + collectors=['dns'] + ), ])), }, } diff --git a/contrib/mixin/config.libsonnet b/contrib/mixin/config.libsonnet index bd24e27f..8a8bdb89 100644 --- a/contrib/mixin/config.libsonnet +++ b/contrib/mixin/config.libsonnet @@ -5,6 +5,9 @@ volumeSelector: 'volume!=""', nicSelector: 'nic!=""', serviceSelector: 'service!=""', + taskSelector: 'task!=""', + // ClusterPerformanceHistory is the volume of the cluster's own health history. + sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"', rateInterval: '5m', // Opt in only for targets with the ad collector enabled. Sets collectors.ad. @@ -43,14 +46,19 @@ collectors: { ad: $._config.enableActiveDirectory, cpu: true, + dhcp: false, diskdrive: false, + dns: false, gpu: false, hyperv: false, + license: false, logical_disk: true, memory: true, + mscluster: false, mssql: false, net: true, os: true, + pagefile: false, physical_disk: true, process: false, scheduled_task: false, @@ -94,6 +102,23 @@ WindowsADReplicationFailures: true, WindowsClockNotSynchronising: true, WindowsClockSkewDetected: true, + WindowsClockSourceLocal: true, + WindowsDiskDriveNotOK: true, + WindowsPagefileAlmostFull: true, + WindowsLicenseNotGenuine: true, + WindowsScheduledTaskFailed: true, + WindowsUpdatesPending: true, + WindowsHyperVWMIUnhealthy: true, + WindowsHyperVReplicationCritical: true, + WindowsHyperVReplicationWarning: true, + WindowsClusterNodeDown: true, + WindowsClusterResourceFailed: true, + WindowsClusterResourceGroupFailed: true, + WindowsClusterSharedVolumeAlmostFull: true, + WindowsStoragePoolUnhealthy: true, + WindowsStorageVirtualDiskUnhealthy: true, + WindowsDHCPScopeAlmostExhausted: true, + WindowsDNSZoneTransferFailures: true, }, runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/', @@ -127,5 +152,14 @@ infoSeverity: 'info', adPendingReplicationThreshold: 50, adReplicationFailureRateThreshold: 0, + // Percent of the page file in use. + pagefileUsageThreshold: 80, + // Percent of free addresses in an active DHCP scope. + dhcpScopeFreeWarningThreshold: 10, + dhcpScopeFreeCriticalThreshold: 2, + // How long updates may wait for installation before WindowsUpdatesPending fires. + updatePendingDuration: '7d', + // Failed zone transfers per second. + dnsZoneTransferFailureRateThreshold: 0, }, } diff --git a/contrib/mixin/tests/alerts.test.jsonnet b/contrib/mixin/tests/alerts.test.jsonnet index 5c5f7c35..2c0fa460 100644 --- a/contrib/mixin/tests/alerts.test.jsonnet +++ b/contrib/mixin/tests/alerts.test.jsonnet @@ -2,7 +2,8 @@ // keep_firing_for and the group layout. local mixin = import '../mixin.libsonnet'; local config = (import '../config.libsonnet')._config; -local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true } }; +local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true, collectors: { [name]: true for name in std.objectFields(config.collectors) } } }; +local optionalGroups = ['windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns']; local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]); local groupNames(alerts) = [group.name for group in alerts.groups]; @@ -12,7 +13,13 @@ assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs fro // Groups follow the collector areas; opt-in groups appear only when enabled. assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts)); -assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory', 'windows-time']; +assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + optionalGroups : std.manifestJson(groupNames(everything.prometheusAlerts)); +// Opt-in collectors add their alerts to the default groups of their area too. +local adOnly = mixin { _config+:: { enableActiveDirectory: true } }; +assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory']; +local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } }; +assert groupNames(pagefile.prometheusAlerts) == groupNames(mixin.prometheusAlerts); +assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull']; // Disabling a collector removes its alerts and an empty group; the alerts of a // rule that needs two collectors go with either. @@ -34,7 +41,28 @@ assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAle // keep_firing_for applies to threshold alerts only and can be turned off. local rules(alerts) = [rule for group in alerts.groups for rule in group.rules]; -local stateAlerts = ['WindowsExporterDown', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsScrapeQueueTimeout', 'WindowsServiceNotRunning', 'WindowsHostRebooted', 'WindowsClockNotSynchronising']; +local stateAlerts = [ + 'WindowsExporterDown', + 'WindowsCollectorFailed', + 'WindowsCollectorTimeout', + 'WindowsScrapeQueueTimeout', + 'WindowsServiceNotRunning', + 'WindowsHostRebooted', + 'WindowsClockNotSynchronising', + 'WindowsClockSourceLocal', + 'WindowsDiskDriveNotOK', + 'WindowsLicenseNotGenuine', + 'WindowsScheduledTaskFailed', + 'WindowsUpdatesPending', + 'WindowsHyperVWMIUnhealthy', + 'WindowsHyperVReplicationCritical', + 'WindowsHyperVReplicationWarning', + 'WindowsClusterNodeDown', + 'WindowsClusterResourceFailed', + 'WindowsClusterResourceGroupFailed', + 'WindowsStoragePoolUnhealthy', + 'WindowsStorageVirtualDiskUnhealthy', +]; assert std.all([ std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert) for rule in rules(everything.prometheusAlerts) @@ -46,5 +74,7 @@ assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.object // The info severity label is configurable. local ticket = mixin { _config+:: { infoSeverity: 'ticket' } }; assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket']; +local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } }; +assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']]; { result: 'Alert configuration checks passed for %d alerts' % std.length(all) } diff --git a/contrib/mixin/tests/config.libsonnet b/contrib/mixin/tests/config.libsonnet index 7bd565c8..a8379b3d 100644 --- a/contrib/mixin/tests/config.libsonnet +++ b/contrib/mixin/tests/config.libsonnet @@ -19,5 +19,23 @@ clockOffsetThresholdSeconds: 0.1, adPendingReplicationThreshold: 10, adReplicationFailureRateThreshold: 0.5, + collectors+: { + dhcp: true, + diskdrive: true, + dns: true, + hyperv: true, + license: true, + mscluster: true, + pagefile: true, + scheduled_task: true, + storage_spaces: true, + update: true, + }, + pagefileUsageThreshold: 75, + dhcpScopeFreeWarningThreshold: 20, + dhcpScopeFreeCriticalThreshold: 5, + updatePendingDuration: '2d', + dnsZoneTransferFailureRateThreshold: 0.05, + taskSelector: 'task!~"/Ignored.*"', }, } diff --git a/contrib/mixin/tests/optional.test.yaml b/contrib/mixin/tests/optional.test.yaml new file mode 100644 index 00000000..912f71e9 --- /dev/null +++ b/contrib/mixin/tests/optional.test.yaml @@ -0,0 +1,508 @@ +# Alerts of the opt-in collectors, rendered with tests/config.libsonnet. +rule_files: + - custom_rules.yaml + - custom_alerts.yaml +evaluation_interval: 1m +group_eval_order: + - windows-exporter-recording + - windows-memory + - windows-disk + - windows-system + - windows-time + - windows-scheduled-task + - windows-update + - windows-hyperv + - windows-cluster + - windows-storage-spaces + - windows-dhcp + - windows-dns + +tests: + - name: Page file usage fires above the custom threshold and resolves after keep_firing_for + interval: 1m + input_series: + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}' + values: '20x19 50x10' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}' + values: '100x30' + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}' + values: '30x30' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}' + values: '100x30' + - series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}' + values: '0x30' + - series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}' + values: '0x30' + alert_rule_test: + - {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []} + - alertname: WindowsPagefileAlmostFull + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/ + summary: Windows page file is almost full + description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.' + - alertname: WindowsPagefileAlmostFull + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/ + summary: Windows page file is almost full + description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.' + - {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []} + + - name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour + interval: 1m + input_series: + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}' + values: '0x9 1x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}' + values: '1x9 0x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}' + values: '0x20' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}' + values: '1x20' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}' + values: '1x10' + - series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}' + values: '0x10' + - series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}' + values: '0x60' + - series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}' + values: '1x60' + - series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}' + values: '1x60' + - series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}' + values: '0x60' + alert_rule_test: + - {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []} + - alertname: WindowsDiskDriveNotOK + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/ + summary: Windows disk drive is not healthy + description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.' + - alertname: WindowsDiskDriveNotOK + eval_time: 9m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/ + summary: Windows disk drive is not healthy + description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.' + - {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []} + - {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []} + - alertname: WindowsLicenseNotGenuine + eval_time: 60m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/ + summary: Windows license is not genuine + description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.' + + - name: Clock source alert names the local source and ignores NTP and domain sources + interval: 1m + input_series: + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}' + values: '1x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}' + values: '0x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}' + values: '1x10' + - series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}' + values: '0x10' + alert_rule_test: + - {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []} + - alertname: WindowsClockSourceLocal + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/ + summary: Windows clock has no external time source + description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.' + + - name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks + interval: 1m + input_series: + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}' + values: '0x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}' + values: '0x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}' + values: '1x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}' + values: '1x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}' + values: '0x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}' + values: '0x60' + - series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}' + values: '1x60' + alert_rule_test: + - {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []} + - alertname: WindowsScheduledTaskFailed + eval_time: 60m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/ + summary: Windows scheduled task did not succeed + description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/ + summary: Windows scheduled task did not succeed + description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.' + + - name: Pending updates fire as info after the configured duration and count installations only + interval: 1m + input_series: + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}' + values: '1x2880' + - series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}' + values: '1x2880' + alert_rule_test: + - {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []} + - alertname: WindowsUpdatesPending + eval_time: 48h + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsupdatespending/ + summary: Windows updates are waiting for installation + description: 'host:9182 has had 2 updates waiting for installation for 2d.' + + - name: Hyper-V alerts cover the WMI namespace and both replication health levels + interval: 1m + input_series: + - series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}' + values: '0x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}' + values: '3x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}' + values: '2x15' + - series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}' + values: '0x15' + alert_rule_test: + - {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []} + - alertname: WindowsHyperVWMIUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/ + summary: Hyper-V WMI namespace is not responding + description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.' + - {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []} + - alertname: WindowsHyperVReplicationCritical + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/ + summary: Hyper-V replication is critical + description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.' + - {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []} + - alertname: WindowsHyperVReplicationWarning + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/ + summary: Hyper-V replication has a warning + description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.' + + - name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes + interval: 1m + input_series: + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}' + values: '1x30' + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}' + values: '0x30' + - series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}' + values: '1x30' + - series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}' + values: '4x30' + - series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}' + values: '3x30' + - series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}' + values: '2x30' + - series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}' + values: '1x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}' + values: '15x19 50x10' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}' + values: '5x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}' + values: '1x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}' + values: '100x30' + - series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}' + values: '0x30' + - series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}' + values: '0x30' + alert_rule_test: + - {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []} + - alertname: WindowsClusterNodeDown + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/ + summary: Failover cluster node is down + description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/ + summary: Failover cluster node is down + description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.' + - alertname: WindowsClusterResourceFailed + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/ + summary: Failover cluster resource has failed + description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.' + - alertname: WindowsClusterResourceGroupFailed + eval_time: 5m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/ + summary: Failover cluster resource group has failed + description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.' + - {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []} + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + - alertname: WindowsClusterSharedVolumeAlmostFull + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.' + - exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/ + summary: Cluster shared volume is almost full + description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.' + + - name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy + interval: 1m + input_series: + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}' + values: '2x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}' + values: '1x15' + - series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}' + values: '2x15' + - series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}' + values: '1x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}' + values: '2x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}' + values: '1x15' + - series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}' + values: '0x15' + alert_rule_test: + - {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []} + - alertname: WindowsStoragePoolUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/ + summary: Storage pool is unhealthy + description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/ + summary: Storage pool has a health warning + description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.' + - alertname: WindowsStorageVirtualDiskUnhealthy + eval_time: 10m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/ + summary: Storage Spaces virtual disk is unhealthy + description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.' + - exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/ + summary: Storage Spaces virtual disk has a health warning + description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.' + + - name: DHCP scope exhaustion counts active scopes only and uses both thresholds + interval: 1m + input_series: + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}' + values: '10x19 50x10' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}' + values: '90x19 50x10' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}' + values: '1x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}' + values: '2x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}' + values: '98x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}' + values: '1x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}' + values: '100x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}' + values: '1x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}' + values: '0x30' + - series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}' + values: '1x30' + alert_rule_test: + - {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []} + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 15m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 24m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + - alertname: WindowsDHCPScopeAlmostExhausted + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.' + - exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/ + summary: DHCP scope is almost exhausted + description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.' + + - name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for + interval: 1m + input_series: + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}' + values: '0+6x20 120x15' + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}' + values: '7x36' + - series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}' + values: '0+1x36' + alert_rule_test: + - {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []} + - alertname: WindowsDNSZoneTransferFailures + eval_time: 17m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/ + summary: DNS zone transfers are failing + description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.' + - alertname: WindowsDNSZoneTransferFailures + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning} + exp_annotations: + runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/ + summary: DNS zone transfers are failing + description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.' + - {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []} diff --git a/docs/runbooks/_index.md b/docs/runbooks/_index.md index 202fb9e6..15a741d2 100644 --- a/docs/runbooks/_index.md +++ b/docs/runbooks/_index.md @@ -11,4 +11,6 @@ The default thresholds are starting points. Check the generated alert rules for your configured thresholds and durations. The rules are grouped by collector area, and each alert is generated only when the collectors it reads are enabled in the mixin configuration. Active Directory and NTP clock alerts need the `ad` -and `time` collectors, which are off by default. +and `time` collectors, which are off by default, and the alerts for page files, +disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters, +Storage Spaces, DHCP and DNS need the collector of that role. diff --git a/docs/runbooks/windowsclocksourcelocal.md b/docs/runbooks/windowsclocksourcelocal.md new file mode 100644 index 00000000..849c0c11 --- /dev/null +++ b/docs/runbooks/windowsclocksourcelocal.md @@ -0,0 +1,56 @@ +--- +title: WindowsClockSourceLocal +--- + +## Meaning + +Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system +clock for ten minutes. The alert carries the source in its `type` label. It +needs the `time` collector with its `clock_source` subcollector and covers +domain members too, whose source is `NT5DS` when they synchronize from the +domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does +not see those hosts. A host that is meant to be the free-running reference of +an isolated network reports the local clock by design; exclude it with +`alerts+: { WindowsClockSourceLocal: false }` on that configuration. + +## Impact + +The clock drifts without an external source. Clock errors break Kerberos +authentication beyond the five minute tolerance, TLS validation, scheduled +work, log correlation and distributed systems that depend on time order. + +## Diagnosis + +1. Inspect `windows_time_clock_sync_source` for the host, and + `windows_time_computed_time_offset_seconds` to see how far the clock has + already drifted. +2. On the host, check the configured and the effective source: + + ```powershell + w32tm /query /source + w32tm /query /status + w32tm /query /configuration + Get-Service W32Time + ``` + +3. `Local CMOS Clock` after a reboot often means the Windows Time service + started before the network was up or the configured peers were unreachable. + `NoSync` means the time provider is configured not to synchronize. +4. For domain members, check that a domain controller is reachable and that + the PDC emulator synchronizes from an external source. For virtual machines, + check whether hypervisor time integration overrides the configured source. + A Hyper-V guest outside a domain that is kept in time by the integration + service reports `NoSync` while `w32tm /query /source` shows the + `VM IC Time Synchronization Provider`; that is by design, so turn the alert + off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or + configure an NTP source in the guest. + +## Mitigation + +Configure the intended source and restart synchronization with +`w32tm /config /syncfromflags:domhier /update` on domain members or +`w32tm /config /manualpeerlist:"" /syncfromflags:manual /update` on +stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the +peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the +offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/) +if it does not. diff --git a/docs/runbooks/windowsclusternodedown.md b/docs/runbooks/windowsclusternodedown.md new file mode 100644 index 00000000..4c25e856 --- /dev/null +++ b/docs/runbooks/windowsclusternodedown.md @@ -0,0 +1,46 @@ +--- +title: WindowsClusterNodeDown +--- + +## Meaning + +The failover cluster reports the node `node` as down for five minutes. The +alert needs the `mscluster` collector. Every node that runs the exporter +reports the state of all nodes, so the alert fires once per reporting node; +the `instance` label names the reporter, and the `node` label the node that is +down. A paused node, for example during cluster-aware updating, has a +different state and does not raise this alert. + +## Impact + +The roles that the node hosted have failed over to the remaining nodes, with a +short outage during the move. The cluster has less capacity and less +redundancy; with too few nodes for quorum, the whole cluster stops. + +## Diagnosis + +1. Inspect `windows_mscluster_node_state` for all nodes and + `windows_mscluster_cluster_quorum_type` and the dynamic quorum metrics to + see how far the cluster is from losing quorum. Check whether the node's own + exporter target is up. +2. On a working node, read the cluster state: + + ```powershell + Get-ClusterNode + Get-ClusterGroup + Get-ClusterLog -Node '' -TimeSpan 60 -Destination . + Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50 + ``` + +3. Check whether the node is powered on and reachable, whether the Cluster + Service runs on it, and whether the cluster networks between the nodes are + working. Events 1135 and 1177 in the System log name the lost node and the + quorum state. + +## Mitigation + +Bring the node back: power it on, repair its network, or start the Cluster +Service with `Start-ClusterNode`. Evict a node that is removed for good with +`Remove-ClusterNode` so the quorum configuration matches the remaining nodes. +Move the roles back to their preferred owners once the node is up and confirm +the node state returns to up. diff --git a/docs/runbooks/windowsclusterresourcefailed.md b/docs/runbooks/windowsclusterresourcefailed.md new file mode 100644 index 00000000..0e31d1d3 --- /dev/null +++ b/docs/runbooks/windowsclusterresourcefailed.md @@ -0,0 +1,45 @@ +--- +title: WindowsClusterResourceFailed +--- + +## Meaning + +The failover cluster reports the resource `resource` of type `type` in the +group `resource_group` as failed for five minutes. The alert needs the +`mscluster` collector and fires once per reporting node, named by the +`instance` label. A failed resource has exhausted its restart attempts on the +current node, or its group has reached its failover threshold. + +## Impact + +The role that depends on the resource is down: a clustered disk, IP address, +network name, file share, SQL Server instance or virtual machine. Other +resources in the same group that depend on it are offline too. + +## Diagnosis + +1. Inspect `windows_mscluster_resource_state` for the group and + `windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node` + to see where the group is and whether the whole group failed. +2. On a cluster node, read the resource and the reason: + + ```powershell + Get-ClusterResource -Name '' | Format-List * + Get-ClusterResource -Name '' | Get-ClusterParameter + Get-ClusterLog -TimeSpan 30 -Destination . + Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20 + ``` + + Event 1069 names the failed resource and its error code. +3. Check the dependency of the resource: a disk that is not reachable from the + node, an IP address whose network is down, a network name whose computer + object in Active Directory is missing or disabled, or an application whose + service fails to start. + +## Mitigation + +Fix the dependency, then bring the resource online with +`Start-ClusterResource`, or move the group to another node with +`Move-ClusterGroup`. Review the restart and failover thresholds of the +resource if it failed because of a transient problem. Confirm the resource +state returns to online and the group is online. diff --git a/docs/runbooks/windowsclusterresourcegroupfailed.md b/docs/runbooks/windowsclusterresourcegroupfailed.md new file mode 100644 index 00000000..70fde0f6 --- /dev/null +++ b/docs/runbooks/windowsclusterresourcegroupfailed.md @@ -0,0 +1,41 @@ +--- +title: WindowsClusterResourceGroupFailed +--- + +## Meaning + +The failover cluster reports the resource group `resource_group` as failed for +five minutes. The alert needs the `mscluster` collector and fires once per +reporting node, named by the `instance` label. A group fails when a resource in +it fails and the group cannot fail over anymore: it reached its failover +threshold within the failover period, or no other node can host it. + +## Impact + +The whole clustered role is down and stays down until an operator intervenes. + +## Diagnosis + +1. Inspect `windows_mscluster_resource_state` for the resources of the group + to find the failed resource, and `windows_mscluster_resourcegroup_owner_node` + to see on which node the group stopped. +2. On a cluster node, read the group and its history: + + ```powershell + Get-ClusterGroup -Name '' | Format-List * + Get-ClusterGroup -Name '' | Get-ClusterResource + Get-ClusterLog -TimeSpan 60 -Destination . + Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20 + ``` + + Event 1254 says the group exceeded its failover threshold. +3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for + the failed resource, and check the possible owners of the group. + +## Mitigation + +Fix the failed resource, then bring the group online with `Start-ClusterGroup` +on a node that can host it. A group that failed over repeatedly may need a +higher failover threshold or a longer failover period, but a resource that +fails on every node needs its cause fixed first. Confirm the group state +returns to online. diff --git a/docs/runbooks/windowsclustersharedvolumealmostfull.md b/docs/runbooks/windowsclustersharedvolumealmostfull.md new file mode 100644 index 00000000..1172c72c --- /dev/null +++ b/docs/runbooks/windowsclustersharedvolumealmostfull.md @@ -0,0 +1,44 @@ +--- +title: WindowsClusterSharedVolumeAlmostFull +--- + +## Meaning + +The Cluster Shared Volume `shared_volume` has less free space than the +configured threshold for 15 minutes. The warning fires below +`diskFreeWarningThreshold` (10 percent by default) and the critical alert +below `diskFreeCriticalThreshold` (5 percent); both fire below the critical +threshold, so inhibit the warning in Alertmanager. The alert needs the +`mscluster` collector and fires once per reporting node, named by the +`instance` label. `sharedVolumeSelector` excludes the +`ClusterPerformanceHistory` volume of the cluster's own health history. + +## Impact + +Virtual machines and databases on the volume cannot grow their files. Dynamic +virtual hard disks pause the virtual machine when the volume is full, Hyper-V +checkpoints and backups fail, and SQL Server stops writing. + +## Diagnosis + +1. Compare `windows_mscluster_shared_volumes_free_bytes` and + `windows_mscluster_shared_volumes_total_bytes` over the last days to see + whether the volume fills steadily or filled suddenly. +2. On a cluster node, find the large files on the volume: + + ```powershell + Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}} + Get-ChildItem 'C:\ClusterStorage\' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}} + ``` + +3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard + disks, backup snapshots that were not cleaned up, and dynamic disks that + grew. + +## Mitigation + +Delete or merge checkpoints, remove orphaned files, move virtual machines to +another volume with storage migration, or extend the volume on the storage +array and then in the cluster with `Resize-Partition`. Confirm free space is +above the threshold. Raise the thresholds only for volumes whose growth is +known and planned. diff --git a/docs/runbooks/windowsdhcpscopealmostexhausted.md b/docs/runbooks/windowsdhcpscopealmostexhausted.md new file mode 100644 index 00000000..9500cf27 --- /dev/null +++ b/docs/runbooks/windowsdhcpscopealmostexhausted.md @@ -0,0 +1,48 @@ +--- +title: WindowsDHCPScopeAlmostExhausted +--- + +## Meaning + +The enabled DHCP scope `scope` has fewer free addresses than the configured +share of its address pool for 15 minutes. The warning fires below +`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert +below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the +critical threshold, so inhibit the warning in Alertmanager. The share divides +`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses, +and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled. +On a failover pair the free and in-use counts are the totals of both servers. +The alert needs the `dhcp` collector. + +## Impact + +Once the scope is exhausted, new clients receive no address and cannot use +the network. Existing clients keep their lease until it expires and then lose +their address too. + +## Diagnosis + +1. Compare the free and in-use addresses of the scope over the last days to + see whether the exhaustion is a trend or a sudden burst, and inspect + `windows_dhcp_scope_reserved_address` and + `windows_dhcp_scope_pending_offers` for reservations and offers in flight. +2. On the server, read the scope and its leases: + + ```powershell + Get-DhcpServerv4ScopeStatistics -ScopeId + Get-DhcpServerv4Scope -ScopeId | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State + Get-DhcpServerv4Lease -ScopeId | Group-Object AddressState + Get-DhcpServerv4ExclusionRange -ScopeId + ``` + +3. Look for a long lease duration on a network with many short-lived clients, + large exclusion ranges, a flood of leases from one device that changes its + MAC address, or a failover partner that holds the free addresses. + +## Mitigation + +Shorten the lease duration for networks with transient clients, remove +exclusion ranges that are not needed, add a range through a superscope or a +larger subnet, or reclaim bad leases. On a failover pair, check the split of +the addresses between the partners. Confirm the free share stays above the +threshold. diff --git a/docs/runbooks/windowsdiskdrivenotok.md b/docs/runbooks/windowsdiskdrivenotok.md new file mode 100644 index 00000000..62e3f2f2 --- /dev/null +++ b/docs/runbooks/windowsdiskdrivenotok.md @@ -0,0 +1,46 @@ +--- +title: WindowsDiskDriveNotOK +--- + +## Meaning + +The physical drive `device_id` reports an operational status other than `OK` +for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded` +or `Error`, in its `status` label. It needs the `diskdrive` collector, which +reads the status that the drive and its driver report through WMI. The +`Unknown` status does not raise the alert: some USB and virtual drives report +it permanently. Transient statuses such as `Starting` or `Stopping` raise it +only when they last longer than five minutes. + +## Impact + +`Pred Fail` is the drive's own prediction that it is about to fail. Any other +status means the drive is not fully working. Data on a single drive is at risk; +a drive in a mirror or parity set leaves the set without redundancy until it is +replaced. + +## Diagnosis + +1. Inspect `windows_diskdrive_status` for the host and check whether the + volumes on the drive still report normal latency with + `windows:logical_disk_read_latency:seconds` and + `windows:logical_disk_write_latency:seconds`. +2. On the host, read the drive and its reliability counters: + + ```powershell + Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status + Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear + ``` + +3. Check the System event log for disk, storahci, stornvme or RAID controller + events. On a virtual machine, check the hypervisor and the datastore of the + virtual disk. + +## Mitigation + +Back up the data on the drive first. Replace a drive that predicts its failure +or reports errors through the hardware or virtualization team. For a drive in +a storage pool or RAID set, follow the vendor's replacement and rebuild +procedure and confirm the set is redundant again. The alert resolves when the +status returns to `OK`. diff --git a/docs/runbooks/windowsdnszonetransferfailures.md b/docs/runbooks/windowsdnszonetransferfailures.md new file mode 100644 index 00000000..ef692e66 --- /dev/null +++ b/docs/runbooks/windowsdnszonetransferfailures.md @@ -0,0 +1,54 @@ +--- +title: WindowsDNSZoneTransferFailures +--- + +## Meaning + +The DNS server has failed zone transfers for 15 minutes, measured as the rate +of `windows_dns_zone_transfer_failures_total` over `rateInterval` above +`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts +transfers this server sends as the primary to its secondaries, including +requests it refuses, and it has no zone label. The alert needs the `dns` +collector and keeps firing for `keepFiringFor` after the failures stop. Raise +the threshold on servers that refuse transfer requests from scanners all the +time. + +## Impact + +The secondary servers of the zone serve stale records. When the zone on a +secondary expires, it stops answering for the zone altogether. Active +Directory integrated zones replicate through Active Directory and are not +affected unless they also have secondaries. + +## Diagnosis + +1. Inspect `windows_dns_zone_transfer_failures_total` next to + `windows_dns_zone_transfer_requests_received_total` and + `windows_dns_zone_transfer_success_sent_total` to see what share of the + transfers fail, and whether the failures started with a change. +2. On this primary, find the refused or failed requests and the zone + settings; the DNS Server log records a refused transfer request with the + address of the requester: + + ```powershell + Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer' + Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy + Get-DnsServerZone -Name '' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers + ``` + + On the secondary, events 6522 to 6527 of the DNS Server log say why its + transfer of the zone failed: refused by the primary, invalid response or + an expired zone. + +3. Check that the requesting secondary is in the zone transfer list, that TCP + port 53 is open between the servers, and that the zone is not paused or + broken. Failures from unknown addresses are refused transfers and may be + expected. + +## Mitigation + +Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries` +and the server list, fix the network path, or remove a decommissioned +secondary from the zone's notify list so it stops requesting transfers. +Confirm the failure counter stops rising and the secondaries show a current +serial number. diff --git a/docs/runbooks/windowshypervreplicationcritical.md b/docs/runbooks/windowshypervreplicationcritical.md new file mode 100644 index 00000000..72132afc --- /dev/null +++ b/docs/runbooks/windowshypervreplicationcritical.md @@ -0,0 +1,47 @@ +--- +title: WindowsHyperVReplicationCritical +--- + +## Meaning + +Hyper-V Replica reports critical health for the virtual machine `vm` for five +minutes. The alert carries the `relationship` label, `primary` for the normal +replication and `extended` for an extended replica, and needs the `hyperv` +collector with its `replica_vm` subcollector. Critical health means that +replication has not succeeded for long enough that the replica misses the +recovery point objective, or that replication is paused because of an error. + +## Impact + +The replica of the virtual machine is stale. A failover to it loses the +changes since the last successful replication, and the recovery point +objective of the service is not met. If the primary host fails now, data is +lost. + +## Diagnosis + +1. Inspect `windows_hyperv_replica_vm_health` and + `windows_hyperv_replica_vm_state` for the virtual machine; the state tells + whether replication is suspended, resynchronizing or in error, and + `windows_hyperv_replica_vm_replication_latency_seconds` how long the last + cycle took. +2. On the primary host, read the replication details: + + ```powershell + Measure-VMReplication -VMName '' | Format-List * + Get-VMReplication -VMName '' | Format-List State, Health, LastReplicationTime, ReplicationMode + Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '**' + ``` + +3. Check the replica server: free space on the replica storage, the + Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication, + and the network path between the hosts. Check whether a snapshot merge or a + storage migration on the primary blocks replication. + +## Mitigation + +Fix the cause on the replica server or the network, then resume replication +with `Resume-VMReplication -VMName ''`; add `-Resynchronize` when the +state asks for a resynchronization. Confirm the health returns to normal and +the last replication time advances. If the replica cannot be recovered, remove +and re-enable replication, which sends a full initial copy. diff --git a/docs/runbooks/windowshypervreplicationwarning.md b/docs/runbooks/windowshypervreplicationwarning.md new file mode 100644 index 00000000..e8e36490 --- /dev/null +++ b/docs/runbooks/windowshypervreplicationwarning.md @@ -0,0 +1,43 @@ +--- +title: WindowsHyperVReplicationWarning +--- + +## Meaning + +Hyper-V Replica reports warning health for the virtual machine `vm` for 15 +minutes. Warning health means that some replication cycles missed their +interval, that more than a fifth of the cycles in the last hour failed, or +that the replication is waiting for an initial copy or a resynchronization. +The alert carries the `relationship` label and needs the `hyperv` collector +with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/) +follows when replication stops working. + +## Impact + +The replica lags behind the primary more than planned. The recovery point +objective is at risk, and a failover would lose more changes than expected. + +## Diagnosis + +1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state` + and `windows_hyperv_replica_vm_replication_latency_seconds` for the + virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes` + to see whether the change rate of the virtual machine has grown. +2. On the primary host, read the statistics of the last hour: + + ```powershell + Measure-VMReplication -VMName '' | Format-List * + Get-VMReplication -VMName '' | Format-List State, Health, LastReplicationTime, FrequencySec + ``` + +3. Check the bandwidth between the hosts, the load on the replica storage, + and whether the replication interval is too short for the change rate of + the virtual machine. + +## Mitigation + +Relieve the bottleneck: more bandwidth, less load on the replica storage, or a +longer replication interval through `Set-VMReplication -ReplicationFrequencySec`. +Resume a waiting replication with `Resume-VMReplication`. Reset the statistics +with `Reset-VMReplicationStatistics` after the fix; the health is computed +over the last hour and takes up to an hour to return to normal. diff --git a/docs/runbooks/windowshypervwmiunhealthy.md b/docs/runbooks/windowshypervwmiunhealthy.md new file mode 100644 index 00000000..d13b248d --- /dev/null +++ b/docs/runbooks/windowshypervwmiunhealthy.md @@ -0,0 +1,43 @@ +--- +title: WindowsHyperVWMIUnhealthy +--- + +## Meaning + +The Hyper-V WMI namespace `root/virtualization/v2` on the host has not +answered a minimal query for ten minutes. The alert needs the `hyperv` +collector with its `wmi_health` subcollector. While the namespace is broken, +the virtual machine, replication and virtual hard disk metrics that come from +WMI are missing, so the other Hyper-V alerts cannot fire. + +## Impact + +Management tools such as Hyper-V Manager, Failover Cluster Manager and System +Center cannot manage the virtual machines on the host. Backups that use the +Hyper-V VSS writer fail. The virtual machines themselves usually keep running. + +## Diagnosis + +1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}` + for the host. +2. On the host, test the namespace and the management service: + + ```powershell + Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1 + Get-Service vmms, vmcompute, winmgmt + winmgmt /verifyrepository + ``` + +3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of + the Virtual Machine Management service, and the WMI-Activity log for + provider failures. A stuck `vmms` process or a corrupt WMI repository are + the usual causes. + +## Mitigation + +Restart the Virtual Machine Management service with `Restart-Service vmms`; +running virtual machines are not affected. If the repository is inconsistent, +run `winmgmt /salvagerepository`, and register the Hyper-V provider again with +`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a +restart of the host if the service cannot be recovered. Confirm the health +metric returns to 1 and the Hyper-V metrics appear again. diff --git a/docs/runbooks/windowslicensenotgenuine.md b/docs/runbooks/windowslicensenotgenuine.md new file mode 100644 index 00000000..1fb33ff0 --- /dev/null +++ b/docs/runbooks/windowslicensenotgenuine.md @@ -0,0 +1,45 @@ +--- +title: WindowsLicenseNotGenuine +--- + +## Meaning + +Windows reports a license status other than `genuine` for one hour. The alert +carries the status in its `state` label and needs the `license` collector. The +status comes from the Software Licensing service and changes when an +activation expires, a KMS host is unreachable for more than the grace period, +the hardware changed, or the key was blocked. `offline` means the genuine +validation could not reach Microsoft; on hosts without internet access that +is the permanent state, so turn the alert off there with +`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter +selector. + +## Impact + +An unlicensed Windows shows activation notices, may limit personalization and, +on some editions, restarts on a schedule. A Windows Server that leaves its +activation grace period is out of compliance. + +## Diagnosis + +1. The `state` label of the alert names the status; `windows_license_status` + shows all states of the host. +2. On the host, read the license details: + + ```powershell + slmgr.vbs /dlv + slmgr.vbs /xpr + Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining + ``` + +3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp` + DNS record and answers on TCP port 1688. For Active Directory based + activation, check the activation object in the forest. For MAK or retail + keys, check whether the activation count is exhausted. + +## Mitigation + +Restore reachability to the KMS host or the activation object, then run +`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk ` if the +key was wrong, or reactivate after a hardware change. Confirm the status +returns to `genuine`. diff --git a/docs/runbooks/windowspagefilealmostfull.md b/docs/runbooks/windowspagefilealmostfull.md new file mode 100644 index 00000000..62ebbea4 --- /dev/null +++ b/docs/runbooks/windowspagefilealmostfull.md @@ -0,0 +1,41 @@ +--- +title: WindowsPagefileAlmostFull +--- + +## Meaning + +The page file `file` has been more than the configured share in use for 15 +minutes. The default `pagefileUsageThreshold` is 80 percent of +`windows_pagefile_limit_bytes`, the current size of the file. The alert needs +the `pagefile` collector. A system-managed page file grows up to its maximum +when it fills, so a high usage of the current size on such a system shows that +the file has already grown. + +## Impact + +When the page file is full, Windows cannot commit more memory. Allocations +fail, processes crash with out-of-memory errors, and the system can become +unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/) +usually fires at the same time. + +## Diagnosis + +1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes` + with `windows:memory_committed:ratio` and `windows_memory_available_bytes` + to see whether physical memory is exhausted too. +2. Find the processes that hold the commit charge: + + ```powershell + Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}} + Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage + ``` + +3. Check for a recent change: a new workload, a memory leak that grows over + days, or a page file that was made smaller or fixed in size. + +## Mitigation + +Stop or restart the process that leaks memory, or move workload off the host. +Add physical memory or enlarge the page file through the normal change +process; a fixed page file needs enough room for the peak commit charge of the +host. Confirm usage falls below the threshold and stays there. diff --git a/docs/runbooks/windowsscheduledtaskfailed.md b/docs/runbooks/windowsscheduledtaskfailed.md new file mode 100644 index 00000000..366a0eea --- /dev/null +++ b/docs/runbooks/windowsscheduledtaskfailed.md @@ -0,0 +1,51 @@ +--- +title: WindowsScheduledTaskFailed +--- + +## Meaning + +The last run of the scheduled task `task` ended with the result `unknown` or +`terminated`, and the task has not completed successfully since for one hour. +The alert needs the `scheduled_task` collector. `terminated` means the run was +stopped by a user or by the scheduler's time limit. `unknown` is every result +code without a named status, which includes the exit code of the task's +program and the error codes of the Task Scheduler; the documentation of the +collector lists the named codes. A program that exits with a code other than +zero on success raises this alert on every run; exclude such tasks with +`taskSelector` or with the include and exclude flags of the collector. A +disabled task keeps its last result but does not raise the alert. + +## Impact + +The work of the task did not happen: backups, cleanups, reports, certificate +renewals or synchronizations that are scheduled on this host. The impact +depends on the task. + +## Diagnosis + +1. Inspect `windows_scheduled_task_last_result_status` for the task and + `windows_scheduled_task_state` to see whether the task is still enabled + and queued. +2. On the host, read the last result and the history: + + ```powershell + Get-ScheduledTaskInfo -TaskName '' -TaskPath '' + Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '**' + ``` + + The history log is off by default; enable it with + `wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`. +3. Translate the result code: `0x1` is a generic failure of the program, `0x2` + a missing file, `0x80070005` an access denied, and codes of the form + `0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other + codes between `0x41300` and `0x41325` are named statuses, such as `ready` + or `running`, and do not raise this alert. +4. Run the program by hand under the task's account to reproduce the failure. + +## Mitigation + +Fix the program, its arguments, its working directory or the permissions of +the account the task runs as. Start the task once with `Start-ScheduledTask` +and confirm the result turns to `success`. Disable or delete tasks that are +not needed anymore instead of leaving them failing; disabling a task clears +the alert. diff --git a/docs/runbooks/windowsstoragepoolunhealthy.md b/docs/runbooks/windowsstoragepoolunhealthy.md new file mode 100644 index 00000000..b8cdd118 --- /dev/null +++ b/docs/runbooks/windowsstoragepoolunhealthy.md @@ -0,0 +1,48 @@ +--- +title: WindowsStoragePoolUnhealthy +--- + +## Meaning + +The Storage Spaces pool `pool` reports a health status of warning or unhealthy +for ten minutes. The critical alert fires for unhealthy, the warning for +warning. The alert needs the `storage_spaces` collector and joins +`windows_storage_spaces_pool_info` so the primordial pool, which only groups +the disks that are not in a pool yet, does not raise it. The pool's +operational status, exposed as `windows_storage_spaces_pool_operational_status`, +tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every +node reports the clustered pool, so the alert fires once per node; the +`instance` label names the reporter. + +## Impact + +A pool in warning state has lost a physical disk or redundancy; the virtual +disks on it still work but have no tolerance left for another failure, or +cannot allocate more space. An unhealthy pool has lost more disks than its +resiliency tolerates, so virtual disks on it are detached or read-only. + +## Diagnosis + +1. Inspect `windows_storage_spaces_pool_health_status`, + `windows_storage_spaces_pool_operational_status` and + `windows_storage_spaces_virtual_disk_health_status` for the pool and its + virtual disks. +2. On the host, find the failed disk: + + ```powershell + Get-StoragePool -FriendlyName '' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly + Get-StoragePool -FriendlyName '' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage + Get-StoragePool -FriendlyName '' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-StorageJob + ``` + +3. Check the System log for disk and storage events, and whether a repair job + is already running. + +## Mitigation + +Replace the failed physical disk: retire it with +`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`, +repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk +with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the +health returns to healthy once the repair jobs finish. diff --git a/docs/runbooks/windowsstoragevirtualdiskunhealthy.md b/docs/runbooks/windowsstoragevirtualdiskunhealthy.md new file mode 100644 index 00000000..37a15904 --- /dev/null +++ b/docs/runbooks/windowsstoragevirtualdiskunhealthy.md @@ -0,0 +1,43 @@ +--- +title: WindowsStorageVirtualDiskUnhealthy +--- + +## Meaning + +The Storage Spaces virtual disk `virtual_disk` reports a health status of +warning or unhealthy for ten minutes. The critical alert fires for unhealthy, +the warning for warning. The alert needs the `storage_spaces` collector. A +virtual disk is in warning state when it is degraded or incomplete and still +accessible, and unhealthy when it is detached or has lost data. On Storage +Spaces Direct every node reports the clustered virtual disks, so the alert +fires once per node; the `instance` label names the reporter. + +## Impact + +A degraded virtual disk still serves data but has no redundancy left; a +further disk failure loses the data. A detached virtual disk takes its volumes +offline, and everything stored on them is unavailable. + +## Diagnosis + +1. Inspect `windows_storage_spaces_virtual_disk_health_status` and + `windows_storage_spaces_pool_health_status` to see whether the pool is + affected too; [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/) + then names the pool. +2. On the host, read the virtual disk, its footprint and the repair jobs: + + ```powershell + Get-VirtualDisk -FriendlyName '' | Select-Object FriendlyName, HealthStatus, OperationalStatus, ResiliencySettingName, DetachedReason + Get-VirtualDisk -FriendlyName '' | Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus + Get-StorageJob + ``` + +3. Check for a failed physical disk, a pool that ran out of space for + allocation, or a disk that was detached by a policy after an error. + +## Mitigation + +Replace the failed physical disk as in [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/) +and repair the virtual disk with `Repair-VirtualDisk`. Attach a detached disk +with `Connect-VirtualDisk` once the cause is fixed. Confirm the health returns +to healthy once the repair jobs finish. diff --git a/docs/runbooks/windowsupdatespending.md b/docs/runbooks/windowsupdatespending.md new file mode 100644 index 00000000..37d25411 --- /dev/null +++ b/docs/runbooks/windowsupdatespending.md @@ -0,0 +1,50 @@ +--- +title: WindowsUpdatesPending +--- + +## Meaning + +The host has updates waiting for installation for longer than +`updatePendingDuration`, seven days by default. The alert is an info alert, its +value is the number of pending updates, and it needs the `update` collector, +which queries the Windows Update agent in the background at the interval of +its scrape setting. Updates waiting for uninstallation do not count. Prometheus counts the +duration and restores it after a restart shorter than its +`--rules.alert.for-outage-tolerance`, one hour by default; a longer outage or +a new server starts the wait again. + +## Impact + +Security updates that wait longer than the patch window leave known +vulnerabilities open. Other updates carry fixes for reliability problems. The +host may also be waiting for a restart to finish an installation. + +## Diagnosis + +1. List the pending updates with their category and severity: + + ```promql + windows_update_pending_info{job="windows_exporter", instance="", deployment_action="installation"} + ``` + + `windows_update_pending_published_timestamp` shows how old each update is. +2. On the host, check the update client and the last installation attempts: + + ```powershell + Get-WindowsUpdateLog + Get-WinEvent -LogName Microsoft-Windows-WindowsUpdateClient/Operational -MaxEvents 50 + Get-CimInstance -Namespace root/cimv2 -ClassName Win32_QuickFixEngineering | Sort-Object InstalledOn -Descending | Select-Object -First 5 + ``` + +3. Check whether the host is in a maintenance window, a deferral period or a + WSUS or Configuration Manager deployment that has not approved the updates + yet, and whether a pending restart blocks the installation. + +## Mitigation + +Install the updates in the next maintenance window, or approve them in the +update management tool. Restart the host if an installation waits for it. +Raise `updatePendingDuration` to the patch window of the environment if seven +days is shorter than the policy, or turn the alert off with +`alerts+: { WindowsUpdatesPending: false }` where another system tracks patch +compliance.