sendnrw/windows_exporter
2
0
Fork 0
mirror of https://github.com/prometheus-community/windows_exporter.git synced 2026-10-20 03:29:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

mixin: add alerts for the opt-in collectors

Browse Source 
Every optional collector of the mixin now has alerts, generated only with
that collector enabled in _config.collectors: clock source, disk drive
status, page file usage, license status, scheduled task results, pending
updates, Hyper-V WMI and replication health, failover cluster nodes,
resources, groups and shared volumes, Storage Spaces pool and virtual disk
health, DHCP scope exhaustion and DNS zone transfer failures. The alerts
of the pagefile, diskdrive and license collectors join the memory, disk
and system groups; the other collectors get a group of their own.

New knobs: taskSelector, sharedVolumeSelector, pagefileUsageThreshold,
dhcpScopeFreeWarningThreshold, dhcpScopeFreeCriticalThreshold and
updatePendingDuration. The test configuration enables every optional
collector and tests/optional.test.yaml covers the new alerts.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-17 02:28:35 +00:00
committed by Claude
parent d6afd48819
commit d819362150
24 changed files with 1688 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/mixin/README.md
+57 -7
View File
@@ -85,7 +85,10 @@ apply to both recording rules and alerts. All target labels, including `job`,
averages idle rates across cores and removes only `core` and `mode`. Disk,
network and service rules also retain `volume`, `nic` and `service`
respectively, filtered by `volumeSelector`, `nicSelector` and
`serviceSelector`. The CPU mode recording retains `mode` and removes only
`serviceSelector`. The scheduled task and cluster shared volume alerts
filter their `task` and `shared_volume` labels with `taskSelector`
(`task!=""`) and `sharedVolumeSelector`
(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only
`core`.
`rateInterval` is the fixed window for recording rules and Active Directory
@@ -179,6 +182,27 @@ percentages except where noted.
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning |
| `windows-memory` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
| `windows-disk` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
| `windows-system` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning |
| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info |
| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning |
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical |
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning |
| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical |
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning |
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical |
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning |
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning |
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical |
| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning |
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
@@ -188,9 +212,11 @@ alerts are all turned off is not generated.
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
value that oscillates around a threshold does not flap. The state alerts
`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`,
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning` and
`WindowsClockNotSynchronising`, and the one-shot `WindowsHostRebooted`, have no
`keep_firing_for` and resolve as soon as their condition clears.
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`,
`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors
(clock source, drive status, license, task result, pending updates, Hyper-V,
cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`,
have no `keep_firing_for` and resolve as soon as their condition clears.
`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for
older servers.
@@ -206,9 +232,33 @@ notification should be sent.
Active Directory alerts need `collectors.ad`, which `enableActiveDirectory`
sets. Clock alerts need `collectors.time`, which `enableTime` sets. A zero NTP
source count may be expected on hosts using another time provider. The offset
compares the host to its chosen source and does not independently verify that
source. Disk latency, paging and network thresholds should be tuned for the
source count may be expected on hosts using another time provider;
`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and
fires only for the local clock or no synchronization. The offset compares the
host to its chosen source and does not independently verify that source.
The other opt-in alerts follow `_config.collectors`: enable `pagefile`,
`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`,
`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are
generated, in the group of their area or in a group of their own.
`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code
without a name, including the exit codes of the task's program, as a failure;
narrow the tasks with `taskSelector`. Disabling a task clears the alert.
`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and
virtual drives report permanently. `WindowsLicenseNotGenuine` reports the
`state` the host has; hosts without internet access can stay in `offline`,
so turn the alert off there. Every failover cluster node reports the state of
the whole cluster, and every Storage Spaces Direct node reports the clustered
pools and virtual disks, so the cluster and storage alerts fire once per node;
group them by their `node`, `resource`, `shared_volume`, `pool` or
`virtual_disk` labels in Alertmanager. The storage pool alert joins
`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP
scope alert only counts scopes in an enabled state, and a scope without
addresses produces no sample. `WindowsUpdatesPending` is an info alert that
waits `updatePendingDuration` for the normal maintenance window. Prometheus
restores the waiting time after a restart shorter than its
`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage
or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the
storage tier, traffic volume, and application requirements.
`WindowsServiceNotRunning` watches services with the `auto` start mode and
contrib/mixin/alerts/alerts.libsonnet
+244
View File
@@ -25,6 +25,12 @@
} + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
] else [],
local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
// Percent of free addresses in a scope; a scope without addresses has no sample.
local dhcpScopeFree = '(%s * 100)' % q.ratio(
q.metric('windows_dhcp_scope_addresses_free'),
'(%s + %s)' % [q.metric('windows_dhcp_scope_addresses_free'), q.metric('windows_dhcp_scope_addresses_in_use')]
),
local dhcpScopeActive = q.metric('windows_dhcp_scope_state', ['state=~"Enabled|EnabledSwitched"']),
prometheusAlerts+:: {
groups:
group('windows-exporter', std.flattenArrays([
@@ -113,6 +119,18 @@
'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold,
collectors=['memory']
),
alert(
'WindowsPagefileAlmostFull',
'(1 - %s) * 100 > %s' % [
q.ratio(q.metric('windows_pagefile_free_bytes'), q.metric('windows_pagefile_limit_bytes')),
c.pagefileUsageThreshold,
],
'15m',
'warning',
'Windows page file is almost full',
'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %s%% in use for 15 minutes.' % c.pagefileUsageThreshold,
collectors=['pagefile']
),
]))
+ group('windows-disk', std.flattenArrays([
alert(
@@ -154,6 +172,19 @@
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours,
collectors=['logical_disk']
),
alert(
'WindowsDiskDriveNotOK',
// One series per status with the current status set to 1, so the
// alert carries the status the drive reports. Some USB and virtual
// drives report Unknown for good.
q.metric('windows_diskdrive_status', ['status!~"OK|Unknown"']) + ' == 1',
'5m',
c.criticalSeverity,
'Windows disk drive is not healthy',
'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
collectors=['diskdrive'],
threshold=false
),
]))
+ group('windows-network', std.flattenArrays([
alert(
@@ -256,6 +287,16 @@
collectors=['system'],
threshold=false
),
alert(
'WindowsLicenseNotGenuine',
q.metric('windows_license_status', ['state!="genuine"']) + ' == 1',
'1h',
'warning',
'Windows license is not genuine',
'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
collectors=['license'],
threshold=false
),
]))
+ group('windows-active-directory', std.flattenArrays([
alert(
@@ -297,6 +338,209 @@
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds,
collectors=['time']
),
alert(
'WindowsClockSourceLocal',
// Covers domain members (NT5DS) too, which the NTP source count misses.
q.metric('windows_time_clock_sync_source', ['type=~"Local CMOS Clock|NoSync"']) + ' == 1',
'10m',
'warning',
'Windows clock has no external time source',
'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
collectors=['time'],
threshold=false
),
]))
+ group('windows-scheduled-task', std.flattenArrays([
alert(
'WindowsScheduledTaskFailed',
// unknown is any result code without a named status, including the
// exit codes of the task's program.
// A disabled task keeps its last result, so disabling a broken task
// clears the alert.
'%s == 1 unless ignoring (status, state) (%s == 1)' % [
q.metric('windows_scheduled_task_last_result_status', [c.taskSelector, 'status=~"unknown|terminated"']),
q.metric('windows_scheduled_task_state', [c.taskSelector, 'state="disabled"']),
],
'1h',
'warning',
'Windows scheduled task did not succeed',
'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
collectors=['scheduled_task'],
threshold=false
),
]))
+ group('windows-update', std.flattenArrays([
alert(
'WindowsUpdatesPending',
'count without (id, revision, category, severity, title) (%s) > 0' % q.metric('windows_update_pending_info', ['deployment_action="installation"']),
c.updatePendingDuration,
c.infoSeverity,
'Windows updates are waiting for installation',
'{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %s.' % c.updatePendingDuration,
collectors=['update'],
threshold=false
),
]))
+ group('windows-hyperv', std.flattenArrays([
alert(
'WindowsHyperVWMIUnhealthy',
q.metric('windows_hyperv_wmi_health') + ' == 0',
'10m',
'warning',
'Hyper-V WMI namespace is not responding',
'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationCritical',
q.metric('windows_hyperv_replica_vm_health') + ' == 3',
'5m',
c.criticalSeverity,
'Hyper-V replication is critical',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
collectors=['hyperv'],
threshold=false
),
alert(
'WindowsHyperVReplicationWarning',
q.metric('windows_hyperv_replica_vm_health') + ' == 2',
'15m',
'warning',
'Hyper-V replication has a warning',
'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
collectors=['hyperv'],
threshold=false
),
]))
// Every cluster node reports the state of the whole cluster, so these
// alerts fire once per node; group them by cluster labels in Alertmanager.
+ group('windows-cluster', std.flattenArrays([
alert(
'WindowsClusterNodeDown',
q.metric('windows_mscluster_node_state') + ' == 1',
'5m',
c.criticalSeverity,
'Failover cluster node is down',
'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceFailed',
q.metric('windows_mscluster_resource_state') + ' == 4',
'5m',
c.criticalSeverity,
'Failover cluster resource has failed',
'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterResourceGroupFailed',
q.metric('windows_mscluster_resourcegroup_state') + ' == 2',
'5m',
c.criticalSeverity,
'Failover cluster resource group has failed',
'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
collectors=['mscluster'],
threshold=false
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeWarningThreshold,
'15m',
'warning',
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeWarningThreshold,
collectors=['mscluster']
),
alert(
'WindowsClusterSharedVolumeAlmostFull',
q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeCriticalThreshold,
'15m',
c.criticalSeverity,
'Cluster shared volume is almost full',
'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeCriticalThreshold,
collectors=['mscluster']
),
]))
// On Storage Spaces Direct every node reports the clustered pools and
// virtual disks, so these alerts fire once per node as well.
+ group('windows-storage-spaces', std.flattenArrays([
// The primordial pool holds the disks that are not in a concrete pool
// yet; only the info metric tells it apart.
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 2) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
c.criticalSeverity,
'Storage pool is unhealthy',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStoragePoolUnhealthy',
'(%s == 1) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])],
'10m',
'warning',
'Storage pool has a health warning',
'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 2',
'10m',
c.criticalSeverity,
'Storage Spaces virtual disk is unhealthy',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
alert(
'WindowsStorageVirtualDiskUnhealthy',
q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 1',
'10m',
'warning',
'Storage Spaces virtual disk has a health warning',
'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
collectors=['storage_spaces'],
threshold=false
),
]))
+ group('windows-dhcp', std.flattenArrays([
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeWarningThreshold, dhcpScopeActive],
'15m',
'warning',
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeWarningThreshold,
collectors=['dhcp']
),
alert(
'WindowsDHCPScopeAlmostExhausted',
dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeCriticalThreshold, dhcpScopeActive],
'15m',
c.criticalSeverity,
'DHCP scope is almost exhausted',
'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeCriticalThreshold,
collectors=['dhcp']
),
]))
+ group('windows-dns', std.flattenArrays([
alert(
'WindowsDNSZoneTransferFailures',
q.rate('windows_dns_zone_transfer_failures_total') + ' > %g' % c.dnsZoneTransferFailureRateThreshold,
'15m',
'warning',
'DNS zone transfers are failing',
'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
collectors=['dns']
),
])),
},
}
contrib/mixin/config.libsonnet
+34
View File
@@ -5,6 +5,9 @@
volumeSelector: 'volume!=""',
nicSelector: 'nic!=""',
serviceSelector: 'service!=""',
taskSelector: 'task!=""',
// ClusterPerformanceHistory is the volume of the cluster's own health history.
sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"',
rateInterval: '5m',
// Opt in only for targets with the ad collector enabled. Sets collectors.ad.
@@ -43,14 +46,19 @@
collectors: {
ad: $._config.enableActiveDirectory,
cpu: true,
dhcp: false,
diskdrive: false,
dns: false,
gpu: false,
hyperv: false,
license: false,
logical_disk: true,
memory: true,
mscluster: false,
mssql: false,
net: true,
os: true,
pagefile: false,
physical_disk: true,
process: false,
scheduled_task: false,
@@ -94,6 +102,23 @@
WindowsADReplicationFailures: true,
WindowsClockNotSynchronising: true,
WindowsClockSkewDetected: true,
WindowsClockSourceLocal: true,
WindowsDiskDriveNotOK: true,
WindowsPagefileAlmostFull: true,
WindowsLicenseNotGenuine: true,
WindowsScheduledTaskFailed: true,
WindowsUpdatesPending: true,
WindowsHyperVWMIUnhealthy: true,
WindowsHyperVReplicationCritical: true,
WindowsHyperVReplicationWarning: true,
WindowsClusterNodeDown: true,
WindowsClusterResourceFailed: true,
WindowsClusterResourceGroupFailed: true,
WindowsClusterSharedVolumeAlmostFull: true,
WindowsStoragePoolUnhealthy: true,
WindowsStorageVirtualDiskUnhealthy: true,
WindowsDHCPScopeAlmostExhausted: true,
WindowsDNSZoneTransferFailures: true,
},
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
@@ -127,5 +152,14 @@
infoSeverity: 'info',
adPendingReplicationThreshold: 50,
adReplicationFailureRateThreshold: 0,
// Percent of the page file in use.
pagefileUsageThreshold: 80,
// Percent of free addresses in an active DHCP scope.
dhcpScopeFreeWarningThreshold: 10,
dhcpScopeFreeCriticalThreshold: 2,
// How long updates may wait for installation before WindowsUpdatesPending fires.
updatePendingDuration: '7d',
// Failed zone transfers per second.
dnsZoneTransferFailureRateThreshold: 0,
},
}
contrib/mixin/tests/alerts.test.jsonnet
+33 -3
View File
@@ -2,7 +2,8 @@
// keep_firing_for and the group layout.
local mixin = import '../mixin.libsonnet';
local config = (import '../config.libsonnet')._config;
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true } };
local everything = mixin { _config+:: { enableActiveDirectory: true, enableTime: true, collectors: { [name]: true for name in std.objectFields(config.collectors) } } };
local optionalGroups = ['windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
local groupNames(alerts) = [group.name for group in alerts.groups];
@@ -12,7 +13,13 @@ assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs fro
// Groups follow the collector areas; opt-in groups appear only when enabled.
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory', 'windows-time'];
assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + optionalGroups : std.manifestJson(groupNames(everything.prometheusAlerts));
// Opt-in collectors add their alerts to the default groups of their area too.
local adOnly = mixin { _config+:: { enableActiveDirectory: true } };
assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory'];
local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } };
assert groupNames(pagefile.prometheusAlerts) == groupNames(mixin.prometheusAlerts);
assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull'];
// Disabling a collector removes its alerts and an empty group; the alerts of a
// rule that needs two collectors go with either.
@@ -34,7 +41,28 @@ assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAle
// keep_firing_for applies to threshold alerts only and can be turned off.
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
local stateAlerts = ['WindowsExporterDown', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsScrapeQueueTimeout', 'WindowsServiceNotRunning', 'WindowsHostRebooted', 'WindowsClockNotSynchronising'];
local stateAlerts = [
'WindowsExporterDown',
'WindowsCollectorFailed',
'WindowsCollectorTimeout',
'WindowsScrapeQueueTimeout',
'WindowsServiceNotRunning',
'WindowsHostRebooted',
'WindowsClockNotSynchronising',
'WindowsClockSourceLocal',
'WindowsDiskDriveNotOK',
'WindowsLicenseNotGenuine',
'WindowsScheduledTaskFailed',
'WindowsUpdatesPending',
'WindowsHyperVWMIUnhealthy',
'WindowsHyperVReplicationCritical',
'WindowsHyperVReplicationWarning',
'WindowsClusterNodeDown',
'WindowsClusterResourceFailed',
'WindowsClusterResourceGroupFailed',
'WindowsStoragePoolUnhealthy',
'WindowsStorageVirtualDiskUnhealthy',
];
assert std.all([
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
for rule in rules(everything.prometheusAlerts)
@@ -46,5 +74,7 @@ assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.object
// The info severity label is configurable.
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } };
assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']];
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
contrib/mixin/tests/config.libsonnet
+18
View File
@@ -19,5 +19,23 @@
clockOffsetThresholdSeconds: 0.1,
adPendingReplicationThreshold: 10,
adReplicationFailureRateThreshold: 0.5,
collectors+: {
dhcp: true,
diskdrive: true,
dns: true,
hyperv: true,
license: true,
mscluster: true,
pagefile: true,
scheduled_task: true,
storage_spaces: true,
update: true,
},
pagefileUsageThreshold: 75,
dhcpScopeFreeWarningThreshold: 20,
dhcpScopeFreeCriticalThreshold: 5,
updatePendingDuration: '2d',
dnsZoneTransferFailureRateThreshold: 0.05,
taskSelector: 'task!~"/Ignored.*"',
},
}
contrib/mixin/tests/optional.test.yaml
+508
View File
@@ -0,0 +1,508 @@
# Alerts of the opt-in collectors, rendered with tests/config.libsonnet.
rule_files:
- custom_rules.yaml
- custom_alerts.yaml
evaluation_interval: 1m
group_eval_order:
- windows-exporter-recording
- windows-memory
- windows-disk
- windows-system
- windows-time
- windows-scheduled-task
- windows-update
- windows-hyperv
- windows-cluster
- windows-storage-spaces
- windows-dhcp
- windows-dns
tests:
- name: Page file usage fires above the custom threshold and resolves after keep_firing_for
interval: 1m
input_series:
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
values: '20x19 50x10'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
values: '100x30'
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
values: '30x30'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
values: '100x30'
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
values: '0x30'
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
values: '0x30'
alert_rule_test:
- {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []}
- alertname: WindowsPagefileAlmostFull
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
summary: Windows page file is almost full
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
- alertname: WindowsPagefileAlmostFull
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
summary: Windows page file is almost full
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
- {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []}
- name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour
interval: 1m
input_series:
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}'
values: '0x9 1x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}'
values: '1x9 0x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}'
values: '0x20'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}'
values: '1x20'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}'
values: '1x10'
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}'
values: '0x10'
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}'
values: '0x60'
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}'
values: '1x60'
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}'
values: '1x60'
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}'
values: '0x60'
alert_rule_test:
- {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []}
- alertname: WindowsDiskDriveNotOK
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
summary: Windows disk drive is not healthy
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
- alertname: WindowsDiskDriveNotOK
eval_time: 9m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
summary: Windows disk drive is not healthy
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
- {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []}
- {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []}
- alertname: WindowsLicenseNotGenuine
eval_time: 60m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/
summary: Windows license is not genuine
description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.'
- name: Clock source alert names the local source and ignores NTP and domain sources
interval: 1m
input_series:
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}'
values: '1x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}'
values: '0x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}'
values: '1x10'
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}'
values: '0x10'
alert_rule_test:
- {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []}
- alertname: WindowsClockSourceLocal
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/
summary: Windows clock has no external time source
description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.'
- name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks
interval: 1m
input_series:
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}'
values: '0x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}'
values: '0x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}'
values: '1x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}'
values: '1x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}'
values: '0x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}'
values: '0x60'
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}'
values: '1x60'
alert_rule_test:
- {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []}
- alertname: WindowsScheduledTaskFailed
eval_time: 60m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
summary: Windows scheduled task did not succeed
description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
summary: Windows scheduled task did not succeed
description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.'
- name: Pending updates fire as info after the configured duration and count installations only
interval: 1m
input_series:
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}'
values: '1x2880'
- series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}'
values: '1x2880'
alert_rule_test:
- {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []}
- alertname: WindowsUpdatesPending
eval_time: 48h
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsupdatespending/
summary: Windows updates are waiting for installation
description: 'host:9182 has had 2 updates waiting for installation for 2d.'
- name: Hyper-V alerts cover the WMI namespace and both replication health levels
interval: 1m
input_series:
- series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}'
values: '0x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}'
values: '3x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}'
values: '2x15'
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}'
values: '0x15'
alert_rule_test:
- {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []}
- alertname: WindowsHyperVWMIUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/
summary: Hyper-V WMI namespace is not responding
description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.'
- {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []}
- alertname: WindowsHyperVReplicationCritical
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/
summary: Hyper-V replication is critical
description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.'
- {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []}
- alertname: WindowsHyperVReplicationWarning
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/
summary: Hyper-V replication has a warning
description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.'
- name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes
interval: 1m
input_series:
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}'
values: '1x30'
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}'
values: '0x30'
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}'
values: '1x30'
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}'
values: '4x30'
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}'
values: '3x30'
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}'
values: '2x30'
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}'
values: '1x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
values: '15x19 50x10'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
values: '5x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
values: '1x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
values: '100x30'
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
values: '0x30'
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
values: '0x30'
alert_rule_test:
- {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []}
- alertname: WindowsClusterNodeDown
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
summary: Failover cluster node is down
description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
summary: Failover cluster node is down
description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.'
- alertname: WindowsClusterResourceFailed
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/
summary: Failover cluster resource has failed
description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
- alertname: WindowsClusterResourceGroupFailed
eval_time: 5m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/
summary: Failover cluster resource group has failed
description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
- {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []}
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- alertname: WindowsClusterSharedVolumeAlmostFull
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
summary: Cluster shared volume is almost full
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
- name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy
interval: 1m
input_series:
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}'
values: '2x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}'
values: '1x15'
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}'
values: '2x15'
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}'
values: '1x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}'
values: '2x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}'
values: '1x15'
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}'
values: '0x15'
alert_rule_test:
- {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []}
- alertname: WindowsStoragePoolUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
summary: Storage pool is unhealthy
description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
summary: Storage pool has a health warning
description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.'
- alertname: WindowsStorageVirtualDiskUnhealthy
eval_time: 10m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
summary: Storage Spaces virtual disk is unhealthy
description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.'
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
summary: Storage Spaces virtual disk has a health warning
description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.'
- name: DHCP scope exhaustion counts active scopes only and uses both thresholds
interval: 1m
input_series:
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
values: '10x19 50x10'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
values: '90x19 50x10'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}'
values: '1x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
values: '2x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
values: '98x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}'
values: '1x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
values: '100x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}'
values: '1x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
values: '0x30'
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}'
values: '1x30'
alert_rule_test:
- {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []}
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 15m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 24m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- alertname: WindowsDHCPScopeAlmostExhausted
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
summary: DHCP scope is almost exhausted
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
- name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for
interval: 1m
input_series:
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}'
values: '0+6x20 120x15'
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}'
values: '7x36'
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}'
values: '0+1x36'
alert_rule_test:
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []}
- alertname: WindowsDNSZoneTransferFailures
eval_time: 17m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
summary: DNS zone transfers are failing
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
- alertname: WindowsDNSZoneTransferFailures
eval_time: 25m
exp_alerts:
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
exp_annotations:
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
summary: DNS zone transfers are failing
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []}