From 51a579e03128a39b034838bf62b76af413fe2225 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan-Otto=20Kr=C3=B6pke?= Date: Sat, 17 Oct 2026 14:53:14 +0000 Subject: [PATCH] mixin: one alert file per collector with plain PromQL MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split alerts/alerts.libsonnet into one file per collector under alerts/, like kubernetes-mixin. Each rule is a plain Prometheus alerting rule with the PromQL written out in an expr block and %(name)s placeholders for the _config values; lib/alerts.libsonnet keeps the collector and per-alert gating and keep_firing_for. The pagefile, diskdrive and license alerts get groups of their own so that one file is one group. The rendered default rules are unchanged apart from expr block scalars. Signed-off-by: Jan-Otto Kröpke --- contrib/mixin/AGENTS.md | 49 +- contrib/mixin/README.md | 17 +- contrib/mixin/alerts/ad.libsonnet | 32 + contrib/mixin/alerts/alerts.libsonnet | 560 +----------------- contrib/mixin/alerts/cpu.libsonnet | 33 ++ contrib/mixin/alerts/dhcp.libsonnet | 36 ++ contrib/mixin/alerts/diskdrive.libsonnet | 24 + contrib/mixin/alerts/dns.libsonnet | 20 + contrib/mixin/alerts/exporter.libsonnet | 60 ++ contrib/mixin/alerts/hyperv.libsonnet | 47 ++ contrib/mixin/alerts/license.libsonnet | 21 + contrib/mixin/alerts/logical_disk.libsonnet | 69 +++ contrib/mixin/alerts/memory.libsonnet | 44 ++ contrib/mixin/alerts/mscluster.libsonnet | 73 +++ contrib/mixin/alerts/net.libsonnet | 63 ++ contrib/mixin/alerts/pagefile.libsonnet | 20 + contrib/mixin/alerts/scheduled_task.libsonnet | 26 + contrib/mixin/alerts/service.libsonnet | 38 ++ contrib/mixin/alerts/storage_spaces.libsonnet | 66 +++ contrib/mixin/alerts/system.libsonnet | 25 + contrib/mixin/alerts/time.libsonnet | 47 ++ contrib/mixin/alerts/update.libsonnet | 21 + contrib/mixin/lib/alerts.libsonnet | 23 + contrib/mixin/tests/alerts.test.jsonnet | 9 +- contrib/mixin/windows_alerts.yaml | 74 ++- docs/upgrade/0.32.md | 2 +- 26 files changed, 915 insertions(+), 584 deletions(-) create mode 100644 contrib/mixin/alerts/ad.libsonnet create mode 100644 contrib/mixin/alerts/cpu.libsonnet create mode 100644 contrib/mixin/alerts/dhcp.libsonnet create mode 100644 contrib/mixin/alerts/diskdrive.libsonnet create mode 100644 contrib/mixin/alerts/dns.libsonnet create mode 100644 contrib/mixin/alerts/exporter.libsonnet create mode 100644 contrib/mixin/alerts/hyperv.libsonnet create mode 100644 contrib/mixin/alerts/license.libsonnet create mode 100644 contrib/mixin/alerts/logical_disk.libsonnet create mode 100644 contrib/mixin/alerts/memory.libsonnet create mode 100644 contrib/mixin/alerts/mscluster.libsonnet create mode 100644 contrib/mixin/alerts/net.libsonnet create mode 100644 contrib/mixin/alerts/pagefile.libsonnet create mode 100644 contrib/mixin/alerts/scheduled_task.libsonnet create mode 100644 contrib/mixin/alerts/service.libsonnet create mode 100644 contrib/mixin/alerts/storage_spaces.libsonnet create mode 100644 contrib/mixin/alerts/system.libsonnet create mode 100644 contrib/mixin/alerts/time.libsonnet create mode 100644 contrib/mixin/alerts/update.libsonnet create mode 100644 contrib/mixin/lib/alerts.libsonnet diff --git a/contrib/mixin/AGENTS.md b/contrib/mixin/AGENTS.md index fb3f54ac..3e1644cd 100644 --- a/contrib/mixin/AGENTS.md +++ b/contrib/mixin/AGENTS.md @@ -7,23 +7,48 @@ keep it in sync with every change. ## Alerts -`alerts/alerts.libsonnet` renders one rule group per collector area. Every -alert names the collectors it reads (`collectors=[...]`) and is generated only -with those enabled in `_config.collectors`; `_config.alerts` switches single -alerts off by name. Threshold alerts get `keep_firing_for` from -`keepFiringFor`; pass `threshold=false` for state alerts (`== 0`, `== 1`) and -one-shot alerts, which must resolve at once. +`alerts/` holds one file per collector, each rendering one rule group, and +`alerts/alerts.libsonnet` imports them in the order of the generated file. +A rule is a plain Prometheus alerting rule, like in kubernetes-mixin: -A new alert touches five places, and CI checks all of them: +```jsonnet +{ + alert: 'WindowsCPUHighUsage', + expr: ||| + windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { summary: '...', description: '...' }, +} +``` -1. The rule in `alerts/alerts.libsonnet`, in the group of its area, with - `{{ $labels. }}` templates that use the 0.32 label keys (`service`, - `volume`, `nic`, `task`, `device_id`, never `name`). +Write the PromQL out in the `expr` block with `%(name)s` placeholders for the +`_config` values; selectors go inside the braces as +`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet` +provides `a.group(name, collectors, rules)`, which keeps a rule only when every +collector in `collectors` is enabled in `_config.collectors` and the alert is +not switched off in `_config.alerts`, and adds `keep_firing_for` from +`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]` +when a rule reads other collectors than its group (the CPU saturation alert +reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`, +`== 1`) and one-shot alerts, which must resolve at once and get no +`keep_firing_for`. A one-shot alert has no `for`. + +A new alert touches six places, and CI checks all of them: + +1. The rule in `alerts/.libsonnet`, with `{{ $labels. }}` + templates that use the 0.32 label keys (`service`, `volume`, `nic`, + `task`, `device_id`, never `name`). Alerts for a new collector go in a new + file, imported from `alerts/alerts.libsonnet`, with the collector added to + `_config.collectors` in `config.libsonnet`; the group expectations in + `tests/alerts.test.jsonnet` list every group. 2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet` fails when the map and the generated alerts differ. Thresholds are `_config` knobs named `Threshold[Unit]`. -3. A unit test in `tests/alerts.test.yaml` that checks the first firing - evaluation, the labels and annotations, and the recovery. With +3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for + opt-in collectors, which `tests/config.libsonnet` enables) that checks the + first firing evaluation, the labels and annotations, and the recovery. With `keep_firing_for` a threshold alert resolves `keepFiringFor` after its condition clears. `rate()` extrapolates while a window has few samples, so ratios of two rates activate earlier than a single rate; probe the diff --git a/contrib/mixin/README.md b/contrib/mixin/README.md index 273a6e0e..4a9064bf 100644 --- a/contrib/mixin/README.md +++ b/contrib/mixin/README.md @@ -151,10 +151,11 @@ pagefile traffic. ## Alerts -The alerts are grouped by collector area. Each group lists the collectors its -rules read; a group is generated only with those collectors enabled in -`_config.collectors`, and a rule that needs two collectors, such as the CPU -saturation alert, only with both. Threshold configuration values below are +The alerts come in one rule group per collector, from one file per collector +in `alerts/` (`alerts/cpu.libsonnet`, `alerts/service.libsonnet`, and so on). +Each group lists the collectors its rules read; a group is generated only with +those collectors enabled in `_config.collectors`, and a rule that needs two +collectors, such as the CPU saturation alert, only with both. Threshold configuration values below are percentages except where noted. | Group | Collectors | Alert | Default condition | Duration | Severity | @@ -185,9 +186,9 @@ percentages except where noted. | `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning | | `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning | | `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning | -| `windows-memory` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning | -| `windows-disk` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical | -| `windows-system` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning | +| `windows-pagefile` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning | +| `windows-diskdrive` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical | +| `windows-license` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning | | `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning | | `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info | | `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning | @@ -244,7 +245,7 @@ host to its chosen source and does not independently verify that source. The other opt-in alerts follow `_config.collectors`: enable `pagefile`, `diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`, `storage_spaces`, `dhcp` or `dns` and the alerts of that collector are -generated, in the group of their area or in a group of their own. +generated, each collector in a group of its own. `WindowsScheduledTaskFailed` treats `unknown`, the status of every result code without a name, including the exit codes of the task's program, as a failure; narrow the tasks with `taskSelector`. Disabling a task clears the alert. diff --git a/contrib/mixin/alerts/ad.libsonnet b/contrib/mixin/alerts/ad.libsonnet new file mode 100644 index 00000000..82bfce6c --- /dev/null +++ b/contrib/mixin/alerts/ad.libsonnet @@ -0,0 +1,32 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-active-directory', collectors=['ad'], rules=[ + { + alert: 'WindowsADPendingReplication', + expr: ||| + windows_ad_replication_pending_operations{%(windowsExporterSelector)s} > %(adPendingReplicationThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Active Directory replication queue is high', + description: 'Pending replication operations on {{ $labels.instance }} have exceeded %(adPendingReplicationThreshold)s for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsADReplicationFailures', + expr: ||| + (rate(windows_ad_replication_sync_requests_total{%(windowsExporterSelector)s}[%(rateInterval)s]) - rate(windows_ad_replication_sync_requests_success_total{%(windowsExporterSelector)s}[%(rateInterval)s])) > %(adReplicationFailureRateThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Active Directory replication requests are failing', + description: 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.', + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/alerts.libsonnet b/contrib/mixin/alerts/alerts.libsonnet index cc70a943..44d04292 100644 --- a/contrib/mixin/alerts/alerts.libsonnet +++ b/contrib/mixin/alerts/alerts.libsonnet @@ -1,6 +1,8 @@ +// One file per collector, each adding one rule group. To add alerts for a +// collector, copy one of these files, name it after the collector and import +// it here; the group order below is the order in the generated file. { local c = self._config, - local q = (import '../lib/queries.libsonnet')(c), local known = std.objectFields((import '../config.libsonnet')._config.alerts), local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)), assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown), @@ -8,541 +10,23 @@ 'networkErrorRateThreshold and networkDiscardRateThreshold (packets per second) were replaced by networkErrorRatioThreshold and networkDiscardRatioThreshold (percent of all packets)', assert !std.objectHas(c, 'enableActiveDirectory') && !std.objectHas(c, 'enableTime') : 'enableActiveDirectory and enableTime were replaced by collectors+: { ad: true } and collectors+: { time: true }', - // A collector missing from a replaced collectors object counts as disabled; - // an alert missing from a replaced alerts object counts as enabled. - local collectorEnabled(collector) = std.get(c.collectors, collector, false), - local alertEnabled(name) = std.get(c.alerts, name, true), - local keepFiring = if c.keepFiringFor != '' then { keep_firing_for: c.keepFiringFor } else {}, - // alert renders a rule when every collector it needs is enabled and the - // alert is not turned off. Threshold alerts set keepFiringFor so a value - // oscillating around the threshold does not flap; state alerts and - // one-shot alerts leave it out. - local alert(name, expr, duration, severity, summary, description, collectors=[], threshold=true) = - if std.all([collectorEnabled(collector) for collector in collectors]) && alertEnabled(name) then [ - { - alert: name, - expr: expr, - labels: { severity: severity }, - annotations: { summary: summary, description: description }, - } + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}), - ] else [], - local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [], - // Percent of free addresses in a scope; a scope without addresses has no sample. - local dhcpScopeFree = '(%s * 100)' % q.ratio( - q.metric('windows_dhcp_scope_addresses_free'), - '(%s + %s)' % [q.metric('windows_dhcp_scope_addresses_free'), q.metric('windows_dhcp_scope_addresses_in_use')] - ), - local dhcpScopeActive = q.metric('windows_dhcp_scope_state', ['state=~"Enabled|EnabledSwitched"']), - prometheusAlerts+:: { - groups: - group('windows-exporter', std.flattenArrays([ - alert( - 'WindowsExporterDown', - q.metric('up') + ' == 0', - '5m', - c.criticalSeverity, - 'Windows exporter is unreachable', - 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).', - threshold=false - ), - alert( - 'WindowsCollectorFailed', - q.metric('windows_exporter_collector_success') + ' == 0', - '5m', - 'warning', - 'Windows collector failed', - 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.', - threshold=false - ), - alert( - 'WindowsCollectorTimeout', - q.metric('windows_exporter_collector_timeout') + ' == 1', - '10m', - 'warning', - 'Windows collector exceeds the scrape timeout', - 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.', - threshold=false - ), - alert( - 'WindowsScrapeQueueTimeout', - q.metric('windows_exporter_scrape_queue_timeout') + ' == 1', - '10m', - 'warning', - 'Windows exporter scrapes time out in the queue', - 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.', - threshold=false - ), - ])) - + group('windows-cpu', std.flattenArrays([ - alert( - 'WindowsCPUHighUsage', - q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold, - '15m', - 'warning', - 'Windows CPU usage is high', - 'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold, - collectors=['cpu'] - ), - alert( - 'WindowsCPUSaturation', - q.metric('windows:cpu_queue_per_core:ratio') + ' > %s' % c.cpuQueuePerCoreThreshold, - '15m', - 'warning', - 'Windows processor queue is long', - 'More than %s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c.cpuQueuePerCoreThreshold, - collectors=['cpu', 'system'] - ), - ])) - + group('windows-memory', std.flattenArrays([ - alert( - 'WindowsMemoryHighUsage', - q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold, - '15m', - 'warning', - 'Windows memory usage is high', - 'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold, - collectors=['memory'] - ), - alert( - 'WindowsMemoryCommitHighUsage', - q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold, - '15m', - 'warning', - 'Windows memory commit usage is high', - 'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold, - collectors=['memory'] - ), - alert( - 'WindowsMemoryPagingHigh', - q.metric('windows:memory_swap_page_operations:rate') + ' > %s' % c.memoryPagingThreshold, - '15m', - 'warning', - 'Windows memory paging is high', - 'Paging on {{ $labels.instance }} has exceeded %s pages per second for 15 minutes.' % c.memoryPagingThreshold, - collectors=['memory'] - ), - alert( - 'WindowsPagefileAlmostFull', - '(1 - %s) * 100 > %s' % [ - q.ratio(q.metric('windows_pagefile_free_bytes'), q.metric('windows_pagefile_limit_bytes')), - c.pagefileUsageThreshold, - ], - '15m', - 'warning', - 'Windows page file is almost full', - 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %s%% in use for 15 minutes.' % c.pagefileUsageThreshold, - collectors=['pagefile'] - ), - ])) - + group('windows-disk', std.flattenArrays([ - alert( - 'WindowsDisk%sLatencyHigh' % direction[0], - q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds, - '15m', - 'warning', - 'Windows disk %s latency is high' % direction[1], - 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]], - collectors=['logical_disk'] - ) - for direction in [['Read', 'read'], ['Write', 'write']] - ] + [ - alert( - 'WindowsDiskAlmostFull', - q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1], - '15m', - entry[0], - 'Windows disk space is low', - 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1], - collectors=['logical_disk'] - ) - for entry in [ - ['warning', c.diskFreeWarningThreshold], - [c.criticalSeverity, c.diskFreeCriticalThreshold], - ] - ] + [ - alert( - 'WindowsDiskFillingUp', - q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [ - c.diskFreeWarningThreshold, - q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]), - c.diskPredictionWindow, - c.diskPredictionHours * 3600, - ], - '1h', - 'warning', - 'Windows disk is predicted to fill up', - 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours, - collectors=['logical_disk'] - ), - alert( - 'WindowsDiskDriveNotOK', - // One series per status with the current status set to 1, so the - // alert carries the status the drive reports. Some USB and virtual - // drives report Unknown for good. - q.metric('windows_diskdrive_status', ['status!~"OK|Unknown"']) + ' == 1', - '5m', - c.criticalSeverity, - 'Windows disk drive is not healthy', - 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.', - collectors=['diskdrive'], - threshold=false - ), - ])) - + group('windows-network', std.flattenArrays([ - alert( - 'WindowsNetwork%s' % outcome[0], - q.ratio( - '(%s + %s)' % [ - q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]), - q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]), - ], - '(%s + %s)' % [ - q.metric('windows:net_received_packets:rate', [c.nicSelector]), - q.metric('windows:net_sent_packets:rate', [c.nicSelector]), - ] - ) + ' * 100 > %g' % outcome[2], - '15m', - 'warning', - 'Windows network packet %s are high' % outcome[3], - 'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g%% of all packets for 15 minutes.' % [outcome[3], outcome[2]], - collectors=['net'] - ) - for outcome in [ - ['Errors', 'errors', c.networkErrorRatioThreshold, 'errors'], - ['Discards', 'discarded', c.networkDiscardRatioThreshold, 'discards'], - ] - ] + [ - alert( - 'WindowsNetworkUtilizationHigh', - '(%s * 100 > %s) or (%s * 100 > %s)' % [ - q.metric('windows:net_received_utilization:ratio', [c.nicSelector]), - c.networkUtilizationThreshold, - q.metric('windows:net_sent_utilization:ratio', [c.nicSelector]), - c.networkUtilizationThreshold, - ], - '15m', - 'warning', - 'Windows network interface utilization is high', - 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %s%% of its bandwidth in at least one direction for 15 minutes.' % c.networkUtilizationThreshold, - collectors=['net'] - ), - alert( - 'WindowsNetworkInterfaceFlapping', - 'max without (status) (changes(%s[%s])) > 2' % [ - q.metric('windows_net_nic_operation_status', [c.nicSelector, 'status="up"']), - c.rateInterval, - ], - '2m', - 'warning', - 'Windows network interface is flapping', - 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %s.' % c.rateInterval, - collectors=['net'] - ), - ])) - + group('windows-service', std.flattenArrays([ - alert( - 'WindowsServiceNotRunning', - // The matches ignore only the labels of the right-hand metric, so - // every target label, including ones a selector adds, takes part. - std.join(' ', [ - 'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']), - 'and ignoring (start_mode) (%s == 1)' % q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"']), - 'unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (%s == 1)' % q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"']), - ]), - '15m', - 'warning', - 'Windows automatic service is not running', - 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.', - collectors=['service'], - threshold=false - ), - alert( - 'WindowsServiceCrashLooping', - // Every service process has its own series, so the series seen in the - // window count the processes the service ran as. - 'count without (process_id) (count_over_time(%s[15m])) - 1 > %s' % [ - q.metric('windows_service_process', [c.serviceSelector]), - c.serviceRestartThreshold, - ], - '0m', - 'warning', - 'Windows service is restarting repeatedly', - 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %s times in 15 minutes.' % c.serviceRestartThreshold, - collectors=['service'] - ), - ])) - + group('windows-system', std.flattenArrays([ - alert( - 'WindowsHostRebooted', - // Fires once per reboot: for the first ten minutes after the boot, - // and only on a host that was monitored before it. The exporter is - // down while the host reboots, so the second operand takes the last - // boot time of the hour before, not the sample of ten minutes ago. - '(time() - %s) < 600 and (time() - last_over_time(%s[1h] offset 10m)) > 600' % [ - q.metric('windows_system_boot_time_timestamp'), - q.metric('windows_system_boot_time_timestamp'), - ], - '0m', - c.infoSeverity, - 'Windows host has rebooted', - '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.', - collectors=['system'], - threshold=false - ), - alert( - 'WindowsLicenseNotGenuine', - q.metric('windows_license_status', ['state!="genuine"']) + ' == 1', - '1h', - 'warning', - 'Windows license is not genuine', - 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.', - collectors=['license'], - threshold=false - ), - ])) - + group('windows-active-directory', std.flattenArrays([ - alert( - 'WindowsADPendingReplication', - q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold, - '15m', - 'warning', - 'Active Directory replication queue is high', - 'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold, - collectors=['ad'] - ), - alert( - 'WindowsADReplicationFailures', - '(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold, - '15m', - 'warning', - 'Active Directory replication requests are failing', - 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.', - collectors=['ad'] - ), - ])) - + group('windows-time', std.flattenArrays([ - alert( - 'WindowsClockNotSynchronising', - q.metric('windows_time_ntp_client_time_sources') + ' == 0', - '10m', - 'warning', - 'Windows clock has no NTP time source', - 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.', - collectors=['time'], - threshold=false - ), - alert( - 'WindowsClockSkewDetected', - 'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds], - '10m', - 'warning', - 'Windows clock offset is high', - 'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds, - collectors=['time'] - ), - alert( - 'WindowsClockSourceLocal', - // Covers domain members (NT5DS) too, which the NTP source count misses. - q.metric('windows_time_clock_sync_source', ['type=~"Local CMOS Clock|NoSync"']) + ' == 1', - '10m', - 'warning', - 'Windows clock has no external time source', - 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.', - collectors=['time'], - threshold=false - ), - ])) - + group('windows-scheduled-task', std.flattenArrays([ - alert( - 'WindowsScheduledTaskFailed', - // unknown is any result code without a named status, including the - // exit codes of the task's program. - // A disabled task keeps its last result, so disabling a broken task - // clears the alert. - '%s == 1 unless ignoring (status, state) (%s == 1)' % [ - q.metric('windows_scheduled_task_last_result_status', [c.taskSelector, 'status=~"unknown|terminated"']), - q.metric('windows_scheduled_task_state', [c.taskSelector, 'state="disabled"']), - ], - '1h', - 'warning', - 'Windows scheduled task did not succeed', - 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.', - collectors=['scheduled_task'], - threshold=false - ), - ])) - + group('windows-update', std.flattenArrays([ - alert( - 'WindowsUpdatesPending', - 'count without (id, revision, category, severity, title) (%s) > 0' % q.metric('windows_update_pending_info', ['deployment_action="installation"']), - c.updatePendingDuration, - c.infoSeverity, - 'Windows updates are waiting for installation', - '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %s.' % c.updatePendingDuration, - collectors=['update'], - threshold=false - ), - ])) - + group('windows-hyperv', std.flattenArrays([ - alert( - 'WindowsHyperVWMIUnhealthy', - q.metric('windows_hyperv_wmi_health') + ' == 0', - '10m', - 'warning', - 'Hyper-V WMI namespace is not responding', - 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.', - collectors=['hyperv'], - threshold=false - ), - alert( - 'WindowsHyperVReplicationCritical', - q.metric('windows_hyperv_replica_vm_health') + ' == 3', - '5m', - c.criticalSeverity, - 'Hyper-V replication is critical', - 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.', - collectors=['hyperv'], - threshold=false - ), - alert( - 'WindowsHyperVReplicationWarning', - q.metric('windows_hyperv_replica_vm_health') + ' == 2', - '15m', - 'warning', - 'Hyper-V replication has a warning', - 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.', - collectors=['hyperv'], - threshold=false - ), - ])) - // Every cluster node reports the state of the whole cluster, so these - // alerts fire once per node; group them by cluster labels in Alertmanager. - + group('windows-cluster', std.flattenArrays([ - alert( - 'WindowsClusterNodeDown', - q.metric('windows_mscluster_node_state') + ' == 1', - '5m', - c.criticalSeverity, - 'Failover cluster node is down', - 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.', - collectors=['mscluster'], - threshold=false - ), - alert( - 'WindowsClusterResourceFailed', - q.metric('windows_mscluster_resource_state') + ' == 4', - '5m', - c.criticalSeverity, - 'Failover cluster resource has failed', - 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', - collectors=['mscluster'], - threshold=false - ), - alert( - 'WindowsClusterResourceGroupFailed', - q.metric('windows_mscluster_resourcegroup_state') + ' == 2', - '5m', - c.criticalSeverity, - 'Failover cluster resource group has failed', - 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', - collectors=['mscluster'], - threshold=false - ), - alert( - 'WindowsClusterSharedVolumeAlmostFull', - q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeWarningThreshold, - '15m', - 'warning', - 'Cluster shared volume is almost full', - 'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeWarningThreshold, - collectors=['mscluster'] - ), - alert( - 'WindowsClusterSharedVolumeAlmostFull', - q.ratio(q.metric('windows_mscluster_shared_volumes_free_bytes', [c.sharedVolumeSelector]), q.metric('windows_mscluster_shared_volumes_total_bytes', [c.sharedVolumeSelector])) + ' * 100 < %s' % c.diskFreeCriticalThreshold, - '15m', - c.criticalSeverity, - 'Cluster shared volume is almost full', - 'Cluster shared volume {{ $labels.shared_volume }} has had less than %s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c.diskFreeCriticalThreshold, - collectors=['mscluster'] - ), - ])) - // On Storage Spaces Direct every node reports the clustered pools and - // virtual disks, so these alerts fire once per node as well. - + group('windows-storage-spaces', std.flattenArrays([ - // The primordial pool holds the disks that are not in a concrete pool - // yet; only the info metric tells it apart. - alert( - 'WindowsStoragePoolUnhealthy', - '(%s == 2) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])], - '10m', - c.criticalSeverity, - 'Storage pool is unhealthy', - 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', - collectors=['storage_spaces'], - threshold=false - ), - alert( - 'WindowsStoragePoolUnhealthy', - '(%s == 1) and ignoring (primordial) %s' % [q.metric('windows_storage_spaces_pool_health_status'), q.metric('windows_storage_spaces_pool_info', ['primordial="false"'])], - '10m', - 'warning', - 'Storage pool has a health warning', - 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.', - collectors=['storage_spaces'], - threshold=false - ), - alert( - 'WindowsStorageVirtualDiskUnhealthy', - q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 2', - '10m', - c.criticalSeverity, - 'Storage Spaces virtual disk is unhealthy', - 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', - collectors=['storage_spaces'], - threshold=false - ), - alert( - 'WindowsStorageVirtualDiskUnhealthy', - q.metric('windows_storage_spaces_virtual_disk_health_status') + ' == 1', - '10m', - 'warning', - 'Storage Spaces virtual disk has a health warning', - 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.', - collectors=['storage_spaces'], - threshold=false - ), - ])) - + group('windows-dhcp', std.flattenArrays([ - alert( - 'WindowsDHCPScopeAlmostExhausted', - dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeWarningThreshold, dhcpScopeActive], - '15m', - 'warning', - 'DHCP scope is almost exhausted', - 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeWarningThreshold, - collectors=['dhcp'] - ), - alert( - 'WindowsDHCPScopeAlmostExhausted', - dhcpScopeFree + ' < %s and ignoring (state) (%s == 1)' % [c.dhcpScopeFreeCriticalThreshold, dhcpScopeActive], - '15m', - c.criticalSeverity, - 'DHCP scope is almost exhausted', - 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %s%% free addresses for 15 minutes.' % c.dhcpScopeFreeCriticalThreshold, - collectors=['dhcp'] - ), - ])) - + group('windows-dns', std.flattenArrays([ - alert( - 'WindowsDNSZoneTransferFailures', - q.rate('windows_dns_zone_transfer_failures_total') + ' > %g' % c.dnsZoneTransferFailureRateThreshold, - '15m', - 'warning', - 'DNS zone transfers are failing', - 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.', - collectors=['dns'] - ), - ])), - }, } ++ (import 'exporter.libsonnet') ++ (import 'cpu.libsonnet') ++ (import 'memory.libsonnet') ++ (import 'pagefile.libsonnet') ++ (import 'logical_disk.libsonnet') ++ (import 'diskdrive.libsonnet') ++ (import 'net.libsonnet') ++ (import 'service.libsonnet') ++ (import 'system.libsonnet') ++ (import 'license.libsonnet') ++ (import 'ad.libsonnet') ++ (import 'time.libsonnet') ++ (import 'scheduled_task.libsonnet') ++ (import 'update.libsonnet') ++ (import 'hyperv.libsonnet') ++ (import 'mscluster.libsonnet') ++ (import 'storage_spaces.libsonnet') ++ (import 'dhcp.libsonnet') ++ (import 'dns.libsonnet') diff --git a/contrib/mixin/alerts/cpu.libsonnet b/contrib/mixin/alerts/cpu.libsonnet new file mode 100644 index 00000000..35c5887e --- /dev/null +++ b/contrib/mixin/alerts/cpu.libsonnet @@ -0,0 +1,33 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-cpu', collectors=['cpu'], rules=[ + { + alert: 'WindowsCPUHighUsage', + expr: ||| + windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows CPU usage is high', + description: 'CPU usage on {{ $labels.instance }} has exceeded %(cpuHighUsageThreshold)s%% for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsCPUSaturation', + expr: ||| + windows:cpu_queue_per_core:ratio{%(windowsExporterSelector)s} > %(cpuQueuePerCoreThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows processor queue is long', + description: 'More than %(cpuQueuePerCoreThreshold)s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c, + }, + collectors:: ['cpu', 'system'], + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/dhcp.libsonnet b/contrib/mixin/alerts/dhcp.libsonnet new file mode 100644 index 00000000..508534a0 --- /dev/null +++ b/contrib/mixin/alerts/dhcp.libsonnet @@ -0,0 +1,36 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-dhcp', collectors=['dhcp'], rules=[ + // Percent of free addresses in a scope; a scope without addresses has + // no sample, and only scopes in an enabled state count. + { + alert: 'WindowsDHCPScopeAlmostExhausted', + expr: ||| + ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeWarningThreshold)s + and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'DHCP scope is almost exhausted', + description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeWarningThreshold)s%% free addresses for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDHCPScopeAlmostExhausted', + expr: ||| + ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeCriticalThreshold)s + and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'DHCP scope is almost exhausted', + description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeCriticalThreshold)s%% free addresses for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/diskdrive.libsonnet b/contrib/mixin/alerts/diskdrive.libsonnet new file mode 100644 index 00000000..89f21179 --- /dev/null +++ b/contrib/mixin/alerts/diskdrive.libsonnet @@ -0,0 +1,24 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-diskdrive', collectors=['diskdrive'], rules=[ + { + alert: 'WindowsDiskDriveNotOK', + // One series per status with the current status set to 1, so the + // alert carries the status the drive reports. Some USB and virtual + // drives report Unknown for good. + expr: ||| + windows_diskdrive_status{%(windowsExporterSelector)s, status!~"OK|Unknown"} == 1 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows disk drive is not healthy', + description: 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/dns.libsonnet b/contrib/mixin/alerts/dns.libsonnet new file mode 100644 index 00000000..4c9abdf2 --- /dev/null +++ b/contrib/mixin/alerts/dns.libsonnet @@ -0,0 +1,20 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-dns', collectors=['dns'], rules=[ + { + alert: 'WindowsDNSZoneTransferFailures', + expr: ||| + rate(windows_dns_zone_transfer_failures_total{%(windowsExporterSelector)s}[%(rateInterval)s]) > %(dnsZoneTransferFailureRateThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'DNS zone transfers are failing', + description: 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.', + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/exporter.libsonnet b/contrib/mixin/alerts/exporter.libsonnet new file mode 100644 index 00000000..61924c74 --- /dev/null +++ b/contrib/mixin/alerts/exporter.libsonnet @@ -0,0 +1,60 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-exporter', collectors=[], rules=[ + { + alert: 'WindowsExporterDown', + expr: ||| + up{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows exporter is unreachable', + description: 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).', + }, + state:: true, + }, + { + alert: 'WindowsCollectorFailed', + expr: ||| + windows_exporter_collector_success{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '5m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows collector failed', + description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsCollectorTimeout', + expr: ||| + windows_exporter_collector_timeout{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows collector exceeds the scrape timeout', + description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsScrapeQueueTimeout', + expr: ||| + windows_exporter_scrape_queue_timeout{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows exporter scrapes time out in the queue', + description: 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/hyperv.libsonnet b/contrib/mixin/alerts/hyperv.libsonnet new file mode 100644 index 00000000..81559c00 --- /dev/null +++ b/contrib/mixin/alerts/hyperv.libsonnet @@ -0,0 +1,47 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-hyperv', collectors=['hyperv'], rules=[ + { + alert: 'WindowsHyperVWMIUnhealthy', + expr: ||| + windows_hyperv_wmi_health{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Hyper-V WMI namespace is not responding', + description: 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.', + }, + state:: true, + }, + { + alert: 'WindowsHyperVReplicationCritical', + expr: ||| + windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 3 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Hyper-V replication is critical', + description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsHyperVReplicationWarning', + expr: ||| + windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Hyper-V replication has a warning', + description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/license.libsonnet b/contrib/mixin/alerts/license.libsonnet new file mode 100644 index 00000000..6874dca6 --- /dev/null +++ b/contrib/mixin/alerts/license.libsonnet @@ -0,0 +1,21 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-license', collectors=['license'], rules=[ + { + alert: 'WindowsLicenseNotGenuine', + expr: ||| + windows_license_status{%(windowsExporterSelector)s, state!="genuine"} == 1 + ||| % c, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows license is not genuine', + description: 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/logical_disk.libsonnet b/contrib/mixin/alerts/logical_disk.libsonnet new file mode 100644 index 00000000..394108ee --- /dev/null +++ b/contrib/mixin/alerts/logical_disk.libsonnet @@ -0,0 +1,69 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-disk', collectors=['logical_disk'], rules=[ + { + alert: 'WindowsDiskReadLatencyHigh', + expr: ||| + windows:logical_disk_read_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk read latency is high', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per read operation for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskWriteLatencyHigh', + expr: ||| + windows:logical_disk_write_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk write latency is high', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per write operation for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskAlmostFull', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk space is low', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeWarningThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskAlmostFull', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeCriticalThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Windows disk space is low', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeCriticalThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c, + }, + }, + { + alert: 'WindowsDiskFillingUp', + expr: ||| + windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s + and predict_linear(windows_logical_disk_free_bytes{%(windowsExporterSelector)s, %(volumeSelector)s}[%(diskPredictionWindow)s], %(diskPredictionSeconds)s) < 0 + ||| % c { diskPredictionSeconds: c.diskPredictionHours * 3600 }, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows disk is predicted to fill up', + description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %(diskPredictionHours)s hours.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/memory.libsonnet b/contrib/mixin/alerts/memory.libsonnet new file mode 100644 index 00000000..80512211 --- /dev/null +++ b/contrib/mixin/alerts/memory.libsonnet @@ -0,0 +1,44 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-memory', collectors=['memory'], rules=[ + { + alert: 'WindowsMemoryHighUsage', + expr: ||| + windows:memory_usage:ratio{%(windowsExporterSelector)s} * 100 > %(memoryHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory usage is high', + description: 'Physical memory usage on {{ $labels.instance }} has exceeded %(memoryHighUsageThreshold)s%% for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsMemoryCommitHighUsage', + expr: ||| + windows:memory_committed:ratio{%(windowsExporterSelector)s} * 100 > %(memoryCommitHighUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory commit usage is high', + description: 'Committed memory on {{ $labels.instance }} has exceeded %(memoryCommitHighUsageThreshold)s%% of the commit limit for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsMemoryPagingHigh', + expr: ||| + windows:memory_swap_page_operations:rate{%(windowsExporterSelector)s} > %(memoryPagingThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows memory paging is high', + description: 'Paging on {{ $labels.instance }} has exceeded %(memoryPagingThreshold)s pages per second for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/mscluster.libsonnet b/contrib/mixin/alerts/mscluster.libsonnet new file mode 100644 index 00000000..64b72f8c --- /dev/null +++ b/contrib/mixin/alerts/mscluster.libsonnet @@ -0,0 +1,73 @@ +// Every cluster node reports the state of the whole cluster, so these +// alerts fire once per node; group them by cluster labels in Alertmanager. +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-cluster', collectors=['mscluster'], rules=[ + { + alert: 'WindowsClusterNodeDown', + expr: ||| + windows_mscluster_node_state{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster node is down', + description: 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterResourceFailed', + expr: ||| + windows_mscluster_resource_state{%(windowsExporterSelector)s} == 4 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster resource has failed', + description: 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterResourceGroupFailed', + expr: ||| + windows_mscluster_resourcegroup_state{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '5m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Failover cluster resource group has failed', + description: 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.', + }, + state:: true, + }, + { + alert: 'WindowsClusterSharedVolumeAlmostFull', + expr: ||| + (windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeWarningThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Cluster shared volume is almost full', + description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeWarningThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c, + }, + }, + { + alert: 'WindowsClusterSharedVolumeAlmostFull', + expr: ||| + (windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeCriticalThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Cluster shared volume is almost full', + description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeCriticalThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/net.libsonnet b/contrib/mixin/alerts/net.libsonnet new file mode 100644 index 00000000..798fe27f --- /dev/null +++ b/contrib/mixin/alerts/net.libsonnet @@ -0,0 +1,63 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-network', collectors=['net'], rules=[ + { + alert: 'WindowsNetworkErrors', + expr: ||| + ( + (windows:net_received_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s}) + / ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0) + ) * 100 > %(networkErrorRatioThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network packet errors are high', + description: 'Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkErrorRatioThreshold)g%% of all packets for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkDiscards', + expr: ||| + ( + (windows:net_received_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s}) + / ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0) + ) * 100 > %(networkDiscardRatioThreshold)g + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network packet discards are high', + description: 'Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkDiscardRatioThreshold)g%% of all packets for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkUtilizationHigh', + expr: ||| + (windows:net_received_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s) + or (windows:net_sent_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network interface utilization is high', + description: 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %(networkUtilizationThreshold)s%% of its bandwidth in at least one direction for 15 minutes.' % c, + }, + }, + { + alert: 'WindowsNetworkInterfaceFlapping', + expr: ||| + max without (status) (changes(windows_net_nic_operation_status{%(windowsExporterSelector)s, %(nicSelector)s, status="up"}[%(rateInterval)s])) > 2 + ||| % c, + 'for': '2m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows network interface is flapping', + description: 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %(rateInterval)s.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/pagefile.libsonnet b/contrib/mixin/alerts/pagefile.libsonnet new file mode 100644 index 00000000..b751a165 --- /dev/null +++ b/contrib/mixin/alerts/pagefile.libsonnet @@ -0,0 +1,20 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-pagefile', collectors=['pagefile'], rules=[ + { + alert: 'WindowsPagefileAlmostFull', + expr: ||| + (1 - (windows_pagefile_free_bytes{%(windowsExporterSelector)s} / (windows_pagefile_limit_bytes{%(windowsExporterSelector)s} > 0))) * 100 > %(pagefileUsageThreshold)s + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows page file is almost full', + description: 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %(pagefileUsageThreshold)s%% in use for 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/scheduled_task.libsonnet b/contrib/mixin/alerts/scheduled_task.libsonnet new file mode 100644 index 00000000..9699c941 --- /dev/null +++ b/contrib/mixin/alerts/scheduled_task.libsonnet @@ -0,0 +1,26 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-scheduled-task', collectors=['scheduled_task'], rules=[ + { + alert: 'WindowsScheduledTaskFailed', + // unknown is any result code without a named status, including the + // exit codes of the task's program. + // A disabled task keeps its last result, so disabling a broken task + // clears the alert. + expr: ||| + windows_scheduled_task_last_result_status{%(windowsExporterSelector)s, %(taskSelector)s, status=~"unknown|terminated"} == 1 + unless ignoring (status, state) (windows_scheduled_task_state{%(windowsExporterSelector)s, %(taskSelector)s, state="disabled"} == 1) + ||| % c, + 'for': '1h', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows scheduled task did not succeed', + description: 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/service.libsonnet b/contrib/mixin/alerts/service.libsonnet new file mode 100644 index 00000000..49d99bbc --- /dev/null +++ b/contrib/mixin/alerts/service.libsonnet @@ -0,0 +1,38 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-service', collectors=['service'], rules=[ + { + alert: 'WindowsServiceNotRunning', + // The matches ignore only the labels of the right-hand metric, so + // every target label, including ones a selector adds, takes part. + expr: ||| + max without (state) (windows_service_state{%(windowsExporterSelector)s, %(serviceSelector)s, state="running"} == 0) + and ignoring (start_mode) (windows_service_start_mode{%(windowsExporterSelector)s, %(serviceSelector)s, start_mode="auto"} == 1) + unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{%(windowsExporterSelector)s, %(serviceSelector)s, trigger_start="true"} == 1) + ||| % c, + 'for': '15m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows automatic service is not running', + description: 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsServiceCrashLooping', + // Every service process has its own series, so the series seen in the + // window count the processes the service ran as. + expr: ||| + count without (process_id) (count_over_time(windows_service_process{%(windowsExporterSelector)s, %(serviceSelector)s}[15m])) - 1 > %(serviceRestartThreshold)s + ||| % c, + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows service is restarting repeatedly', + description: 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %(serviceRestartThreshold)s times in 15 minutes.' % c, + }, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/storage_spaces.libsonnet b/contrib/mixin/alerts/storage_spaces.libsonnet new file mode 100644 index 00000000..b3d9dc01 --- /dev/null +++ b/contrib/mixin/alerts/storage_spaces.libsonnet @@ -0,0 +1,66 @@ +// On Storage Spaces Direct every node reports the clustered pools and +// virtual disks, so these alerts fire once per node as well. +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-storage-spaces', collectors=['storage_spaces'], rules=[ + { + alert: 'WindowsStoragePoolUnhealthy', + // The primordial pool holds the disks that are not in a concrete pool + // yet; only the info metric tells it apart. + expr: ||| + (windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 2) + and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"} + ||| % c, + 'for': '10m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Storage pool is unhealthy', + description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStoragePoolUnhealthy', + expr: ||| + (windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 1) + and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"} + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Storage pool has a health warning', + description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStorageVirtualDiskUnhealthy', + expr: ||| + windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 2 + ||| % c, + 'for': '10m', + labels: { severity: c.criticalSeverity }, + annotations: { + summary: 'Storage Spaces virtual disk is unhealthy', + description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsStorageVirtualDiskUnhealthy', + expr: ||| + windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Storage Spaces virtual disk has a health warning', + description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/system.libsonnet b/contrib/mixin/alerts/system.libsonnet new file mode 100644 index 00000000..5c8843c1 --- /dev/null +++ b/contrib/mixin/alerts/system.libsonnet @@ -0,0 +1,25 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-system', collectors=['system'], rules=[ + { + alert: 'WindowsHostRebooted', + // Fires once per reboot: for the first ten minutes after the boot, + // and only on a host that was monitored before it. The exporter is + // down while the host reboots, so the second operand takes the last + // boot time of the hour before, not the sample of ten minutes ago. + expr: ||| + (time() - windows_system_boot_time_timestamp{%(windowsExporterSelector)s}) < 600 + and (time() - last_over_time(windows_system_boot_time_timestamp{%(windowsExporterSelector)s}[1h] offset 10m)) > 600 + ||| % c, + labels: { severity: c.infoSeverity }, + annotations: { + summary: 'Windows host has rebooted', + description: '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/time.libsonnet b/contrib/mixin/alerts/time.libsonnet new file mode 100644 index 00000000..8955f8b6 --- /dev/null +++ b/contrib/mixin/alerts/time.libsonnet @@ -0,0 +1,47 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-time', collectors=['time'], rules=[ + { + alert: 'WindowsClockNotSynchronising', + expr: ||| + windows_time_ntp_client_time_sources{%(windowsExporterSelector)s} == 0 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock has no NTP time source', + description: 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.', + }, + state:: true, + }, + { + alert: 'WindowsClockSkewDetected', + expr: ||| + abs(windows_time_computed_time_offset_seconds{%(windowsExporterSelector)s}) > %(clockOffsetThresholdSeconds)g + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock offset is high', + description: 'Windows Time on {{ $labels.instance }} reports a clock offset above %(clockOffsetThresholdSeconds)g seconds for 10 minutes.' % c, + }, + }, + { + alert: 'WindowsClockSourceLocal', + // Covers domain members (NT5DS) too, which the NTP source count misses. + expr: ||| + windows_time_clock_sync_source{%(windowsExporterSelector)s, type=~"Local CMOS Clock|NoSync"} == 1 + ||| % c, + 'for': '10m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Windows clock has no external time source', + description: 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.', + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/alerts/update.libsonnet b/contrib/mixin/alerts/update.libsonnet new file mode 100644 index 00000000..7eff2968 --- /dev/null +++ b/contrib/mixin/alerts/update.libsonnet @@ -0,0 +1,21 @@ +{ + local c = self._config, + local a = (import '../lib/alerts.libsonnet')(c), + prometheusAlerts+:: { + groups+: a.group('windows-update', collectors=['update'], rules=[ + { + alert: 'WindowsUpdatesPending', + expr: ||| + count without (id, revision, category, severity, title) (windows_update_pending_info{%(windowsExporterSelector)s, deployment_action="installation"}) > 0 + ||| % c, + 'for': c.updatePendingDuration, + labels: { severity: c.infoSeverity }, + annotations: { + summary: 'Windows updates are waiting for installation', + description: '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %(updatePendingDuration)s.' % c, + }, + state:: true, + }, + ]), + }, +} diff --git a/contrib/mixin/lib/alerts.libsonnet b/contrib/mixin/lib/alerts.libsonnet new file mode 100644 index 00000000..8f0868ea --- /dev/null +++ b/contrib/mixin/lib/alerts.libsonnet @@ -0,0 +1,23 @@ +// Helpers shared by the alert files in alerts/. Takes the mixin _config. +function(c) + { + // A collector missing from a replaced collectors object counts as disabled; + // an alert missing from a replaced alerts object counts as enabled. + collectorEnabled(collector): std.get(c.collectors, collector, false), + alertEnabled(name): std.get(c.alerts, name, true), + // group renders the rules of one collector as a rule group, or nothing + // when none of them is enabled. Each rule is a plain Prometheus alerting + // rule plus two hidden fields: collectors:: lists the collectors the rule + // reads when it differs from the group's, and state:: true marks state + // alerts (== 0, == 1) and one-shot alerts, which must resolve at once. + // Every other alert is a threshold alert and keeps firing for + // keepFiringFor after its condition clears, so a value oscillating around + // the threshold does not flap. + group(name, collectors, rules): + local enabled = [ + rule + (if c.keepFiringFor != '' && !std.get(rule, 'state', false) then { keep_firing_for: c.keepFiringFor } else {}) + for rule in rules + if self.alertEnabled(rule.alert) && std.all([self.collectorEnabled(collector) for collector in std.get(rule, 'collectors', collectors)]) + ]; + if std.length(enabled) > 0 then [{ name: name, rules: enabled }] else [], + } diff --git a/contrib/mixin/tests/alerts.test.jsonnet b/contrib/mixin/tests/alerts.test.jsonnet index 6ba30c13..513f6f84 100644 --- a/contrib/mixin/tests/alerts.test.jsonnet +++ b/contrib/mixin/tests/alerts.test.jsonnet @@ -3,7 +3,7 @@ local mixin = import '../mixin.libsonnet'; local config = (import '../config.libsonnet')._config; local everything = mixin { _config+:: { collectors: { [name]: true for name in std.objectFields(config.collectors) } } }; -local optionalGroups = ['windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns']; +local optionalGroups = ['windows-pagefile', 'windows-diskdrive', 'windows-license', 'windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns']; local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]); local groupNames(alerts) = [group.name for group in alerts.groups]; @@ -13,12 +13,13 @@ assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs fro // Groups follow the collector areas; opt-in groups appear only when enabled. assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts)); -assert groupNames(everything.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + optionalGroups : std.manifestJson(groupNames(everything.prometheusAlerts)); -// Opt-in collectors add their alerts to the default groups of their area too. +// One file per collector, one group per file: the groups of every enabled +// collector come in the order of the imports in alerts/alerts.libsonnet. +assert std.set(groupNames(everything.prometheusAlerts)) == std.set(groupNames(mixin.prometheusAlerts) + optionalGroups) : std.manifestJson(groupNames(everything.prometheusAlerts)); +assert groupNames(everything.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-pagefile', 'windows-disk', 'windows-diskdrive', 'windows-network', 'windows-service', 'windows-system', 'windows-license'] + optionalGroups[3:] : std.manifestJson(groupNames(everything.prometheusAlerts)); local adOnly = mixin { _config+:: { collectors+: { ad: true } } }; assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory']; local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } }; -assert groupNames(pagefile.prometheusAlerts) == groupNames(mixin.prometheusAlerts); assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull']; // Disabling a collector removes its alerts and an empty group; the alerts of a diff --git a/contrib/mixin/windows_alerts.yaml b/contrib/mixin/windows_alerts.yaml index 21e37edc..e621e892 100644 --- a/contrib/mixin/windows_alerts.yaml +++ b/contrib/mixin/windows_alerts.yaml @@ -6,7 +6,8 @@ groups: description: "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/" summary: "Windows exporter is unreachable" - expr: "up{job=\"windows_exporter\"} == 0" + expr: | + up{job="windows_exporter"} == 0 for: "5m" labels: severity: "critical" @@ -15,7 +16,8 @@ groups: description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/" summary: "Windows collector failed" - expr: "windows_exporter_collector_success{job=\"windows_exporter\"} == 0" + expr: | + windows_exporter_collector_success{job="windows_exporter"} == 0 for: "5m" labels: severity: "warning" @@ -24,7 +26,8 @@ groups: description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/" summary: "Windows collector exceeds the scrape timeout" - expr: "windows_exporter_collector_timeout{job=\"windows_exporter\"} == 1" + expr: | + windows_exporter_collector_timeout{job="windows_exporter"} == 1 for: "10m" labels: severity: "warning" @@ -33,7 +36,8 @@ groups: description: "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/" summary: "Windows exporter scrapes time out in the queue" - expr: "windows_exporter_scrape_queue_timeout{job=\"windows_exporter\"} == 1" + expr: | + windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1 for: "10m" labels: severity: "warning" @@ -44,7 +48,8 @@ groups: description: "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/" summary: "Windows CPU usage is high" - expr: "windows:cpu_usage:ratio{job=\"windows_exporter\"} * 100 > 90" + expr: | + windows:cpu_usage:ratio{job="windows_exporter"} * 100 > 90 for: "15m" keep_firing_for: "5m" labels: @@ -54,7 +59,8 @@ groups: description: "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/" summary: "Windows processor queue is long" - expr: "windows:cpu_queue_per_core:ratio{job=\"windows_exporter\"} > 2" + expr: | + windows:cpu_queue_per_core:ratio{job="windows_exporter"} > 2 for: "15m" keep_firing_for: "5m" labels: @@ -66,7 +72,8 @@ groups: description: "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/" summary: "Windows memory usage is high" - expr: "windows:memory_usage:ratio{job=\"windows_exporter\"} * 100 > 90" + expr: | + windows:memory_usage:ratio{job="windows_exporter"} * 100 > 90 for: "15m" keep_firing_for: "5m" labels: @@ -76,7 +83,8 @@ groups: description: "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/" summary: "Windows memory commit usage is high" - expr: "windows:memory_committed:ratio{job=\"windows_exporter\"} * 100 > 90" + expr: | + windows:memory_committed:ratio{job="windows_exporter"} * 100 > 90 for: "15m" keep_firing_for: "5m" labels: @@ -86,7 +94,8 @@ groups: description: "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/" summary: "Windows memory paging is high" - expr: "windows:memory_swap_page_operations:rate{job=\"windows_exporter\"} > 1000" + expr: | + windows:memory_swap_page_operations:rate{job="windows_exporter"} > 1000 for: "15m" keep_firing_for: "5m" labels: @@ -98,7 +107,8 @@ groups: description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/" summary: "Windows disk read latency is high" - expr: "windows:logical_disk_read_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05" + expr: | + windows:logical_disk_read_latency:seconds{job="windows_exporter", volume!=""} > 0.05 for: "15m" keep_firing_for: "5m" labels: @@ -108,7 +118,8 @@ groups: description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/" summary: "Windows disk write latency is high" - expr: "windows:logical_disk_write_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05" + expr: | + windows:logical_disk_write_latency:seconds{job="windows_exporter", volume!=""} > 0.05 for: "15m" keep_firing_for: "5m" labels: @@ -118,7 +129,8 @@ groups: description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/" summary: "Windows disk space is low" - expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10 for: "15m" keep_firing_for: "5m" labels: @@ -128,7 +140,8 @@ groups: description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/" summary: "Windows disk space is low" - expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 5" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 5 for: "15m" keep_firing_for: "5m" labels: @@ -138,7 +151,9 @@ groups: description: "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/" summary: "Windows disk is predicted to fill up" - expr: "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10 and predict_linear(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}[6h], 86400) < 0" + expr: | + windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10 + and predict_linear(windows_logical_disk_free_bytes{job="windows_exporter", volume!=""}[6h], 86400) < 0 for: "1h" keep_firing_for: "5m" labels: @@ -150,7 +165,11 @@ groups: description: "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/" summary: "Windows network packet errors are high" - expr: "((windows:net_received_errors:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_errors:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1" + expr: | + ( + (windows:net_received_errors:rate{job="windows_exporter", nic!=""} + windows:net_outbound_errors:rate{job="windows_exporter", nic!=""}) + / ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0) + ) * 100 > 1 for: "15m" keep_firing_for: "5m" labels: @@ -160,7 +179,11 @@ groups: description: "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/" summary: "Windows network packet discards are high" - expr: "((windows:net_received_discarded:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_discarded:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1" + expr: | + ( + (windows:net_received_discarded:rate{job="windows_exporter", nic!=""} + windows:net_outbound_discarded:rate{job="windows_exporter", nic!=""}) + / ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0) + ) * 100 > 1 for: "15m" keep_firing_for: "5m" labels: @@ -170,7 +193,9 @@ groups: description: "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/" summary: "Windows network interface utilization is high" - expr: "(windows:net_received_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90) or (windows:net_sent_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90)" + expr: | + (windows:net_received_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90) + or (windows:net_sent_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90) for: "15m" keep_firing_for: "5m" labels: @@ -180,7 +205,8 @@ groups: description: "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/" summary: "Windows network interface is flapping" - expr: "max without (status) (changes(windows_net_nic_operation_status{job=\"windows_exporter\", nic!=\"\", status=\"up\"}[5m])) > 2" + expr: | + max without (status) (changes(windows_net_nic_operation_status{job="windows_exporter", nic!="", status="up"}[5m])) > 2 for: "2m" keep_firing_for: "5m" labels: @@ -192,7 +218,10 @@ groups: description: "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/" summary: "Windows automatic service is not running" - expr: "max without (state) (windows_service_state{job=\"windows_exporter\", service!=\"\", state=\"running\"} == 0) and ignoring (start_mode) (windows_service_start_mode{job=\"windows_exporter\", service!=\"\", start_mode=\"auto\"} == 1) unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job=\"windows_exporter\", service!=\"\", trigger_start=\"true\"} == 1)" + expr: | + max without (state) (windows_service_state{job="windows_exporter", service!="", state="running"} == 0) + and ignoring (start_mode) (windows_service_start_mode{job="windows_exporter", service!="", start_mode="auto"} == 1) + unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job="windows_exporter", service!="", trigger_start="true"} == 1) for: "15m" labels: severity: "warning" @@ -201,7 +230,8 @@ groups: description: "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/" summary: "Windows service is restarting repeatedly" - expr: "count without (process_id) (count_over_time(windows_service_process{job=\"windows_exporter\", service!=\"\"}[15m])) - 1 > 2" + expr: | + count without (process_id) (count_over_time(windows_service_process{job="windows_exporter", service!=""}[15m])) - 1 > 2 keep_firing_for: "5m" labels: severity: "warning" @@ -212,6 +242,8 @@ groups: description: "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago." runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/" summary: "Windows host has rebooted" - expr: "(time() - windows_system_boot_time_timestamp{job=\"windows_exporter\"}) < 600 and (time() - last_over_time(windows_system_boot_time_timestamp{job=\"windows_exporter\"}[1h] offset 10m)) > 600" + expr: | + (time() - windows_system_boot_time_timestamp{job="windows_exporter"}) < 600 + and (time() - last_over_time(windows_system_boot_time_timestamp{job="windows_exporter"}[1h] offset 10m)) > 600 labels: severity: "info" diff --git a/docs/upgrade/0.32.md b/docs/upgrade/0.32.md index aab18db2..137691a6 100644 --- a/docs/upgrade/0.32.md +++ b/docs/upgrade/0.32.md @@ -116,7 +116,7 @@ grep -rnwFf old-names.txt --exclude=windows_exporter-0.32.prom path/to/dashboard This doesn't find the label renames and the metrics without a compatibility rule; check those against the tables. -If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/alerts.libsonnet`](../../contrib/mixin/alerts/alerts.libsonnet) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too. +If you maintain your own alerts or recording rules for windows_exporter, consider contributing them to the mixin, in [`alerts/`](../../contrib/mixin/alerts) (one file per collector) or [`rules/rules.libsonnet`](../../contrib/mixin/rules/rules.libsonnet). The maintainers keep the mixin in step with every metric change, so you won't have to migrate those rules yourself at the next rename, and other users benefit from them too. ### 5. Remove the compatibility rules