diff --git a/contrib/mixin/AGENTS.md b/contrib/mixin/AGENTS.md index cd3e9224..0603edb1 100644 --- a/contrib/mixin/AGENTS.md +++ b/contrib/mixin/AGENTS.md @@ -35,7 +35,12 @@ A new alert touches five places, and CI checks all of them: Metrics with a state label (`windows_service_state{state="running"} == 0`) keep that label in the alert; drop it with `max without (state) (...)` so the alert does not say `state="running"` for a stopped service. Join a state metric -with its info metric `on (job, instance, )`. +with its info metric with `ignoring ()`, never +`on (job, instance, )`: `on` drops the other target labels, so two +targets with the same `instance` in different clusters match each other. +A target can be scraped again after a gap (a reboot, a restart of the +exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset +10m)`); a plain `m offset 10m` has no sample while the gap covers that point. ## Recording rules diff --git a/contrib/mixin/README.md b/contrib/mixin/README.md index 4115ab49..d0258f42 100644 --- a/contrib/mixin/README.md +++ b/contrib/mixin/README.md @@ -64,7 +64,7 @@ modifying its source. For example, create `custom.libsonnet` in this directory: cpuHighUsageThreshold: 95, enableActiveDirectory: true, enableTime: true, - collectors+: { pagefile: true }, + collectors+: { process: true }, alerts+: { WindowsHostRebooted: false }, keepFiringFor: '10m', runbookURLPattern: 'https://runbooks.example.com/windows/%s/', @@ -186,10 +186,18 @@ build, and a name missing from a replaced object stays enabled. A group whose alerts are all turned off is not generated. Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a -value that oscillates around a threshold does not flap. State alerts such as -`WindowsExporterDown` and `WindowsServiceNotRunning` resolve as soon as their -condition clears. `keep_firing_for` needs Prometheus 2.42 or later; set -`keepFiringFor: ''` for older servers. +value that oscillates around a threshold does not flap. The state alerts +`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`, +`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning` and +`WindowsClockNotSynchronising`, and the one-shot `WindowsHostRebooted`, have no +`keep_firing_for` and resolve as soon as their condition clears. +`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for +older servers. + +The network error and discard thresholds are percentages of all packets on the +interface. Earlier versions of the mixin had `networkErrorRateThreshold` and +`networkDiscardRateThreshold` in packets per second; setting them now fails +the build with a message that names the replacements. `criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override them to match your routing labels. Below the critical disk threshold, both disk @@ -209,8 +217,10 @@ skips trigger-start services, which stop by design, through the included; the 15 minute duration covers their later start after a reboot. `WindowsServiceCrashLooping` counts the `windows_service_process` series of a service within 15 minutes, one per process the service ran as. Exclude services -with `serviceSelector`. `WindowsHostRebooted` fires once for about ten minutes -after a reboot and resolves by itself. +with `serviceSelector`, for example automatic services that exit by design +after their work, such as the software protection or updater services of some +products. `WindowsHostRebooted` fires once for about ten minutes after a +reboot and resolves by itself. Windows disk size/free-space counters may lag by 10–15 minutes. Predictions depend on a sustained trend and are intentionally delayed for an hour. A target diff --git a/contrib/mixin/alerts/alerts.libsonnet b/contrib/mixin/alerts/alerts.libsonnet index c38d36d3..c70bb63c 100644 --- a/contrib/mixin/alerts/alerts.libsonnet +++ b/contrib/mixin/alerts/alerts.libsonnet @@ -4,6 +4,8 @@ local known = std.objectFields((import '../config.libsonnet')._config.alerts), local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)), assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown), + assert !std.objectHas(c, 'networkErrorRateThreshold') && !std.objectHas(c, 'networkDiscardRateThreshold') : + 'networkErrorRateThreshold and networkDiscardRateThreshold (packets per second) were replaced by networkErrorRatioThreshold and networkDiscardRatioThreshold (percent of all packets)', // A collector missing from a replaced collectors object counts as disabled; // an alert missing from a replaced alerts object counts as enabled. local collectorEnabled(collector) = std.get(c.collectors, collector, false), @@ -23,7 +25,6 @@ } + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}), ] else [], local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [], - local onService = 'on (job, instance, service)', prometheusAlerts+:: { groups: group('windows-exporter', std.flattenArrays([ @@ -208,10 +209,12 @@ + group('windows-service', std.flattenArrays([ alert( 'WindowsServiceNotRunning', + // The matches ignore only the labels of the right-hand metric, so + // every target label, including ones a selector adds, takes part. std.join(' ', [ 'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']), - 'and %s (%s == 1)' % [onService, q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"'])], - 'unless %s (%s == 1)' % [onService, q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"'])], + 'and ignoring (start_mode) (%s == 1)' % q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"']), + 'unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (%s == 1)' % q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"']), ]), '15m', 'warning', @@ -239,8 +242,10 @@ alert( 'WindowsHostRebooted', // Fires once per reboot: for the first ten minutes after the boot, - // while the boot time known ten minutes ago was an older one. - '(time() - %s) < 600 and (time() - %s offset 10m) > 600' % [ + // and only on a host that was monitored before it. The exporter is + // down while the host reboots, so the second operand takes the last + // boot time of the hour before, not the sample of ten minutes ago. + '(time() - %s) < 600 and (time() - last_over_time(%s[1h] offset 10m)) > 600' % [ q.metric('windows_system_boot_time_timestamp'), q.metric('windows_system_boot_time_timestamp'), ], diff --git a/contrib/mixin/tests/alerts.test.yaml b/contrib/mixin/tests/alerts.test.yaml index cc5c01b9..831101a4 100644 --- a/contrib/mixin/tests/alerts.test.yaml +++ b/contrib/mixin/tests/alerts.test.yaml @@ -381,7 +381,15 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/ summary: Windows memory paging is high description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.' - - {alertname: WindowsMemoryPagingHigh, eval_time: 35m, exp_alerts: []} + - alertname: WindowsMemoryPagingHigh + eval_time: 26m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/ + summary: Windows memory paging is high + description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.' + - {alertname: WindowsMemoryPagingHigh, eval_time: 27m, exp_alerts: []} - name: Network utilization fires in one direction and interface flapping counts status changes interval: 1m @@ -408,7 +416,15 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/ summary: Windows network interface utilization is high description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.' - - {alertname: WindowsNetworkUtilizationHigh, eval_time: 35m, exp_alerts: []} + - alertname: WindowsNetworkUtilizationHigh + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/ + summary: Windows network interface utilization is high + description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.' + - {alertname: WindowsNetworkUtilizationHigh, eval_time: 26m, exp_alerts: []} - {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []} - alertname: WindowsNetworkInterfaceFlapping eval_time: 5m @@ -424,33 +440,33 @@ tests: interval: 1m input_series: - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}' - values: '0x15 1x5' + values: '0x15 1x8' - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}' - values: '1x20' + values: '1x23' - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}' - values: '1x20' + values: '1x23' - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}' - values: '0x20' + values: '0x23' - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}' - values: '1x20' + values: '1x23' - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}' - values: '1x20' + values: '1x23' - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}' - values: '0x20' + values: '0x23' - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}' - values: '1x20' + values: '1x23' - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}' - values: '0x20' + values: '0x23' - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}' - values: '1000x4 _x16' + values: '1000x4 _x19' - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}' - values: '_x5 2000x4 _x11' + values: '_x5 2000x4 _x14' - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}' - values: '_x10 3000x4 _x6' + values: '_x10 3000x4 _x9' - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}' - values: '_x15 4000x6' + values: '_x15 4000x9' - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}' - values: '_x16 5000x5' + values: '_x16 5000x8' alert_rule_test: - {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []} - alertname: WindowsServiceNotRunning @@ -471,6 +487,46 @@ tests: runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/ summary: Windows service is restarting repeatedly description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.' + - alertname: WindowsServiceCrashLooping + eval_time: 23m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/ + summary: Windows service is restarting repeatedly + description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.' + - {alertname: WindowsServiceCrashLooping, eval_time: 24m, exp_alerts: []} + + - name: Service alert matches within a cluster when instance names repeat across clusters + interval: 1m + input_series: + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", state="running"}' + values: '0x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", trigger_start="false", delayed_start="false"}' + values: '1x20' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", state="running"}' + values: '1x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", trigger_start="false", delayed_start="false"}' + values: '1x20' + - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", state="running"}' + values: '0x20' + - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", start_mode="auto"}' + values: '1x20' + - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", trigger_start="true", delayed_start="false"}' + values: '1x20' + alert_rule_test: + - alertname: WindowsServiceNotRunning + eval_time: 20m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, service: Spooler, severity: warning} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/ + summary: Windows automatic service is not running + description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.' - name: Host reboot fires once for the first ten minutes after a new boot time interval: 1m @@ -498,3 +554,28 @@ tests: summary: Windows host has rebooted description: 'host:9182 rebooted 9m 0s ago.' - {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []} + + - name: Host reboot fires after a scrape gap longer than ten minutes + interval: 1m + input_series: + - series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}' + values: '-100000x5 stale _x11 960x20' + alert_rule_test: + - {alertname: WindowsHostRebooted, eval_time: 17m, exp_alerts: []} + - alertname: WindowsHostRebooted + eval_time: 18m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 2m 0s ago.' + - alertname: WindowsHostRebooted + eval_time: 25m + exp_alerts: + - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info} + exp_annotations: + runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/ + summary: Windows host has rebooted + description: 'host:9182 rebooted 9m 0s ago.' + - {alertname: WindowsHostRebooted, eval_time: 26m, exp_alerts: []} diff --git a/docs/runbooks/windowsnetworkdiscards.md b/docs/runbooks/windowsnetworkdiscards.md index d96cdaca..ffa562c8 100644 --- a/docs/runbooks/windowsnetworkdiscards.md +++ b/docs/runbooks/windowsnetworkdiscards.md @@ -4,9 +4,15 @@ title: WindowsNetworkDiscards ## Meaning -The sum of inbound and outbound discarded packet rates exceeds the configured -threshold for 15 minutes. The default is one discard per second per interface. -Discards count packets dropped without a packet error, such as buffer exhaustion. +Discarded inbound and outbound packets make up more than the configured share +of all packets on an interface for 15 minutes. The default +`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates +(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`) +by the packet rates (`windows:net_received_packets:rate` plus +`windows:net_sent_packets:rate`). Discards count packets dropped without a packet +error, such as buffer exhaustion. Interfaces that carry no packets produce no +sample and cannot alert. The alert keeps firing for `keepFiringFor` after the +ratio drops below the threshold. ## Impact @@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy. 1. Inspect `windows:net_received_discarded:rate` and `windows:net_outbound_discarded:rate` to identify the direction and affected - `nic`. + `nic`, and compare them with the packet rate recordings to see whether a low + packet rate inflates the ratio. 2. Compare traffic, directional utilization ratios, and `windows_net_output_queue_length_packets` with the normal baseline. 3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers, @@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy. Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver, or virtual switch problems when evidence identifies them. Review intentional -drop policies with the network owner before changing thresholds. Confirm discard -rates and application retransmissions return to the expected baseline. +drop policies with the network owner before raising +`networkDiscardRatioThreshold`. Confirm discard rates and application +retransmissions return to the expected baseline. diff --git a/docs/runbooks/windowsnetworkerrors.md b/docs/runbooks/windowsnetworkerrors.md index 882cb728..b5ba471c 100644 --- a/docs/runbooks/windowsnetworkerrors.md +++ b/docs/runbooks/windowsnetworkerrors.md @@ -4,9 +4,16 @@ title: WindowsNetworkErrors ## Meaning -The sum of inbound and outbound packet error rates exceeds the configured -threshold for 15 minutes. The default is one error per second per interface. -The alert identifies the interface using its `nic` label. +Packets with inbound or outbound errors make up more than the configured share +of all packets on an interface for 15 minutes. The default +`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates +(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by +the packet rates (`windows:net_received_packets:rate` plus +`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors +alerts while a busy interface with the same error count does not. Interfaces +that carry no packets produce no sample and cannot alert. The alert identifies +the interface using its `nic` label and keeps firing for `keepFiringFor` after +the ratio drops below the threshold. ## Impact @@ -16,7 +23,10 @@ connection failures. ## Diagnosis 1. Inspect `windows:net_received_errors:rate` and - `windows:net_outbound_errors:rate` separately to identify the direction. + `windows:net_outbound_errors:rate` separately to identify the direction, and + compare them with `windows:net_received_packets:rate` and + `windows:net_sent_packets:rate` to see whether a low packet rate inflates the + ratio. 2. Compare traffic and `windows:net_received_utilization:ratio` and `windows:net_sent_utilization:ratio` with the normal baseline. These ratios use bandwidth in bytes per second and keep each direction separate. @@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration problems. Apply driver and firmware fixes through the normal change process. Relieve congestion if it contributes. Verify errors stop increasing and affected application connections recover; cumulative counters do not need to return to -zero for the rate alert to resolve. +zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for +interfaces whose error share is known and accepted. diff --git a/docs/runbooks/windowsservicecrashlooping.md b/docs/runbooks/windowsservicecrashlooping.md index dbb1d583..5a050726 100644 --- a/docs/runbooks/windowsservicecrashlooping.md +++ b/docs/runbooks/windowsservicecrashlooping.md @@ -9,6 +9,13 @@ the configured number of restarts allows. The default is two restarts. Every service process has its own `windows_service_process` series, identified by `process_id`, so the series seen within 15 minutes count the restarts. +Three limits apply. Windows reuses process IDs, so a restart that lands on the +same ID is not counted. Services that share a process, such as `svchost.exe` +groups, share a `process_id`: one crash of the shared process alerts for every +service in it. Services without a `windows_service_process` series, for example +stopped services or services the exporter cannot read the process of, are +invisible to this alert. + ## Impact The service crashes or is stopped and started repeatedly. It is unavailable @@ -24,7 +31,9 @@ give up after a configurable number of failures, after which count_over_time(windows_service_process{job="windows_exporter", service=""}[15m]) ``` - The value of each series is the start time of the process. + The value of each series is the start time of the process. If several + services report the same `process_id`, the shared host process crashed and + the faulty service is one of that group. 2. Check the Application log for crashes of the service executable (events 1000 and 1026, .NET Runtime errors) and the System log for Service Control Manager events 7031 and 7034, which also show the recovery action. diff --git a/docs/runbooks/windowsservicenotrunning.md b/docs/runbooks/windowsservicenotrunning.md index 64b5bdeb..52b32be7 100644 --- a/docs/runbooks/windowsservicenotrunning.md +++ b/docs/runbooks/windowsservicenotrunning.md @@ -40,4 +40,6 @@ Start the service after fixing the cause: the account, a missing dependency, a configuration error or a crash. Set recovery options so Windows restarts the service itself. Change the start mode to `manual` or `disabled` for services that should not run, or exclude them with the `serviceSelector` when the mixin -should not watch them. Confirm the service stays running. +should not watch them. For example +`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that +exit on their own by design. Confirm the service stays running.