mixin: address review findings on tier 1 alerts

WindowsHostRebooted compares against the last boot time of the previous
hour instead of the sample of ten minutes ago, so a scrape gap during
the reboot no longer suppresses the alert. WindowsServiceNotRunning
joins with ignoring() so instances with the same name in different
clusters do not match each other. Replacing the removed
networkErrorRateThreshold or networkDiscardRateThreshold knobs now
fails with a message naming the ratio thresholds.

The network error and discard runbooks describe the packet ratio, the
service runbooks explain PID reuse, shared host processes and selector
exclusions, and the tests cover alert recovery, a reboot after a scrape
gap and a service name repeated across clusters.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

contrib/mixin/AGENTS.md

@@ -35,7 +35,12 @@ A new alert touches five places, and CI checks all of them:
 Metrics with a state label (`windows_service_state{state="running"} == 0`)
 keep that label in the alert; drop it with `max without (state) (...)` so the
 alert does not say `state="running"` for a stopped service. Join a state metric
-with its info metric `on (job, instance, <key>)`.
+with its info metric with `ignoring (<labels of the info metric>)`, never
+`on (job, instance, <key>)`: `on` drops the other target labels, so two
+targets with the same `instance` in different clusters match each other.
+A target can be scraped again after a gap (a reboot, a restart of the
+exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset
+10m)`); a plain `m offset 10m` has no sample while the gap covers that point.
 
 ## Recording rules
 

contrib/mixin/README.md

@@ -64,7 +64,7 @@ modifying its source. For example, create `custom.libsonnet` in this directory:
     cpuHighUsageThreshold: 95,
     enableActiveDirectory: true,
     enableTime: true,
-    collectors+: { pagefile: true },
+    collectors+: { process: true },
     alerts+: { WindowsHostRebooted: false },
     keepFiringFor: '10m',
     runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
@@ -186,10 +186,18 @@ build, and a name missing from a replaced object stays enabled. A group whose
 alerts are all turned off is not generated.
 
 Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
-value that oscillates around a threshold does not flap. State alerts such as
-`WindowsExporterDown` and `WindowsServiceNotRunning` resolve as soon as their
-condition clears. `keep_firing_for` needs Prometheus 2.42 or later; set
-`keepFiringFor: ''` for older servers.
+value that oscillates around a threshold does not flap. The state alerts
+`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`,
+`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning` and
+`WindowsClockNotSynchronising`, and the one-shot `WindowsHostRebooted`, have no
+`keep_firing_for` and resolve as soon as their condition clears.
+`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for
+older servers.
+
+The network error and discard thresholds are percentages of all packets on the
+interface. Earlier versions of the mixin had `networkErrorRateThreshold` and
+`networkDiscardRateThreshold` in packets per second; setting them now fails
+the build with a message that names the replacements.
 
 `criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override
 them to match your routing labels. Below the critical disk threshold, both disk
@@ -209,8 +217,10 @@ skips trigger-start services, which stop by design, through the
 included; the 15 minute duration covers their later start after a reboot.
 `WindowsServiceCrashLooping` counts the `windows_service_process` series of a
 service within 15 minutes, one per process the service ran as. Exclude services
-with `serviceSelector`. `WindowsHostRebooted` fires once for about ten minutes
-after a reboot and resolves by itself.
+with `serviceSelector`, for example automatic services that exit by design
+after their work, such as the software protection or updater services of some
+products. `WindowsHostRebooted` fires once for about ten minutes after a
+reboot and resolves by itself.
 
 Windows disk size/free-space counters may lag by 10–15 minutes. Predictions
 depend on a sustained trend and are intentionally delayed for an hour. A target

contrib/mixin/alerts/alerts.libsonnet

@@ -4,6 +4,8 @@
   local known = std.objectFields((import '../config.libsonnet')._config.alerts),
   local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
   assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
+  assert !std.objectHas(c, 'networkErrorRateThreshold') && !std.objectHas(c, 'networkDiscardRateThreshold') :
+         'networkErrorRateThreshold and networkDiscardRateThreshold (packets per second) were replaced by networkErrorRatioThreshold and networkDiscardRatioThreshold (percent of all packets)',
   // A collector missing from a replaced collectors object counts as disabled;
   // an alert missing from a replaced alerts object counts as enabled.
   local collectorEnabled(collector) = std.get(c.collectors, collector, false),
@@ -23,7 +25,6 @@
       } + (if duration != '0m' then { 'for': duration } else {}) + (if threshold then keepFiring else {}),
     ] else [],
   local group(name, rules) = if std.length(rules) > 0 then [{ name: name, rules: rules }] else [],
-  local onService = 'on (job, instance, service)',
   prometheusAlerts+:: {
     groups:
       group('windows-exporter', std.flattenArrays([
@@ -208,10 +209,12 @@
       + group('windows-service', std.flattenArrays([
         alert(
           'WindowsServiceNotRunning',
+          // The matches ignore only the labels of the right-hand metric, so
+          // every target label, including ones a selector adds, takes part.
           std.join(' ', [
             'max without (state) (%s == 0)' % q.metric('windows_service_state', [c.serviceSelector, 'state="running"']),
-            'and %s (%s == 1)' % [onService, q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"'])],
-            'unless %s (%s == 1)' % [onService, q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"'])],
+            'and ignoring (start_mode) (%s == 1)' % q.metric('windows_service_start_mode', [c.serviceSelector, 'start_mode="auto"']),
+            'unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (%s == 1)' % q.metric('windows_service_info', [c.serviceSelector, 'trigger_start="true"']),
           ]),
           '15m',
           'warning',
@@ -239,8 +242,10 @@
         alert(
           'WindowsHostRebooted',
           // Fires once per reboot: for the first ten minutes after the boot,
-          // while the boot time known ten minutes ago was an older one.
-          '(time() - %s) < 600 and (time() - %s offset 10m) > 600' % [
+          // and only on a host that was monitored before it. The exporter is
+          // down while the host reboots, so the second operand takes the last
+          // boot time of the hour before, not the sample of ten minutes ago.
+          '(time() - %s) < 600 and (time() - last_over_time(%s[1h] offset 10m)) > 600' % [
             q.metric('windows_system_boot_time_timestamp'),
             q.metric('windows_system_boot_time_timestamp'),
           ],

contrib/mixin/tests/alerts.test.yaml

@@ -381,7 +381,15 @@ tests:
               runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
               summary: Windows memory paging is high
               description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
-      - {alertname: WindowsMemoryPagingHigh, eval_time: 35m, exp_alerts: []}
+      - alertname: WindowsMemoryPagingHigh
+        eval_time: 26m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
+              summary: Windows memory paging is high
+              description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
+      - {alertname: WindowsMemoryPagingHigh, eval_time: 27m, exp_alerts: []}
 
   - name: Network utilization fires in one direction and interface flapping counts status changes
     interval: 1m
@@ -408,7 +416,15 @@ tests:
               runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
               summary: Windows network interface utilization is high
               description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
-      - {alertname: WindowsNetworkUtilizationHigh, eval_time: 35m, exp_alerts: []}
+      - alertname: WindowsNetworkUtilizationHigh
+        eval_time: 25m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
+              summary: Windows network interface utilization is high
+              description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
+      - {alertname: WindowsNetworkUtilizationHigh, eval_time: 26m, exp_alerts: []}
       - {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []}
       - alertname: WindowsNetworkInterfaceFlapping
         eval_time: 5m
@@ -424,33 +440,33 @@ tests:
     interval: 1m
     input_series:
       - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}'
-        values: '0x15 1x5'
+        values: '0x15 1x8'
       - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}'
-        values: '1x20'
+        values: '1x23'
       - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}'
-        values: '1x20'
+        values: '1x23'
       - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}'
-        values: '0x20'
+        values: '0x23'
       - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}'
-        values: '1x20'
+        values: '1x23'
       - series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}'
-        values: '1x20'
+        values: '1x23'
       - series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}'
-        values: '0x20'
+        values: '0x23'
       - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}'
-        values: '1x20'
+        values: '1x23'
       - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}'
-        values: '0x20'
+        values: '0x23'
       - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}'
-        values: '1000x4 _x16'
+        values: '1000x4 _x19'
       - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}'
-        values: '_x5 2000x4 _x11'
+        values: '_x5 2000x4 _x14'
       - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}'
-        values: '_x10 3000x4 _x6'
+        values: '_x10 3000x4 _x9'
       - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}'
-        values: '_x15 4000x6'
+        values: '_x15 4000x9'
       - series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}'
-        values: '_x16 5000x5'
+        values: '_x16 5000x8'
     alert_rule_test:
       - {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []}
       - alertname: WindowsServiceNotRunning
@@ -471,6 +487,46 @@ tests:
               runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
               summary: Windows service is restarting repeatedly
               description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
+      - alertname: WindowsServiceCrashLooping
+        eval_time: 23m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
+              summary: Windows service is restarting repeatedly
+              description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
+      - {alertname: WindowsServiceCrashLooping, eval_time: 24m, exp_alerts: []}
+
+  - name: Service alert matches within a cluster when instance names repeat across clusters
+    interval: 1m
+    input_series:
+      - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", state="running"}'
+        values: '0x20'
+      - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", start_mode="auto"}'
+        values: '1x20'
+      - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", trigger_start="false", delayed_start="false"}'
+        values: '1x20'
+      - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", state="running"}'
+        values: '1x20'
+      - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", start_mode="auto"}'
+        values: '1x20'
+      - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", trigger_start="false", delayed_start="false"}'
+        values: '1x20'
+      - series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", state="running"}'
+        values: '0x20'
+      - series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", start_mode="auto"}'
+        values: '1x20'
+      - series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", trigger_start="true", delayed_start="false"}'
+        values: '1x20'
+    alert_rule_test:
+      - alertname: WindowsServiceNotRunning
+        eval_time: 20m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, service: Spooler, severity: warning}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
+              summary: Windows automatic service is not running
+              description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
 
   - name: Host reboot fires once for the first ten minutes after a new boot time
     interval: 1m
@@ -498,3 +554,28 @@ tests:
               summary: Windows host has rebooted
               description: 'host:9182 rebooted 9m 0s ago.'
       - {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []}
+
+  - name: Host reboot fires after a scrape gap longer than ten minutes
+    interval: 1m
+    input_series:
+      - series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
+        values: '-100000x5 stale _x11 960x20'
+    alert_rule_test:
+      - {alertname: WindowsHostRebooted, eval_time: 17m, exp_alerts: []}
+      - alertname: WindowsHostRebooted
+        eval_time: 18m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
+              summary: Windows host has rebooted
+              description: 'host:9182 rebooted 2m 0s ago.'
+      - alertname: WindowsHostRebooted
+        eval_time: 25m
+        exp_alerts:
+          - exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
+            exp_annotations:
+              runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
+              summary: Windows host has rebooted
+              description: 'host:9182 rebooted 9m 0s ago.'
+      - {alertname: WindowsHostRebooted, eval_time: 26m, exp_alerts: []}

docs/runbooks/windowsnetworkdiscards.md

@@ -4,9 +4,15 @@ title: WindowsNetworkDiscards
 
 ## Meaning
 
-The sum of inbound and outbound discarded packet rates exceeds the configured
-threshold for 15 minutes. The default is one discard per second per interface.
-Discards count packets dropped without a packet error, such as buffer exhaustion.
+Discarded inbound and outbound packets make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates
+(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`)
+by the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`). Discards count packets dropped without a packet
+error, such as buffer exhaustion. Interfaces that carry no packets produce no
+sample and cannot alert. The alert keeps firing for `keepFiringFor` after the
+ratio drops below the threshold.
 
 ## Impact
 
@@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 1. Inspect `windows:net_received_discarded:rate` and
    `windows:net_outbound_discarded:rate` to identify the direction and affected
-   `nic`.
+   `nic`, and compare them with the packet rate recordings to see whether a low
+   packet rate inflates the ratio.
 2. Compare traffic, directional utilization ratios, and
    `windows_net_output_queue_length_packets` with the normal baseline.
 3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers,
@@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver,
 or virtual switch problems when evidence identifies them. Review intentional
-drop policies with the network owner before changing thresholds. Confirm discard
-rates and application retransmissions return to the expected baseline.
+drop policies with the network owner before raising
+`networkDiscardRatioThreshold`. Confirm discard rates and application
+retransmissions return to the expected baseline.

docs/runbooks/windowsnetworkerrors.md

@@ -4,9 +4,16 @@ title: WindowsNetworkErrors
 
 ## Meaning
 
-The sum of inbound and outbound packet error rates exceeds the configured
-threshold for 15 minutes. The default is one error per second per interface.
-The alert identifies the interface using its `nic` label.
+Packets with inbound or outbound errors make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates
+(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by
+the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors
+alerts while a busy interface with the same error count does not. Interfaces
+that carry no packets produce no sample and cannot alert. The alert identifies
+the interface using its `nic` label and keeps firing for `keepFiringFor` after
+the ratio drops below the threshold.
 
 ## Impact
 
@@ -16,7 +23,10 @@ connection failures.
 ## Diagnosis
 
 1. Inspect `windows:net_received_errors:rate` and
-   `windows:net_outbound_errors:rate` separately to identify the direction.
+   `windows:net_outbound_errors:rate` separately to identify the direction, and
+   compare them with `windows:net_received_packets:rate` and
+   `windows:net_sent_packets:rate` to see whether a low packet rate inflates the
+   ratio.
 2. Compare traffic and `windows:net_received_utilization:ratio` and
    `windows:net_sent_utilization:ratio` with the normal baseline. These ratios
    use bandwidth in bytes per second and keep each direction separate.
@@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration
 problems. Apply driver and firmware fixes through the normal change process.
 Relieve congestion if it contributes. Verify errors stop increasing and affected
 application connections recover; cumulative counters do not need to return to
-zero for the rate alert to resolve.
+zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for
+interfaces whose error share is known and accepted.

docs/runbooks/windowsservicecrashlooping.md

@@ -9,6 +9,13 @@ the configured number of restarts allows. The default is two restarts. Every
 service process has its own `windows_service_process` series, identified by
 `process_id`, so the series seen within 15 minutes count the restarts.
 
+Three limits apply. Windows reuses process IDs, so a restart that lands on the
+same ID is not counted. Services that share a process, such as `svchost.exe`
+groups, share a `process_id`: one crash of the shared process alerts for every
+service in it. Services without a `windows_service_process` series, for example
+stopped services or services the exporter cannot read the process of, are
+invisible to this alert.
+
 ## Impact
 
 The service crashes or is stopped and started repeatedly. It is unavailable
@@ -24,7 +31,9 @@ give up after a configurable number of failures, after which
    count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
    ```
 
-   The value of each series is the start time of the process.
+   The value of each series is the start time of the process. If several
+   services report the same `process_id`, the shared host process crashed and
+   the faulty service is one of that group.
 2. Check the Application log for crashes of the service executable (events
    1000 and 1026, .NET Runtime errors) and the System log for Service Control
    Manager events 7031 and 7034, which also show the recovery action.

docs/runbooks/windowsservicenotrunning.md

@@ -40,4 +40,6 @@ Start the service after fixing the cause: the account, a missing dependency, a
 configuration error or a crash. Set recovery options so Windows restarts the
 service itself. Change the start mode to `manual` or `disabled` for services
 that should not run, or exclude them with the `serviceSelector` when the mixin
-should not watch them. Confirm the service stays running.
+should not watch them. For example
+`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that
+exit on their own by design. Confirm the service stays running.