mixin: address review findings on tier 1 alerts

WindowsHostRebooted compares against the last boot time of the previous
hour instead of the sample of ten minutes ago, so a scrape gap during
the reboot no longer suppresses the alert. WindowsServiceNotRunning
joins with ignoring() so instances with the same name in different
clusters do not match each other. Replacing the removed
networkErrorRateThreshold or networkDiscardRateThreshold knobs now
fails with a message naming the ratio thresholds.

The network error and discard runbooks describe the packet ratio, the
service runbooks explain PID reuse, shared host processes and selector
exclusions, and the tests cover alert recovery, a reboot after a scrape
gap and a service name repeated across clusters.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

docs/runbooks/windowsnetworkdiscards.md

@@ -4,9 +4,15 @@ title: WindowsNetworkDiscards
 
 ## Meaning
 
-The sum of inbound and outbound discarded packet rates exceeds the configured
-threshold for 15 minutes. The default is one discard per second per interface.
-Discards count packets dropped without a packet error, such as buffer exhaustion.
+Discarded inbound and outbound packets make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates
+(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`)
+by the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`). Discards count packets dropped without a packet
+error, such as buffer exhaustion. Interfaces that carry no packets produce no
+sample and cannot alert. The alert keeps firing for `keepFiringFor` after the
+ratio drops below the threshold.
 
 ## Impact
 
@@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 1. Inspect `windows:net_received_discarded:rate` and
    `windows:net_outbound_discarded:rate` to identify the direction and affected
-   `nic`.
+   `nic`, and compare them with the packet rate recordings to see whether a low
+   packet rate inflates the ratio.
 2. Compare traffic, directional utilization ratios, and
    `windows_net_output_queue_length_packets` with the normal baseline.
 3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers,
@@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver,
 or virtual switch problems when evidence identifies them. Review intentional
-drop policies with the network owner before changing thresholds. Confirm discard
-rates and application retransmissions return to the expected baseline.
+drop policies with the network owner before raising
+`networkDiscardRatioThreshold`. Confirm discard rates and application
+retransmissions return to the expected baseline.

docs/runbooks/windowsnetworkerrors.md

@@ -4,9 +4,16 @@ title: WindowsNetworkErrors
 
 ## Meaning
 
-The sum of inbound and outbound packet error rates exceeds the configured
-threshold for 15 minutes. The default is one error per second per interface.
-The alert identifies the interface using its `nic` label.
+Packets with inbound or outbound errors make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates
+(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by
+the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors
+alerts while a busy interface with the same error count does not. Interfaces
+that carry no packets produce no sample and cannot alert. The alert identifies
+the interface using its `nic` label and keeps firing for `keepFiringFor` after
+the ratio drops below the threshold.
 
 ## Impact
 
@@ -16,7 +23,10 @@ connection failures.
 ## Diagnosis
 
 1. Inspect `windows:net_received_errors:rate` and
-   `windows:net_outbound_errors:rate` separately to identify the direction.
+   `windows:net_outbound_errors:rate` separately to identify the direction, and
+   compare them with `windows:net_received_packets:rate` and
+   `windows:net_sent_packets:rate` to see whether a low packet rate inflates the
+   ratio.
 2. Compare traffic and `windows:net_received_utilization:ratio` and
    `windows:net_sent_utilization:ratio` with the normal baseline. These ratios
    use bandwidth in bytes per second and keep each direction separate.
@@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration
 problems. Apply driver and firmware fixes through the normal change process.
 Relieve congestion if it contributes. Verify errors stop increasing and affected
 application connections recover; cumulative counters do not need to return to
-zero for the rate alert to resolve.
+zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for
+interfaces whose error share is known and accepted.

docs/runbooks/windowsservicecrashlooping.md

@@ -9,6 +9,13 @@ the configured number of restarts allows. The default is two restarts. Every
 service process has its own `windows_service_process` series, identified by
 `process_id`, so the series seen within 15 minutes count the restarts.
 
+Three limits apply. Windows reuses process IDs, so a restart that lands on the
+same ID is not counted. Services that share a process, such as `svchost.exe`
+groups, share a `process_id`: one crash of the shared process alerts for every
+service in it. Services without a `windows_service_process` series, for example
+stopped services or services the exporter cannot read the process of, are
+invisible to this alert.
+
 ## Impact
 
 The service crashes or is stopped and started repeatedly. It is unavailable
@@ -24,7 +31,9 @@ give up after a configurable number of failures, after which
    count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
    ```
 
-   The value of each series is the start time of the process.
+   The value of each series is the start time of the process. If several
+   services report the same `process_id`, the shared host process crashed and
+   the faulty service is one of that group.
 2. Check the Application log for crashes of the service executable (events
    1000 and 1026, .NET Runtime errors) and the System log for Service Control
    Manager events 7031 and 7034, which also show the recovery action.

docs/runbooks/windowsservicenotrunning.md

@@ -40,4 +40,6 @@ Start the service after fixing the cause: the account, a missing dependency, a
 configuration error or a crash. Set recovery options so Windows restarts the
 service itself. Change the start mode to `manual` or `disabled` for services
 that should not run, or exclude them with the `serviceSelector` when the mixin
-should not watch them. Confirm the service stays running.
+should not watch them. For example
+`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that
+exit on their own by design. Confirm the service stays running.