Jan-Otto Kröpke
d6afd48819
WindowsHostRebooted compares against the last boot time of the previous hour instead of the sample of ten minutes ago, so a scrape gap during the reboot no longer suppresses the alert. WindowsServiceNotRunning joins with ignoring() so instances with the same name in different clusters do not match each other. Replacing the removed networkErrorRateThreshold or networkDiscardRateThreshold knobs now fails with a message naming the ratio thresholds. The network error and discard runbooks describe the packet ratio, the service runbooks explain PID reuse, shared host processes and selector exclusions, and the tests cover alert recovery, a reboot after a scrape gap and a service name repeated across clusters. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
2.1 KiB
title
| title |
|---|
| WindowsNetworkErrors |
Meaning
Packets with inbound or outbound errors make up more than the configured share
of all packets on an interface for 15 minutes. The default
networkErrorRatioThreshold is 1 percent. The ratio divides the error rates
(windows:net_received_errors:rate plus windows:net_outbound_errors:rate) by
the packet rates (windows:net_received_packets:rate plus
windows:net_sent_packets:rate), so a quiet interface with a handful of errors
alerts while a busy interface with the same error count does not. Interfaces
that carry no packets produce no sample and cannot alert. The alert identifies
the interface using its nic label and keeps firing for keepFiringFor after
the ratio drops below the threshold.
Impact
Packet errors may cause retransmissions, lower throughput, and intermittent connection failures.
Diagnosis
-
Inspect
windows:net_received_errors:rateandwindows:net_outbound_errors:rateseparately to identify the direction, and compare them withwindows:net_received_packets:rateandwindows:net_sent_packets:rateto see whether a low packet rate inflates the ratio. -
Compare traffic and
windows:net_received_utilization:ratioandwindows:net_sent_utilization:ratiowith the normal baseline. These ratios use bandwidth in bytes per second and keep each direction separate. -
Inspect local adapter state and counters:
Get-NetAdapter Get-NetAdapterStatistics -
Inspect switch-port counters, cabling, link negotiation, NIC drivers, and System event logs. For virtual NICs, inspect the host and virtual switch.
Mitigation
Repair faulty links or switch ports and correct driver or adapter configuration
problems. Apply driver and firmware fixes through the normal change process.
Relieve congestion if it contributes. Verify errors stop increasing and affected
application connections recover; cumulative counters do not need to return to
zero for the ratio alert to resolve. Raise networkErrorRatioThreshold only for
interfaces whose error share is known and accepted.