Jan-Otto Kröpke
d6afd48819
WindowsHostRebooted compares against the last boot time of the previous hour instead of the sample of ten minutes ago, so a scrape gap during the reboot no longer suppresses the alert. WindowsServiceNotRunning joins with ignoring() so instances with the same name in different clusters do not match each other. Replacing the removed networkErrorRateThreshold or networkDiscardRateThreshold knobs now fails with a message naming the ratio thresholds. The network error and discard runbooks describe the packet ratio, the service runbooks explain PID reuse, shared host processes and selector exclusions, and the tests cover alert recovery, a reboot after a scrape gap and a service name repeated across clusters. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
2.2 KiB
title
| title |
|---|
| WindowsServiceCrashLooping |
Meaning
The service service has run as more processes in the last 15 minutes than
the configured number of restarts allows. The default is two restarts. Every
service process has its own windows_service_process series, identified by
process_id, so the series seen within 15 minutes count the restarts.
Three limits apply. Windows reuses process IDs, so a restart that lands on the
same ID is not counted. Services that share a process, such as svchost.exe
groups, share a process_id: one crash of the shared process alerts for every
service in it. Services without a windows_service_process series, for example
stopped services or services the exporter cannot read the process of, are
invisible to this alert.
Impact
The service crashes or is stopped and started repeatedly. It is unavailable
during every restart, and the recovery options of the Service Control Manager
give up after a configurable number of failures, after which
WindowsServiceNotRunning follows.
Diagnosis
-
List the processes the service ran as:
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])The value of each series is the start time of the process. If several services report the same
process_id, the shared host process crashed and the faulty service is one of that group. -
Check the Application log for crashes of the service executable (events 1000 and 1026, .NET Runtime errors) and the System log for Service Control Manager events 7031 and 7034, which also show the recovery action.
-
Check whether a deployment, configuration push, scheduled task or another automation restarts the service on purpose.
-
Check for resources the service fails to acquire at start, such as a port that is in use, a certificate, a database or a license.
Mitigation
Fix the crash or the start failure, roll back the deployment that caused it, or stop the automation that restarts the service. Keep the service stopped until the cause is known if every restart has side effects. Confirm the service keeps one process and stays running. Raise the threshold for services that restart by design, for example worker services that recycle themselves.