mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-18 10:39:05 +02:00
Jan-Otto Kröpke
d6afd48819
WindowsHostRebooted compares against the last boot time of the previous hour instead of the sample of ten minutes ago, so a scrape gap during the reboot no longer suppresses the alert. WindowsServiceNotRunning joins with ignoring() so instances with the same name in different clusters do not match each other. Replacing the removed networkErrorRateThreshold or networkDiscardRateThreshold knobs now fails with a message naming the ratio thresholds. The network error and discard runbooks describe the packet ratio, the service runbooks explain PID reuse, shared host processes and selector exclusions, and the tests cover alert recovery, a reboot after a scrape gap and a service name repeated across clusters. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
52 lines
2.2 KiB
Markdown
52 lines
2.2 KiB
Markdown
---
|
|
title: WindowsServiceCrashLooping
|
|
---
|
|
|
|
## Meaning
|
|
|
|
The service `service` has run as more processes in the last 15 minutes than
|
|
the configured number of restarts allows. The default is two restarts. Every
|
|
service process has its own `windows_service_process` series, identified by
|
|
`process_id`, so the series seen within 15 minutes count the restarts.
|
|
|
|
Three limits apply. Windows reuses process IDs, so a restart that lands on the
|
|
same ID is not counted. Services that share a process, such as `svchost.exe`
|
|
groups, share a `process_id`: one crash of the shared process alerts for every
|
|
service in it. Services without a `windows_service_process` series, for example
|
|
stopped services or services the exporter cannot read the process of, are
|
|
invisible to this alert.
|
|
|
|
## Impact
|
|
|
|
The service crashes or is stopped and started repeatedly. It is unavailable
|
|
during every restart, and the recovery options of the Service Control Manager
|
|
give up after a configurable number of failures, after which
|
|
`WindowsServiceNotRunning` follows.
|
|
|
|
## Diagnosis
|
|
|
|
1. List the processes the service ran as:
|
|
|
|
```promql
|
|
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
|
|
```
|
|
|
|
The value of each series is the start time of the process. If several
|
|
services report the same `process_id`, the shared host process crashed and
|
|
the faulty service is one of that group.
|
|
2. Check the Application log for crashes of the service executable (events
|
|
1000 and 1026, .NET Runtime errors) and the System log for Service Control
|
|
Manager events 7031 and 7034, which also show the recovery action.
|
|
3. Check whether a deployment, configuration push, scheduled task or another
|
|
automation restarts the service on purpose.
|
|
4. Check for resources the service fails to acquire at start, such as a port
|
|
that is in use, a certificate, a database or a license.
|
|
|
|
## Mitigation
|
|
|
|
Fix the crash or the start failure, roll back the deployment that caused it, or
|
|
stop the automation that restarts the service. Keep the service stopped until
|
|
the cause is known if every restart has side effects. Confirm the service keeps
|
|
one process and stays running. Raise the threshold for services that restart by
|
|
design, for example worker services that recycle themselves.
|