mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-20 03:29:05 +02:00
Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
43 lines
1.7 KiB
Markdown
43 lines
1.7 KiB
Markdown
---
|
|
title: WindowsServiceCrashLooping
|
|
---
|
|
|
|
## Meaning
|
|
|
|
The service `service` has run as more processes in the last 15 minutes than
|
|
the configured number of restarts allows. The default is two restarts. Every
|
|
service process has its own `windows_service_process` series, identified by
|
|
`process_id`, so the series seen within 15 minutes count the restarts.
|
|
|
|
## Impact
|
|
|
|
The service crashes or is stopped and started repeatedly. It is unavailable
|
|
during every restart, and the recovery options of the Service Control Manager
|
|
give up after a configurable number of failures, after which
|
|
`WindowsServiceNotRunning` follows.
|
|
|
|
## Diagnosis
|
|
|
|
1. List the processes the service ran as:
|
|
|
|
```promql
|
|
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
|
|
```
|
|
|
|
The value of each series is the start time of the process.
|
|
2. Check the Application log for crashes of the service executable (events
|
|
1000 and 1026, .NET Runtime errors) and the System log for Service Control
|
|
Manager events 7031 and 7034, which also show the recovery action.
|
|
3. Check whether a deployment, configuration push, scheduled task or another
|
|
automation restarts the service on purpose.
|
|
4. Check for resources the service fails to acquire at start, such as a port
|
|
that is in use, a certificate, a database or a license.
|
|
|
|
## Mitigation
|
|
|
|
Fix the crash or the start failure, roll back the deployment that caused it, or
|
|
stop the automation that restarts the service. Keep the service stopped until
|
|
the cause is known if every restart has side effects. Confirm the service keeps
|
|
one process and stays running. Raise the threshold for services that restart by
|
|
design, for example worker services that recycle themselves.
|