Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.7 KiB
title
| title |
|---|
| WindowsServiceCrashLooping |
Meaning
The service service has run as more processes in the last 15 minutes than
the configured number of restarts allows. The default is two restarts. Every
service process has its own windows_service_process series, identified by
process_id, so the series seen within 15 minutes count the restarts.
Impact
The service crashes or is stopped and started repeatedly. It is unavailable
during every restart, and the recovery options of the Service Control Manager
give up after a configurable number of failures, after which
WindowsServiceNotRunning follows.
Diagnosis
-
List the processes the service ran as:
count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])The value of each series is the start time of the process.
-
Check the Application log for crashes of the service executable (events 1000 and 1026, .NET Runtime errors) and the System log for Service Control Manager events 7031 and 7034, which also show the recovery action.
-
Check whether a deployment, configuration push, scheduled task or another automation restarts the service on purpose.
-
Check for resources the service fails to acquire at start, such as a port that is in use, a certificate, a database or a license.
Mitigation
Fix the crash or the start failure, roll back the deployment that caused it, or stop the automation that restarts the service. Keep the service stopped until the cause is known if every restart has side effects. Confirm the service keeps one process and stays running. Raise the threshold for services that restart by design, for example worker services that recycle themselves.