Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.7 KiB
title
| title |
|---|
| WindowsServiceNotRunning |
Meaning
The service service has the start mode auto but has not been in the
running state for 15 minutes. Trigger-start services, which Windows stops by
design when their trigger condition ends, are left out through the
trigger_start label of windows_service_info. Delayed-start services are
included; the 15 minute duration covers their later start after a reboot.
Impact
A service that should always run is stopped, so the function it provides is unavailable, for example printing, time synchronization, a web site or a database instance. Windows restarts a crashed service only when its recovery options say so.
Diagnosis
-
Inspect the current state of the service and when it changed:
windows_service_state{job="windows_exporter", service="<service>"} == 1 -
Check the System event log for Service Control Manager events 7000, 7009, 7031 and 7034, and the Application log for crashes of the service process.
-
Check
windows_service_infofor the account inrun_as; an expired password or a removed right to log on as a service stops the service at the next start.windows_service_processshows whether a process is running. -
Check whether the service was stopped on purpose for maintenance or disabled by policy, and whether it depends on another service that is also stopped.
Mitigation
Start the service after fixing the cause: the account, a missing dependency, a
configuration error or a crash. Set recovery options so Windows restarts the
service itself. Change the start mode to manual or disabled for services
that should not run, or exclude them with the serviceSelector when the mixin
should not watch them. Confirm the service stays running.