Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.5 KiB
title
| title |
|---|
| WindowsHostRebooted |
Meaning
The boot time of the host, windows_system_boot_time_timestamp, is less than
ten minutes old and differs from the boot time known ten minutes earlier. The
alert fires once for about ten minutes after a reboot and then resolves by
itself. It has the info severity and is meant for a dashboard, a ticket or a
chat notification, not for a page.
Impact
Every service of the host restarted, connections to it were dropped and caches are cold. A reboot outside a maintenance window can mean a crash, an unattended Windows Update restart or a power event.
Diagnosis
- Compare the boot time with the maintenance calendar and the Windows Update
schedule. With the
updatecollector enabled,windows_update_pending_infoshows updates that were pending before the reboot. - Check the System event log: event 1074 names the user or process that requested the shutdown, event 6008 marks an unexpected shutdown and event 41 a reset without a clean shutdown. Check for a memory dump when it was a crash.
- Confirm the host came back with the expected services; the
WindowsServiceNotRunningalert covers automatic services that did not start.
Mitigation
No action is needed for a planned reboot. For an unplanned one, find the cause
in the event log and fix the driver, update setting or power problem. Turn the
alert off with alerts+: { WindowsHostRebooted: false } when reboots are not
worth a notification.