The rule files rendered from the default configuration are committed next
to the sources so they can be used without Jsonnet. mise run check-rules
regenerates them and fails on a difference; CI and mise run test run it,
like the dashboard drift check.
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
collectors.ad and collectors.time are the only switches for the Active
Directory and clock alerts, like every other opt-in collector. Setting
the removed keys fails the build with a message naming the replacement.
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Every optional collector of the mixin now has alerts, generated only with
that collector enabled in _config.collectors: clock source, disk drive
status, page file usage, license status, scheduled task results, pending
updates, Hyper-V WMI and replication health, failover cluster nodes,
resources, groups and shared volumes, Storage Spaces pool and virtual disk
health, DHCP scope exhaustion and DNS zone transfer failures. The alerts
of the pagefile, diskdrive and license collectors join the memory, disk
and system groups; the other collectors get a group of their own.
New knobs: taskSelector, sharedVolumeSelector, pagefileUsageThreshold,
dhcpScopeFreeWarningThreshold, dhcpScopeFreeCriticalThreshold and
updatePendingDuration. The test configuration enables every optional
collector and tests/optional.test.yaml covers the new alerts.
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
WindowsHostRebooted compares against the last boot time of the previous
hour instead of the sample of ten minutes ago, so a scrape gap during
the reboot no longer suppresses the alert. WindowsServiceNotRunning
joins with ignoring() so instances with the same name in different
clusters do not match each other. Replacing the removed
networkErrorRateThreshold or networkDiscardRateThreshold knobs now
fails with a message naming the ratio thresholds.
The network error and discard runbooks describe the packet ratio, the
service runbooks explain PID reuse, shared host processes and selector
exclusions, and the tests cover alert recovery, a reboot after a scrape
gap and a service name repeated across clusters.
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
The alerts of the mixin now come in one rule group per collector area
(windows-exporter, windows-cpu, windows-memory, windows-disk,
windows-network, windows-service, windows-system, windows-active-directory,
windows-time). Every alert names the collectors it reads and is generated
only when _config.collectors enables them, the same switch the dashboard
uses; enableActiveDirectory and enableTime set collectors.ad and
collectors.time. _config.alerts switches single alerts off by name, and an
unknown name fails the build.
New alerts for the default collectors: WindowsCollectorTimeout,
WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core),
WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh,
WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic
services without triggers), WindowsServiceCrashLooping (processes per
service in 15 minutes) and WindowsHostRebooted (info, fires once per
reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error
and discard rates with the packet rate (networkErrorRatioThreshold and
networkDiscardRatioThreshold, in percent) instead of an absolute packet
rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new
infoSeverity and serviceSelector knobs follow the existing ones.
New recordings: windows:cpu_queue_per_core:ratio,
windows:net_received_packets:rate and windows:net_sent_packets:rate.
Each new alert has a runbook, a README row and a unit test;
tests/alerts.test.jsonnet checks the collector gating, the alert switches
and the group layout. The runbook check renders the alerts of every
collector. contrib/mixin/AGENTS.md documents the checklist for new alerts.
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Jan-Otto Kröpke