Jan-Otto Kröpke
ae5d0cca2a
The alerts of the mixin now come in one rule group per collector area (windows-exporter, windows-cpu, windows-memory, windows-disk, windows-network, windows-service, windows-system, windows-active-directory, windows-time). Every alert names the collectors it reads and is generated only when _config.collectors enables them, the same switch the dashboard uses; enableActiveDirectory and enableTime set collectors.ad and collectors.time. _config.alerts switches single alerts off by name, and an unknown name fails the build. New alerts for the default collectors: WindowsCollectorTimeout, WindowsScrapeQueueTimeout, WindowsCPUSaturation (processor queue per core), WindowsMemoryPagingHigh, WindowsNetworkUtilizationHigh, WindowsNetworkInterfaceFlapping, WindowsServiceNotRunning (automatic services without triggers), WindowsServiceCrashLooping (processes per service in 15 minutes) and WindowsHostRebooted (info, fires once per reboot). WindowsNetworkErrors and WindowsNetworkDiscards compare the error and discard rates with the packet rate (networkErrorRatioThreshold and networkDiscardRatioThreshold, in percent) instead of an absolute packet rate. Threshold alerts set keep_firing_for from keepFiringFor (5m); the new infoSeverity and serviceSelector knobs follow the existing ones. New recordings: windows:cpu_queue_per_core:ratio, windows:net_received_packets:rate and windows:net_sent_packets:rate. Each new alert has a runbook, a README row and a unit test; tests/alerts.test.jsonnet checks the collector gating, the alert switches and the group layout. The runbook check renders the alerts of every collector. contrib/mixin/AGENTS.md documents the checklist for new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
2.5 KiB
Mixin instructions
Read the root AGENTS.md first. This file covers
contrib/mixin: Jsonnet recording rules, alerts and dashboard sources.
README.md documents the configuration and the generated rules;
keep it in sync with every change.
Alerts
alerts/alerts.libsonnet renders one rule group per collector area. Every
alert names the collectors it reads (collectors=[...]) and is generated only
with those enabled in _config.collectors; _config.alerts switches single
alerts off by name. Threshold alerts get keep_firing_for from
keepFiringFor; pass threshold=false for state alerts (== 0, == 1) and
one-shot alerts, which must resolve at once.
A new alert touches five places, and CI checks all of them:
- The rule in
alerts/alerts.libsonnet, in the group of its area, with{{ $labels.<key> }}templates that use the 0.32 label keys (service,volume,nic,task,device_id, nevername). - Its switch in
_config.alertsinconfig.libsonnet;tests/alerts.test.jsonnetfails when the map and the generated alerts differ. Thresholds are_configknobs named<area><Condition>Threshold[Unit]. - A unit test in
tests/alerts.test.yamlthat checks the first firing evaluation, the labels and annotations, and the recovery. Withkeep_firing_fora threshold alert resolveskeepFiringForafter its condition clears.rate()extrapolates while a window has few samples, so ratios of two rates activate earlier than a single rate; probe the activation time withALERTS_FOR_STATEinstead of guessing it. - A runbook
docs/runbooks/<lowercase-alert-name>.mdwith theMeaning,Impact,DiagnosisandMitigationsections. - A row in the alerts table of
README.md.
Metrics with a state label (windows_service_state{state="running"} == 0)
keep that label in the alert; drop it with max without (state) (...) so the
alert does not say state="running" for a stopped service. Join a state metric
with its info metric on (job, instance, <key>).
Recording rules
New recordings go in rules/rules.libsonnet with the shared expressions in
lib/queries.libsonnet, a row in the README table and a test in
tests/rules.test.yaml. Use q.ratio for divisions so idle devices and zero
capacities produce no sample instead of NaN or +Inf.
Checks
mise run test runs everything CI runs. Without mise, the tools from
mise.toml (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
the lint, check-runbooks and test tasks do the same.