Jan-Otto Kröpke
d6afd48819
WindowsHostRebooted compares against the last boot time of the previous hour instead of the sample of ten minutes ago, so a scrape gap during the reboot no longer suppresses the alert. WindowsServiceNotRunning joins with ignoring() so instances with the same name in different clusters do not match each other. Replacing the removed networkErrorRateThreshold or networkDiscardRateThreshold knobs now fails with a message naming the ratio thresholds. The network error and discard runbooks describe the packet ratio, the service runbooks explain PID reuse, shared host processes and selector exclusions, and the tests cover alert recovery, a reboot after a scrape gap and a service name repeated across clusters. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
2.9 KiB
Mixin instructions
Read the root AGENTS.md first. This file covers
contrib/mixin: Jsonnet recording rules, alerts and dashboard sources.
README.md documents the configuration and the generated rules;
keep it in sync with every change.
Alerts
alerts/alerts.libsonnet renders one rule group per collector area. Every
alert names the collectors it reads (collectors=[...]) and is generated only
with those enabled in _config.collectors; _config.alerts switches single
alerts off by name. Threshold alerts get keep_firing_for from
keepFiringFor; pass threshold=false for state alerts (== 0, == 1) and
one-shot alerts, which must resolve at once.
A new alert touches five places, and CI checks all of them:
- The rule in
alerts/alerts.libsonnet, in the group of its area, with{{ $labels.<key> }}templates that use the 0.32 label keys (service,volume,nic,task,device_id, nevername). - Its switch in
_config.alertsinconfig.libsonnet;tests/alerts.test.jsonnetfails when the map and the generated alerts differ. Thresholds are_configknobs named<area><Condition>Threshold[Unit]. - A unit test in
tests/alerts.test.yamlthat checks the first firing evaluation, the labels and annotations, and the recovery. Withkeep_firing_fora threshold alert resolveskeepFiringForafter its condition clears.rate()extrapolates while a window has few samples, so ratios of two rates activate earlier than a single rate; probe the activation time withALERTS_FOR_STATEinstead of guessing it. - A runbook
docs/runbooks/<lowercase-alert-name>.mdwith theMeaning,Impact,DiagnosisandMitigationsections. - A row in the alerts table of
README.md.
Metrics with a state label (windows_service_state{state="running"} == 0)
keep that label in the alert; drop it with max without (state) (...) so the
alert does not say state="running" for a stopped service. Join a state metric
with its info metric with ignoring (<labels of the info metric>), never
on (job, instance, <key>): on drops the other target labels, so two
targets with the same instance in different clusters match each other.
A target can be scraped again after a gap (a reboot, a restart of the
exporter), so an offset lookup needs a range (last_over_time(m[1h] offset 10m)); a plain m offset 10m has no sample while the gap covers that point.
Recording rules
New recordings go in rules/rules.libsonnet with the shared expressions in
lib/queries.libsonnet, a row in the README.md table and a test in
tests/rules.test.yaml. Use q.ratio for divisions so idle devices and zero
capacities produce no sample instead of NaN or +Inf.
Checks
mise run test runs everything CI runs. Without mise, the tools from
mise.toml (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
the lint, check-runbooks and test tasks do the same.