mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-19 19:19:04 +02:00
Jan-Otto Kröpke
018ec75223
kubernetes/ moves to docs/kubernetes/, dashboard/ to contrib/dashboard/, and the rendered Prometheus rules and alerts to contrib/monitoring/ with a README that says how to use and regenerate them. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
89 lines
4.5 KiB
Markdown
89 lines
4.5 KiB
Markdown
# Mixin instructions
|
|
|
|
Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers
|
|
`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources.
|
|
[`README.md`](README.md) documents the configuration and the generated rules;
|
|
keep it in sync with every change.
|
|
|
|
## Alerts
|
|
|
|
`alerts/` holds one file per collector, each rendering one rule group, and
|
|
`alerts/alerts.libsonnet` imports them in the order of the generated file.
|
|
A rule is a plain Prometheus alerting rule, like in kubernetes-mixin:
|
|
|
|
```jsonnet
|
|
{
|
|
alert: 'WindowsCPUHighUsage',
|
|
expr: |||
|
|
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
|
|
||| % c,
|
|
'for': '15m',
|
|
labels: { severity: 'warning' },
|
|
annotations: { summary: '...', description: '...' },
|
|
}
|
|
```
|
|
|
|
Write the PromQL out in the `expr` block with `%(name)s` placeholders for the
|
|
`_config` values; selectors go inside the braces as
|
|
`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet`
|
|
provides `a.group(name, collectors, rules)`, which keeps a rule only when every
|
|
collector in `collectors` is enabled in `_config.collectors` and the alert is
|
|
not switched off in `_config.alerts`, and adds `keep_firing_for` from
|
|
`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]`
|
|
when a rule reads other collectors than its group (the CPU saturation alert
|
|
reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`,
|
|
`== 1`) and one-shot alerts, which must resolve at once and get no
|
|
`keep_firing_for`. A one-shot alert has no `for`.
|
|
|
|
A new alert touches six places, and CI checks all of them:
|
|
|
|
1. The rule in `alerts/<collector>.libsonnet`, with `{{ $labels.<key> }}`
|
|
templates that use the 0.32 label keys (`service`, `volume`, `nic`,
|
|
`task`, `device_id`, never `name`). Alerts for a new collector go in a new
|
|
file, imported from `alerts/alerts.libsonnet`, with the collector added to
|
|
`_config.collectors` in `config.libsonnet`; the group expectations in
|
|
`tests/alerts.test.jsonnet` list every group.
|
|
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
|
|
fails when the map and the generated alerts differ. Thresholds are
|
|
`_config` knobs named `<area><Condition>Threshold[Unit]`.
|
|
3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for
|
|
opt-in collectors, which `tests/config.libsonnet` enables) that checks the
|
|
first firing evaluation, the labels and annotations, and the recovery. With
|
|
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
|
|
condition clears. `rate()` extrapolates while a window has few samples, so
|
|
ratios of two rates activate earlier than a single rate; probe the
|
|
activation time with `ALERTS_FOR_STATE` instead of guessing it.
|
|
4. A runbook `docs/runbooks/<lowercase-alert-name>.md` with the `Meaning`,
|
|
`Impact`, `Diagnosis` and `Mitigation` sections.
|
|
`go test ./internal/metriccontract/` checks every `windows_*` metric name
|
|
a runbook mentions against the metrics the exporter emits; the names in
|
|
the collector docs are not always current, so check the collector source.
|
|
5. A row in the alerts table of `README.md`.
|
|
6. The committed `contrib/monitoring/windows_rules.yaml` and
|
|
`windows_alerts.yaml`, regenerated
|
|
with `mise run rules`; `mise run check-rules` fails in CI when they are
|
|
stale.
|
|
|
|
Metrics with a state label (`windows_service_state{state="running"} == 0`)
|
|
keep that label in the alert; drop it with `max without (state) (...)` so the
|
|
alert does not say `state="running"` for a stopped service. Join a state metric
|
|
with its info metric with `ignoring (<labels of the info metric>)`, never
|
|
`on (job, instance, <key>)`: `on` drops the other target labels, so two
|
|
targets with the same `instance` in different clusters match each other.
|
|
A target can be scraped again after a gap (a reboot, a restart of the
|
|
exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset
|
|
10m)`); a plain `m offset 10m` has no sample while the gap covers that point.
|
|
|
|
## Recording rules
|
|
|
|
New recordings go in `rules/rules.libsonnet` with the shared expressions in
|
|
`lib/queries.libsonnet`, a row in the `README.md` table and a test in
|
|
`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero
|
|
capacities produce no sample instead of `NaN` or `+Inf`.
|
|
|
|
## Checks
|
|
|
|
`mise run test` runs everything CI runs. Without mise, the tools from
|
|
`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
|
|
the `lint`, `check-runbooks` and `test` tasks do the same.
|