mixin: commit the rendered rules and alerts

The rule files rendered from the default configuration are committed next
to the sources so they can be used without Jsonnet. mise run check-rules
regenerates them and fails on a difference; CI and mise run test run it,
like the dashboard drift check.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-17 07:50:01 +00:00
parent f8f2a71d77
commit fcb7cf32be
7 changed files with 305 additions and 12 deletions
+3
View File
@@ -64,6 +64,9 @@ jobs:
- name: Build dashboard and check generated changes
run: mise run check-dashboards
- name: Generate rules and check generated changes
run: mise run check-rules
- name: Lint dashboard and test rules
run: mise run test
-2
View File
@@ -1,5 +1,3 @@
/vendor/
/windows_alerts.yaml
/windows_rules.yaml
/tests/*_alerts.yaml
/tests/*_rules.yaml
+3
View File
@@ -34,6 +34,9 @@ A new alert touches five places, and CI checks all of them:
a runbook mentions against the metrics the exporter emits; the names in
the collector docs are not always current, so check the collector source.
5. A row in the alerts table of `README.md`.
6. The committed `windows_rules.yaml` and `windows_alerts.yaml`, regenerated
with `mise run rules`; `mise run check-rules` fails in CI when they are
stale.
Metrics with a state label (`windows_service_state{state="running"} == 0`)
keep that label in the alert; drop it with `max without (state) (...)` so the
+11 -7
View File
@@ -23,18 +23,22 @@ Mise installs release binaries with checksum verification and builds Jsonnet
Bundler using Go. `mise run` makes the pinned tools available automatically.
Only the dashboard depends on Jsonnet libraries, so `jb install` is not required
to generate rules. Generated rule files are ignored by Git:
to generate rules. The rules rendered from the default configuration are
committed next to the sources, so they can be used without Jsonnet:
- `windows_rules.yaml`: recording rules.
- `windows_alerts.yaml`: alerting rules.
`mise run rules` regenerates them after a source change, and CI fails when the
committed files are out of date (`mise run check-rules`).
The default selector is `job="windows_exporter"`. Change it to match your scrape
job before generating rules. The rules read the default `cpu`, `memory`,
`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors`
lists which collectors the targets run, and the alerts of a collector that is
turned off are not generated. Enable `ad` on domain controllers when opting
into Active Directory alerts. Clock alerts require the `time` collector with
its `ntp` subcollector and are intended for hosts using NTP synchronization.
turned off are not generated. Enable `ad` on domain controllers for the Active
Directory alerts and `time` for the clock alerts, which need the `ntp` and
`clock_source` subcollectors.
Copy both generated files into your Prometheus rule directory and reference
them in `prometheus.yml`:
@@ -468,7 +472,7 @@ mise run test
This checks Jsonnet formatting, runs `jsonnet-lint` on the Jsonnet sources to
find unused variables and references to missing fields, checks that the
committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
committed dashboards and rule files are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
`promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic
time series.
Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`,
@@ -513,8 +517,8 @@ Then run `pint --config /path/to/local.pint.hcl lint windows_rules.yaml windows_
without `--offline`. See the [pint documentation](https://cloudflare.github.io/pint/)
for available checks.
Run `mise run fmt` after source changes and `mise run clean` to remove generated rule
files. For downstream composition with Jsonnet Bundler, import
Run `mise run fmt` and `mise run rules` after source changes, and `mise run clean`
to remove the rule files generated for the tests. For downstream composition with Jsonnet Bundler, import
`github.com/prometheus-community/windows_exporter/contrib/mixin/mixin.libsonnet`
and extend `_config`. See the [monitoring mixin documentation](https://github.com/monitoring-mixins/docs)
for the standard composition workflow.
+12 -3
View File
@@ -39,6 +39,15 @@ rm -f {{vars.dashboards}}/windows-exporter-*.json
cp "$out"/*.json {{vars.dashboards}}/
'''
[tasks.check-rules]
description = "Regenerate the rules and alerts and fail on uncommitted changes"
depends = ["rules"]
run = '''
set -eu
git add --intent-to-add -- windows_rules.yaml windows_alerts.yaml
git diff --exit-code -- windows_rules.yaml windows_alerts.yaml
'''
[tasks.check-dashboards]
description = "Regenerate the dashboards and fail on uncommitted changes"
depends = ["dashboard"]
@@ -99,7 +108,7 @@ run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [na
[tasks.test]
description = "Run all mixin lint, runbook, and rule tests"
depends = ["lint", "check-runbooks", "vendor"]
depends = ["lint", "check-rules", "check-runbooks", "vendor"]
run = [
"jsonnet tests/runbooks.test.jsonnet",
"jsonnet tests/alerts.test.jsonnet",
@@ -108,5 +117,5 @@ run = [
]
[tasks.clean]
description = "Remove generated rule and alert files"
run = "rm -f windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml"
description = "Remove the generated test rule and alert files"
run = "rm -f tests/custom_rules.yaml tests/custom_alerts.yaml"
+217
View File
@@ -0,0 +1,217 @@
"groups":
- "name": "windows-exporter"
"rules":
- "alert": "WindowsExporterDown"
"annotations":
"description": "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/"
"summary": "Windows exporter is unreachable"
"expr": "up{job=\"windows_exporter\"} == 0"
"for": "5m"
"labels":
"severity": "critical"
- "alert": "WindowsCollectorFailed"
"annotations":
"description": "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/"
"summary": "Windows collector failed"
"expr": "windows_exporter_collector_success{job=\"windows_exporter\"} == 0"
"for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsCollectorTimeout"
"annotations":
"description": "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/"
"summary": "Windows collector exceeds the scrape timeout"
"expr": "windows_exporter_collector_timeout{job=\"windows_exporter\"} == 1"
"for": "10m"
"labels":
"severity": "warning"
- "alert": "WindowsScrapeQueueTimeout"
"annotations":
"description": "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/"
"summary": "Windows exporter scrapes time out in the queue"
"expr": "windows_exporter_scrape_queue_timeout{job=\"windows_exporter\"} == 1"
"for": "10m"
"labels":
"severity": "warning"
- "name": "windows-cpu"
"rules":
- "alert": "WindowsCPUHighUsage"
"annotations":
"description": "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/"
"summary": "Windows CPU usage is high"
"expr": "windows:cpu_usage:ratio{job=\"windows_exporter\"} * 100 > 90"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsCPUSaturation"
"annotations":
"description": "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/"
"summary": "Windows processor queue is long"
"expr": "windows:cpu_queue_per_core:ratio{job=\"windows_exporter\"} > 2"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "name": "windows-memory"
"rules":
- "alert": "WindowsMemoryHighUsage"
"annotations":
"description": "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/"
"summary": "Windows memory usage is high"
"expr": "windows:memory_usage:ratio{job=\"windows_exporter\"} * 100 > 90"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsMemoryCommitHighUsage"
"annotations":
"description": "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/"
"summary": "Windows memory commit usage is high"
"expr": "windows:memory_committed:ratio{job=\"windows_exporter\"} * 100 > 90"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsMemoryPagingHigh"
"annotations":
"description": "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/"
"summary": "Windows memory paging is high"
"expr": "windows:memory_swap_page_operations:rate{job=\"windows_exporter\"} > 1000"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "name": "windows-disk"
"rules":
- "alert": "WindowsDiskReadLatencyHigh"
"annotations":
"description": "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/"
"summary": "Windows disk read latency is high"
"expr": "windows:logical_disk_read_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsDiskWriteLatencyHigh"
"annotations":
"description": "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/"
"summary": "Windows disk write latency is high"
"expr": "windows:logical_disk_write_latency:seconds{job=\"windows_exporter\", volume!=\"\"} > 0.05"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsDiskAlmostFull"
"annotations":
"description": "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
"summary": "Windows disk space is low"
"expr": "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsDiskAlmostFull"
"annotations":
"description": "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
"summary": "Windows disk space is low"
"expr": "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 5"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "critical"
- "alert": "WindowsDiskFillingUp"
"annotations":
"description": "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/"
"summary": "Windows disk is predicted to fill up"
"expr": "windows:logical_disk_free:ratio{job=\"windows_exporter\", volume!=\"\"} * 100 < 10 and predict_linear(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}[6h], 86400) < 0"
"for": "1h"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "name": "windows-network"
"rules":
- "alert": "WindowsNetworkErrors"
"annotations":
"description": "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/"
"summary": "Windows network packet errors are high"
"expr": "((windows:net_received_errors:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_errors:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsNetworkDiscards"
"annotations":
"description": "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/"
"summary": "Windows network packet discards are high"
"expr": "((windows:net_received_discarded:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_outbound_discarded:rate{job=\"windows_exporter\", nic!=\"\"}) / ((windows:net_received_packets:rate{job=\"windows_exporter\", nic!=\"\"} + windows:net_sent_packets:rate{job=\"windows_exporter\", nic!=\"\"}) > 0)) * 100 > 1"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsNetworkUtilizationHigh"
"annotations":
"description": "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/"
"summary": "Windows network interface utilization is high"
"expr": "(windows:net_received_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90) or (windows:net_sent_utilization:ratio{job=\"windows_exporter\", nic!=\"\"} * 100 > 90)"
"for": "15m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "alert": "WindowsNetworkInterfaceFlapping"
"annotations":
"description": "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/"
"summary": "Windows network interface is flapping"
"expr": "max without (status) (changes(windows_net_nic_operation_status{job=\"windows_exporter\", nic!=\"\", status=\"up\"}[5m])) > 2"
"for": "2m"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "name": "windows-service"
"rules":
- "alert": "WindowsServiceNotRunning"
"annotations":
"description": "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/"
"summary": "Windows automatic service is not running"
"expr": "max without (state) (windows_service_state{job=\"windows_exporter\", service!=\"\", state=\"running\"} == 0) and ignoring (start_mode) (windows_service_start_mode{job=\"windows_exporter\", service!=\"\", start_mode=\"auto\"} == 1) unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job=\"windows_exporter\", service!=\"\", trigger_start=\"true\"} == 1)"
"for": "15m"
"labels":
"severity": "warning"
- "alert": "WindowsServiceCrashLooping"
"annotations":
"description": "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/"
"summary": "Windows service is restarting repeatedly"
"expr": "count without (process_id) (count_over_time(windows_service_process{job=\"windows_exporter\", service!=\"\"}[15m])) - 1 > 2"
"keep_firing_for": "5m"
"labels":
"severity": "warning"
- "name": "windows-system"
"rules":
- "alert": "WindowsHostRebooted"
"annotations":
"description": "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago."
"runbook_url": "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/"
"summary": "Windows host has rebooted"
"expr": "(time() - windows_system_boot_time_timestamp{job=\"windows_exporter\"}) < 600 and (time() - last_over_time(windows_system_boot_time_timestamp{job=\"windows_exporter\"}[1h] offset 10m)) > 600"
"labels":
"severity": "info"
+59
View File
@@ -0,0 +1,59 @@
"groups":
- "name": "windows-exporter-recording"
"rules":
- "expr": "1 - avg without (core, mode) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"}[5m]))"
"record": "windows:cpu_usage:ratio"
- "expr": "count without (core, mode) (windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"})"
"record": "windows:cpu_cores:count"
- "expr": "avg without (core) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\"}[5m]))"
"record": "windows:cpu_time:ratio_rate"
- "expr": "(avg_over_time(windows_system_processor_queue_length{job=\"windows_exporter\"}[5m]) / (windows:cpu_cores:count{job=\"windows_exporter\"} > 0))"
"record": "windows:cpu_queue_per_core:ratio"
- "expr": "1 - (windows_memory_available_bytes{job=\"windows_exporter\"} / (windows_memory_physical_total_bytes{job=\"windows_exporter\"} > 0))"
"record": "windows:memory_usage:ratio"
- "expr": "windows_memory_physical_total_bytes{job=\"windows_exporter\"} - windows_memory_available_bytes{job=\"windows_exporter\"}"
"record": "windows:memory_used:bytes"
- "expr": "windows_memory_cache_bytes{job=\"windows_exporter\"} + windows_memory_modified_page_list_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_core_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_normal_priority_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_reserve_bytes{job=\"windows_exporter\"}"
"record": "windows:memory_cached:bytes"
- "expr": "(windows_memory_committed_bytes{job=\"windows_exporter\"} / (windows_memory_commit_limit_bytes{job=\"windows_exporter\"} > 0))"
"record": "windows:memory_committed:ratio"
- "expr": "rate(windows_memory_swap_page_operations_total{job=\"windows_exporter\"}[5m])"
"record": "windows:memory_swap_page_operations:rate"
- "expr": "(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"} / (windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} > 0))"
"record": "windows:logical_disk_free:ratio"
- "expr": "windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} - windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}"
"record": "windows:logical_disk_used:bytes"
- "expr": "clamp(1 - rate(windows_logical_disk_idle_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]), 0, 1)"
"record": "windows:logical_disk_busy:ratio"
- "expr": "rate(windows_logical_disk_read_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
"record": "windows:logical_disk_read_bytes:rate"
- "expr": "rate(windows_logical_disk_write_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
"record": "windows:logical_disk_write_bytes:rate"
- "expr": "rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
"record": "windows:logical_disk_read_operations:rate"
- "expr": "rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
"record": "windows:logical_disk_write_operations:rate"
- "expr": "(rate(windows_logical_disk_read_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
"record": "windows:logical_disk_read_latency:seconds"
- "expr": "(rate(windows_logical_disk_write_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
"record": "windows:logical_disk_write_latency:seconds"
- "expr": "rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_received_bytes:rate"
- "expr": "rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_sent_bytes:rate"
- "expr": "rate(windows_net_packets_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_received_packets:rate"
- "expr": "rate(windows_net_packets_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_sent_packets:rate"
- "expr": "(rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
"record": "windows:net_received_utilization:ratio"
- "expr": "(rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
"record": "windows:net_sent_utilization:ratio"
- "expr": "rate(windows_net_packets_received_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_received_errors:rate"
- "expr": "rate(windows_net_packets_received_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_received_discarded:rate"
- "expr": "rate(windows_net_packets_outbound_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_outbound_errors:rate"
- "expr": "rate(windows_net_packets_outbound_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
"record": "windows:net_outbound_discarded:rate"