mirror of
https://github.com/prometheus-community/windows_exporter.git
synced 2026-10-20 19:49:03 +02:00
mixin: group alerts by collector area and add alerts for default and opt-in collectors (#2777)
Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
@@ -0,0 +1,133 @@
|
||||
## Sample dashboards for Windows Exporter
|
||||
|
||||
[windows-exporter-dashboard.json](windows-exporter-dashboard.json) is a Grafana dashboard for hosts monitored by windows_exporter, with all tabs described below.
|
||||
Each tab is also a shorter dashboard of its own, such as [windows-exporter-cpu.json](windows-exporter-cpu.json) or [windows-exporter-fleet.json](windows-exporter-fleet.json).
|
||||
It was originally inspired by [this dashboard in Chinese](https://grafana.com/grafana/dashboards/10467-windows-exporter-for-prometheus-dashboard-cn-v20230531/)
|
||||
and takes several ideas from Grafana's [windows-observ-lib](https://github.com/grafana/jsonnet-libs/tree/master/windows-observ-lib).
|
||||
|
||||
### Requirements
|
||||
|
||||
- Grafana 13 or later with a Prometheus data source. The file uses the v2 dashboard schema, which Grafana needs for tabs.
|
||||
- windows_exporter with the default collectors (`cpu`, `logical_disk`, `memory`, `net`, `os`, `physical_disk`, `service`, `system`).
|
||||
The SMB, Processes, Scheduled tasks, Updates, GPU, Hyper-V, Failover Cluster and Time tabs, the MSSQL and Hyper-V Cluster dashboards, the TCP and UDP rows of the Network tab and the Drives and Storage Spaces rows of the Disk tab need the `smb` or `smbclient`, `process`, `scheduled_task`, `update`, `gpu`, `hyperv`, `mscluster`, `time`, `mssql`, `tcp`, `udp`, `diskdrive` and `storage_spaces` collectors, which are not enabled by default.
|
||||
The sample shows every tab; build the dashboard from the [mixin](../mixin#collectors) with your collectors to hide the others.
|
||||
The process names in the GPU tab need the `process` collector.
|
||||
GPU temperature, fan, power and clock panels require WDDM 2.4 or newer and driver support; unsupported sensors are absent.
|
||||
|
||||
Import the JSON files through **Dashboards > New > Import** and pick the Prometheus data source in the **Data source** variable.
|
||||
|
||||
| Dashboard | File | UID |
|
||||
|-----------|------|-----|
|
||||
| Windows Exporter / Full | [windows-exporter-dashboard.json](windows-exporter-dashboard.json) | `windows-exporter-full` |
|
||||
| Windows Exporter / `<tab>` | `windows-exporter-<tab>.json`, for example [windows-exporter-cpu.json](windows-exporter-cpu.json) | `windows-exporter-<tab>`, for example `windows-exporter-cpu` |
|
||||
| Windows Exporter / MSSQL | [windows-exporter-mssql.json](windows-exporter-mssql.json) | `windows-exporter-mssql` |
|
||||
| Windows Exporter / Hyper-V Cluster | [windows-exporter-hyper-v-cluster.json](windows-exporter-hyper-v-cluster.json) | `windows-exporter-hyper-v-cluster` |
|
||||
|
||||
The MSSQL dashboard is not a tab of the full dashboard. The [mixin](../mixin#metadata) adds it as a tab with `separateDashboardTabs: []`.
|
||||
The Hyper-V Cluster dashboard has tabs of its own, see [Hyper-V Cluster dashboard](#hyper-v-cluster-dashboard).
|
||||
|
||||
The tab dashboards show only the variables their panels use.
|
||||
The **Windows Exporter** dropdown in the links of the tab dashboards opens the other imported dashboards and keeps the time range and variables.
|
||||
Before these UIDs, the full dashboard used the UID `Kdaassddw`; importing the new file creates a second dashboard, so delete the old one afterwards.
|
||||
|
||||
### Variables
|
||||
|
||||
| Variable | Description |
|
||||
|-------------------|-----------------------------------------------------------------------------|
|
||||
| Data source | The Prometheus data source to query. |
|
||||
| Job | One or more scrape jobs. |
|
||||
| Hostname | Narrows the fleet overview and the instance list to the selected hostnames. |
|
||||
| Instance | The host shown in all tabs and dashboards except Fleet. |
|
||||
| Volume | Volumes shown in the Overview and Disk tabs. `HarddiskVolume*` are hidden. |
|
||||
| Network interface | Interfaces shown in the Overview and Network tabs. |
|
||||
| Process | One or more process names shown in the Processes tab, or All. |
|
||||
| Top N processes | Maximum processes per graph: 5, 10 (default), or 20. |
|
||||
| SQL Server instance | The SQL Server instance shown in the MSSQL dashboard. |
|
||||
| Database | One or more databases shown in the Databases row of the MSSQL dashboard. |
|
||||
|
||||
The **Reboots** annotation marks each reboot of the selected host on all graphs.
|
||||
|
||||
### Tabs
|
||||
|
||||
| Tab | Content |
|
||||
|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Fleet | Counts of hosts, hosts down and hosts above the CPU, memory and disk alert thresholds or with stopped automatic services; one row per host with OS version, uptime, CPU, memory, commit charge, C: and fullest volume usage and stopped automatic services; graphs of the 25 highest hosts, including the busiest interface per host. |
|
||||
| Overview | Summary of the selected host: operating system, uptime, CPUs, memory, commit charge, volumes, disk and network throughput. |
|
||||
| CPU | Utilization by mode and per core, processor queue length, context switches, interrupts, processes, threads, system calls and the effective CPU frequency. |
|
||||
| Memory | Used and available physical memory, commit charge, hard page faults, kernel pools and the system cache. |
|
||||
| Disk | Volume label, disk, file system, size and free space, usage, busy time, throughput, IOPS, latency and queue length; drive model, size, partitions and status; busy time, throughput, IOPS, latency, queue length and split I/O per physical disk, labelled with the drive model when `diskdrive` is enabled; health, size and usage of Storage Spaces pools and health, size, pool footprint and efficiency of their virtual disks. |
|
||||
| Network | Throughput, utilization, packets, errors and discards per interface; TCP connection states and rates, segments and retransmissions; UDP datagrams and errors. |
|
||||
| SMB | Open files, connections, traffic, requests and opened files per share this host serves; traffic, requests, latency, queue length, credit stalls and metadata requests per remote share this host uses. |
|
||||
| Services | Running, pending and disabled services, automatic services that are not running, and services that started or stopped in the time range. |
|
||||
| Processes | CPU usage, working set, private bytes, I/O throughput and operations, threads, handles and page faults per process, labelled with name and PID. |
|
||||
| Scheduled tasks | Task count, running, disabled and never-run tasks, enabled tasks with an unknown result code or missed runs, tasks that ran in the time range, and all tasks with state, last result and missed runs. |
|
||||
| Updates | Pending updates by severity and category, the oldest pending update, the time since the last Windows Update check, a list of pending updates and the query duration. |
|
||||
| GPU | Utilization by engine type, memory capacity, commitment and locality, temperature, fans, power, clocks, and top processes by GPU utilization and memory. |
|
||||
| Hyper-V | VM count and health, Hyper-V WMI health, VM memory share, slow virtual disks, a table of the host's VMs, host CPU time, CPU usage per VM and the busiest VMs, memory per VM, virtual disks and the slowest of them, VHD capacity, virtual switches, VM network adapters and dropped packets per VM. |
|
||||
| Failover Cluster | Nodes up and not up, failed and offline resource groups, failed resources, cluster networks not up and the quorum type; state, drain status, vote, owned resource groups and build of each node, node state over time and resource groups per node; resource groups and resources that are not online with their owner node; state, role and metric of each cluster network; size, free space and usage of the Cluster Shared Volumes. Every node reports the whole cluster, so any node of a cluster shows it. |
|
||||
| Time | Time zone, clock sync source, NTP time sources, clock offset and NTP round-trip delay. |
|
||||
| MSSQL | A separate dashboard for one SQL Server instance: failed mssql sub-collectors, connections, batch requests, compilations, logins, access methods and errors; wait time by wait class, lock waits, average lock wait, deadlocks and lock timeouts; server memory against its target, memory by consumer, page life expectancy, buffer pool I/O and memory grants; size, log usage, transactions, log flushes and log flush write time per database; tempdb free space, the version store and active transactions; send and redo queues and traffic of Always On availability replicas. |
|
||||
| Exporter | Scrape status and duration, exporter version and uptime, collector status, duration, failures and timeouts, and the exporter's CPU, memory, handles and Go runtime. |
|
||||
|
||||
Click a hostname in the Fleet tab to open that host in the Overview tab. In the Fleet dashboard, it opens the Overview dashboard.
|
||||
|
||||
### Hyper-V Cluster dashboard
|
||||
|
||||
The Hyper-V Cluster dashboard shows the Hyper-V hosts of one failover cluster. It needs the `hyperv` and `mscluster` collectors on all nodes,
|
||||
the `memory` collector for the memory shares and the `storage_spaces` collector for the Storage Spaces Direct row.
|
||||
The **Cluster** variable lists the clusters by the `cluster` label of the mscluster metrics, and the dashboard shows the instances that report that cluster.
|
||||
If your scrape configuration sets its own `cluster` target label, Prometheus renames the exporter's label to `exported_cluster` unless `honor_labels` is set; the variable then lists your label values instead.
|
||||
**Host** selects the host of the Host tab and **VM** the virtual machine of the VM tab, on whichever node runs it.
|
||||
|
||||
| Tab | Content |
|
||||
|----------|---------|
|
||||
| Overview | Nodes up, running VMs, VMs in critical health, virtual processors per logical processor, VM memory as a share of all host memory and of the memory left when the largest host fails, host CPU usage and slow virtual disks; running VMs, vCPU ratio, CPU usage and VM memory share per host; CPU usage, running VMs, VM memory and virtual disk IOPS per host over time. |
|
||||
| Cluster | The Failover Cluster tab for the selected cluster, without the Cluster Shared Volumes, which are in the Storage tab. |
|
||||
| Host | The Hyper-V tab for the selected host. |
|
||||
| VM | The host that runs the VM, virtual processors, assigned memory, RAM in use and committed memory, CPU usage by state and per virtual processor, dynamic memory changes, size, file size and fill of its virtual hard disks, disk IOPS, throughput and latency, and network throughput and dropped packets per adapter. |
|
||||
| VM disks | Count, size, file size and growth if thick of all virtual hard disks, differencing disks and slow disks; capacity by provisioning type; every disk with its VM, host and type; the slowest, busiest and highest-latency disks. |
|
||||
| Storage | Size and free space of the Cluster Shared Volumes, also if every dynamic virtual hard disk on them grew to its full size; usage per volume; virtual hard disks per folder below `C:\ClusterStorage`; health, size and usage of the Storage Spaces pools and health, size, footprint and efficiency of their virtual disks. |
|
||||
|
||||
Each per-host tab starts with a row of current values, each with a sparkline
|
||||
of the time range behind it. Green, orange and red show health against fixed
|
||||
thresholds; values without a meaningful threshold, like throughput or counts,
|
||||
stay in the text color.
|
||||
|
||||
The Processes tab starts with totals for all selected processes: process count,
|
||||
threads, handles, CPU usage across the host, private working set and private bytes.
|
||||
The synthetic Idle process (PID 0) is excluded. Each graph shows at most **Top N**
|
||||
processes, ranked separately at the end of the time range, with a fixed set of
|
||||
process IDs across the graph. Processes that exited before that point may not
|
||||
appear. I/O graphs combine read, write and other operations per process.
|
||||
|
||||
### Changing the dashboard
|
||||
|
||||
Don't edit the JSON files by hand. They are generated from Jsonnet with [grafonnet](https://github.com/grafana/grafonnet) as part of the [windows_exporter mixin](../mixin):
|
||||
|
||||
- [dashboards/windows-exporter.libsonnet](../mixin/dashboards/windows-exporter.libsonnet) composes the dashboards from one file per tab.
|
||||
- [dashboards/builders.libsonnet](../mixin/dashboards/builders.libsonnet) holds the panel, query, and layout builders.
|
||||
- [dashboards/styles.libsonnet](../mixin/dashboards/styles.libsonnet) shares the visualization defaults.
|
||||
|
||||
The generator writes the v2 schema directly, so it needs neither Grafana nor Prometheus.
|
||||
Install [mise](https://mise.jdx.dev/) and Go, then install the pinned tools and
|
||||
regenerate the files in `contrib/mixin` (use WSL on Windows):
|
||||
|
||||
```sh
|
||||
cd contrib/mixin
|
||||
mise trust
|
||||
mise install
|
||||
mise run dashboard
|
||||
```
|
||||
|
||||
`mise run dashboard` runs `jb install` to fetch grafonnet into `contrib/mixin/vendor` and renders the dashboards.
|
||||
`mise run lint` fails if the committed JSON files differ from the generated ones.
|
||||
`mise run check-dashboards` rebuilds them and runs `git diff --exit-code`, as CI does before the mixin tests.
|
||||
Check new or changed queries in Grafana before you open a pull request, for example by importing the files into a local Grafana 13.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 299 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 565 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 162 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,943 @@
|
||||
{
|
||||
"apiVersion": "dashboard.grafana.app/v2",
|
||||
"kind": "Dashboard",
|
||||
"metadata": {
|
||||
"name": "windows-exporter-services"
|
||||
},
|
||||
"spec": {
|
||||
"annotations": [
|
||||
{
|
||||
"kind": "AnnotationQuery",
|
||||
"spec": {
|
||||
"builtIn": true,
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"legacyOptions": {
|
||||
"type": "dashboard"
|
||||
},
|
||||
"name": "Annotations & Alerts",
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "-- Grafana --"
|
||||
},
|
||||
"group": "grafana",
|
||||
"kind": "DataQuery",
|
||||
"spec": {},
|
||||
"version": "v0"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "AnnotationQuery",
|
||||
"spec": {
|
||||
"enable": true,
|
||||
"hide": false,
|
||||
"iconColor": "orange",
|
||||
"legacyOptions": {
|
||||
"expr": "windows_system_boot_time_timestamp{job=~\"$job\", instance=\"$instance\"} * 1000 > $__from < $__to",
|
||||
"step": "1m",
|
||||
"tagKeys": "instance",
|
||||
"textFormat": "",
|
||||
"titleFormat": "Reboot",
|
||||
"useValueForTime": "on"
|
||||
},
|
||||
"name": "Reboots",
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {},
|
||||
"version": "v0"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"cursorSync": "Crosshair",
|
||||
"description": "Service states and automatic services that are not running on one Windows host.",
|
||||
"editable": true,
|
||||
"elements": {
|
||||
"panel-50": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "sum(windows_service_state{job=~\"$job\", instance=\"$instance\", state=\"running\"})",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {
|
||||
"maxDataPoints": 100
|
||||
},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Number of services currently in the running state, reported by the service collector. Compare unexpected drops with the stopped automatic services table and service change history.",
|
||||
"id": 50,
|
||||
"links": [],
|
||||
"title": "Running services",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "text",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-51": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "count(windows_service_state{job=~\"$job\", instance=\"$instance\", state=\"running\"} == 0 and on (service) windows_service_start_mode{job=~\"$job\", instance=\"$instance\", start_mode=\"auto\"} == 1) or vector(0)",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {
|
||||
"maxDataPoints": 100
|
||||
},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Services with start mode \"auto\" that are not running.",
|
||||
"id": 51,
|
||||
"links": [],
|
||||
"title": "Stopped auto services",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-52": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "sum(windows_service_state{job=~\"$job\", instance=\"$instance\", state=~\".* pending\"}) or vector(0)",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {
|
||||
"maxDataPoints": 100
|
||||
},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Services in a start, stop, pause or continue pending state. A service that stays pending is hung.",
|
||||
"id": 52,
|
||||
"links": [],
|
||||
"title": "Pending services",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-53": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "sum(windows_service_start_mode{job=~\"$job\", instance=\"$instance\", start_mode=\"disabled\"})",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {
|
||||
"maxDataPoints": 100
|
||||
},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Number of services configured with disabled startup. A disabled service cannot start until its startup mode changes; disabled services are not necessarily unhealthy.",
|
||||
"id": 53,
|
||||
"links": [],
|
||||
"title": "Disabled services",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "text",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-54": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "max by (service, state) (windows_service_state{job=~\"$job\", instance=\"$instance\"} == 1) and on (service) (windows_service_state{job=~\"$job\", instance=\"$instance\", state=\"running\"} == 0) and on (service) (windows_service_start_mode{job=~\"$job\", instance=\"$instance\", start_mode=\"auto\"} == 1)",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"legendFormat": "__auto",
|
||||
"range": false
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": [
|
||||
{
|
||||
"group": "organize",
|
||||
"kind": "Transformation",
|
||||
"spec": {
|
||||
"options": {
|
||||
"excludeByName": {
|
||||
"Time": true,
|
||||
"Time 1": true,
|
||||
"Value": true
|
||||
},
|
||||
"includeByName": {},
|
||||
"indexByName": {
|
||||
"service": 0,
|
||||
"state": 1
|
||||
},
|
||||
"renameByName": {
|
||||
"service": "Service",
|
||||
"state": "State"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"description": "Services configured to start automatically that are not in the running state. Delayed-start and trigger-start services may show up here until they start.",
|
||||
"id": 54,
|
||||
"links": [],
|
||||
"title": "Automatic services not running",
|
||||
"vizConfig": {
|
||||
"group": "table",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"filterable": false,
|
||||
"inspect": false,
|
||||
"minWidth": 60
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"enablePagination": false,
|
||||
"footer": {
|
||||
"countRows": false,
|
||||
"fields": "",
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"show": false
|
||||
},
|
||||
"showHeader": true,
|
||||
"sortBy": [
|
||||
{
|
||||
"desc": false,
|
||||
"displayName": "Service"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-55": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_service_state{job=~\"$job\", instance=\"$instance\", state=\"running\"} and on (service) (changes(windows_service_state{job=~\"$job\", instance=\"$instance\", state=\"running\"}[$__range] @ end()) > 0)",
|
||||
"instant": false,
|
||||
"legendFormat": "{{service}}",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Services that started or stopped in the selected time range.",
|
||||
"id": 55,
|
||||
"links": [],
|
||||
"title": "Service state changes",
|
||||
"vizConfig": {
|
||||
"group": "state-timeline",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"custom": {
|
||||
"fillOpacity": 90,
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineWidth": 0,
|
||||
"spanNulls": false
|
||||
},
|
||||
"mappings": [
|
||||
{
|
||||
"options": {
|
||||
"0": {
|
||||
"color": "red",
|
||||
"index": 0,
|
||||
"text": "not running"
|
||||
},
|
||||
"1": {
|
||||
"color": "green",
|
||||
"index": 1,
|
||||
"text": "running"
|
||||
}
|
||||
},
|
||||
"type": "value"
|
||||
}
|
||||
],
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"alignValue": "left",
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": false
|
||||
},
|
||||
"mergeValues": false,
|
||||
"rowHeight": 1,
|
||||
"showValue": "never",
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "single",
|
||||
"sort": "none"
|
||||
}
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"layout": {
|
||||
"kind": "RowsLayout",
|
||||
"spec": {
|
||||
"rows": [
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
"collapse": false,
|
||||
"layout": {
|
||||
"kind": "GridLayout",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-50"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-51"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-52"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-53"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"title": "Summary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
"collapse": false,
|
||||
"layout": {
|
||||
"kind": "GridLayout",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-54"
|
||||
},
|
||||
"height": 10,
|
||||
"width": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-55"
|
||||
},
|
||||
"height": 10,
|
||||
"width": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"title": "State"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"links": [
|
||||
{
|
||||
"asDropdown": true,
|
||||
"icon": "external link",
|
||||
"includeVars": true,
|
||||
"keepTime": true,
|
||||
"tags": [
|
||||
"prometheus",
|
||||
"windows",
|
||||
"windows_exporter"
|
||||
],
|
||||
"targetBlank": false,
|
||||
"title": "Windows Exporter",
|
||||
"tooltip": "Dashboards for windows_exporter",
|
||||
"type": "dashboards",
|
||||
"url": ""
|
||||
},
|
||||
{
|
||||
"asDropdown": false,
|
||||
"icon": "doc",
|
||||
"includeVars": false,
|
||||
"keepTime": false,
|
||||
"tags": [],
|
||||
"targetBlank": true,
|
||||
"title": "windows_exporter",
|
||||
"tooltip": "",
|
||||
"type": "link",
|
||||
"url": "https://github.com/prometheus-community/windows_exporter"
|
||||
}
|
||||
],
|
||||
"liveNow": false,
|
||||
"preload": false,
|
||||
"tags": [
|
||||
"prometheus",
|
||||
"windows",
|
||||
"windows_exporter"
|
||||
],
|
||||
"timeSettings": {
|
||||
"autoRefresh": "1m",
|
||||
"autoRefreshIntervals": [
|
||||
"5s",
|
||||
"10s",
|
||||
"30s",
|
||||
"1m",
|
||||
"5m",
|
||||
"15m",
|
||||
"30m",
|
||||
"1h",
|
||||
"2h",
|
||||
"1d"
|
||||
],
|
||||
"fiscalYearStartMonth": 0,
|
||||
"from": "now-3h",
|
||||
"hideTimepicker": false,
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Windows Exporter / Services",
|
||||
"variables": [
|
||||
{
|
||||
"kind": "DatasourceVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"hide": "dontHide",
|
||||
"includeAll": false,
|
||||
"label": "Data source",
|
||||
"multi": false,
|
||||
"name": "datasource",
|
||||
"options": [],
|
||||
"pluginId": "prometheus",
|
||||
"refresh": "onDashboardLoad",
|
||||
"regex": "",
|
||||
"skipUrlSync": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname, job)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": true,
|
||||
"label": "Job",
|
||||
"multi": true,
|
||||
"name": "job",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname, job)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname{job=~\"$job\"}, hostname)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": true,
|
||||
"label": "Hostname",
|
||||
"multi": true,
|
||||
"name": "hostname",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname{job=~\"$job\"}, hostname)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname{job=~\"$job\", hostname=~\"$hostname\"}, instance)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": false,
|
||||
"label": "Instance",
|
||||
"multi": false,
|
||||
"name": "instance",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname{job=~\"$job\", hostname=~\"$hostname\"}, instance)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,833 @@
|
||||
{
|
||||
"apiVersion": "dashboard.grafana.app/v2",
|
||||
"kind": "Dashboard",
|
||||
"metadata": {
|
||||
"name": "windows-exporter-time"
|
||||
},
|
||||
"spec": {
|
||||
"annotations": [
|
||||
{
|
||||
"kind": "AnnotationQuery",
|
||||
"spec": {
|
||||
"builtIn": true,
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"legacyOptions": {
|
||||
"type": "dashboard"
|
||||
},
|
||||
"name": "Annotations & Alerts",
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "-- Grafana --"
|
||||
},
|
||||
"group": "grafana",
|
||||
"kind": "DataQuery",
|
||||
"spec": {},
|
||||
"version": "v0"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "AnnotationQuery",
|
||||
"spec": {
|
||||
"enable": true,
|
||||
"hide": false,
|
||||
"iconColor": "orange",
|
||||
"legacyOptions": {
|
||||
"expr": "windows_system_boot_time_timestamp{job=~\"$job\", instance=\"$instance\"} * 1000 > $__from < $__to",
|
||||
"step": "1m",
|
||||
"tagKeys": "instance",
|
||||
"textFormat": "",
|
||||
"titleFormat": "Reboot",
|
||||
"useValueForTime": "on"
|
||||
},
|
||||
"name": "Reboots",
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {},
|
||||
"version": "v0"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"cursorSync": "Crosshair",
|
||||
"description": "Clock synchronization, NTP sources and clock offset of one Windows host.",
|
||||
"editable": true,
|
||||
"elements": {
|
||||
"panel-84": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_time_timezone{job=~\"$job\", instance=\"$instance\"}",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"legendFormat": "__auto",
|
||||
"range": false
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Needs the time collector, which is not enabled by default.",
|
||||
"id": 84,
|
||||
"links": [],
|
||||
"title": "Time zone",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": ""
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "none",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "/^timezone$/",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-85": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_time_clock_sync_source{job=~\"$job\", instance=\"$instance\"} == 1",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"legendFormat": "__auto",
|
||||
"range": false
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Active Windows Time synchronization provider. A non-NTP provider can be expected on virtual machines; this requires the optional time collector.",
|
||||
"id": 85,
|
||||
"links": [],
|
||||
"title": "Clock sync source",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": ""
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "none",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "/^type$/",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "value",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-86": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_time_ntp_client_time_sources{job=~\"$job\", instance=\"$instance\"}",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {
|
||||
"maxDataPoints": 100
|
||||
},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Number of time sources the NTP client uses. 0 means the clock is not synchronized.",
|
||||
"id": 86,
|
||||
"links": [],
|
||||
"title": "NTP time sources",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-87": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "abs(windows_time_computed_time_offset_seconds{job=~\"$job\", instance=\"$instance\"})",
|
||||
"instant": false,
|
||||
"legendFormat": "__auto",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Absolute offset between the system clock and the selected time source.",
|
||||
"id": 87,
|
||||
"links": [],
|
||||
"title": "Clock offset",
|
||||
"vizConfig": {
|
||||
"group": "stat",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 0.5
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"panel-88": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"data": {
|
||||
"kind": "QueryGroup",
|
||||
"spec": {
|
||||
"queries": [
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_time_computed_time_offset_seconds{job=~\"$job\", instance=\"$instance\"}",
|
||||
"instant": false,
|
||||
"legendFormat": "clock offset",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "A"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "PanelQuery",
|
||||
"spec": {
|
||||
"hidden": false,
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"editorMode": "code",
|
||||
"expr": "windows_time_ntp_round_trip_delay_seconds{job=~\"$job\", instance=\"$instance\"}",
|
||||
"instant": false,
|
||||
"legendFormat": "NTP round-trip delay",
|
||||
"range": true
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refId": "B"
|
||||
}
|
||||
}
|
||||
],
|
||||
"queryOptions": {},
|
||||
"transformations": []
|
||||
}
|
||||
},
|
||||
"description": "Signed offset from the selected time source and the NTP request round-trip delay, in seconds. Persistent offset or increased delay can affect time-sensitive applications; this compares the host to its chosen source and does not independently verify that source.",
|
||||
"id": 88,
|
||||
"links": [],
|
||||
"title": "Clock offset and NTP round-trip delay",
|
||||
"vizConfig": {
|
||||
"group": "timeseries",
|
||||
"kind": "VizConfig",
|
||||
"spec": {
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 4,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max",
|
||||
"lastNotNull"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
}
|
||||
},
|
||||
"version": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"layout": {
|
||||
"kind": "RowsLayout",
|
||||
"spec": {
|
||||
"rows": [
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
"collapse": false,
|
||||
"layout": {
|
||||
"kind": "GridLayout",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-84"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-85"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-86"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-87"
|
||||
},
|
||||
"height": 4,
|
||||
"width": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"title": "Summary"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "RowsLayoutRow",
|
||||
"spec": {
|
||||
"collapse": false,
|
||||
"layout": {
|
||||
"kind": "GridLayout",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"kind": "GridLayoutItem",
|
||||
"spec": {
|
||||
"element": {
|
||||
"kind": "ElementReference",
|
||||
"name": "panel-88"
|
||||
},
|
||||
"height": 8,
|
||||
"width": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"title": "Clock"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"links": [
|
||||
{
|
||||
"asDropdown": true,
|
||||
"icon": "external link",
|
||||
"includeVars": true,
|
||||
"keepTime": true,
|
||||
"tags": [
|
||||
"prometheus",
|
||||
"windows",
|
||||
"windows_exporter"
|
||||
],
|
||||
"targetBlank": false,
|
||||
"title": "Windows Exporter",
|
||||
"tooltip": "Dashboards for windows_exporter",
|
||||
"type": "dashboards",
|
||||
"url": ""
|
||||
},
|
||||
{
|
||||
"asDropdown": false,
|
||||
"icon": "doc",
|
||||
"includeVars": false,
|
||||
"keepTime": false,
|
||||
"tags": [],
|
||||
"targetBlank": true,
|
||||
"title": "windows_exporter",
|
||||
"tooltip": "",
|
||||
"type": "link",
|
||||
"url": "https://github.com/prometheus-community/windows_exporter"
|
||||
}
|
||||
],
|
||||
"liveNow": false,
|
||||
"preload": false,
|
||||
"tags": [
|
||||
"prometheus",
|
||||
"windows",
|
||||
"windows_exporter"
|
||||
],
|
||||
"timeSettings": {
|
||||
"autoRefresh": "1m",
|
||||
"autoRefreshIntervals": [
|
||||
"5s",
|
||||
"10s",
|
||||
"30s",
|
||||
"1m",
|
||||
"5m",
|
||||
"15m",
|
||||
"30m",
|
||||
"1h",
|
||||
"2h",
|
||||
"1d"
|
||||
],
|
||||
"fiscalYearStartMonth": 0,
|
||||
"from": "now-3h",
|
||||
"hideTimepicker": false,
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Windows Exporter / Time",
|
||||
"variables": [
|
||||
{
|
||||
"kind": "DatasourceVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"hide": "dontHide",
|
||||
"includeAll": false,
|
||||
"label": "Data source",
|
||||
"multi": false,
|
||||
"name": "datasource",
|
||||
"options": [],
|
||||
"pluginId": "prometheus",
|
||||
"refresh": "onDashboardLoad",
|
||||
"regex": "",
|
||||
"skipUrlSync": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname, job)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": true,
|
||||
"label": "Job",
|
||||
"multi": true,
|
||||
"name": "job",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname, job)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname{job=~\"$job\"}, hostname)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": true,
|
||||
"label": "Hostname",
|
||||
"multi": true,
|
||||
"name": "hostname",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname{job=~\"$job\"}, hostname)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "QueryVariable",
|
||||
"spec": {
|
||||
"allowCustomValue": true,
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"definition": "label_values(windows_os_hostname{job=~\"$job\", hostname=~\"$hostname\"}, instance)",
|
||||
"hide": "dontHide",
|
||||
"includeAll": false,
|
||||
"label": "Instance",
|
||||
"multi": false,
|
||||
"name": "instance",
|
||||
"options": [],
|
||||
"query": {
|
||||
"datasource": {
|
||||
"name": "${datasource}"
|
||||
},
|
||||
"group": "prometheus",
|
||||
"kind": "DataQuery",
|
||||
"spec": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(windows_os_hostname{job=~\"$job\", hostname=~\"$hostname\"}, instance)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"version": "v0"
|
||||
},
|
||||
"refresh": "onTimeRangeChanged",
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": "alphabeticalAsc"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,5 +1,3 @@
|
||||
/vendor/
|
||||
/windows_alerts.yaml
|
||||
/windows_rules.yaml
|
||||
/tests/*_alerts.yaml
|
||||
/tests/*_rules.yaml
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Mixin instructions
|
||||
|
||||
Read the root [`AGENTS.md`](../../AGENTS.md) first. This file covers
|
||||
`contrib/mixin`: Jsonnet recording rules, alerts and dashboard sources.
|
||||
[`README.md`](README.md) documents the configuration and the generated rules;
|
||||
keep it in sync with every change.
|
||||
|
||||
## Alerts
|
||||
|
||||
`alerts/` holds one file per collector, each rendering one rule group, and
|
||||
`alerts/alerts.libsonnet` imports them in the order of the generated file.
|
||||
A rule is a plain Prometheus alerting rule, like in kubernetes-mixin:
|
||||
|
||||
```jsonnet
|
||||
{
|
||||
alert: 'WindowsCPUHighUsage',
|
||||
expr: |||
|
||||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: { summary: '...', description: '...' },
|
||||
}
|
||||
```
|
||||
|
||||
Write the PromQL out in the `expr` block with `%(name)s` placeholders for the
|
||||
`_config` values; selectors go inside the braces as
|
||||
`{%(windowsExporterSelector)s, %(volumeSelector)s}`. `lib/alerts.libsonnet`
|
||||
provides `a.group(name, collectors, rules)`, which keeps a rule only when every
|
||||
collector in `collectors` is enabled in `_config.collectors` and the alert is
|
||||
not switched off in `_config.alerts`, and adds `keep_firing_for` from
|
||||
`keepFiringFor`. Two hidden fields change that per rule: `collectors:: [...]`
|
||||
when a rule reads other collectors than its group (the CPU saturation alert
|
||||
reads `cpu` and `system`), and `state:: true` for state alerts (`== 0`,
|
||||
`== 1`) and one-shot alerts, which must resolve at once and get no
|
||||
`keep_firing_for`. A one-shot alert has no `for`.
|
||||
|
||||
A new alert touches six places, and CI checks all of them:
|
||||
|
||||
1. The rule in `alerts/<collector>.libsonnet`, with `{{ $labels.<key> }}`
|
||||
templates that use the 0.32 label keys (`service`, `volume`, `nic`,
|
||||
`task`, `device_id`, never `name`). Alerts for a new collector go in a new
|
||||
file, imported from `alerts/alerts.libsonnet`, with the collector added to
|
||||
`_config.collectors` in `config.libsonnet`; the group expectations in
|
||||
`tests/alerts.test.jsonnet` list every group.
|
||||
2. Its switch in `_config.alerts` in `config.libsonnet`; `tests/alerts.test.jsonnet`
|
||||
fails when the map and the generated alerts differ. Thresholds are
|
||||
`_config` knobs named `<area><Condition>Threshold[Unit]`.
|
||||
3. A unit test in `tests/alerts.test.yaml` (`tests/optional.test.yaml` for
|
||||
opt-in collectors, which `tests/config.libsonnet` enables) that checks the
|
||||
first firing evaluation, the labels and annotations, and the recovery. With
|
||||
`keep_firing_for` a threshold alert resolves `keepFiringFor` after its
|
||||
condition clears. `rate()` extrapolates while a window has few samples, so
|
||||
ratios of two rates activate earlier than a single rate; probe the
|
||||
activation time with `ALERTS_FOR_STATE` instead of guessing it.
|
||||
4. A runbook `docs/runbooks/<lowercase-alert-name>.md` with the `Meaning`,
|
||||
`Impact`, `Diagnosis` and `Mitigation` sections.
|
||||
`go test ./internal/metriccontract/` checks every `windows_*` metric name
|
||||
a runbook mentions against the metrics the exporter emits; the names in
|
||||
the collector docs are not always current, so check the collector source.
|
||||
5. A row in the alerts table of `README.md`.
|
||||
6. The committed `contrib/monitoring/windows_rules.yaml` and
|
||||
`windows_alerts.yaml`, regenerated
|
||||
with `mise run rules`; `mise run check-rules` fails in CI when they are
|
||||
stale.
|
||||
|
||||
Metrics with a state label (`windows_service_state{state="running"} == 0`)
|
||||
keep that label in the alert; drop it with `max without (state) (...)` so the
|
||||
alert does not say `state="running"` for a stopped service. Join a state metric
|
||||
with its info metric with `ignoring (<labels of the info metric>)`, never
|
||||
`on (job, instance, <key>)`: `on` drops the other target labels, so two
|
||||
targets with the same `instance` in different clusters match each other.
|
||||
A target can be scraped again after a gap (a reboot, a restart of the
|
||||
exporter), so an `offset` lookup needs a range (`last_over_time(m[1h] offset
|
||||
10m)`); a plain `m offset 10m` has no sample while the gap covers that point.
|
||||
|
||||
## Recording rules
|
||||
|
||||
New recordings go in `rules/rules.libsonnet` with the shared expressions in
|
||||
`lib/queries.libsonnet`, a row in the `README.md` table and a test in
|
||||
`tests/rules.test.yaml`. Use `q.ratio` for divisions so idle devices and zero
|
||||
capacities produce no sample instead of `NaN` or `+Inf`.
|
||||
|
||||
## Checks
|
||||
|
||||
`mise run test` runs everything CI runs. Without mise, the tools from
|
||||
`mise.toml` (jsonnet, jb, promtool, pint, dashboard-linter) and the commands of
|
||||
the `lint`, `check-runbooks` and `test` tasks do the same.
|
||||
+161
-52
@@ -23,16 +23,23 @@ Mise installs release binaries with checksum verification and builds Jsonnet
|
||||
Bundler using Go. `mise run` makes the pinned tools available automatically.
|
||||
|
||||
Only the dashboard depends on Jsonnet libraries, so `jb install` is not required
|
||||
to generate rules. Generated rule files are ignored by Git:
|
||||
to generate rules. The rules rendered from the default configuration are
|
||||
committed in [`contrib/monitoring`](../monitoring), so they can be used
|
||||
without Jsonnet:
|
||||
|
||||
- `windows_rules.yaml`: recording rules.
|
||||
- `windows_alerts.yaml`: alerting rules.
|
||||
|
||||
`mise run rules` regenerates them after a source change, and CI fails when the
|
||||
committed files are out of date (`mise run check-rules`).
|
||||
|
||||
The default selector is `job="windows_exporter"`. Change it to match your scrape
|
||||
job before generating rules. Resource rules use the default `cpu`, `memory`,
|
||||
`logical_disk`, and `net` collectors. Enable `ad` on domain controllers when
|
||||
opting into Active Directory alerts. Clock alerts require the `time` collector
|
||||
with its `ntp` subcollector and are intended for hosts using NTP synchronization.
|
||||
job before generating rules. The rules read the default `cpu`, `memory`,
|
||||
`logical_disk`, `net`, `service` and `system` collectors; `_config.collectors`
|
||||
lists which collectors the targets run, and the alerts of a collector that is
|
||||
turned off are not generated. Enable `ad` on domain controllers for the Active
|
||||
Directory alerts and `time` for the clock alerts, which need the `ntp` and
|
||||
`clock_source` subcollectors.
|
||||
|
||||
Copy both generated files into your Prometheus rule directory and reference
|
||||
them in `prometheus.yml`:
|
||||
@@ -60,8 +67,9 @@ modifying its source. For example, create `custom.libsonnet` in this directory:
|
||||
nicSelector: 'nic!="Loopback"',
|
||||
rateInterval: '5m',
|
||||
cpuHighUsageThreshold: 95,
|
||||
enableActiveDirectory: true,
|
||||
enableTime: true,
|
||||
collectors+: { ad: true, time: true, process: true },
|
||||
alerts+: { WindowsHostRebooted: false },
|
||||
keepFiringFor: '10m',
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
},
|
||||
}
|
||||
@@ -77,9 +85,14 @@ jsonnet -S -e 'std.manifestYamlDoc((import "custom.libsonnet").prometheusAlerts)
|
||||
Selectors are comma-separated Prometheus label matchers without braces. They
|
||||
apply to both recording rules and alerts. All target labels, including `job`,
|
||||
`instance`, `cluster`, and custom selector labels, are preserved. CPU usage
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk and
|
||||
network recordings also retain `volume` and `nic` respectively. The CPU mode
|
||||
recording retains `mode` and removes only `core`.
|
||||
averages idle rates across cores and removes only `core` and `mode`. Disk,
|
||||
network and service rules also retain `volume`, `nic` and `service`
|
||||
respectively, filtered by `volumeSelector`, `nicSelector` and
|
||||
`serviceSelector`. The scheduled task and cluster shared volume alerts
|
||||
filter their `task` and `shared_volume` labels with `taskSelector`
|
||||
(`task!=""`) and `sharedVolumeSelector`
|
||||
(`shared_volume!="ClusterPerformanceHistory"`). The CPU mode recording retains `mode` and removes only
|
||||
`core`.
|
||||
|
||||
`rateInterval` is the fixed window for recording rules and Active Directory
|
||||
failure rates; keep it at least four times your scrape interval. Grafana's
|
||||
@@ -93,6 +106,7 @@ rules.
|
||||
| `windows:cpu_usage:ratio` | Non-idle CPU time averaged across cores | Ratio, 0–1 |
|
||||
| `windows:cpu_cores:count` | Logical processors observed in idle CPU metrics | Count |
|
||||
| `windows:cpu_time:ratio_rate` | CPU time averaged across cores, retaining `mode` | Seconds/second |
|
||||
| `windows:cpu_queue_per_core:ratio` | Threads waiting for a processor, averaged over `rateInterval`, per logical processor | Threads/processor |
|
||||
| `windows:memory_usage:ratio` | Physical memory in use, based on available bytes | Ratio, 0–1 |
|
||||
| `windows:memory_used:bytes` | Total physical memory minus available memory | Bytes |
|
||||
| `windows:memory_cached:bytes` | Cache, modified pages, and three standby lists | Bytes |
|
||||
@@ -109,6 +123,8 @@ rules.
|
||||
| `windows:logical_disk_write_latency:seconds` | Write-time rate divided by write-operation rate | Seconds/operation |
|
||||
| `windows:net_received_bytes:rate` | Received network traffic per interface | Bytes/second |
|
||||
| `windows:net_sent_bytes:rate` | Sent network traffic per interface | Bytes/second |
|
||||
| `windows:net_received_packets:rate` | Received packets per interface | Packets/second |
|
||||
| `windows:net_sent_packets:rate` | Sent packets per interface | Packets/second |
|
||||
| `windows:net_received_utilization:ratio` | Received throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_sent_utilization:ratio` | Sent throughput divided by interface bandwidth | Ratio |
|
||||
| `windows:net_received_errors:rate` | Inbound packet errors | Packets/second |
|
||||
@@ -127,46 +143,137 @@ which can exceed wall time for concurrent I/O. Latency is an average and is
|
||||
omitted when there are no operations. Memory, disk space, commit, and network
|
||||
utilization ratios are omitted when their capacity is zero or missing. Network
|
||||
bandwidth is already in bytes/second; receive and send utilization are separate
|
||||
for full-duplex links.
|
||||
for full-duplex links. The processor queue length is an instantaneous sample,
|
||||
so the queue recording averages it over `rateInterval` before dividing by the
|
||||
core count.
|
||||
The cache recording includes modified pages and is not a measure of immediately
|
||||
reclaimable memory. Paging operations can include file-backed pages as well as
|
||||
pagefile traffic.
|
||||
|
||||
## Alerts
|
||||
|
||||
Threshold configuration values below are percentages except where noted.
|
||||
The alerts come in one rule group per collector, from one file per collector
|
||||
in `alerts/` (`alerts/cpu.libsonnet`, `alerts/service.libsonnet`, and so on).
|
||||
Each group lists the collectors its rules read; a group is generated only with
|
||||
those collectors enabled in `_config.collectors`, and a rule that needs two
|
||||
collectors, such as the CPU saturation alert, only with both. Threshold configuration values below are
|
||||
percentages except where noted.
|
||||
|
||||
| Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- |
|
||||
| `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `WindowsNetworkErrors` | Combined receive/outbound errors above `networkErrorRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsNetworkDiscards` | Combined receive/outbound discards above `networkDiscardRateThreshold` (1 packet/second) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
| Group | Collectors | Alert | Default condition | Duration | Severity |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `windows-exporter` | | `WindowsExporterDown` | Scrape target `up == 0` | 5m | Critical |
|
||||
| `windows-exporter` | | `WindowsCollectorFailed` | `windows_exporter_collector_success == 0` | 5m | Warning |
|
||||
| `windows-exporter` | | `WindowsCollectorTimeout` | `windows_exporter_collector_timeout == 1` | 10m | Warning |
|
||||
| `windows-exporter` | | `WindowsScrapeQueueTimeout` | `windows_exporter_scrape_queue_timeout == 1` | 10m | Warning |
|
||||
| `windows-cpu` | `cpu` | `WindowsCPUHighUsage` | CPU usage above `cpuHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-cpu` | `cpu`, `system` | `WindowsCPUSaturation` | Waiting threads per logical processor above `cpuQueuePerCoreThreshold` (2) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryHighUsage` | Memory usage above `memoryHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryCommitHighUsage` | Commit usage above `memoryCommitHighUsageThreshold` (90%) | 15m | Warning |
|
||||
| `windows-memory` | `memory` | `WindowsMemoryPagingHigh` | Pages read or written per second above `memoryPagingThreshold` (1000) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskReadLatencyHigh` | Average read latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskWriteLatencyHigh` | Average write latency above `diskLatencyThresholdSeconds` (0.05 seconds) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `windows-disk` | `logical_disk` | `WindowsDiskFillingUp` | Below the warning threshold and predicted to fill within `diskPredictionHours` (24h), using `diskPredictionWindow` (6h) | 1h | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkErrors` | Receive and outbound errors above `networkErrorRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkDiscards` | Receive and outbound discards above `networkDiscardRatioThreshold` (1%) of all packets | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkUtilizationHigh` | Received or sent throughput above `networkUtilizationThreshold` (90%) of the link bandwidth | 15m | Warning |
|
||||
| `windows-network` | `net` | `WindowsNetworkInterfaceFlapping` | Operational status changed more than twice within `rateInterval` | 2m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceNotRunning` | Automatic service without triggers not running | 15m | Warning |
|
||||
| `windows-service` | `service` | `WindowsServiceCrashLooping` | More than `serviceRestartThreshold` (2) restarts in 15 minutes | 0m | Warning |
|
||||
| `windows-system` | `system` | `WindowsHostRebooted` | Boot time less than ten minutes old and changed | 0m | Info |
|
||||
| `windows-active-directory` | `ad` | `WindowsADPendingReplication` | Pending operations above `adPendingReplicationThreshold` (50) | 15m | Warning |
|
||||
| `windows-active-directory` | `ad` | `WindowsADReplicationFailures` | Failed synchronization requests above `adReplicationFailureRateThreshold` (0 requests/second) | 15m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockNotSynchronising` | No active NTP time sources | 10m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockSkewDetected` | Absolute reported time offset above `clockOffsetThresholdSeconds` (0.05 seconds) | 10m | Warning |
|
||||
| `windows-time` | `time` | `WindowsClockSourceLocal` | Clock source is `Local CMOS Clock` or `NoSync` | 10m | Warning |
|
||||
| `windows-pagefile` | `pagefile` | `WindowsPagefileAlmostFull` | Page file usage above `pagefileUsageThreshold` (80%) | 15m | Warning |
|
||||
| `windows-diskdrive` | `diskdrive` | `WindowsDiskDriveNotOK` | Drive status other than `OK` or `Unknown` | 5m | Critical |
|
||||
| `windows-license` | `license` | `WindowsLicenseNotGenuine` | License status is not `genuine` | 1h | Warning |
|
||||
| `windows-scheduled-task` | `scheduled_task` | `WindowsScheduledTaskFailed` | Last result of an enabled task matching `taskSelector` is `unknown` or `terminated` | 1h | Warning |
|
||||
| `windows-update` | `update` | `WindowsUpdatesPending` | Updates waiting for installation | `updatePendingDuration` (7d) | Info |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVWMIUnhealthy` | Hyper-V WMI namespace not responding | 10m | Warning |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationCritical` | Replication health of a virtual machine is critical | 5m | Critical |
|
||||
| `windows-hyperv` | `hyperv` | `WindowsHyperVReplicationWarning` | Replication health of a virtual machine is warning | 15m | Warning |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterNodeDown` | Node state is down | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterResourceFailed` | Resource state is failed | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterResourceGroupFailed` | Resource group state is failed | 5m | Critical |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space of a volume matching `sharedVolumeSelector` below `diskFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-cluster` | `mscluster` | `WindowsClusterSharedVolumeAlmostFull` | Free space below `diskFreeCriticalThreshold` (5%) | 15m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is unhealthy | 10m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStoragePoolUnhealthy` | Health of a pool that is not primordial is warning | 10m | Warning |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is unhealthy | 10m | Critical |
|
||||
| `windows-storage-spaces` | `storage_spaces` | `WindowsStorageVirtualDiskUnhealthy` | Health of a virtual disk is warning | 10m | Warning |
|
||||
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeWarningThreshold` (10%) | 15m | Warning |
|
||||
| `windows-dhcp` | `dhcp` | `WindowsDHCPScopeAlmostExhausted` | Free addresses of an active scope below `dhcpScopeFreeCriticalThreshold` (2%) | 15m | Critical |
|
||||
| `windows-dns` | `dns` | `WindowsDNSZoneTransferFailures` | Zone transfer failure rate above `dnsZoneTransferFailureRateThreshold` (0 per second) | 15m | Warning |
|
||||
|
||||
Active Directory alerts are generated only when `enableActiveDirectory` is
|
||||
`true`. `criticalSeverity` defaults to `critical` and can be overridden to match
|
||||
your routing labels. Below the critical disk threshold, both disk severity
|
||||
levels fire; configure Alertmanager inhibition if only the critical
|
||||
`_config.alerts` lists every alert by name. Set one to `false` to leave it out,
|
||||
for example `alerts+: { WindowsHostRebooted: false }`; an unknown name fails the
|
||||
build, and a name missing from a replaced object stays enabled. A group whose
|
||||
alerts are all turned off is not generated.
|
||||
|
||||
Threshold alerts set `keep_firing_for` to `keepFiringFor` (5 minutes), so a
|
||||
value that oscillates around a threshold does not flap. The state alerts
|
||||
`WindowsExporterDown`, `WindowsCollectorFailed`, `WindowsCollectorTimeout`,
|
||||
`WindowsScrapeQueueTimeout`, `WindowsServiceNotRunning`,
|
||||
`WindowsClockNotSynchronising` and the state alerts of the opt-in collectors
|
||||
(clock source, drive status, license, task result, pending updates, Hyper-V,
|
||||
cluster and Storage Spaces health), and the one-shot `WindowsHostRebooted`,
|
||||
have no `keep_firing_for` and resolve as soon as their condition clears.
|
||||
`keep_firing_for` needs Prometheus 2.42 or later; set `keepFiringFor: ''` for
|
||||
older servers.
|
||||
|
||||
The network error and discard thresholds are percentages of all packets on the
|
||||
interface.
|
||||
|
||||
`criticalSeverity` defaults to `critical` and `infoSeverity` to `info`; override
|
||||
them to match your routing labels. Below the critical disk threshold, both disk
|
||||
severity levels fire; configure Alertmanager inhibition if only the critical
|
||||
notification should be sent.
|
||||
|
||||
Clock alerts are generated only when `enableTime` is `true`. A zero NTP source
|
||||
count may be expected on hosts using another time provider. The offset compares
|
||||
the host to its chosen source and does not independently verify that source.
|
||||
Disk latency and network error/discard thresholds should be tuned for the
|
||||
Active Directory alerts need `collectors.ad`, clock alerts `collectors.time`;
|
||||
both are off by default. A zero NTP
|
||||
source count may be expected on hosts using another time provider;
|
||||
`WindowsClockSourceLocal` covers domain members, whose source is `NT5DS`, and
|
||||
fires only for the local clock or no synchronization. The offset compares the
|
||||
host to its chosen source and does not independently verify that source.
|
||||
|
||||
The other opt-in alerts follow `_config.collectors`: enable `pagefile`,
|
||||
`diskdrive`, `license`, `scheduled_task`, `update`, `hyperv`, `mscluster`,
|
||||
`storage_spaces`, `dhcp` or `dns` and the alerts of that collector are
|
||||
generated, each collector in a group of its own.
|
||||
`WindowsScheduledTaskFailed` treats `unknown`, the status of every result code
|
||||
without a name, including the exit codes of the task's program, as a failure;
|
||||
narrow the tasks with `taskSelector`. Disabling a task clears the alert.
|
||||
`WindowsDiskDriveNotOK` skips the `Unknown` status, which some USB and
|
||||
virtual drives report permanently. `WindowsLicenseNotGenuine` reports the
|
||||
`state` the host has; hosts without internet access can stay in `offline`,
|
||||
so turn the alert off there. Every failover cluster node reports the state of
|
||||
the whole cluster, and every Storage Spaces Direct node reports the clustered
|
||||
pools and virtual disks, so the cluster and storage alerts fire once per node;
|
||||
group them by their `node`, `resource`, `shared_volume`, `pool` or
|
||||
`virtual_disk` labels in Alertmanager. The storage pool alert joins
|
||||
`windows_storage_spaces_pool_info` to leave out the primordial pool. The DHCP
|
||||
scope alert only counts scopes in an enabled state, and a scope without
|
||||
addresses produces no sample. `WindowsUpdatesPending` is an info alert that
|
||||
waits `updatePendingDuration` for the normal maintenance window. Prometheus
|
||||
restores the waiting time after a restart shorter than its
|
||||
`--rules.alert.for-outage-tolerance` (one hour by default); a longer outage
|
||||
or a new server starts the wait again. Disk latency, paging and network thresholds should be tuned for the
|
||||
storage tier, traffic volume, and application requirements.
|
||||
|
||||
`WindowsServiceNotRunning` watches services with the `auto` start mode and
|
||||
skips trigger-start services, which stop by design, through the
|
||||
`trigger_start` label of `windows_service_info`. Delayed-start services are
|
||||
included; the 15 minute duration covers their later start after a reboot.
|
||||
`WindowsServiceCrashLooping` counts the `windows_service_process` series of a
|
||||
service within 15 minutes, one per process the service ran as. Exclude services
|
||||
with `serviceSelector`, for example automatic services that exit by design
|
||||
after their work, such as the software protection or updater services of some
|
||||
products. `WindowsHostRebooted` fires once for about ten minutes after a
|
||||
reboot and resolves by itself.
|
||||
|
||||
Windows disk size/free-space counters may lag by 10–15 minutes. Predictions
|
||||
depend on a sustained trend and are intentionally delayed for an hour. A target
|
||||
removed from service discovery has no `up` series and will not trigger
|
||||
@@ -204,10 +311,10 @@ Fork users should override `runbookURLPattern` to match their published URL.
|
||||
|
||||
`grafanaDashboards` contains `windows-exporter.json`, the full sample dashboard
|
||||
with all tabs in
|
||||
[dashboard/windows-exporter-dashboard.json](../../dashboard/windows-exporter-dashboard.json),
|
||||
[contrib/dashboard/windows-exporter-dashboard.json](../dashboard/windows-exporter-dashboard.json),
|
||||
and a shorter dashboard per tab, such as `windows-exporter-cpu.json`, in
|
||||
[dashboard](../../dashboard). They are Grafana v2 dashboards and need Grafana 13
|
||||
or later. See the [dashboard documentation](../../dashboard/README.md) for their
|
||||
[contrib/dashboard](../dashboard). They are Grafana v2 dashboards and need Grafana 13
|
||||
or later. See the [dashboard documentation](../dashboard/README.md) for their
|
||||
content.
|
||||
|
||||
The dashboard uses additive builders with [grafonnet](https://github.com/grafana/grafonnet),
|
||||
@@ -254,7 +361,7 @@ Regenerate the committed files after changing the dashboard source:
|
||||
mise run dashboard
|
||||
```
|
||||
|
||||
This runs `jb install` and writes `../../dashboard/windows-exporter-*.json`.
|
||||
This runs `jb install` and writes `../dashboard/windows-exporter-*.json`.
|
||||
`mise run dashboards` is an alias. `mise run generate` builds both the rules and the
|
||||
dashboard, while `mise run rules` builds only the rules. The same commands work
|
||||
inside WSL; mise supplies the tool binaries on `PATH`.
|
||||
@@ -322,7 +429,7 @@ tab to the full dashboard:
|
||||
names, such as `cpu`, `physical_disk`, `diskdrive`, `smb` or `update`. The
|
||||
dashboard renders only the tabs, rows, panels and variables of enabled
|
||||
collectors, so a tab for a collector you do not run does not appear. The
|
||||
defaults match the exporter's default collectors; `time` follows `enableTime`.
|
||||
defaults match the exporter's default collectors.
|
||||
The `os` collector is required, because the variables select hosts through
|
||||
`windows_os_hostname`. An unknown collector name fails the build. Enable
|
||||
optional collectors on top of the defaults with `collectors+:`; replacing the
|
||||
@@ -366,14 +473,16 @@ mise run test
|
||||
|
||||
This checks Jsonnet formatting, runs `jsonnet-lint` on the Jsonnet sources to
|
||||
find unused variables and references to missing fields, checks that the
|
||||
committed dashboards are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
|
||||
committed dashboards and rule files are up to date, runs `dashboard-linter lint --strict`, validates generated rules with
|
||||
`promtool`, runs `pint --offline lint`, and evaluates the rules against synthetic
|
||||
time series.
|
||||
Tests cover all 25 recordings, alert timing and recovery, threshold
|
||||
boundaries, counter resets, zero denominators, disk forecasts, target-label
|
||||
preservation, and custom selectors/thresholds with Active Directory and clock
|
||||
alerts enabled. Runbook checks include opt-in alerts and verify that their pages
|
||||
and URLs exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
Tests cover all 28 recordings, alert timing and recovery, `keep_firing_for`,
|
||||
threshold boundaries, counter resets, zero denominators, disk forecasts,
|
||||
target-label preservation, and custom selectors/thresholds with Active
|
||||
Directory and clock alerts enabled. `tests/alerts.test.jsonnet` checks the
|
||||
collector gating, the per-alert switches and the group layout. Runbook checks
|
||||
render the alerts of every collector and verify that their pages and URLs
|
||||
exist. CI uses Prometheus 3.15.0 and the same pinned tool versions as
|
||||
`mise install`.
|
||||
|
||||
To list the available tasks:
|
||||
@@ -405,12 +514,12 @@ prometheus "local" {
|
||||
}
|
||||
```
|
||||
|
||||
Then run `pint --config /path/to/local.pint.hcl lint windows_rules.yaml windows_alerts.yaml`
|
||||
Then run `pint --config /path/to/local.pint.hcl lint ../monitoring/windows_rules.yaml ../monitoring/windows_alerts.yaml`
|
||||
without `--offline`. See the [pint documentation](https://cloudflare.github.io/pint/)
|
||||
for available checks.
|
||||
|
||||
Run `mise run fmt` after source changes and `mise run clean` to remove generated rule
|
||||
files. For downstream composition with Jsonnet Bundler, import
|
||||
Run `mise run fmt` and `mise run rules` after source changes, and `mise run clean`
|
||||
to remove the rule files generated for the tests. For downstream composition with Jsonnet Bundler, import
|
||||
`github.com/prometheus-community/windows_exporter/contrib/mixin/mixin.libsonnet`
|
||||
and extend `_config`. See the [monitoring mixin documentation](https://github.com/monitoring-mixins/docs)
|
||||
for the standard composition workflow.
|
||||
|
||||
@@ -1 +1 @@
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts)
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusAlerts, quote_keys=false)
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-active-directory', collectors=['ad'], rules=[
|
||||
{
|
||||
alert: 'WindowsADPendingReplication',
|
||||
expr: |||
|
||||
windows_ad_replication_pending_operations{%(windowsExporterSelector)s} > %(adPendingReplicationThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Active Directory replication queue is high',
|
||||
description: 'Pending replication operations on {{ $labels.instance }} have exceeded %(adPendingReplicationThreshold)s for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsADReplicationFailures',
|
||||
expr: |||
|
||||
(rate(windows_ad_replication_sync_requests_total{%(windowsExporterSelector)s}[%(rateInterval)s]) - rate(windows_ad_replication_sync_requests_success_total{%(windowsExporterSelector)s}[%(rateInterval)s])) > %(adReplicationFailureRateThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Active Directory replication requests are failing',
|
||||
description: 'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.',
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -1,158 +1,28 @@
|
||||
// One file per collector, each adding one rule group. To add alerts for a
|
||||
// collector, copy one of these files, name it after the collector and import
|
||||
// it here; the group order below is the order in the generated file.
|
||||
{
|
||||
local c = self._config,
|
||||
local q = (import '../lib/queries.libsonnet')(c),
|
||||
local alert(name, expr, duration, severity, summary, description) = {
|
||||
alert: name,
|
||||
expr: expr,
|
||||
'for': duration,
|
||||
labels: { severity: severity },
|
||||
annotations: { summary: summary, description: description },
|
||||
},
|
||||
prometheusAlerts+:: {
|
||||
groups: [
|
||||
{
|
||||
name: 'windows-exporter-alerts',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsExporterDown',
|
||||
q.metric('up') + ' == 0',
|
||||
'5m',
|
||||
c.criticalSeverity,
|
||||
'Windows exporter is unreachable',
|
||||
'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).'
|
||||
),
|
||||
alert(
|
||||
'WindowsCollectorFailed',
|
||||
q.metric('windows_exporter_collector_success') + ' == 0',
|
||||
'5m',
|
||||
'warning',
|
||||
'Windows collector failed',
|
||||
'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsCPUHighUsage',
|
||||
q.metric('windows:cpu_usage:ratio') + ' * 100 > %s' % c.cpuHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows CPU usage is high',
|
||||
'CPU usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.cpuHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryHighUsage',
|
||||
q.metric('windows:memory_usage:ratio') + ' * 100 > %s' % c.memoryHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory usage is high',
|
||||
'Physical memory usage on {{ $labels.instance }} has exceeded %s%% for 15 minutes.' % c.memoryHighUsageThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsMemoryCommitHighUsage',
|
||||
q.metric('windows:memory_committed:ratio') + ' * 100 > %s' % c.memoryCommitHighUsageThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows memory commit usage is high',
|
||||
'Committed memory on {{ $labels.instance }} has exceeded %s%% of the commit limit for 15 minutes.' % c.memoryCommitHighUsageThreshold
|
||||
),
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDisk%sLatencyHigh' % direction[0],
|
||||
q.metric('windows:logical_disk_%s_latency:seconds' % direction[1], [c.volumeSelector]) + ' > %g' % c.diskLatencyThresholdSeconds,
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows disk %s latency is high' % direction[1],
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %g seconds per %s operation for 15 minutes.' % [c.diskLatencyThresholdSeconds, direction[1]]
|
||||
)
|
||||
for direction in [['Read', 'read'], ['Write', 'write']]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsNetwork%s' % outcome[0],
|
||||
'(%s + %s) > %g' % [
|
||||
q.metric('windows:net_received_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
q.metric('windows:net_outbound_%s:rate' % outcome[1], [c.nicSelector]),
|
||||
outcome[2],
|
||||
],
|
||||
'15m',
|
||||
'warning',
|
||||
'Windows network packet %s are high' % outcome[3],
|
||||
'Packet %s on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %g per second for 15 minutes.' % [outcome[3], outcome[2]]
|
||||
)
|
||||
for outcome in [
|
||||
['Errors', 'errors', c.networkErrorRateThreshold, 'errors'],
|
||||
['Discards', 'discarded', c.networkDiscardRateThreshold, 'discards'],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskAlmostFull',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s' % entry[1],
|
||||
'15m',
|
||||
entry[0],
|
||||
'Windows disk space is low',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %s%% free space. Windows may delay space metrics by 10–15 minutes.' % entry[1]
|
||||
)
|
||||
for entry in [
|
||||
['warning', c.diskFreeWarningThreshold],
|
||||
[c.criticalSeverity, c.diskFreeCriticalThreshold],
|
||||
]
|
||||
] + [
|
||||
alert(
|
||||
'WindowsDiskFillingUp',
|
||||
q.metric('windows:logical_disk_free:ratio', [c.volumeSelector]) + ' * 100 < %s and predict_linear(%s[%s], %s) < 0' % [
|
||||
c.diskFreeWarningThreshold,
|
||||
q.metric('windows_logical_disk_free_bytes', [c.volumeSelector]),
|
||||
c.diskPredictionWindow,
|
||||
c.diskPredictionHours * 3600,
|
||||
],
|
||||
'1h',
|
||||
'warning',
|
||||
'Windows disk is predicted to fill up',
|
||||
'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %s hours.' % c.diskPredictionHours
|
||||
),
|
||||
],
|
||||
},
|
||||
] + (if c.enableActiveDirectory then [
|
||||
{
|
||||
name: 'windows-active-directory',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsADPendingReplication',
|
||||
q.metric('windows_ad_replication_pending_operations') + ' > %s' % c.adPendingReplicationThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication queue is high',
|
||||
'Pending replication operations on {{ $labels.instance }} have exceeded %s for 15 minutes.' % c.adPendingReplicationThreshold
|
||||
),
|
||||
alert(
|
||||
'WindowsADReplicationFailures',
|
||||
'(' + q.rate('windows_ad_replication_sync_requests_total') + ' - ' + q.rate('windows_ad_replication_sync_requests_success_total') + ') > %s' % c.adReplicationFailureRateThreshold,
|
||||
'15m',
|
||||
'warning',
|
||||
'Active Directory replication requests are failing',
|
||||
'Replication synchronization requests on {{ $labels.instance }} have been failing for 15 minutes.'
|
||||
),
|
||||
],
|
||||
},
|
||||
] else []) + if c.enableTime then [
|
||||
{
|
||||
name: 'windows-time',
|
||||
rules: [
|
||||
alert(
|
||||
'WindowsClockNotSynchronising',
|
||||
q.metric('windows_time_ntp_client_time_sources') + ' == 0',
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock has no NTP time source',
|
||||
'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.'
|
||||
),
|
||||
alert(
|
||||
'WindowsClockSkewDetected',
|
||||
'abs(%s) > %g' % [q.metric('windows_time_computed_time_offset_seconds'), c.clockOffsetThresholdSeconds],
|
||||
'10m',
|
||||
'warning',
|
||||
'Windows clock offset is high',
|
||||
'Windows Time on {{ $labels.instance }} reports a clock offset above %g seconds for 10 minutes.' % c.clockOffsetThresholdSeconds
|
||||
),
|
||||
],
|
||||
},
|
||||
] else [],
|
||||
},
|
||||
local known = std.objectFields((import '../config.libsonnet')._config.alerts),
|
||||
local unknown = std.setDiff(std.set(std.objectFields(c.alerts)), std.set(known)),
|
||||
assert std.length(unknown) == 0 : 'Unknown alerts in _config.alerts: ' + std.join(', ', unknown),
|
||||
}
|
||||
+ (import 'exporter.libsonnet')
|
||||
+ (import 'cpu.libsonnet')
|
||||
+ (import 'memory.libsonnet')
|
||||
+ (import 'pagefile.libsonnet')
|
||||
+ (import 'logical_disk.libsonnet')
|
||||
+ (import 'diskdrive.libsonnet')
|
||||
+ (import 'net.libsonnet')
|
||||
+ (import 'service.libsonnet')
|
||||
+ (import 'system.libsonnet')
|
||||
+ (import 'license.libsonnet')
|
||||
+ (import 'ad.libsonnet')
|
||||
+ (import 'time.libsonnet')
|
||||
+ (import 'scheduled_task.libsonnet')
|
||||
+ (import 'update.libsonnet')
|
||||
+ (import 'hyperv.libsonnet')
|
||||
+ (import 'mscluster.libsonnet')
|
||||
+ (import 'storage_spaces.libsonnet')
|
||||
+ (import 'dhcp.libsonnet')
|
||||
+ (import 'dns.libsonnet')
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-cpu', collectors=['cpu'], rules=[
|
||||
{
|
||||
alert: 'WindowsCPUHighUsage',
|
||||
expr: |||
|
||||
windows:cpu_usage:ratio{%(windowsExporterSelector)s} * 100 > %(cpuHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows CPU usage is high',
|
||||
description: 'CPU usage on {{ $labels.instance }} has exceeded %(cpuHighUsageThreshold)s%% for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCPUSaturation',
|
||||
expr: |||
|
||||
windows:cpu_queue_per_core:ratio{%(windowsExporterSelector)s} > %(cpuQueuePerCoreThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows processor queue is long',
|
||||
description: 'More than %(cpuQueuePerCoreThreshold)s threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes.' % c,
|
||||
},
|
||||
collectors:: ['cpu', 'system'],
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-dhcp', collectors=['dhcp'], rules=[
|
||||
// Percent of free addresses in a scope; a scope without addresses has
|
||||
// no sample, and only scopes in an enabled state count.
|
||||
{
|
||||
alert: 'WindowsDHCPScopeAlmostExhausted',
|
||||
expr: |||
|
||||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeWarningThreshold)s
|
||||
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'DHCP scope is almost exhausted',
|
||||
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeWarningThreshold)s%% free addresses for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDHCPScopeAlmostExhausted',
|
||||
expr: |||
|
||||
((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} / ((windows_dhcp_scope_addresses_free{%(windowsExporterSelector)s} + windows_dhcp_scope_addresses_in_use{%(windowsExporterSelector)s}) > 0)) * 100) < %(dhcpScopeFreeCriticalThreshold)s
|
||||
and ignoring (state) (windows_dhcp_scope_state{%(windowsExporterSelector)s, state=~"Enabled|EnabledSwitched"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'DHCP scope is almost exhausted',
|
||||
description: 'DHCP scope {{ $labels.scope }} on {{ $labels.instance }} has had less than %(dhcpScopeFreeCriticalThreshold)s%% free addresses for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-diskdrive', collectors=['diskdrive'], rules=[
|
||||
{
|
||||
alert: 'WindowsDiskDriveNotOK',
|
||||
// One series per status with the current status set to 1, so the
|
||||
// alert carries the status the drive reports. Some USB and virtual
|
||||
// drives report Unknown for good.
|
||||
expr: |||
|
||||
windows_diskdrive_status{%(windowsExporterSelector)s, status!~"OK|Unknown"} == 1
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows disk drive is not healthy',
|
||||
description: 'Disk drive {{ $labels.device_id }} on {{ $labels.instance }} has reported status {{ $labels.status }} for 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-dns', collectors=['dns'], rules=[
|
||||
{
|
||||
alert: 'WindowsDNSZoneTransferFailures',
|
||||
expr: |||
|
||||
rate(windows_dns_zone_transfer_failures_total{%(windowsExporterSelector)s}[%(rateInterval)s]) > %(dnsZoneTransferFailureRateThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'DNS zone transfers are failing',
|
||||
description: 'Zone transfers from DNS server {{ $labels.instance }} have been failing for 15 minutes.',
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-exporter', collectors=[], rules=[
|
||||
{
|
||||
alert: 'WindowsExporterDown',
|
||||
expr: |||
|
||||
up{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows exporter is unreachable',
|
||||
description: 'Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }}).',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCollectorFailed',
|
||||
expr: |||
|
||||
windows_exporter_collector_success{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows collector failed',
|
||||
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsCollectorTimeout',
|
||||
expr: |||
|
||||
windows_exporter_collector_timeout{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows collector exceeds the scrape timeout',
|
||||
description: 'Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsScrapeQueueTimeout',
|
||||
expr: |||
|
||||
windows_exporter_scrape_queue_timeout{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows exporter scrapes time out in the queue',
|
||||
description: 'Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-hyperv', collectors=['hyperv'], rules=[
|
||||
{
|
||||
alert: 'WindowsHyperVWMIUnhealthy',
|
||||
expr: |||
|
||||
windows_hyperv_wmi_health{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Hyper-V WMI namespace is not responding',
|
||||
description: 'The Hyper-V WMI namespace on {{ $labels.instance }} has not responded for 10 minutes; Hyper-V metrics are missing.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsHyperVReplicationCritical',
|
||||
expr: |||
|
||||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 3
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Hyper-V replication is critical',
|
||||
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in critical health for 5 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsHyperVReplicationWarning',
|
||||
expr: |||
|
||||
windows_hyperv_replica_vm_health{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Hyper-V replication has a warning',
|
||||
description: 'Replication of virtual machine {{ $labels.vm }} ({{ $labels.relationship }}) on {{ $labels.instance }} has been in warning health for 15 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-license', collectors=['license'], rules=[
|
||||
{
|
||||
alert: 'WindowsLicenseNotGenuine',
|
||||
expr: |||
|
||||
windows_license_status{%(windowsExporterSelector)s, state!="genuine"} == 1
|
||||
||| % c,
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows license is not genuine',
|
||||
description: 'Windows on {{ $labels.instance }} has reported license status {{ $labels.state }} for 1 hour.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-disk', collectors=['logical_disk'], rules=[
|
||||
{
|
||||
alert: 'WindowsDiskReadLatencyHigh',
|
||||
expr: |||
|
||||
windows:logical_disk_read_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk read latency is high',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per read operation for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskWriteLatencyHigh',
|
||||
expr: |||
|
||||
windows:logical_disk_write_latency:seconds{%(windowsExporterSelector)s, %(volumeSelector)s} > %(diskLatencyThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk write latency is high',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than %(diskLatencyThresholdSeconds)g seconds per write operation for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskAlmostFull',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk space is low',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeWarningThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskAlmostFull',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeCriticalThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows disk space is low',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} has less than %(diskFreeCriticalThreshold)s%% free space. Windows may delay space metrics by 10–15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsDiskFillingUp',
|
||||
expr: |||
|
||||
windows:logical_disk_free:ratio{%(windowsExporterSelector)s, %(volumeSelector)s} * 100 < %(diskFreeWarningThreshold)s
|
||||
and predict_linear(windows_logical_disk_free_bytes{%(windowsExporterSelector)s, %(volumeSelector)s}[%(diskPredictionWindow)s], %(diskPredictionSeconds)s) < 0
|
||||
||| % c { diskPredictionSeconds: c.diskPredictionHours * 3600 },
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows disk is predicted to fill up',
|
||||
description: 'Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within %(diskPredictionHours)s hours.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-memory', collectors=['memory'], rules=[
|
||||
{
|
||||
alert: 'WindowsMemoryHighUsage',
|
||||
expr: |||
|
||||
windows:memory_usage:ratio{%(windowsExporterSelector)s} * 100 > %(memoryHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory usage is high',
|
||||
description: 'Physical memory usage on {{ $labels.instance }} has exceeded %(memoryHighUsageThreshold)s%% for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsMemoryCommitHighUsage',
|
||||
expr: |||
|
||||
windows:memory_committed:ratio{%(windowsExporterSelector)s} * 100 > %(memoryCommitHighUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory commit usage is high',
|
||||
description: 'Committed memory on {{ $labels.instance }} has exceeded %(memoryCommitHighUsageThreshold)s%% of the commit limit for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsMemoryPagingHigh',
|
||||
expr: |||
|
||||
windows:memory_swap_page_operations:rate{%(windowsExporterSelector)s} > %(memoryPagingThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows memory paging is high',
|
||||
description: 'Paging on {{ $labels.instance }} has exceeded %(memoryPagingThreshold)s pages per second for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
// Every cluster node reports the state of the whole cluster, so these
|
||||
// alerts fire once per node; group them by cluster labels in Alertmanager.
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-cluster', collectors=['mscluster'], rules=[
|
||||
{
|
||||
alert: 'WindowsClusterNodeDown',
|
||||
expr: |||
|
||||
windows_mscluster_node_state{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster node is down',
|
||||
description: 'Cluster node {{ $labels.node }} has been down for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterResourceFailed',
|
||||
expr: |||
|
||||
windows_mscluster_resource_state{%(windowsExporterSelector)s} == 4
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster resource has failed',
|
||||
description: 'Cluster resource {{ $labels.resource }} ({{ $labels.type }}) in group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterResourceGroupFailed',
|
||||
expr: |||
|
||||
windows_mscluster_resourcegroup_state{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '5m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Failover cluster resource group has failed',
|
||||
description: 'Cluster resource group {{ $labels.resource_group }} has been in the failed state for 5 minutes, as reported by {{ $labels.instance }}.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterSharedVolumeAlmostFull',
|
||||
expr: |||
|
||||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeWarningThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Cluster shared volume is almost full',
|
||||
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeWarningThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClusterSharedVolumeAlmostFull',
|
||||
expr: |||
|
||||
(windows_mscluster_shared_volumes_free_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} / (windows_mscluster_shared_volumes_total_bytes{%(windowsExporterSelector)s, %(sharedVolumeSelector)s} > 0)) * 100 < %(diskFreeCriticalThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Cluster shared volume is almost full',
|
||||
description: 'Cluster shared volume {{ $labels.shared_volume }} has had less than %(diskFreeCriticalThreshold)s%% free space for 15 minutes, as reported by {{ $labels.instance }}.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-network', collectors=['net'], rules=[
|
||||
{
|
||||
alert: 'WindowsNetworkErrors',
|
||||
expr: |||
|
||||
(
|
||||
(windows:net_received_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_errors:rate{%(windowsExporterSelector)s, %(nicSelector)s})
|
||||
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
|
||||
) * 100 > %(networkErrorRatioThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network packet errors are high',
|
||||
description: 'Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkErrorRatioThreshold)g%% of all packets for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkDiscards',
|
||||
expr: |||
|
||||
(
|
||||
(windows:net_received_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_outbound_discarded:rate{%(windowsExporterSelector)s, %(nicSelector)s})
|
||||
/ ((windows:net_received_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s} + windows:net_sent_packets:rate{%(windowsExporterSelector)s, %(nicSelector)s}) > 0)
|
||||
) * 100 > %(networkDiscardRatioThreshold)g
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network packet discards are high',
|
||||
description: 'Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded %(networkDiscardRatioThreshold)g%% of all packets for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkUtilizationHigh',
|
||||
expr: |||
|
||||
(windows:net_received_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
|
||||
or (windows:net_sent_utilization:ratio{%(windowsExporterSelector)s, %(nicSelector)s} * 100 > %(networkUtilizationThreshold)s)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network interface utilization is high',
|
||||
description: 'Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded %(networkUtilizationThreshold)s%% of its bandwidth in at least one direction for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsNetworkInterfaceFlapping',
|
||||
expr: |||
|
||||
max without (status) (changes(windows_net_nic_operation_status{%(windowsExporterSelector)s, %(nicSelector)s, status="up"}[%(rateInterval)s])) > 2
|
||||
||| % c,
|
||||
'for': '2m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows network interface is flapping',
|
||||
description: 'Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in %(rateInterval)s.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-pagefile', collectors=['pagefile'], rules=[
|
||||
{
|
||||
alert: 'WindowsPagefileAlmostFull',
|
||||
expr: |||
|
||||
(1 - (windows_pagefile_free_bytes{%(windowsExporterSelector)s} / (windows_pagefile_limit_bytes{%(windowsExporterSelector)s} > 0))) * 100 > %(pagefileUsageThreshold)s
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows page file is almost full',
|
||||
description: 'Page file {{ $labels.file }} on {{ $labels.instance }} has been more than %(pagefileUsageThreshold)s%% in use for 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-scheduled-task', collectors=['scheduled_task'], rules=[
|
||||
{
|
||||
alert: 'WindowsScheduledTaskFailed',
|
||||
// unknown is any result code without a named status, including the
|
||||
// exit codes of the task's program.
|
||||
// A disabled task keeps its last result, so disabling a broken task
|
||||
// clears the alert.
|
||||
expr: |||
|
||||
windows_scheduled_task_last_result_status{%(windowsExporterSelector)s, %(taskSelector)s, status=~"unknown|terminated"} == 1
|
||||
unless ignoring (status, state) (windows_scheduled_task_state{%(windowsExporterSelector)s, %(taskSelector)s, state="disabled"} == 1)
|
||||
||| % c,
|
||||
'for': '1h',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows scheduled task did not succeed',
|
||||
description: 'The last run of task {{ $labels.task }} on {{ $labels.instance }} ended with result {{ $labels.status }}, and the task has not succeeded in the hour since.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-service', collectors=['service'], rules=[
|
||||
{
|
||||
alert: 'WindowsServiceNotRunning',
|
||||
// The matches ignore only the labels of the right-hand metric, so
|
||||
// every target label, including ones a selector adds, takes part.
|
||||
expr: |||
|
||||
max without (state) (windows_service_state{%(windowsExporterSelector)s, %(serviceSelector)s, state="running"} == 0)
|
||||
and ignoring (start_mode) (windows_service_start_mode{%(windowsExporterSelector)s, %(serviceSelector)s, start_mode="auto"} == 1)
|
||||
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{%(windowsExporterSelector)s, %(serviceSelector)s, trigger_start="true"} == 1)
|
||||
||| % c,
|
||||
'for': '15m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows automatic service is not running',
|
||||
description: 'Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsServiceCrashLooping',
|
||||
// Every service process has its own series, so the series seen in the
|
||||
// window count the processes the service ran as.
|
||||
expr: |||
|
||||
count without (process_id) (count_over_time(windows_service_process{%(windowsExporterSelector)s, %(serviceSelector)s}[15m])) - 1 > %(serviceRestartThreshold)s
|
||||
||| % c,
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows service is restarting repeatedly',
|
||||
description: 'Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than %(serviceRestartThreshold)s times in 15 minutes.' % c,
|
||||
},
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// On Storage Spaces Direct every node reports the clustered pools and
|
||||
// virtual disks, so these alerts fire once per node as well.
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-storage-spaces', collectors=['storage_spaces'], rules=[
|
||||
{
|
||||
alert: 'WindowsStoragePoolUnhealthy',
|
||||
// The primordial pool holds the disks that are not in a concrete pool
|
||||
// yet; only the info metric tells it apart.
|
||||
expr: |||
|
||||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 2)
|
||||
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Storage pool is unhealthy',
|
||||
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStoragePoolUnhealthy',
|
||||
expr: |||
|
||||
(windows_storage_spaces_pool_health_status{%(windowsExporterSelector)s} == 1)
|
||||
and ignoring (primordial) windows_storage_spaces_pool_info{%(windowsExporterSelector)s, primordial="false"}
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Storage pool has a health warning',
|
||||
description: 'Storage pool {{ $labels.pool }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStorageVirtualDiskUnhealthy',
|
||||
expr: |||
|
||||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 2
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: c.criticalSeverity },
|
||||
annotations: {
|
||||
summary: 'Storage Spaces virtual disk is unhealthy',
|
||||
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported unhealthy status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsStorageVirtualDiskUnhealthy',
|
||||
expr: |||
|
||||
windows_storage_spaces_virtual_disk_health_status{%(windowsExporterSelector)s} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Storage Spaces virtual disk has a health warning',
|
||||
description: 'Virtual disk {{ $labels.virtual_disk }} on {{ $labels.instance }} has reported warning status for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-system', collectors=['system'], rules=[
|
||||
{
|
||||
alert: 'WindowsHostRebooted',
|
||||
// Fires once per reboot: for the first ten minutes after the boot,
|
||||
// and only on a host that was monitored before it. The exporter is
|
||||
// down while the host reboots, so the second operand takes the last
|
||||
// boot time of the hour before, not the sample of ten minutes ago.
|
||||
expr: |||
|
||||
(time() - windows_system_boot_time_timestamp{%(windowsExporterSelector)s}) < 600
|
||||
and (time() - last_over_time(windows_system_boot_time_timestamp{%(windowsExporterSelector)s}[1h] offset 10m)) > 600
|
||||
||| % c,
|
||||
labels: { severity: c.infoSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows host has rebooted',
|
||||
description: '{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-time', collectors=['time'], rules=[
|
||||
{
|
||||
alert: 'WindowsClockNotSynchronising',
|
||||
expr: |||
|
||||
windows_time_ntp_client_time_sources{%(windowsExporterSelector)s} == 0
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock has no NTP time source',
|
||||
description: 'Windows Time on {{ $labels.instance }} has had no active NTP time source for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClockSkewDetected',
|
||||
expr: |||
|
||||
abs(windows_time_computed_time_offset_seconds{%(windowsExporterSelector)s}) > %(clockOffsetThresholdSeconds)g
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock offset is high',
|
||||
description: 'Windows Time on {{ $labels.instance }} reports a clock offset above %(clockOffsetThresholdSeconds)g seconds for 10 minutes.' % c,
|
||||
},
|
||||
},
|
||||
{
|
||||
alert: 'WindowsClockSourceLocal',
|
||||
// Covers domain members (NT5DS) too, which the NTP source count misses.
|
||||
expr: |||
|
||||
windows_time_clock_sync_source{%(windowsExporterSelector)s, type=~"Local CMOS Clock|NoSync"} == 1
|
||||
||| % c,
|
||||
'for': '10m',
|
||||
labels: { severity: 'warning' },
|
||||
annotations: {
|
||||
summary: 'Windows clock has no external time source',
|
||||
description: 'Windows Time on {{ $labels.instance }} has used {{ $labels.type }} as its clock source for 10 minutes.',
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
local c = self._config,
|
||||
local a = (import '../lib/alerts.libsonnet')(c),
|
||||
prometheusAlerts+:: {
|
||||
groups+: a.group('windows-update', collectors=['update'], rules=[
|
||||
{
|
||||
alert: 'WindowsUpdatesPending',
|
||||
expr: |||
|
||||
count without (id, revision, category, severity, title) (windows_update_pending_info{%(windowsExporterSelector)s, deployment_action="installation"}) > 0
|
||||
||| % c,
|
||||
'for': c.updatePendingDuration,
|
||||
labels: { severity: c.infoSeverity },
|
||||
annotations: {
|
||||
summary: 'Windows updates are waiting for installation',
|
||||
description: '{{ $labels.instance }} has had {{ $value }} updates waiting for installation for %(updatePendingDuration)s.' % c,
|
||||
},
|
||||
state:: true,
|
||||
},
|
||||
]),
|
||||
},
|
||||
}
|
||||
@@ -4,13 +4,12 @@
|
||||
windowsExporterSelector: 'job="windows_exporter"',
|
||||
volumeSelector: 'volume!=""',
|
||||
nicSelector: 'nic!=""',
|
||||
serviceSelector: 'service!=""',
|
||||
taskSelector: 'task!=""',
|
||||
// ClusterPerformanceHistory is the volume of the cluster's own health history.
|
||||
sharedVolumeSelector: 'shared_volume!="ClusterPerformanceHistory"',
|
||||
rateInterval: '5m',
|
||||
|
||||
// Opt in only for targets with the ad collector enabled.
|
||||
enableActiveDirectory: false,
|
||||
// Requires the time collector with its ntp subcollector enabled.
|
||||
enableTime: false,
|
||||
|
||||
// Grafana dashboard metadata. A UID is the v2 metadata.name; change the
|
||||
// UIDs to import the dashboards next to the sample ones.
|
||||
// Every tab also renders as a dashboard of its own with the UID
|
||||
@@ -37,18 +36,25 @@
|
||||
|
||||
// Collectors enabled on the windows_exporter targets, by collector name.
|
||||
// The dashboard only renders the tabs, rows and panels of enabled
|
||||
// collectors. The defaults match the exporter's default collectors.
|
||||
// collectors, and the alerts only the rules whose collectors are enabled.
|
||||
// The defaults match the exporter's default collectors.
|
||||
collectors: {
|
||||
// Only for targets with the ad collector enabled, such as domain controllers.
|
||||
ad: false,
|
||||
cpu: true,
|
||||
dhcp: false,
|
||||
diskdrive: false,
|
||||
dns: false,
|
||||
gpu: false,
|
||||
hyperv: false,
|
||||
license: false,
|
||||
logical_disk: true,
|
||||
memory: true,
|
||||
mscluster: false,
|
||||
mssql: false,
|
||||
net: true,
|
||||
os: true,
|
||||
pagefile: false,
|
||||
physical_disk: true,
|
||||
process: false,
|
||||
scheduled_task: false,
|
||||
@@ -58,28 +64,101 @@
|
||||
storage_spaces: false,
|
||||
system: true,
|
||||
tcp: false,
|
||||
time: $._config.enableTime,
|
||||
// Needs the time collector with its ntp and clock_source subcollectors.
|
||||
time: false,
|
||||
udp: false,
|
||||
update: false,
|
||||
},
|
||||
|
||||
// Alerts by name. Set one to false to leave it out of the generated
|
||||
// rules, for example alerts+: { WindowsHostRebooted: false }. Names that
|
||||
// are not listed here fail the build; names left out of a replaced object
|
||||
// stay enabled.
|
||||
alerts: {
|
||||
WindowsExporterDown: true,
|
||||
WindowsCollectorFailed: true,
|
||||
WindowsCollectorTimeout: true,
|
||||
WindowsScrapeQueueTimeout: true,
|
||||
WindowsCPUHighUsage: true,
|
||||
WindowsCPUSaturation: true,
|
||||
WindowsMemoryHighUsage: true,
|
||||
WindowsMemoryCommitHighUsage: true,
|
||||
WindowsMemoryPagingHigh: true,
|
||||
WindowsDiskReadLatencyHigh: true,
|
||||
WindowsDiskWriteLatencyHigh: true,
|
||||
WindowsDiskAlmostFull: true,
|
||||
WindowsDiskFillingUp: true,
|
||||
WindowsNetworkErrors: true,
|
||||
WindowsNetworkDiscards: true,
|
||||
WindowsNetworkUtilizationHigh: true,
|
||||
WindowsNetworkInterfaceFlapping: true,
|
||||
WindowsServiceNotRunning: true,
|
||||
WindowsServiceCrashLooping: true,
|
||||
WindowsHostRebooted: true,
|
||||
WindowsADPendingReplication: true,
|
||||
WindowsADReplicationFailures: true,
|
||||
WindowsClockNotSynchronising: true,
|
||||
WindowsClockSkewDetected: true,
|
||||
WindowsClockSourceLocal: true,
|
||||
WindowsDiskDriveNotOK: true,
|
||||
WindowsPagefileAlmostFull: true,
|
||||
WindowsLicenseNotGenuine: true,
|
||||
WindowsScheduledTaskFailed: true,
|
||||
WindowsUpdatesPending: true,
|
||||
WindowsHyperVWMIUnhealthy: true,
|
||||
WindowsHyperVReplicationCritical: true,
|
||||
WindowsHyperVReplicationWarning: true,
|
||||
WindowsClusterNodeDown: true,
|
||||
WindowsClusterResourceFailed: true,
|
||||
WindowsClusterResourceGroupFailed: true,
|
||||
WindowsClusterSharedVolumeAlmostFull: true,
|
||||
WindowsStoragePoolUnhealthy: true,
|
||||
WindowsStorageVirtualDiskUnhealthy: true,
|
||||
WindowsDHCPScopeAlmostExhausted: true,
|
||||
WindowsDNSZoneTransferFailures: true,
|
||||
},
|
||||
|
||||
runbookURLPattern: 'https://prometheus-community.github.io/windows_exporter/runbooks/%s/',
|
||||
|
||||
// Threshold alerts keep firing this long after their condition clears,
|
||||
// so a value oscillating around a threshold does not flap. Needs
|
||||
// Prometheus 2.42 or later; set it to '' to leave keep_firing_for out.
|
||||
keepFiringFor: '5m',
|
||||
|
||||
cpuHighUsageThreshold: 90,
|
||||
// Threads waiting for a processor, per logical processor.
|
||||
cpuQueuePerCoreThreshold: 2,
|
||||
memoryHighUsageThreshold: 90,
|
||||
memoryCommitHighUsageThreshold: 90,
|
||||
// Pages read from or written to disk per second.
|
||||
memoryPagingThreshold: 1000,
|
||||
diskLatencyThresholdSeconds: 0.05,
|
||||
// Average latency per I/O above which a Hyper-V virtual disk counts as slow.
|
||||
hypervDiskLatencyThresholdSeconds: 0.01,
|
||||
networkErrorRateThreshold: 1,
|
||||
networkDiscardRateThreshold: 1,
|
||||
// Percent of all packets on the interface.
|
||||
networkErrorRatioThreshold: 1,
|
||||
networkDiscardRatioThreshold: 1,
|
||||
// Percent of the interface bandwidth, in either direction.
|
||||
networkUtilizationThreshold: 90,
|
||||
clockOffsetThresholdSeconds: 0.05,
|
||||
diskFreeWarningThreshold: 10,
|
||||
diskFreeCriticalThreshold: 5,
|
||||
diskPredictionWindow: '6h',
|
||||
diskPredictionHours: 24,
|
||||
// Restarts of one service within 15 minutes.
|
||||
serviceRestartThreshold: 2,
|
||||
criticalSeverity: 'critical',
|
||||
infoSeverity: 'info',
|
||||
adPendingReplicationThreshold: 50,
|
||||
adReplicationFailureRateThreshold: 0,
|
||||
// Percent of the page file in use.
|
||||
pagefileUsageThreshold: 80,
|
||||
// Percent of free addresses in an active DHCP scope.
|
||||
dhcpScopeFreeWarningThreshold: 10,
|
||||
dhcpScopeFreeCriticalThreshold: 2,
|
||||
// How long updates may wait for installation before WindowsUpdatesPending fires.
|
||||
updatePendingDuration: '7d',
|
||||
// Failed zone transfers per second.
|
||||
dnsZoneTransferFailureRateThreshold: 0,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// Renders the sample dashboards into ../../dashboard. Run with
|
||||
// Renders the sample dashboards into ../dashboard. Run with
|
||||
// jsonnet -J vendor -S -m <directory>. The sample dashboards show every
|
||||
// collector; mixin users select theirs with _config.collectors.
|
||||
local manifest = import 'lib/manifest.libsonnet';
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
// Helpers shared by the alert files in alerts/. Takes the mixin _config.
|
||||
function(c)
|
||||
{
|
||||
// A collector missing from a replaced collectors object counts as disabled;
|
||||
// an alert missing from a replaced alerts object counts as enabled.
|
||||
collectorEnabled(collector): std.get(c.collectors, collector, false),
|
||||
alertEnabled(name): std.get(c.alerts, name, true),
|
||||
// group renders the rules of one collector as a rule group, or nothing
|
||||
// when none of them is enabled. Each rule is a plain Prometheus alerting
|
||||
// rule plus two hidden fields: collectors:: lists the collectors the rule
|
||||
// reads when it differs from the group's, and state:: true marks state
|
||||
// alerts (== 0, == 1) and one-shot alerts, which must resolve at once.
|
||||
// Every other alert is a threshold alert and keeps firing for
|
||||
// keepFiringFor after its condition clears, so a value oscillating around
|
||||
// the threshold does not flap.
|
||||
group(name, collectors, rules):
|
||||
local enabled = [
|
||||
rule + (if c.keepFiringFor != '' && !std.get(rule, 'state', false) then { keep_firing_for: c.keepFiringFor } else {})
|
||||
for rule in rules
|
||||
if self.alertEnabled(rule.alert) && std.all([self.collectorEnabled(collector) for collector in std.get(rule, 'collectors', collectors)])
|
||||
];
|
||||
if std.length(enabled) > 0 then [{ name: name, rules: enabled }] else [],
|
||||
}
|
||||
@@ -7,6 +7,13 @@ function(c)
|
||||
ratio(numerator, denominator): '(%s / (%s > 0))' % [numerator, denominator],
|
||||
cpuUsage:
|
||||
'1 - avg without (core, mode) (%s)' % self.rate('windows_cpu_time_seconds_total', ['mode="idle"']),
|
||||
// Threads waiting for a processor, averaged over the rate interval, per
|
||||
// logical processor. The queue length is an instantaneous sample.
|
||||
cpuQueuePerCore:
|
||||
self.ratio(
|
||||
'avg_over_time(%s[%s])' % [self.metric('windows_system_processor_queue_length'), c.rateInterval],
|
||||
self.metric('windows:cpu_cores:count')
|
||||
),
|
||||
memoryUsage:
|
||||
'1 - %s' % self.ratio(
|
||||
self.metric('windows_memory_available_bytes'),
|
||||
|
||||
+23
-12
@@ -8,7 +8,8 @@
|
||||
[vars]
|
||||
sources = "*.libsonnet *.jsonnet lib/*.libsonnet rules/*.libsonnet alerts/*.libsonnet dashboards/*.libsonnet tests/*.libsonnet tests/*.jsonnet"
|
||||
fmt_flags = "-n 2 --max-blank-lines 2 --string-style s --comment-style s"
|
||||
dashboards = "../../dashboard"
|
||||
dashboards = "../dashboard"
|
||||
rules = "../monitoring"
|
||||
|
||||
[tasks.generate]
|
||||
description = "Generate Prometheus rules, alerts, and the dashboard"
|
||||
@@ -16,10 +17,10 @@ alias = "default"
|
||||
depends = ["rules", "dashboard"]
|
||||
|
||||
[tasks.rules]
|
||||
description = "Generate Prometheus rules and alerts"
|
||||
description = "Generate the Prometheus rules and alerts in contrib/monitoring"
|
||||
run = [
|
||||
"jsonnet -S rules.jsonnet > windows_rules.yaml",
|
||||
"jsonnet -S alerts.jsonnet > windows_alerts.yaml",
|
||||
"jsonnet -S rules.jsonnet > {{vars.rules}}/windows_rules.yaml",
|
||||
"jsonnet -S alerts.jsonnet > {{vars.rules}}/windows_alerts.yaml",
|
||||
]
|
||||
|
||||
[tasks.vendor]
|
||||
@@ -39,6 +40,15 @@ rm -f {{vars.dashboards}}/windows-exporter-*.json
|
||||
cp "$out"/*.json {{vars.dashboards}}/
|
||||
'''
|
||||
|
||||
[tasks.check-rules]
|
||||
description = "Regenerate the rules and alerts in contrib/monitoring and fail on uncommitted changes"
|
||||
depends = ["rules"]
|
||||
run = '''
|
||||
set -eu
|
||||
git add --intent-to-add -- {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml
|
||||
git diff --exit-code -- {{vars.rules}}
|
||||
'''
|
||||
|
||||
[tasks.check-dashboards]
|
||||
description = "Regenerate the dashboards and fail on uncommitted changes"
|
||||
depends = ["dashboard"]
|
||||
@@ -65,8 +75,8 @@ run = "jsonnetfmt {{vars.fmt_flags}} -i {{vars.sources}}"
|
||||
description = "Generate rules and alerts with the test configuration"
|
||||
hide = true
|
||||
run = [
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules)' > tests/custom_rules.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts)' > tests/custom_alerts.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusRules, quote_keys=false)' > tests/custom_rules.yaml''',
|
||||
'''jsonnet -S -e 'std.manifestYamlDoc((import "tests/config.libsonnet").prometheusAlerts, quote_keys=false)' > tests/custom_alerts.yaml''',
|
||||
]
|
||||
|
||||
[tasks.lint]
|
||||
@@ -88,25 +98,26 @@ for file in $(cat "$out/generated"); do
|
||||
diff -u {{vars.dashboards}}/"$file" "$out/$file" \
|
||||
|| { echo "{{vars.dashboards}}/$file is out of date; run mise run dashboard"; exit 1; }
|
||||
done
|
||||
promtool check rules windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom
|
||||
pint --offline lint windows_rules.yaml windows_alerts.yaml
|
||||
promtool check rules {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml ../../docs/upgrade/0.32.prom
|
||||
pint --offline lint {{vars.rules}}/windows_rules.yaml {{vars.rules}}/windows_alerts.yaml
|
||||
pint --offline lint tests/custom_rules.yaml tests/custom_alerts.yaml
|
||||
'''
|
||||
|
||||
[tasks.check-runbooks]
|
||||
description = "Check runbook coverage for all alerts, including opt-in alerts"
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { enableActiveDirectory: true, enableTime: true } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
run = '''jsonnet -e '(import "mixin.libsonnet") { _config+:: { collectors: { [name]: true for name in std.objectFields(super.collectors) } } }.prometheusAlerts' | python3 tests/check_runbooks.py ../../docs/runbooks'''
|
||||
|
||||
[tasks.test]
|
||||
description = "Run all mixin lint, runbook, and rule tests"
|
||||
depends = ["lint", "check-runbooks", "vendor"]
|
||||
depends = ["lint", "check-rules", "check-runbooks", "vendor"]
|
||||
run = [
|
||||
"jsonnet tests/runbooks.test.jsonnet",
|
||||
"jsonnet tests/hyperv.test.jsonnet > /dev/null",
|
||||
"jsonnet tests/alerts.test.jsonnet",
|
||||
"jsonnet -J vendor tests/dashboard.test.jsonnet > /dev/null",
|
||||
"promtool test rules tests/*.test.yaml",
|
||||
]
|
||||
|
||||
[tasks.clean]
|
||||
description = "Remove generated rule and alert files"
|
||||
run = "rm -f windows_rules.yaml windows_alerts.yaml tests/custom_rules.yaml tests/custom_alerts.yaml"
|
||||
description = "Remove the generated test rule and alert files"
|
||||
run = "rm -f tests/custom_rules.yaml tests/custom_alerts.yaml"
|
||||
|
||||
@@ -1 +1 @@
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules)
|
||||
std.manifestYamlDoc((import 'mixin.libsonnet').prometheusRules, quote_keys=false)
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
{ record: 'windows:cpu_usage:ratio', expr: q.cpuUsage },
|
||||
{ record: 'windows:cpu_cores:count', expr: 'count without (core, mode) (%s)' % q.metric('windows_cpu_time_seconds_total', ['mode="idle"']) },
|
||||
{ record: 'windows:cpu_time:ratio_rate', expr: 'avg without (core) (%s)' % q.rate('windows_cpu_time_seconds_total') },
|
||||
{ record: 'windows:cpu_queue_per_core:ratio', expr: q.cpuQueuePerCore },
|
||||
{ record: 'windows:memory_usage:ratio', expr: q.memoryUsage },
|
||||
{
|
||||
record: 'windows:memory_used:bytes',
|
||||
@@ -56,6 +57,12 @@
|
||||
expr: q.rate('windows_net_bytes_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{
|
||||
record: 'windows:net_%s_packets:rate' % direction,
|
||||
expr: q.rate('windows_net_packets_%s_total' % direction, [c.nicSelector]),
|
||||
}
|
||||
for direction in ['received', 'sent']
|
||||
] + [
|
||||
{ record: 'windows:net_%s_utilization:ratio' % direction, expr: q.networkUtilization(direction) }
|
||||
for direction in ['received', 'sent']
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
// Checks the alert configuration: collector gating, per-alert switches,
|
||||
// keep_firing_for and the group layout.
|
||||
local mixin = import '../mixin.libsonnet';
|
||||
local config = (import '../config.libsonnet')._config;
|
||||
local everything = mixin { _config+:: { collectors: { [name]: true for name in std.objectFields(config.collectors) } } };
|
||||
local optionalGroups = ['windows-pagefile', 'windows-diskdrive', 'windows-license', 'windows-active-directory', 'windows-time', 'windows-scheduled-task', 'windows-update', 'windows-hyperv', 'windows-cluster', 'windows-storage-spaces', 'windows-dhcp', 'windows-dns'];
|
||||
local names(alerts) = std.sort([rule.alert for group in alerts.groups for rule in group.rules]);
|
||||
local groupNames(alerts) = [group.name for group in alerts.groups];
|
||||
|
||||
// Every configured switch belongs to a generated alert, and every alert has a switch.
|
||||
local all = std.set(names(everything.prometheusAlerts));
|
||||
assert all == std.set(std.objectFields(config.alerts)) : 'alerts map differs from generated alerts: %s' % std.manifestJson([std.setDiff(all, std.set(std.objectFields(config.alerts))), std.setDiff(std.set(std.objectFields(config.alerts)), all)]);
|
||||
|
||||
// Groups follow the collector areas; opt-in groups appear only when enabled.
|
||||
assert groupNames(mixin.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-disk', 'windows-network', 'windows-service', 'windows-system'] : std.manifestJson(groupNames(mixin.prometheusAlerts));
|
||||
// One file per collector, one group per file: the groups of every enabled
|
||||
// collector come in the order of the imports in alerts/alerts.libsonnet.
|
||||
assert std.set(groupNames(everything.prometheusAlerts)) == std.set(groupNames(mixin.prometheusAlerts) + optionalGroups) : std.manifestJson(groupNames(everything.prometheusAlerts));
|
||||
assert groupNames(everything.prometheusAlerts) == ['windows-exporter', 'windows-cpu', 'windows-memory', 'windows-pagefile', 'windows-disk', 'windows-diskdrive', 'windows-network', 'windows-service', 'windows-system', 'windows-license'] + optionalGroups[3:] : std.manifestJson(groupNames(everything.prometheusAlerts));
|
||||
local adOnly = mixin { _config+:: { collectors+: { ad: true } } };
|
||||
assert groupNames(adOnly.prometheusAlerts) == groupNames(mixin.prometheusAlerts) + ['windows-active-directory'];
|
||||
local pagefile = mixin { _config+:: { collectors+: { pagefile: true, diskdrive: true, license: true } } };
|
||||
assert std.setDiff(names(pagefile.prometheusAlerts), names(mixin.prometheusAlerts)) == ['WindowsDiskDriveNotOK', 'WindowsLicenseNotGenuine', 'WindowsPagefileAlmostFull'];
|
||||
|
||||
// Disabling a collector removes its alerts and an empty group; the alerts of a
|
||||
// rule that needs two collectors go with either.
|
||||
local noNet = mixin { _config+:: { collectors+: { net: false } } };
|
||||
assert !std.member(groupNames(noNet.prometheusAlerts), 'windows-network');
|
||||
local noSystem = mixin { _config+:: { collectors+: { system: false } } };
|
||||
assert names(noSystem.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUSaturation', 'WindowsHostRebooted']);
|
||||
assert std.member(names(noSystem.prometheusAlerts), 'WindowsCPUHighUsage');
|
||||
// A replaced collectors object counts missing collectors as disabled.
|
||||
local onlyCPU = mixin { _config+:: { collectors: { cpu: true } } };
|
||||
assert names(onlyCPU.prometheusAlerts) == ['WindowsCPUHighUsage', 'WindowsCollectorFailed', 'WindowsCollectorTimeout', 'WindowsExporterDown', 'WindowsScrapeQueueTimeout'] : std.manifestJson(names(onlyCPU.prometheusAlerts));
|
||||
|
||||
// Single alerts switch off by name, with alerts+: or a replaced object.
|
||||
local noReboot = mixin { _config+:: { alerts+: { WindowsHostRebooted: false } } };
|
||||
assert names(noReboot.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsHostRebooted']);
|
||||
assert !std.member(groupNames(noReboot.prometheusAlerts), 'windows-system');
|
||||
local replaced = mixin { _config+:: { alerts: { WindowsCPUHighUsage: false } } };
|
||||
assert names(replaced.prometheusAlerts) == std.setDiff(names(mixin.prometheusAlerts), ['WindowsCPUHighUsage']);
|
||||
|
||||
// keep_firing_for applies to threshold alerts only and can be turned off.
|
||||
local rules(alerts) = [rule for group in alerts.groups for rule in group.rules];
|
||||
local stateAlerts = [
|
||||
'WindowsExporterDown',
|
||||
'WindowsCollectorFailed',
|
||||
'WindowsCollectorTimeout',
|
||||
'WindowsScrapeQueueTimeout',
|
||||
'WindowsServiceNotRunning',
|
||||
'WindowsHostRebooted',
|
||||
'WindowsClockNotSynchronising',
|
||||
'WindowsClockSourceLocal',
|
||||
'WindowsDiskDriveNotOK',
|
||||
'WindowsLicenseNotGenuine',
|
||||
'WindowsScheduledTaskFailed',
|
||||
'WindowsUpdatesPending',
|
||||
'WindowsHyperVWMIUnhealthy',
|
||||
'WindowsHyperVReplicationCritical',
|
||||
'WindowsHyperVReplicationWarning',
|
||||
'WindowsClusterNodeDown',
|
||||
'WindowsClusterResourceFailed',
|
||||
'WindowsClusterResourceGroupFailed',
|
||||
'WindowsStoragePoolUnhealthy',
|
||||
'WindowsStorageVirtualDiskUnhealthy',
|
||||
];
|
||||
assert std.all([
|
||||
std.objectHas(rule, 'keep_firing_for') == !std.member(stateAlerts, rule.alert)
|
||||
for rule in rules(everything.prometheusAlerts)
|
||||
]) : 'keep_firing_for set on a state alert or missing on a threshold alert';
|
||||
assert std.all([rule.keep_firing_for == '5m' for rule in rules(everything.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]);
|
||||
local noKeep = mixin { _config+:: { keepFiringFor: '' } };
|
||||
assert std.length([rule for rule in rules(noKeep.prometheusAlerts) if std.objectHas(rule, 'keep_firing_for')]) == 0;
|
||||
|
||||
// The info severity label is configurable.
|
||||
local ticket = mixin { _config+:: { infoSeverity: 'ticket' } };
|
||||
assert [rule.labels.severity for rule in rules(ticket.prometheusAlerts) if rule.alert == 'WindowsHostRebooted'] == ['ticket'];
|
||||
local updates = everything { _config+:: { infoSeverity: 'ticket', updatePendingDuration: '3d' } };
|
||||
assert [[rule.labels.severity, rule['for']] for rule in rules(updates.prometheusAlerts) if rule.alert == 'WindowsUpdatesPending'] == [['ticket', '3d']];
|
||||
|
||||
{ result: 'Alert configuration checks passed for %d alerts' % std.length(all) }
|
||||
@@ -1,10 +1,18 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../windows_alerts.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
- ../../monitoring/windows_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Commit pressure, disk latency and network failures wait and recover
|
||||
@@ -30,6 +38,10 @@ tests:
|
||||
values: '0x31'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+120x20 2520x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+3000x31'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
@@ -40,7 +52,8 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/
|
||||
summary: Windows memory commit usage is high
|
||||
description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 21m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 25m, exp_alerts: [{exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}, exp_annotations: {runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/, summary: Windows memory commit usage is high, description: 'Committed memory on host:9182 has exceeded 90% of the commit limit for 15 minutes.'}}]}
|
||||
- {alertname: WindowsMemoryCommitHighUsage, eval_time: 26m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskReadLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -50,7 +63,7 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/
|
||||
summary: Windows disk read latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per read operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskReadLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsDiskWriteLatencyHigh
|
||||
eval_time: 16m
|
||||
@@ -60,27 +73,27 @@ tests:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/
|
||||
summary: Windows disk write latency is high
|
||||
description: 'Volume C: on host:9182 has averaged more than 0.05 seconds per write operation for 15 minutes.'
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 17m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskWriteLatencyHigh, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkErrors
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 25m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 17m, exp_alerts: []}
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkErrors, eval_time: 30m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 15m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 18m
|
||||
eval_time: 16m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1 per second for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 25m, exp_alerts: []}
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 1% of all packets for 15 minutes.'
|
||||
- {alertname: WindowsNetworkDiscards, eval_time: 30m, exp_alerts: []}
|
||||
|
||||
- name: New resource alerts stay quiet at exact thresholds and clock alerts are disabled
|
||||
interval: 1m
|
||||
@@ -105,6 +118,10 @@ tests:
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+30x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+3600x20'
|
||||
- series: 'windows_time_ntp_client_time_sources{job="windows_exporter", instance="host:9182"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_time_computed_time_offset_seconds{job="windows_exporter", instance="host:9182"}'
|
||||
@@ -178,7 +195,7 @@ tests:
|
||||
summary: Windows CPU usage is high
|
||||
description: 'CPU usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsCPUHighUsage
|
||||
eval_time: 25m
|
||||
eval_time: 30m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 14m
|
||||
@@ -192,16 +209,16 @@ tests:
|
||||
summary: Windows memory usage is high
|
||||
description: 'Physical memory usage on host:9182 has exceeded 90% for 15 minutes.'
|
||||
- alertname: WindowsMemoryHighUsage
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Disk alerts retain volume labels and fire at both severity levels
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_logical_disk_free_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '3x15 30x5'
|
||||
values: '3x15 30x10'
|
||||
- series: 'windows_logical_disk_size_bytes{job="windows_exporter", instance="host:9182", volume="C:"}'
|
||||
values: '100x21'
|
||||
values: '100x26'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 14m
|
||||
@@ -220,7 +237,7 @@ tests:
|
||||
summary: Windows disk space is low
|
||||
description: 'Volume C: on host:9182 has less than 5% free space. Windows may delay space metrics by 10–15 minutes.'
|
||||
- alertname: WindowsDiskAlmostFull
|
||||
eval_time: 16m
|
||||
eval_time: 21m
|
||||
exp_alerts: []
|
||||
|
||||
- name: Healthy resources at the exact thresholds do not fire
|
||||
@@ -291,3 +308,274 @@ tests:
|
||||
- {alertname: WindowsMemoryHighUsage, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskAlmostFull, eval_time: 70m, exp_alerts: []}
|
||||
- {alertname: WindowsDiskFillingUp, eval_time: 70m, exp_alerts: []}
|
||||
|
||||
- name: Collector and scrape queue timeouts wait ten minutes and recover without keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="service"}'
|
||||
values: '1x10 0x5'
|
||||
- series: 'windows_exporter_collector_timeout{job="windows_exporter", instance="host:9182", collector="cpu"}'
|
||||
values: '0x15'
|
||||
- series: 'windows_exporter_scrape_queue_timeout{job="windows_exporter", instance="host:9182"}'
|
||||
values: '1x10 0x5'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsCollectorTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', collector: service, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/
|
||||
summary: Windows collector exceeds the scrape timeout
|
||||
description: 'Collector service on host:9182 has not finished within the scrape timeout for 10 minutes.'
|
||||
- {alertname: WindowsCollectorTimeout, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsScrapeQueueTimeout
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/
|
||||
summary: Windows exporter scrapes time out in the queue
|
||||
description: 'Scrapes of host:9182 have timed out waiting for an earlier scrape to finish for 10 minutes.'
|
||||
- {alertname: WindowsScrapeQueueTimeout, eval_time: 11m, exp_alerts: []}
|
||||
|
||||
- name: CPU saturation and paging use recordings, wait fifteen minutes and keep firing five
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '9x25 0x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x35'
|
||||
- series: 'windows_memory_swap_page_operations_total{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '0+90000x20 1800000x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- alertname: WindowsCPUSaturation
|
||||
eval_time: 31m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/
|
||||
summary: Windows processor queue is long
|
||||
description: 'More than 2 threads per logical processor have been waiting for CPU time on host:9182 for 15 minutes.'
|
||||
- {alertname: WindowsCPUSaturation, eval_time: 32m, exp_alerts: []}
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 18m, exp_alerts: []}
|
||||
- alertname: WindowsMemoryPagingHigh
|
||||
eval_time: 19m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
|
||||
summary: Windows memory paging is high
|
||||
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
|
||||
- alertname: WindowsMemoryPagingHigh
|
||||
eval_time: 26m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/
|
||||
summary: Windows memory paging is high
|
||||
description: 'Paging on host:9182 has exceeded 1000 pages per second for 15 minutes.'
|
||||
- {alertname: WindowsMemoryPagingHigh, eval_time: 27m, exp_alerts: []}
|
||||
|
||||
- name: Network utilization fires in one direction and interface flapping counts status changes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_net_bytes_received_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0+6000x20 120000x15'
|
||||
- series: 'windows_net_bytes_sent_total{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '0x36'
|
||||
- series: 'windows_net_current_bandwidth_bytes{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
values: '100x36'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="up"}'
|
||||
values: '1 0 1 0 1 0 1 0 1 0 1x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Ethernet", status="down"}'
|
||||
values: '0 1 0 1 0 1 0 1 0 1 0x10'
|
||||
- series: 'windows_net_nic_operation_status{job="windows_exporter", instance="host:9182", nic="Wi-Fi", status="up"}'
|
||||
values: '0x21'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 19m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkUtilizationHigh
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
|
||||
summary: Windows network interface utilization is high
|
||||
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
|
||||
- alertname: WindowsNetworkUtilizationHigh
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/
|
||||
summary: Windows network interface utilization is high
|
||||
description: 'Traffic on interface Ethernet on host:9182 has exceeded 90% of its bandwidth in at least one direction for 15 minutes.'
|
||||
- {alertname: WindowsNetworkUtilizationHigh, eval_time: 26m, exp_alerts: []}
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsNetworkInterfaceFlapping
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', nic: Ethernet, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/
|
||||
summary: Windows network interface is flapping
|
||||
description: 'Interface Ethernet on host:9182 has changed its operational status more than twice in 5m.'
|
||||
- {alertname: WindowsNetworkInterfaceFlapping, eval_time: 17m, exp_alerts: []}
|
||||
|
||||
- name: Service alerts skip trigger-start and manual services and count restarts by process
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="Spooler", state="running"}'
|
||||
values: '0x15 1x8'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="Spooler", start_mode="auto"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="dcsvc", state="running"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="dcsvc", start_mode="auto"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", service="dcsvc", trigger_start="true", delayed_start="false"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", service="RemoteRegistry", state="running"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="manual"}'
|
||||
values: '1x23'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", service="RemoteRegistry", start_mode="auto"}'
|
||||
values: '0x23'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="100"}'
|
||||
values: '1000x4 _x19'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="200"}'
|
||||
values: '_x5 2000x4 _x14'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="300"}'
|
||||
values: '_x10 3000x4 _x9'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="W3SVC", process_id="400"}'
|
||||
values: '_x15 4000x9'
|
||||
- series: 'windows_service_process{job="windows_exporter", instance="host:9182", service="Spooler", process_id="500"}'
|
||||
values: '_x16 5000x8'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceNotRunning
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: Spooler, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
|
||||
summary: Windows automatic service is not running
|
||||
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
|
||||
- {alertname: WindowsServiceNotRunning, eval_time: 16m, exp_alerts: []}
|
||||
- {alertname: WindowsServiceCrashLooping, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsServiceCrashLooping
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
|
||||
summary: Windows service is restarting repeatedly
|
||||
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
|
||||
- alertname: WindowsServiceCrashLooping
|
||||
eval_time: 23m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', service: W3SVC, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/
|
||||
summary: Windows service is restarting repeatedly
|
||||
description: 'Service W3SVC on host:9182 has restarted more than 2 times in 15 minutes.'
|
||||
- {alertname: WindowsServiceCrashLooping, eval_time: 24m, exp_alerts: []}
|
||||
|
||||
- name: Service alert matches within a cluster when instance names repeat across clusters
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="a", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", state="running"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="b", service="Spooler", trigger_start="false", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_state{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", state="running"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_service_start_mode{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", start_mode="auto"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_service_info{job="windows_exporter", instance="host:9182", cluster="c", service="Spooler", trigger_start="true", delayed_start="false"}'
|
||||
values: '1x20'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsServiceNotRunning
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', cluster: a, service: Spooler, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/
|
||||
summary: Windows automatic service is not running
|
||||
description: 'Service Spooler on host:9182 starts automatically but has not been running for 15 minutes.'
|
||||
|
||||
- name: Host reboot fires once for the first ten minutes after a new boot time
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
|
||||
values: '-100000x10 540x15'
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="steady:9182"}'
|
||||
values: '-100000x25'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHostRebooted, eval_time: 10m, exp_alerts: []}
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 11m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 2m 0s ago.'
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 18m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 9m 0s ago.'
|
||||
- {alertname: WindowsHostRebooted, eval_time: 19m, exp_alerts: []}
|
||||
|
||||
- name: Host reboot fires after a scrape gap longer than ten minutes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_boot_time_timestamp{job="windows_exporter", instance="host:9182"}'
|
||||
values: '-100000x5 stale _x11 960x20'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHostRebooted, eval_time: 17m, exp_alerts: []}
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 18m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 2m 0s ago.'
|
||||
- alertname: WindowsHostRebooted
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows_exporter, instance: 'host:9182', severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/
|
||||
summary: Windows host has rebooted
|
||||
description: 'host:9182 rebooted 9m 0s ago.'
|
||||
- {alertname: WindowsHostRebooted, eval_time: 26m, exp_alerts: []}
|
||||
|
||||
@@ -9,15 +9,33 @@
|
||||
diskFreeWarningThreshold: 20,
|
||||
diskFreeCriticalThreshold: 10,
|
||||
criticalSeverity: 'page',
|
||||
enableActiveDirectory: true,
|
||||
enableTime: true,
|
||||
runbookURLPattern: 'https://runbooks.example.com/windows/%s/',
|
||||
memoryCommitHighUsageThreshold: 80,
|
||||
diskLatencyThresholdSeconds: 0.02,
|
||||
networkErrorRateThreshold: 0.5,
|
||||
networkDiscardRateThreshold: 0.5,
|
||||
networkErrorRatioThreshold: 0.5,
|
||||
networkDiscardRatioThreshold: 0.5,
|
||||
clockOffsetThresholdSeconds: 0.1,
|
||||
adPendingReplicationThreshold: 10,
|
||||
adReplicationFailureRateThreshold: 0.5,
|
||||
collectors+: {
|
||||
ad: true,
|
||||
dhcp: true,
|
||||
diskdrive: true,
|
||||
dns: true,
|
||||
hyperv: true,
|
||||
license: true,
|
||||
mscluster: true,
|
||||
pagefile: true,
|
||||
scheduled_task: true,
|
||||
storage_spaces: true,
|
||||
time: true,
|
||||
update: true,
|
||||
},
|
||||
pagefileUsageThreshold: 75,
|
||||
dhcpScopeFreeWarningThreshold: 20,
|
||||
dhcpScopeFreeCriticalThreshold: 5,
|
||||
updatePendingDuration: '2d',
|
||||
dnsZoneTransferFailureRateThreshold: 0.05,
|
||||
taskSelector: 'task!~"/Ignored.*"',
|
||||
},
|
||||
}
|
||||
|
||||
@@ -4,7 +4,13 @@ rule_files:
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
@@ -52,6 +58,14 @@ tests:
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_outbound_discarded_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Ethernet"}'
|
||||
values: '0+3000x20'
|
||||
- series: 'windows_net_packets_received_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
- series: 'windows_net_packets_sent_total{job="windows-custom", instance="host:9182", cluster="test", nic="Loopback"}'
|
||||
values: '0+120x20'
|
||||
alert_rule_test:
|
||||
- alertname: WindowsMemoryCommitHighUsage
|
||||
eval_time: 20m
|
||||
@@ -84,7 +98,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkerrors/
|
||||
summary: Windows network packet errors are high
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet errors on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
- alertname: WindowsNetworkDiscards
|
||||
eval_time: 20m
|
||||
exp_alerts:
|
||||
@@ -92,7 +106,7 @@ tests:
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsnetworkdiscards/
|
||||
summary: Windows network packet discards are high
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5 per second for 15 minutes.'
|
||||
description: 'Packet discards on interface Ethernet on host:9182 have exceeded 0.5% of all packets for 15 minutes.'
|
||||
|
||||
- name: Clock alerts opt in, use absolute offset, custom URLs and recover
|
||||
interval: 1m
|
||||
@@ -129,7 +143,7 @@ tests:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclockskewdetected/
|
||||
summary: Windows clock offset is high
|
||||
description: 'Windows Time on host:9182 reports a clock offset above 0.1 seconds for 10 minutes.'
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 11m, exp_alerts: []}
|
||||
- {alertname: WindowsClockSkewDetected, eval_time: 16m, exp_alerts: []}
|
||||
- name: Custom selectors, thresholds and critical severity propagate to alerts
|
||||
interval: 1m
|
||||
input_series:
|
||||
@@ -239,7 +253,7 @@ tests:
|
||||
summary: Active Directory replication queue is high
|
||||
description: 'Pending replication operations on host:9182 have exceeded 10 for 15 minutes.'
|
||||
- alertname: WindowsADPendingReplication
|
||||
eval_time: 21m
|
||||
eval_time: 26m
|
||||
exp_alerts: []
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 16m
|
||||
@@ -253,5 +267,5 @@ tests:
|
||||
summary: Active Directory replication requests are failing
|
||||
description: 'Replication synchronization requests on host:9182 have been failing for 15 minutes.'
|
||||
- alertname: WindowsADReplicationFailures
|
||||
eval_time: 23m
|
||||
eval_time: 28m
|
||||
exp_alerts: []
|
||||
|
||||
@@ -0,0 +1,508 @@
|
||||
# Alerts of the opt-in collectors, rendered with tests/config.libsonnet.
|
||||
rule_files:
|
||||
- custom_rules.yaml
|
||||
- custom_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-system
|
||||
- windows-time
|
||||
- windows-scheduled-task
|
||||
- windows-update
|
||||
- windows-hyperv
|
||||
- windows-cluster
|
||||
- windows-storage-spaces
|
||||
- windows-dhcp
|
||||
- windows-dns
|
||||
|
||||
tests:
|
||||
- name: Page file usage fires above the custom threshold and resolves after keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
|
||||
values: '20x19 50x10'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="C:\\pagefile.sys"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
|
||||
values: '30x30'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="D:\\pagefile.sys"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_pagefile_free_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_pagefile_limit_bytes{job="windows-custom", instance="host:9182", cluster="test", file="E:\\pagefile.sys"}'
|
||||
values: '0x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsPagefileAlmostFull, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsPagefileAlmostFull
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
|
||||
summary: Windows page file is almost full
|
||||
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
|
||||
- alertname: WindowsPagefileAlmostFull
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, file: 'C:\pagefile.sys', severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowspagefilealmostfull/
|
||||
summary: Windows page file is almost full
|
||||
description: 'Page file C:\pagefile.sys on host:9182 has been more than 75% in use for 15 minutes.'
|
||||
- {alertname: WindowsPagefileAlmostFull, eval_time: 25m, exp_alerts: []}
|
||||
|
||||
- name: Disk drive status reports the status the drive has, resolves at once and skips Unknown, and the license alert waits an hour
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="OK"}'
|
||||
values: '0x9 1x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE0", status="Pred Fail"}'
|
||||
values: '1x9 0x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="OK"}'
|
||||
values: '0x20'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE2", status="Unknown"}'
|
||||
values: '1x20'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="OK"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_diskdrive_status{job="windows-custom", instance="host:9182", cluster="test", device_id="\\\\.\\PHYSICALDRIVE1", status="Pred Fail"}'
|
||||
values: '0x10'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="genuine"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="host:9182", cluster="test", state="invalid_license"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="genuine"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_license_status{job="windows-custom", instance="licensed:9182", cluster="test", state="invalid_license"}'
|
||||
values: '0x60'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDiskDriveNotOK, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsDiskDriveNotOK
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
|
||||
summary: Windows disk drive is not healthy
|
||||
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
|
||||
- alertname: WindowsDiskDriveNotOK
|
||||
eval_time: 9m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, device_id: '\\.\PHYSICALDRIVE0', status: Pred Fail, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdiskdrivenotok/
|
||||
summary: Windows disk drive is not healthy
|
||||
description: 'Disk drive \\.\PHYSICALDRIVE0 on host:9182 has reported status Pred Fail for 5 minutes.'
|
||||
- {alertname: WindowsDiskDriveNotOK, eval_time: 10m, exp_alerts: []}
|
||||
- {alertname: WindowsLicenseNotGenuine, eval_time: 59m, exp_alerts: []}
|
||||
- alertname: WindowsLicenseNotGenuine
|
||||
eval_time: 60m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, state: invalid_license, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowslicensenotgenuine/
|
||||
summary: Windows license is not genuine
|
||||
description: 'Windows on host:9182 has reported license status invalid_license for 1 hour.'
|
||||
|
||||
- name: Clock source alert names the local source and ignores NTP and domain sources
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="Local CMOS Clock"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="host:9182", cluster="test", type="NTP"}'
|
||||
values: '0x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NT5DS"}'
|
||||
values: '1x10'
|
||||
- series: 'windows_time_clock_sync_source{job="windows-custom", instance="dc:9182", cluster="test", type="NoSync"}'
|
||||
values: '0x10'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsClockSourceLocal, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsClockSourceLocal
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, type: Local CMOS Clock, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclocksourcelocal/
|
||||
summary: Windows clock has no external time source
|
||||
description: 'Windows Time on host:9182 has used Local CMOS Clock as its clock source for 10 minutes.'
|
||||
|
||||
- name: Scheduled task alert fires for unmapped and terminated results after an hour and skips disabled and deselected tasks
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", status="success"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Cleanup", status="terminated"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="has_not_run"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Report", status="unknown"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Ignored/Flaky", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_last_result_status{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", status="unknown"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="disabled"}'
|
||||
values: '1x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Retired", state="ready"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="disabled"}'
|
||||
values: '0x60'
|
||||
- series: 'windows_scheduled_task_state{job="windows-custom", instance="host:9182", cluster="test", task="/Backup", state="ready"}'
|
||||
values: '1x60'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsScheduledTaskFailed, eval_time: 59m, exp_alerts: []}
|
||||
- alertname: WindowsScheduledTaskFailed
|
||||
eval_time: 60m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Backup, status: unknown, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
|
||||
summary: Windows scheduled task did not succeed
|
||||
description: 'The last run of task /Backup on host:9182 ended with result unknown, and the task has not succeeded in the hour since.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, task: /Cleanup, status: terminated, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsscheduledtaskfailed/
|
||||
summary: Windows scheduled task did not succeed
|
||||
description: 'The last run of task /Cleanup on host:9182 ended with result terminated, and the task has not succeeded in the hour since.'
|
||||
|
||||
- name: Pending updates fire as info after the configured duration and count installations only
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="a", revision="1", category="Security Updates", severity="Critical", title="KB1", deployment_action="installation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="b", revision="1", category="Updates", severity="", title="KB2", deployment_action="installation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="host:9182", cluster="test", id="c", revision="1", category="Updates", severity="", title="KB3", deployment_action="uninstallation"}'
|
||||
values: '1x2880'
|
||||
- series: 'windows_update_pending_info{job="windows-custom", instance="clean:9182", cluster="test", id="d", revision="1", category="Updates", severity="", title="KB4", deployment_action="uninstallation"}'
|
||||
values: '1x2880'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsUpdatesPending, eval_time: 47h, exp_alerts: []}
|
||||
- alertname: WindowsUpdatesPending
|
||||
eval_time: 48h
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, deployment_action: installation, severity: info}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsupdatespending/
|
||||
summary: Windows updates are waiting for installation
|
||||
description: 'host:9182 has had 2 updates waiting for installation for 2d.'
|
||||
|
||||
- name: Hyper-V alerts cover the WMI namespace and both replication health levels
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_hyperv_wmi_health{job="windows-custom", instance="hv:9182", cluster="test"}'
|
||||
values: '0x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="web", vm_id="1", relationship="primary"}'
|
||||
values: '3x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="db", vm_id="2", relationship="extended"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_hyperv_replica_vm_health{job="windows-custom", instance="hv:9182", cluster="test", vm="dev", vm_id="3", relationship="primary"}'
|
||||
values: '0x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsHyperVWMIUnhealthy, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVWMIUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervwmiunhealthy/
|
||||
summary: Hyper-V WMI namespace is not responding
|
||||
description: 'The Hyper-V WMI namespace on hv:9182 has not responded for 10 minutes; Hyper-V metrics are missing.'
|
||||
- {alertname: WindowsHyperVReplicationCritical, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVReplicationCritical
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: web, vm_id: '1', relationship: primary, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationcritical/
|
||||
summary: Hyper-V replication is critical
|
||||
description: 'Replication of virtual machine web (primary) on hv:9182 has been in critical health for 5 minutes.'
|
||||
- {alertname: WindowsHyperVReplicationWarning, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsHyperVReplicationWarning
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'hv:9182', cluster: test, vm: db, vm_id: '2', relationship: extended, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowshypervreplicationwarning/
|
||||
summary: Hyper-V replication has a warning
|
||||
description: 'Replication of virtual machine db (extended) on hv:9182 has been in warning health for 15 minutes.'
|
||||
|
||||
- name: Cluster alerts fire for down nodes, failed resources and groups, and nearly full shared volumes
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node2"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node1:9182", cluster="test", node="node1"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_mscluster_node_state{job="windows-custom", instance="node3:9182", cluster="test", node="node2"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="IP Address", resource_group="SQL", resource="SQL IP"}'
|
||||
values: '4x30'
|
||||
- series: 'windows_mscluster_resource_state{job="windows-custom", instance="node1:9182", cluster="test", type="Network Name", resource_group="SQL", resource="SQL Name"}'
|
||||
values: '3x30'
|
||||
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="SQL"}'
|
||||
values: '2x30'
|
||||
- series: 'windows_mscluster_resourcegroup_state{job="windows-custom", instance="node1:9182", cluster="test", resource_group="Cluster Group"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
|
||||
values: '15x19 50x10'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume1", volume_guid="g1"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
|
||||
values: '5x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Volume2", volume_guid="g2"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="ClusterPerformanceHistory", volume_guid="g3"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_mscluster_shared_volumes_free_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_mscluster_shared_volumes_total_bytes{job="windows-custom", instance="node1:9182", cluster="test", shared_volume="Empty", volume_guid="g4"}'
|
||||
values: '0x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsClusterNodeDown, eval_time: 4m, exp_alerts: []}
|
||||
- alertname: WindowsClusterNodeDown
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, node: node2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
|
||||
summary: Failover cluster node is down
|
||||
description: 'Cluster node node2 has been down for 5 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node3:9182', cluster: test, node: node2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusternodedown/
|
||||
summary: Failover cluster node is down
|
||||
description: 'Cluster node node2 has been down for 5 minutes, as reported by node3:9182.'
|
||||
- alertname: WindowsClusterResourceFailed
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, type: IP Address, resource_group: SQL, resource: SQL IP, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcefailed/
|
||||
summary: Failover cluster resource has failed
|
||||
description: 'Cluster resource SQL IP (IP Address) in group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterResourceGroupFailed
|
||||
eval_time: 5m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, resource_group: SQL, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclusterresourcegroupfailed/
|
||||
summary: Failover cluster resource group has failed
|
||||
description: 'Cluster resource group SQL has been in the failed state for 5 minutes, as reported by node1:9182.'
|
||||
- {alertname: WindowsClusterSharedVolumeAlmostFull, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume1, volume_guid: g1, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume1 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
- alertname: WindowsClusterSharedVolumeAlmostFull
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 20% free space for 15 minutes, as reported by node1:9182.'
|
||||
- exp_labels: {job: windows-custom, instance: 'node1:9182', cluster: test, shared_volume: Volume2, volume_guid: g2, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsclustersharedvolumealmostfull/
|
||||
summary: Cluster shared volume is almost full
|
||||
description: 'Cluster shared volume Volume2 has had less than 10% free space for 15 minutes, as reported by node1:9182.'
|
||||
|
||||
- name: Storage Spaces alerts skip the primordial pool and separate warning from unhealthy
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool A", unique_id="a", primordial="false"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Pool B", unique_id="b", primordial="false"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_pool_health_status{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_pool_info{job="windows-custom", instance="host:9182", cluster="test", pool="Primordial", unique_id="p", primordial="true"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Data", unique_id="v1"}'
|
||||
values: '2x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Logs", unique_id="v2"}'
|
||||
values: '1x15'
|
||||
- series: 'windows_storage_spaces_virtual_disk_health_status{job="windows-custom", instance="host:9182", cluster="test", virtual_disk="Scratch", unique_id="v3"}'
|
||||
values: '0x15'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsStoragePoolUnhealthy, eval_time: 9m, exp_alerts: []}
|
||||
- alertname: WindowsStoragePoolUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool A, unique_id: a, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
|
||||
summary: Storage pool is unhealthy
|
||||
description: 'Storage pool Pool A on host:9182 has reported unhealthy status for 10 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, pool: Pool B, unique_id: b, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragepoolunhealthy/
|
||||
summary: Storage pool has a health warning
|
||||
description: 'Storage pool Pool B on host:9182 has reported warning status for 10 minutes.'
|
||||
- alertname: WindowsStorageVirtualDiskUnhealthy
|
||||
eval_time: 10m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Data, unique_id: v1, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
|
||||
summary: Storage Spaces virtual disk is unhealthy
|
||||
description: 'Virtual disk Data on host:9182 has reported unhealthy status for 10 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'host:9182', cluster: test, virtual_disk: Logs, unique_id: v2, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsstoragevirtualdiskunhealthy/
|
||||
summary: Storage Spaces virtual disk has a health warning
|
||||
description: 'Virtual disk Logs on host:9182 has reported warning status for 10 minutes.'
|
||||
|
||||
- name: DHCP scope exhaustion counts active scopes only and uses both thresholds
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
|
||||
values: '10x19 50x10'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24"}'
|
||||
values: '90x19 50x10'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Enabled"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.0.0/24", state="Disabled"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
|
||||
values: '2x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24"}'
|
||||
values: '98x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.1.0/24", state="EnabledSwitched"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24"}'
|
||||
values: '100x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Disabled"}'
|
||||
values: '1x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.2.0/24", state="Enabled"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_free{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_addresses_in_use{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24"}'
|
||||
values: '0x30'
|
||||
- series: 'windows_dhcp_scope_state{job="windows-custom", instance="dhcp:9182", cluster="test", scope="10.0.3.0/24", state="Enabled"}'
|
||||
values: '1x30'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDHCPScopeAlmostExhausted, eval_time: 14m, exp_alerts: []}
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 15m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 24m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.0.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.0.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
- alertname: WindowsDHCPScopeAlmostExhausted
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 20% free addresses for 15 minutes.'
|
||||
- exp_labels: {job: windows-custom, instance: 'dhcp:9182', cluster: test, scope: 10.0.1.0/24, severity: page}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdhcpscopealmostexhausted/
|
||||
summary: DHCP scope is almost exhausted
|
||||
description: 'DHCP scope 10.0.1.0/24 on dhcp:9182 has had less than 5% free addresses for 15 minutes.'
|
||||
|
||||
- name: DNS zone transfer failures fire on a rising counter and resolve after keep_firing_for
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="dns:9182", cluster="test"}'
|
||||
values: '0+6x20 120x15'
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="quiet:9182", cluster="test"}'
|
||||
values: '7x36'
|
||||
- series: 'windows_dns_zone_transfer_failures_total{job="windows-custom", instance="scanned:9182", cluster="test"}'
|
||||
values: '0+1x36'
|
||||
alert_rule_test:
|
||||
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 16m, exp_alerts: []}
|
||||
- alertname: WindowsDNSZoneTransferFailures
|
||||
eval_time: 17m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
|
||||
summary: DNS zone transfers are failing
|
||||
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
|
||||
- alertname: WindowsDNSZoneTransferFailures
|
||||
eval_time: 25m
|
||||
exp_alerts:
|
||||
- exp_labels: {job: windows-custom, instance: 'dns:9182', cluster: test, severity: warning}
|
||||
exp_annotations:
|
||||
runbook_url: https://runbooks.example.com/windows/windowsdnszonetransferfailures/
|
||||
summary: DNS zone transfers are failing
|
||||
description: 'Zone transfers from DNS server dns:9182 have been failing for 15 minutes.'
|
||||
- {alertname: WindowsDNSZoneTransferFailures, eval_time: 26m, exp_alerts: []}
|
||||
@@ -1,10 +1,18 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../windows_alerts.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
- ../../monitoring/windows_alerts.yaml
|
||||
evaluation_interval: 1m
|
||||
group_eval_order:
|
||||
- windows-exporter-recording
|
||||
- windows-exporter-alerts
|
||||
- windows-exporter
|
||||
- windows-cpu
|
||||
- windows-memory
|
||||
- windows-disk
|
||||
- windows-network
|
||||
- windows-service
|
||||
- windows-system
|
||||
- windows-active-directory
|
||||
- windows-time
|
||||
|
||||
tests:
|
||||
- name: Memory cache includes all lists, commit is a ratio and paging is a rate
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
rule_files:
|
||||
- ../windows_rules.yaml
|
||||
- ../../monitoring/windows_rules.yaml
|
||||
evaluation_interval: 1m
|
||||
fuzzy_compare: true
|
||||
|
||||
@@ -111,3 +111,44 @@ tests:
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_bytes:rate{job="windows_exporter", instance="host:9182", nic="Ethernet"}'
|
||||
value: 30
|
||||
|
||||
- name: Processor queue per core averages the queue and packet rates keep the nic label
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
values: '12x5 0x5'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="0", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="1", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="2", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_cpu_time_seconds_total{job="windows_exporter", instance="host:9182", cluster="a", core="3", mode="idle"}'
|
||||
values: '0+60x10'
|
||||
- series: 'windows_system_processor_queue_length{job="windows_exporter", instance="nocpu:9182", cluster="a"}'
|
||||
values: '12x10'
|
||||
- series: 'windows_net_packets_received_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+600x10'
|
||||
- series: 'windows_net_packets_sent_total{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
values: '0+1200x10'
|
||||
promql_expr_test:
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 3
|
||||
- expr: windows:cpu_queue_per_core:ratio
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:cpu_queue_per_core:ratio{job="windows_exporter", instance="host:9182", cluster="a"}'
|
||||
value: 0
|
||||
- expr: windows:net_received_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_received_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 10
|
||||
- expr: windows:net_sent_packets:rate
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: 'windows:net_sent_packets:rate{job="windows_exporter", instance="host:9182", cluster="a", nic="Ethernet"}'
|
||||
value: 20
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Prometheus rules for windows_exporter
|
||||
|
||||
Ready-to-use Prometheus rule files, rendered from the
|
||||
[mixin](../mixin/README.md) with its default configuration:
|
||||
|
||||
- [`windows_rules.yaml`](windows_rules.yaml): recording rules.
|
||||
- [`windows_alerts.yaml`](windows_alerts.yaml): alerting rules for the default
|
||||
collectors.
|
||||
|
||||
The alerts build on the recording rules, so load both files:
|
||||
|
||||
```yaml
|
||||
rule_files:
|
||||
- /etc/prometheus/rules/windows_rules.yaml
|
||||
- /etc/prometheus/rules/windows_alerts.yaml
|
||||
```
|
||||
|
||||
The files expect the exporter under the job name `windows_exporter`. For other
|
||||
job names, thresholds, selectors or the alerts of opt-in collectors, render
|
||||
the mixin with your own configuration as the
|
||||
[mixin documentation](../mixin/README.md#customize-or-compose) describes. Every alert links
|
||||
to its [runbook](https://prometheus-community.github.io/windows_exporter/runbooks/).
|
||||
|
||||
Do not edit these files by hand: `mise run rules` in `contrib/mixin`
|
||||
regenerates them, and CI fails when they are out of date.
|
||||
@@ -0,0 +1,249 @@
|
||||
groups:
|
||||
- name: "windows-exporter"
|
||||
rules:
|
||||
- alert: "WindowsExporterDown"
|
||||
annotations:
|
||||
description: "Prometheus cannot scrape Windows exporter on {{ $labels.instance }} (job {{ $labels.job }})."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsexporterdown/"
|
||||
summary: "Windows exporter is unreachable"
|
||||
expr: |
|
||||
up{job="windows_exporter"} == 0
|
||||
for: "5m"
|
||||
labels:
|
||||
severity: "critical"
|
||||
- alert: "WindowsCollectorFailed"
|
||||
annotations:
|
||||
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has failed for at least 5 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectorfailed/"
|
||||
summary: "Windows collector failed"
|
||||
expr: |
|
||||
windows_exporter_collector_success{job="windows_exporter"} == 0
|
||||
for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsCollectorTimeout"
|
||||
annotations:
|
||||
description: "Collector {{ $labels.collector }} on {{ $labels.instance }} has not finished within the scrape timeout for 10 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscollectortimeout/"
|
||||
summary: "Windows collector exceeds the scrape timeout"
|
||||
expr: |
|
||||
windows_exporter_collector_timeout{job="windows_exporter"} == 1
|
||||
for: "10m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsScrapeQueueTimeout"
|
||||
annotations:
|
||||
description: "Scrapes of {{ $labels.instance }} have timed out waiting for an earlier scrape to finish for 10 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsscrapequeuetimeout/"
|
||||
summary: "Windows exporter scrapes time out in the queue"
|
||||
expr: |
|
||||
windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
|
||||
for: "10m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-cpu"
|
||||
rules:
|
||||
- alert: "WindowsCPUHighUsage"
|
||||
annotations:
|
||||
description: "CPU usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpuhighusage/"
|
||||
summary: "Windows CPU usage is high"
|
||||
expr: |
|
||||
windows:cpu_usage:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsCPUSaturation"
|
||||
annotations:
|
||||
description: "More than 2 threads per logical processor have been waiting for CPU time on {{ $labels.instance }} for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowscpusaturation/"
|
||||
summary: "Windows processor queue is long"
|
||||
expr: |
|
||||
windows:cpu_queue_per_core:ratio{job="windows_exporter"} > 2
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-memory"
|
||||
rules:
|
||||
- alert: "WindowsMemoryHighUsage"
|
||||
annotations:
|
||||
description: "Physical memory usage on {{ $labels.instance }} has exceeded 90% for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemoryhighusage/"
|
||||
summary: "Windows memory usage is high"
|
||||
expr: |
|
||||
windows:memory_usage:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsMemoryCommitHighUsage"
|
||||
annotations:
|
||||
description: "Committed memory on {{ $labels.instance }} has exceeded 90% of the commit limit for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorycommithighusage/"
|
||||
summary: "Windows memory commit usage is high"
|
||||
expr: |
|
||||
windows:memory_committed:ratio{job="windows_exporter"} * 100 > 90
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsMemoryPagingHigh"
|
||||
annotations:
|
||||
description: "Paging on {{ $labels.instance }} has exceeded 1000 pages per second for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsmemorypaginghigh/"
|
||||
summary: "Windows memory paging is high"
|
||||
expr: |
|
||||
windows:memory_swap_page_operations:rate{job="windows_exporter"} > 1000
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-disk"
|
||||
rules:
|
||||
- alert: "WindowsDiskReadLatencyHigh"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per read operation for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskreadlatencyhigh/"
|
||||
summary: "Windows disk read latency is high"
|
||||
expr: |
|
||||
windows:logical_disk_read_latency:seconds{job="windows_exporter", volume!=""} > 0.05
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskWriteLatencyHigh"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has averaged more than 0.05 seconds per write operation for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskwritelatencyhigh/"
|
||||
summary: "Windows disk write latency is high"
|
||||
expr: |
|
||||
windows:logical_disk_write_latency:seconds{job="windows_exporter", volume!=""} > 0.05
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskAlmostFull"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 10% free space. Windows may delay space metrics by 10–15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
|
||||
summary: "Windows disk space is low"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsDiskAlmostFull"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} has less than 5% free space. Windows may delay space metrics by 10–15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskalmostfull/"
|
||||
summary: "Windows disk space is low"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 5
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "critical"
|
||||
- alert: "WindowsDiskFillingUp"
|
||||
annotations:
|
||||
description: "Volume {{ $labels.volume }} on {{ $labels.instance }} is predicted to fill within 24 hours."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsdiskfillingup/"
|
||||
summary: "Windows disk is predicted to fill up"
|
||||
expr: |
|
||||
windows:logical_disk_free:ratio{job="windows_exporter", volume!=""} * 100 < 10
|
||||
and predict_linear(windows_logical_disk_free_bytes{job="windows_exporter", volume!=""}[6h], 86400) < 0
|
||||
for: "1h"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-network"
|
||||
rules:
|
||||
- alert: "WindowsNetworkErrors"
|
||||
annotations:
|
||||
description: "Packet errors on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkerrors/"
|
||||
summary: "Windows network packet errors are high"
|
||||
expr: |
|
||||
(
|
||||
(windows:net_received_errors:rate{job="windows_exporter", nic!=""} + windows:net_outbound_errors:rate{job="windows_exporter", nic!=""})
|
||||
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
|
||||
) * 100 > 1
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkDiscards"
|
||||
annotations:
|
||||
description: "Packet discards on interface {{ $labels.nic }} on {{ $labels.instance }} have exceeded 1% of all packets for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkdiscards/"
|
||||
summary: "Windows network packet discards are high"
|
||||
expr: |
|
||||
(
|
||||
(windows:net_received_discarded:rate{job="windows_exporter", nic!=""} + windows:net_outbound_discarded:rate{job="windows_exporter", nic!=""})
|
||||
/ ((windows:net_received_packets:rate{job="windows_exporter", nic!=""} + windows:net_sent_packets:rate{job="windows_exporter", nic!=""}) > 0)
|
||||
) * 100 > 1
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkUtilizationHigh"
|
||||
annotations:
|
||||
description: "Traffic on interface {{ $labels.nic }} on {{ $labels.instance }} has exceeded 90% of its bandwidth in at least one direction for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkutilizationhigh/"
|
||||
summary: "Windows network interface utilization is high"
|
||||
expr: |
|
||||
(windows:net_received_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
|
||||
or (windows:net_sent_utilization:ratio{job="windows_exporter", nic!=""} * 100 > 90)
|
||||
for: "15m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsNetworkInterfaceFlapping"
|
||||
annotations:
|
||||
description: "Interface {{ $labels.nic }} on {{ $labels.instance }} has changed its operational status more than twice in 5m."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsnetworkinterfaceflapping/"
|
||||
summary: "Windows network interface is flapping"
|
||||
expr: |
|
||||
max without (status) (changes(windows_net_nic_operation_status{job="windows_exporter", nic!="", status="up"}[5m])) > 2
|
||||
for: "2m"
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-service"
|
||||
rules:
|
||||
- alert: "WindowsServiceNotRunning"
|
||||
annotations:
|
||||
description: "Service {{ $labels.service }} on {{ $labels.instance }} starts automatically but has not been running for 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicenotrunning/"
|
||||
summary: "Windows automatic service is not running"
|
||||
expr: |
|
||||
max without (state) (windows_service_state{job="windows_exporter", service!="", state="running"} == 0)
|
||||
and ignoring (start_mode) (windows_service_start_mode{job="windows_exporter", service!="", start_mode="auto"} == 1)
|
||||
unless ignoring (display_name, run_as, path_name, delayed_start, trigger_start) (windows_service_info{job="windows_exporter", service!="", trigger_start="true"} == 1)
|
||||
for: "15m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- alert: "WindowsServiceCrashLooping"
|
||||
annotations:
|
||||
description: "Service {{ $labels.service }} on {{ $labels.instance }} has restarted more than 2 times in 15 minutes."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowsservicecrashlooping/"
|
||||
summary: "Windows service is restarting repeatedly"
|
||||
expr: |
|
||||
count without (process_id) (count_over_time(windows_service_process{job="windows_exporter", service!=""}[15m])) - 1 > 2
|
||||
keep_firing_for: "5m"
|
||||
labels:
|
||||
severity: "warning"
|
||||
- name: "windows-system"
|
||||
rules:
|
||||
- alert: "WindowsHostRebooted"
|
||||
annotations:
|
||||
description: "{{ $labels.instance }} rebooted {{ $value | humanizeDuration }} ago."
|
||||
runbook_url: "https://prometheus-community.github.io/windows_exporter/runbooks/windowshostrebooted/"
|
||||
summary: "Windows host has rebooted"
|
||||
expr: |
|
||||
(time() - windows_system_boot_time_timestamp{job="windows_exporter"}) < 600
|
||||
and (time() - last_over_time(windows_system_boot_time_timestamp{job="windows_exporter"}[1h] offset 10m)) > 600
|
||||
labels:
|
||||
severity: "info"
|
||||
@@ -0,0 +1,59 @@
|
||||
groups:
|
||||
- name: "windows-exporter-recording"
|
||||
rules:
|
||||
- expr: "1 - avg without (core, mode) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"}[5m]))"
|
||||
record: "windows:cpu_usage:ratio"
|
||||
- expr: "count without (core, mode) (windows_cpu_time_seconds_total{job=\"windows_exporter\", mode=\"idle\"})"
|
||||
record: "windows:cpu_cores:count"
|
||||
- expr: "avg without (core) (rate(windows_cpu_time_seconds_total{job=\"windows_exporter\"}[5m]))"
|
||||
record: "windows:cpu_time:ratio_rate"
|
||||
- expr: "(avg_over_time(windows_system_processor_queue_length{job=\"windows_exporter\"}[5m]) / (windows:cpu_cores:count{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:cpu_queue_per_core:ratio"
|
||||
- expr: "1 - (windows_memory_available_bytes{job=\"windows_exporter\"} / (windows_memory_physical_total_bytes{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:memory_usage:ratio"
|
||||
- expr: "windows_memory_physical_total_bytes{job=\"windows_exporter\"} - windows_memory_available_bytes{job=\"windows_exporter\"}"
|
||||
record: "windows:memory_used:bytes"
|
||||
- expr: "windows_memory_cache_bytes{job=\"windows_exporter\"} + windows_memory_modified_page_list_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_core_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_normal_priority_bytes{job=\"windows_exporter\"} + windows_memory_standby_cache_reserve_bytes{job=\"windows_exporter\"}"
|
||||
record: "windows:memory_cached:bytes"
|
||||
- expr: "(windows_memory_committed_bytes{job=\"windows_exporter\"} / (windows_memory_commit_limit_bytes{job=\"windows_exporter\"} > 0))"
|
||||
record: "windows:memory_committed:ratio"
|
||||
- expr: "rate(windows_memory_swap_page_operations_total{job=\"windows_exporter\"}[5m])"
|
||||
record: "windows:memory_swap_page_operations:rate"
|
||||
- expr: "(windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"} / (windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} > 0))"
|
||||
record: "windows:logical_disk_free:ratio"
|
||||
- expr: "windows_logical_disk_size_bytes{job=\"windows_exporter\", volume!=\"\"} - windows_logical_disk_free_bytes{job=\"windows_exporter\", volume!=\"\"}"
|
||||
record: "windows:logical_disk_used:bytes"
|
||||
- expr: "clamp(1 - rate(windows_logical_disk_idle_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]), 0, 1)"
|
||||
record: "windows:logical_disk_busy:ratio"
|
||||
- expr: "rate(windows_logical_disk_read_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_read_bytes:rate"
|
||||
- expr: "rate(windows_logical_disk_write_bytes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_write_bytes:rate"
|
||||
- expr: "rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_read_operations:rate"
|
||||
- expr: "rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m])"
|
||||
record: "windows:logical_disk_write_operations:rate"
|
||||
- expr: "(rate(windows_logical_disk_read_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_reads_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
|
||||
record: "windows:logical_disk_read_latency:seconds"
|
||||
- expr: "(rate(windows_logical_disk_write_seconds_total{job=\"windows_exporter\", volume!=\"\"}[5m]) / (rate(windows_logical_disk_writes_total{job=\"windows_exporter\", volume!=\"\"}[5m]) > 0))"
|
||||
record: "windows:logical_disk_write_latency:seconds"
|
||||
- expr: "rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_bytes:rate"
|
||||
- expr: "rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_sent_bytes:rate"
|
||||
- expr: "rate(windows_net_packets_received_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_packets:rate"
|
||||
- expr: "rate(windows_net_packets_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_sent_packets:rate"
|
||||
- expr: "(rate(windows_net_bytes_received_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
|
||||
record: "windows:net_received_utilization:ratio"
|
||||
- expr: "(rate(windows_net_bytes_sent_total{job=\"windows_exporter\", nic!=\"\"}[5m]) / (windows_net_current_bandwidth_bytes{job=\"windows_exporter\", nic!=\"\"} > 0))"
|
||||
record: "windows:net_sent_utilization:ratio"
|
||||
- expr: "rate(windows_net_packets_received_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_errors:rate"
|
||||
- expr: "rate(windows_net_packets_received_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_received_discarded:rate"
|
||||
- expr: "rate(windows_net_packets_outbound_errors_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_outbound_errors:rate"
|
||||
- expr: "rate(windows_net_packets_outbound_discarded_total{job=\"windows_exporter\", nic!=\"\"}[5m])"
|
||||
record: "windows:net_outbound_discarded:rate"
|
||||
Reference in New Issue
Block a user