mixin: group alerts by collector area and add alerts for default and opt-in collectors (#2777)

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

docs/runbooks/_index.md

@@ -8,5 +8,9 @@ the affected host. Disk alerts also identify a `volume`; network alerts identify
 a `nic`. Adjust example PromQL selectors to match your scrape configuration.
 
 The default thresholds are starting points. Check the generated alert rules for
-your configured thresholds and durations. Active Directory and NTP clock alerts
-are opt-in and require their collectors to be enabled.
+your configured thresholds and durations. The rules are grouped by collector
+area, and each alert is generated only when the collectors it reads are enabled
+in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
+and `time` collectors, which are off by default, and the alerts for page files,
+disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters,
+Storage Spaces, DHCP and DNS need the collector of that role.

docs/runbooks/windowsadpendingreplication.md

@@ -6,7 +6,7 @@ title: WindowsADPendingReplication
 
 Pending Active Directory replication operations exceed the configured threshold
 for 15 minutes. The default is 50 operations. This alert is generated only with
-`enableActiveDirectory: true` and requires the `ad` collector on domain controllers.
+`collectors+: { ad: true }` and requires the `ad` collector on domain controllers.
 
 ## Impact
 

docs/runbooks/windowsclocknotsynchronising.md

@@ -5,7 +5,7 @@ title: WindowsClockNotSynchronising
 ## Meaning
 
 Windows Time reports no active NTP time source for ten minutes. This alert is
-generated only with `enableTime: true` and requires the `time` collector and its
+generated only with `collectors+: { time: true }` and requires the `time` collector and its
 `ntp` subcollector. Enable it on hosts expected to synchronize through NTP;
 hosts intentionally using another clock source may legitimately report zero.
 

docs/runbooks/windowsclocksourcelocal.md

@@ -0,0 +1,56 @@
+---
+title: WindowsClockSourceLocal
+---
+
+## Meaning
+
+Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system
+clock for ten minutes. The alert carries the source in its `type` label. It
+needs the `time` collector with its `clock_source` subcollector and covers
+domain members too, whose source is `NT5DS` when they synchronize from the
+domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does
+not see those hosts. A host that is meant to be the free-running reference of
+an isolated network reports the local clock by design; exclude it with
+`alerts+: { WindowsClockSourceLocal: false }` on that configuration.
+
+## Impact
+
+The clock drifts without an external source. Clock errors break Kerberos
+authentication beyond the five minute tolerance, TLS validation, scheduled
+work, log correlation and distributed systems that depend on time order.
+
+## Diagnosis
+
+1. Inspect `windows_time_clock_sync_source` for the host, and
+   `windows_time_computed_time_offset_seconds` to see how far the clock has
+   already drifted.
+2. On the host, check the configured and the effective source:
+
+   ```powershell
+   w32tm /query /source
+   w32tm /query /status
+   w32tm /query /configuration
+   Get-Service W32Time
+   ```
+
+3. `Local CMOS Clock` after a reboot often means the Windows Time service
+   started before the network was up or the configured peers were unreachable.
+   `NoSync` means the time provider is configured not to synchronize.
+4. For domain members, check that a domain controller is reachable and that
+   the PDC emulator synchronizes from an external source. For virtual machines,
+   check whether hypervisor time integration overrides the configured source.
+   A Hyper-V guest outside a domain that is kept in time by the integration
+   service reports `NoSync` while `w32tm /query /source` shows the
+   `VM IC Time Synchronization Provider`; that is by design, so turn the alert
+   off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or
+   configure an NTP source in the guest.
+
+## Mitigation
+
+Configure the intended source and restart synchronization with
+`w32tm /config /syncfromflags:domhier /update` on domain members or
+`w32tm /config /manualpeerlist:"<servers>" /syncfromflags:manual /update` on
+stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the
+peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the
+offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/)
+if it does not.

docs/runbooks/windowsclusternodedown.md

@@ -0,0 +1,46 @@
+---
+title: WindowsClusterNodeDown
+---
+
+## Meaning
+
+The failover cluster reports the node `node` as down for five minutes. The
+alert needs the `mscluster` collector. Every node that runs the exporter
+reports the state of all nodes, so the alert fires once per reporting node;
+the `instance` label names the reporter, and the `node` label the node that is
+down. A paused node, for example during cluster-aware updating, has a
+different state and does not raise this alert.
+
+## Impact
+
+The roles that the node hosted have failed over to the remaining nodes, with a
+short outage during the move. The cluster has less capacity and less
+redundancy; with too few nodes for quorum, the whole cluster stops.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_node_state` for all nodes and
+   `windows_mscluster_cluster_quorum_type_value` and the dynamic quorum metrics to
+   see how far the cluster is from losing quorum. Check whether the node's own
+   exporter target is up.
+2. On a working node, read the cluster state:
+
+   ```powershell
+   Get-ClusterNode
+   Get-ClusterGroup
+   Get-ClusterLog -Node '<node>' -TimeSpan 60 -Destination .
+   Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50
+   ```
+
+3. Check whether the node is powered on and reachable, whether the Cluster
+   Service runs on it, and whether the cluster networks between the nodes are
+   working. Events 1135 and 1177 in the System log name the lost node and the
+   quorum state.
+
+## Mitigation
+
+Bring the node back: power it on, repair its network, or start the Cluster
+Service with `Start-ClusterNode`. Evict a node that is removed for good with
+`Remove-ClusterNode` so the quorum configuration matches the remaining nodes.
+Move the roles back to their preferred owners once the node is up and confirm
+the node state returns to up.

docs/runbooks/windowsclusterresourcefailed.md

@@ -0,0 +1,45 @@
+---
+title: WindowsClusterResourceFailed
+---
+
+## Meaning
+
+The failover cluster reports the resource `resource` of type `type` in the
+group `resource_group` as failed for five minutes. The alert needs the
+`mscluster` collector and fires once per reporting node, named by the
+`instance` label. A failed resource has exhausted its restart attempts on the
+current node, or its group has reached its failover threshold.
+
+## Impact
+
+The role that depends on the resource is down: a clustered disk, IP address,
+network name, file share, SQL Server instance or virtual machine. Other
+resources in the same group that depend on it are offline too.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_resource_state` for the group and
+   `windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node`
+   to see where the group is and whether the whole group failed.
+2. On a cluster node, read the resource and the reason:
+
+   ```powershell
+   Get-ClusterResource -Name '<resource>' | Format-List *
+   Get-ClusterResource -Name '<resource>' | Get-ClusterParameter
+   Get-ClusterLog -TimeSpan 30 -Destination .
+   Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20
+   ```
+
+   Event 1069 names the failed resource and its error code.
+3. Check the dependency of the resource: a disk that is not reachable from the
+   node, an IP address whose network is down, a network name whose computer
+   object in Active Directory is missing or disabled, or an application whose
+   service fails to start.
+
+## Mitigation
+
+Fix the dependency, then bring the resource online with
+`Start-ClusterResource`, or move the group to another node with
+`Move-ClusterGroup`. Review the restart and failover thresholds of the
+resource if it failed because of a transient problem. Confirm the resource
+state returns to online and the group is online.

docs/runbooks/windowsclusterresourcegroupfailed.md

@@ -0,0 +1,41 @@
+---
+title: WindowsClusterResourceGroupFailed
+---
+
+## Meaning
+
+The failover cluster reports the resource group `resource_group` as failed for
+five minutes. The alert needs the `mscluster` collector and fires once per
+reporting node, named by the `instance` label. A group fails when a resource in
+it fails and the group cannot fail over anymore: it reached its failover
+threshold within the failover period, or no other node can host it.
+
+## Impact
+
+The whole clustered role is down and stays down until an operator intervenes.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_resource_state` for the resources of the group
+   to find the failed resource, and `windows_mscluster_resourcegroup_owner_node`
+   to see on which node the group stopped.
+2. On a cluster node, read the group and its history:
+
+   ```powershell
+   Get-ClusterGroup -Name '<group>' | Format-List *
+   Get-ClusterGroup -Name '<group>' | Get-ClusterResource
+   Get-ClusterLog -TimeSpan 60 -Destination .
+   Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20
+   ```
+
+   Event 1254 says the group exceeded its failover threshold.
+3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for
+   the failed resource, and check the possible owners of the group.
+
+## Mitigation
+
+Fix the failed resource, then bring the group online with `Start-ClusterGroup`
+on a node that can host it. A group that failed over repeatedly may need a
+higher failover threshold or a longer failover period, but a resource that
+fails on every node needs its cause fixed first. Confirm the group state
+returns to online.

docs/runbooks/windowsclustersharedvolumealmostfull.md

@@ -0,0 +1,44 @@
+---
+title: WindowsClusterSharedVolumeAlmostFull
+---
+
+## Meaning
+
+The Cluster Shared Volume `shared_volume` has less free space than the
+configured threshold for 15 minutes. The warning fires below
+`diskFreeWarningThreshold` (10 percent by default) and the critical alert
+below `diskFreeCriticalThreshold` (5 percent); both fire below the critical
+threshold, so inhibit the warning in Alertmanager. The alert needs the
+`mscluster` collector and fires once per reporting node, named by the
+`instance` label. `sharedVolumeSelector` excludes the
+`ClusterPerformanceHistory` volume of the cluster's own health history.
+
+## Impact
+
+Virtual machines and databases on the volume cannot grow their files. Dynamic
+virtual hard disks pause the virtual machine when the volume is full, Hyper-V
+checkpoints and backups fail, and SQL Server stops writing.
+
+## Diagnosis
+
+1. Compare `windows_mscluster_shared_volumes_free_bytes` and
+   `windows_mscluster_shared_volumes_total_bytes` over the last days to see
+   whether the volume fills steadily or filled suddenly.
+2. On a cluster node, find the large files on the volume:
+
+   ```powershell
+   Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}}
+   Get-ChildItem 'C:\ClusterStorage\<volume>' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}}
+   ```
+
+3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard
+   disks, backup snapshots that were not cleaned up, and dynamic disks that
+   grew.
+
+## Mitigation
+
+Delete or merge checkpoints, remove orphaned files, move virtual machines to
+another volume with storage migration, or extend the volume on the storage
+array and then in the cluster with `Resize-Partition`. Confirm free space is
+above the threshold. Raise the thresholds only for volumes whose growth is
+known and planned.

docs/runbooks/windowscollectortimeout.md

@@ -0,0 +1,46 @@
+---
+title: WindowsCollectorTimeout
+---
+
+## Meaning
+
+The collector identified by the `collector` label has reported
+`windows_exporter_collector_timeout == 1` for ten minutes. The collector did
+not finish within the scrape timeout, or its call from an earlier scrape is
+still running and later scrapes skip it until it returns.
+
+## Impact
+
+The metrics of this collector are missing or stale on every scrape that times
+out, so dashboards show gaps and alerts on those metrics stop evaluating. A
+collector that never returns can hold a scrape slot and delay the other
+collectors of the host.
+
+## Diagnosis
+
+1. Find the collector and compare how long it usually takes:
+
+   ```promql
+   windows_exporter_collector_timeout{job="windows_exporter"} == 1
+   windows_exporter_collector_duration_seconds{job="windows_exporter"}
+   ```
+
+2. Compare the duration with the scrape timeout of the job. The exporter
+   subtracts `--scrape.timeout-margin` from the `scrape_timeout` that
+   Prometheus sends.
+3. Check the exporter log for the collector. WMI and Management Infrastructure
+   queries, the Task Scheduler and Windows Update are slow on hosts with many
+   objects or a busy WMI provider; the collector's documentation lists the
+   expensive options.
+4. Check whether the host was under CPU, disk or memory pressure at the same
+   time, and whether a Windows service the collector depends on hangs, for
+   example `Winmgmt` for WMI-based collectors.
+
+## Mitigation
+
+Raise the `scrape_timeout` of the job within its `scrape_interval`, narrow the
+collector with its include and exclude flags, or move the collector to a
+separate job with a longer timeout. Restart a hanging provider service when the
+log identifies it. Disable the collector on hosts where its data is not needed.
+Confirm `windows_exporter_collector_timeout` returns to `0` and the collector's
+metrics are published again.

docs/runbooks/windowscpusaturation.md

@@ -0,0 +1,39 @@
+---
+title: WindowsCPUSaturation
+---
+
+## Meaning
+
+The number of threads waiting for a processor, averaged over the rate window
+and divided by the number of logical processors, exceeds the configured
+threshold for 15 minutes. The default is two waiting threads per logical
+processor. The metric is `windows:cpu_queue_per_core:ratio`, built from
+`windows_system_processor_queue_length` and `windows:cpu_cores:count`.
+
+## Impact
+
+Runnable threads wait for CPU time, so applications respond slowly even when
+their own CPU usage looks moderate. A long queue with CPU usage below 100%
+points at a few busy cores, a single-threaded bottleneck, or processor
+scheduling problems such as parked cores or hypervisor contention.
+
+## Diagnosis
+
+1. Inspect `windows:cpu_queue_per_core:ratio` next to `windows:cpu_usage:ratio`
+   for the host. Saturation with high usage is a capacity problem; saturation
+   with low usage points at scheduling.
+2. Compare the per-core rates of `windows_cpu_time_seconds_total`. One or two
+   cores at 100% while others idle indicate a thread that cannot spread.
+3. Check `windows_cpu_parking_status`, `windows_cpu_core_frequency_hertz` and
+   the power plan. On virtual machines check the hypervisor for CPU ready time
+   or steal.
+4. Use Task Manager or Performance Monitor (`System\Processor Queue Length`,
+   `Process\Thread Count`) to find the processes with many runnable threads.
+
+## Mitigation
+
+Reduce or redistribute the workload, add logical processors, or fix the power
+plan and parking settings. Give the virtual machine more CPU share or move it to
+a less loaded host. Tune the threshold for hosts whose workload keeps a long
+queue by design, such as build servers. Confirm the queue length per core and
+application latency return to normal.

docs/runbooks/windowsdhcpscopealmostexhausted.md

@@ -0,0 +1,48 @@
+---
+title: WindowsDHCPScopeAlmostExhausted
+---
+
+## Meaning
+
+The enabled DHCP scope `scope` has fewer free addresses than the configured
+share of its address pool for 15 minutes. The warning fires below
+`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert
+below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the
+critical threshold, so inhibit the warning in Alertmanager. The share divides
+`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses,
+and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled.
+On a failover pair the free and in-use counts are the totals of both servers.
+The alert needs the `dhcp` collector.
+
+## Impact
+
+Once the scope is exhausted, new clients receive no address and cannot use
+the network. Existing clients keep their lease until it expires and then lose
+their address too.
+
+## Diagnosis
+
+1. Compare the free and in-use addresses of the scope over the last days to
+   see whether the exhaustion is a trend or a sudden burst, and inspect
+   `windows_dhcp_scope_reserved_address` and
+   `windows_dhcp_scope_pending_offers` for reservations and offers in flight.
+2. On the server, read the scope and its leases:
+
+   ```powershell
+   Get-DhcpServerv4ScopeStatistics -ScopeId <scope>
+   Get-DhcpServerv4Scope -ScopeId <scope> | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State
+   Get-DhcpServerv4Lease -ScopeId <scope> | Group-Object AddressState
+   Get-DhcpServerv4ExclusionRange -ScopeId <scope>
+   ```
+
+3. Look for a long lease duration on a network with many short-lived clients,
+   large exclusion ranges, a flood of leases from one device that changes its
+   MAC address, or a failover partner that holds the free addresses.
+
+## Mitigation
+
+Shorten the lease duration for networks with transient clients, remove
+exclusion ranges that are not needed, add a range through a superscope or a
+larger subnet, or reclaim bad leases. On a failover pair, check the split of
+the addresses between the partners. Confirm the free share stays above the
+threshold.

docs/runbooks/windowsdiskdrivenotok.md

@@ -0,0 +1,46 @@
+---
+title: WindowsDiskDriveNotOK
+---
+
+## Meaning
+
+The physical drive `device_id` reports an operational status other than `OK`
+for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded`
+or `Error`, in its `status` label. It needs the `diskdrive` collector, which
+reads the status that the drive and its driver report through WMI. The
+`Unknown` status does not raise the alert: some USB and virtual drives report
+it permanently. Transient statuses such as `Starting` or `Stopping` raise it
+only when they last longer than five minutes.
+
+## Impact
+
+`Pred Fail` is the drive's own prediction that it is about to fail. Any other
+status means the drive is not fully working. Data on a single drive is at risk;
+a drive in a mirror or parity set leaves the set without redundancy until it is
+replaced.
+
+## Diagnosis
+
+1. Inspect `windows_diskdrive_status` for the host and check whether the
+   volumes on the drive still report normal latency with
+   `windows:logical_disk_read_latency:seconds` and
+   `windows:logical_disk_write_latency:seconds`.
+2. On the host, read the drive and its reliability counters:
+
+   ```powershell
+   Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status
+   Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear
+   ```
+
+3. Check the System event log for disk, storahci, stornvme or RAID controller
+   events. On a virtual machine, check the hypervisor and the datastore of the
+   virtual disk.
+
+## Mitigation
+
+Back up the data on the drive first. Replace a drive that predicts its failure
+or reports errors through the hardware or virtualization team. For a drive in
+a storage pool or RAID set, follow the vendor's replacement and rebuild
+procedure and confirm the set is redundant again. The alert resolves when the
+status returns to `OK`.

docs/runbooks/windowsdnszonetransferfailures.md

@@ -0,0 +1,54 @@
+---
+title: WindowsDNSZoneTransferFailures
+---
+
+## Meaning
+
+The DNS server has failed zone transfers for 15 minutes, measured as the rate
+of `windows_dns_zone_transfer_failures_total` over `rateInterval` above
+`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts
+transfers this server sends as the primary to its secondaries, including
+requests it refuses, and it has no zone label. The alert needs the `dns`
+collector and keeps firing for `keepFiringFor` after the failures stop. Raise
+the threshold on servers that refuse transfer requests from scanners all the
+time.
+
+## Impact
+
+The secondary servers of the zone serve stale records. When the zone on a
+secondary expires, it stops answering for the zone altogether. Active
+Directory integrated zones replicate through Active Directory and are not
+affected unless they also have secondaries.
+
+## Diagnosis
+
+1. Inspect `windows_dns_zone_transfer_failures_total` next to
+   `windows_dns_zone_transfer_requests_received_total` and
+   `windows_dns_zone_transfer_success_sent_total` to see what share of the
+   transfers fail, and whether the failures started with a change.
+2. On this primary, find the refused or failed requests and the zone
+   settings; the DNS Server log records a refused transfer request with the
+   address of the requester:
+
+   ```powershell
+   Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer'
+   Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy
+   Get-DnsServerZone -Name '<zone>' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers
+   ```
+
+   On the secondary, events 6522 to 6527 of the DNS Server log say why its
+   transfer of the zone failed: refused by the primary, invalid response or
+   an expired zone.
+
+3. Check that the requesting secondary is in the zone transfer list, that TCP
+   port 53 is open between the servers, and that the zone is not paused or
+   broken. Failures from unknown addresses are refused transfers and may be
+   expected.
+
+## Mitigation
+
+Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries`
+and the server list, fix the network path, or remove a decommissioned
+secondary from the zone's notify list so it stops requesting transfers.
+Confirm the failure counter stops rising and the secondaries show a current
+serial number.

docs/runbooks/windowshostrebooted.md

@@ -0,0 +1,37 @@
+---
+title: WindowsHostRebooted
+---
+
+## Meaning
+
+The boot time of the host, `windows_system_boot_time_timestamp`, is less than
+ten minutes old and differs from the boot time known ten minutes earlier. The
+alert fires once for about ten minutes after a reboot and then resolves by
+itself. It has the `info` severity and is meant for a dashboard, a ticket or a
+chat notification, not for a page.
+
+## Impact
+
+Every service of the host restarted, connections to it were dropped and
+caches are cold. A reboot outside a maintenance window can mean a crash, an
+unattended Windows Update restart or a power event.
+
+## Diagnosis
+
+1. Compare the boot time with the maintenance calendar and the Windows Update
+   schedule. With the `update` collector enabled, `windows_update_pending_info`
+   shows updates that were pending before the reboot.
+2. Check the System event log: event 1074 names the user or process that
+   requested the shutdown, event 6008 marks an unexpected shutdown and event 41
+   a reset without a clean shutdown. Check for a memory dump when it was a
+   crash.
+3. Confirm the host came back with the expected services; the
+   `WindowsServiceNotRunning` alert covers automatic services that did not
+   start.
+
+## Mitigation
+
+No action is needed for a planned reboot. For an unplanned one, find the cause
+in the event log and fix the driver, update setting or power problem. Turn the
+alert off with `alerts+: { WindowsHostRebooted: false }` when reboots are not
+worth a notification.

docs/runbooks/windowshypervreplicationcritical.md

@@ -0,0 +1,47 @@
+---
+title: WindowsHyperVReplicationCritical
+---
+
+## Meaning
+
+Hyper-V Replica reports critical health for the virtual machine `vm` for five
+minutes. The alert carries the `relationship` label, `primary` for the normal
+replication and `extended` for an extended replica, and needs the `hyperv`
+collector with its `replica_vm` subcollector. Critical health means that
+replication has not succeeded for long enough that the replica misses the
+recovery point objective, or that replication is paused because of an error.
+
+## Impact
+
+The replica of the virtual machine is stale. A failover to it loses the
+changes since the last successful replication, and the recovery point
+objective of the service is not met. If the primary host fails now, data is
+lost.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_replica_vm_health` and
+   `windows_hyperv_replica_vm_state` for the virtual machine; the state tells
+   whether replication is suspended, resynchronizing or in error, and
+   `windows_hyperv_replica_vm_replication_latency_seconds` how long the last
+   cycle took.
+2. On the primary host, read the replication details:
+
+   ```powershell
+   Measure-VMReplication -VMName '<vm>' | Format-List *
+   Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, ReplicationMode
+   Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '*<vm>*'
+   ```
+
+3. Check the replica server: free space on the replica storage, the
+   Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication,
+   and the network path between the hosts. Check whether a snapshot merge or a
+   storage migration on the primary blocks replication.
+
+## Mitigation
+
+Fix the cause on the replica server or the network, then resume replication
+with `Resume-VMReplication -VMName '<vm>'`; add `-Resynchronize` when the
+state asks for a resynchronization. Confirm the health returns to normal and
+the last replication time advances. If the replica cannot be recovered, remove
+and re-enable replication, which sends a full initial copy.

docs/runbooks/windowshypervreplicationwarning.md

@@ -0,0 +1,43 @@
+---
+title: WindowsHyperVReplicationWarning
+---
+
+## Meaning
+
+Hyper-V Replica reports warning health for the virtual machine `vm` for 15
+minutes. Warning health means that some replication cycles missed their
+interval, that more than a fifth of the cycles in the last hour failed, or
+that the replication is waiting for an initial copy or a resynchronization.
+The alert carries the `relationship` label and needs the `hyperv` collector
+with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/)
+follows when replication stops working.
+
+## Impact
+
+The replica lags behind the primary more than planned. The recovery point
+objective is at risk, and a failover would lose more changes than expected.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state`
+   and `windows_hyperv_replica_vm_replication_latency_seconds` for the
+   virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes`
+   to see whether the change rate of the virtual machine has grown.
+2. On the primary host, read the statistics of the last hour:
+
+   ```powershell
+   Measure-VMReplication -VMName '<vm>' | Format-List *
+   Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, FrequencySec
+   ```
+
+3. Check the bandwidth between the hosts, the load on the replica storage,
+   and whether the replication interval is too short for the change rate of
+   the virtual machine.
+
+## Mitigation
+
+Relieve the bottleneck: more bandwidth, less load on the replica storage, or a
+longer replication interval through `Set-VMReplication -ReplicationFrequencySec`.
+Resume a waiting replication with `Resume-VMReplication`. Reset the statistics
+with `Reset-VMReplicationStatistics` after the fix; the health is computed
+over the last hour and takes up to an hour to return to normal.

docs/runbooks/windowshypervwmiunhealthy.md

@@ -0,0 +1,43 @@
+---
+title: WindowsHyperVWMIUnhealthy
+---
+
+## Meaning
+
+The Hyper-V WMI namespace `root/virtualization/v2` on the host has not
+answered a minimal query for ten minutes. The alert needs the `hyperv`
+collector with its `wmi_health` subcollector. While the namespace is broken,
+the virtual machine, replication and virtual hard disk metrics that come from
+WMI are missing, so the other Hyper-V alerts cannot fire.
+
+## Impact
+
+Management tools such as Hyper-V Manager, Failover Cluster Manager and System
+Center cannot manage the virtual machines on the host. Backups that use the
+Hyper-V VSS writer fail. The virtual machines themselves usually keep running.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}`
+   for the host.
+2. On the host, test the namespace and the management service:
+
+   ```powershell
+   Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1
+   Get-Service vmms, vmcompute, winmgmt
+   winmgmt /verifyrepository
+   ```
+
+3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of
+   the Virtual Machine Management service, and the WMI-Activity log for
+   provider failures. A stuck `vmms` process or a corrupt WMI repository are
+   the usual causes.
+
+## Mitigation
+
+Restart the Virtual Machine Management service with `Restart-Service vmms`;
+running virtual machines are not affected. If the repository is inconsistent,
+run `winmgmt /salvagerepository`, and register the Hyper-V provider again with
+`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a
+restart of the host if the service cannot be recovered. Confirm the health
+metric returns to 1 and the Hyper-V metrics appear again.

docs/runbooks/windowslicensenotgenuine.md

@@ -0,0 +1,45 @@
+---
+title: WindowsLicenseNotGenuine
+---
+
+## Meaning
+
+Windows reports a license status other than `genuine` for one hour. The alert
+carries the status in its `state` label and needs the `license` collector. The
+status comes from the Software Licensing service and changes when an
+activation expires, a KMS host is unreachable for more than the grace period,
+the hardware changed, or the key was blocked. `offline` means the genuine
+validation could not reach Microsoft; on hosts without internet access that
+is the permanent state, so turn the alert off there with
+`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter
+selector.
+
+## Impact
+
+An unlicensed Windows shows activation notices, may limit personalization and,
+on some editions, restarts on a schedule. A Windows Server that leaves its
+activation grace period is out of compliance.
+
+## Diagnosis
+
+1. The `state` label of the alert names the status; `windows_license_status`
+   shows all states of the host.
+2. On the host, read the license details:
+
+   ```powershell
+   slmgr.vbs /dlv
+   slmgr.vbs /xpr
+   Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining
+   ```
+
+3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp`
+   DNS record and answers on TCP port 1688. For Active Directory based
+   activation, check the activation object in the forest. For MAK or retail
+   keys, check whether the activation count is exhausted.
+
+## Mitigation
+
+Restore reachability to the KMS host or the activation object, then run
+`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk <key>` if the
+key was wrong, or reactivate after a hardware change. Confirm the status
+returns to `genuine`.

docs/runbooks/windowsmemorypaginghigh.md

@@ -0,0 +1,42 @@
+---
+title: WindowsMemoryPagingHigh
+---
+
+## Meaning
+
+The rate of pages read from or written to disk to resolve hard page faults,
+`windows:memory_swap_page_operations:rate`, exceeds the configured threshold
+for 15 minutes. The default is 1000 pages per second. The counter counts
+pagefile traffic and memory-mapped file I/O alike.
+
+## Impact
+
+The host is short of physical memory and uses the disk as backing store, so
+every affected process slows down to disk speed. Sustained paging also
+competes with application I/O on the system disk.
+
+## Diagnosis
+
+1. Inspect the paging rate together with `windows:memory_usage:ratio`,
+   `windows_memory_available_bytes` and `windows:memory_committed:ratio`. Low
+   available memory confirms memory pressure; a high rate with ample memory
+   points at a process reading a large memory-mapped file, such as a backup or
+   an index rebuild.
+2. Split the rate into reads and writes with
+   `windows_memory_swap_page_reads_total` and
+   `windows_memory_swap_page_writes_total`. Reads dominate when the working
+   sets of processes do not fit; writes dominate when the system trims and
+   writes modified pages.
+3. Find the processes with the largest working sets and page fault rates in
+   Resource Monitor or, with the `process` collector enabled, with
+   `windows_process_working_set_private_bytes` and
+   `windows_process_page_faults_total`.
+4. Check `windows_memory_standby_cache_lifetime_seconds`; a low lifetime means
+   cached pages are repurposed quickly, another sign of memory pressure.
+
+## Mitigation
+
+Stop or move the process that uses the memory, fix a memory leak, or add
+physical memory. Move large sequential file jobs off peak hours when they cause
+the paging. Confirm the paging rate drops and available memory recovers. Raise
+the threshold on hosts whose workload maps large files by design.

docs/runbooks/windowsnetworkdiscards.md

@@ -4,9 +4,15 @@ title: WindowsNetworkDiscards
 
 ## Meaning
 
-The sum of inbound and outbound discarded packet rates exceeds the configured
-threshold for 15 minutes. The default is one discard per second per interface.
-Discards count packets dropped without a packet error, such as buffer exhaustion.
+Discarded inbound and outbound packets make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkDiscardRatioThreshold` is 1 percent. The ratio divides the discard rates
+(`windows:net_received_discarded:rate` plus `windows:net_outbound_discarded:rate`)
+by the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`). Discards count packets dropped without a packet
+error, such as buffer exhaustion. Interfaces that carry no packets produce no
+sample and cannot alert. The alert keeps firing for `keepFiringFor` after the
+ratio drops below the threshold.
 
 ## Impact
 
@@ -17,7 +23,8 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 1. Inspect `windows:net_received_discarded:rate` and
    `windows:net_outbound_discarded:rate` to identify the direction and affected
-   `nic`.
+   `nic`, and compare them with the packet rate recordings to see whether a low
+   packet rate inflates the ratio.
 2. Compare traffic, directional utilization ratios, and
    `windows_net_output_queue_length_packets` with the normal baseline.
 3. Inspect `Get-NetAdapterStatistics`, switch-port drops, adapter buffers,
@@ -30,5 +37,6 @@ timeouts. Some drops may be expected under a deliberate traffic policy.
 
 Relieve congestion, distribute traffic, or add bandwidth. Correct buffer, driver,
 or virtual switch problems when evidence identifies them. Review intentional
-drop policies with the network owner before changing thresholds. Confirm discard
-rates and application retransmissions return to the expected baseline.
+drop policies with the network owner before raising
+`networkDiscardRatioThreshold`. Confirm discard rates and application
+retransmissions return to the expected baseline.

docs/runbooks/windowsnetworkerrors.md

@@ -4,9 +4,16 @@ title: WindowsNetworkErrors
 
 ## Meaning
 
-The sum of inbound and outbound packet error rates exceeds the configured
-threshold for 15 minutes. The default is one error per second per interface.
-The alert identifies the interface using its `nic` label.
+Packets with inbound or outbound errors make up more than the configured share
+of all packets on an interface for 15 minutes. The default
+`networkErrorRatioThreshold` is 1 percent. The ratio divides the error rates
+(`windows:net_received_errors:rate` plus `windows:net_outbound_errors:rate`) by
+the packet rates (`windows:net_received_packets:rate` plus
+`windows:net_sent_packets:rate`), so a quiet interface with a handful of errors
+alerts while a busy interface with the same error count does not. Interfaces
+that carry no packets produce no sample and cannot alert. The alert identifies
+the interface using its `nic` label and keeps firing for `keepFiringFor` after
+the ratio drops below the threshold.
 
 ## Impact
 
@@ -16,7 +23,10 @@ connection failures.
 ## Diagnosis
 
 1. Inspect `windows:net_received_errors:rate` and
-   `windows:net_outbound_errors:rate` separately to identify the direction.
+   `windows:net_outbound_errors:rate` separately to identify the direction, and
+   compare them with `windows:net_received_packets:rate` and
+   `windows:net_sent_packets:rate` to see whether a low packet rate inflates the
+   ratio.
 2. Compare traffic and `windows:net_received_utilization:ratio` and
    `windows:net_sent_utilization:ratio` with the normal baseline. These ratios
    use bandwidth in bytes per second and keep each direction separate.
@@ -36,4 +46,5 @@ Repair faulty links or switch ports and correct driver or adapter configuration
 problems. Apply driver and firmware fixes through the normal change process.
 Relieve congestion if it contributes. Verify errors stop increasing and affected
 application connections recover; cumulative counters do not need to return to
-zero for the rate alert to resolve.
+zero for the ratio alert to resolve. Raise `networkErrorRatioThreshold` only for
+interfaces whose error share is known and accepted.

docs/runbooks/windowsnetworkinterfaceflapping.md

@@ -0,0 +1,35 @@
+---
+title: WindowsNetworkInterfaceFlapping
+---
+
+## Meaning
+
+The operational status of the interface `nic` changed more than twice within
+the rate window, and did so for two minutes. The alert counts the changes of
+`windows_net_nic_operation_status{status="up"}` between `1` and `0`.
+
+## Impact
+
+Every transition drops the connections that use the interface and triggers
+route, DHCP and DNS registration changes. A flapping uplink on a server causes
+intermittent application failures that are hard to attribute.
+
+## Diagnosis
+
+1. Inspect `windows_net_nic_operation_status` for the interface to see the
+   status it flips to; `down` and `lower layer down` point at the link,
+   `dormant` at authentication such as 802.1X.
+2. Check the System event log for NDIS, the adapter driver and
+   `Microsoft-Windows-NetworkProfile` events, and the switch port for link
+   state changes, errors and port security actions.
+3. Check the cable, SFP, speed and duplex negotiation, and whether a teaming or
+   failover configuration switches between members.
+4. On virtual machines, check the hypervisor for virtual switch changes, live
+   migrations, or a host adapter that flaps.
+
+## Mitigation
+
+Replace the cable or transceiver, fix the speed and duplex settings, update the
+adapter driver, or correct the switch port and authentication configuration.
+Exclude an interface that flaps by design, such as a Wi-Fi or VPN adapter,
+with the `nicSelector`. Confirm the status stays `up`.

docs/runbooks/windowsnetworkutilizationhigh.md

@@ -0,0 +1,40 @@
+---
+title: WindowsNetworkUtilizationHigh
+---
+
+## Meaning
+
+Received or sent throughput on the interface `nic` exceeds the configured share
+of its link bandwidth for 15 minutes. The default is 90%. The recordings
+`windows:net_received_utilization:ratio` and
+`windows:net_sent_utilization:ratio` divide the byte rate of each direction by
+`windows_net_current_bandwidth_bytes`.
+
+## Impact
+
+The link is close to saturation in at least one direction. Further traffic
+queues at the adapter, which raises latency and causes discards and TCP
+retransmissions for every application on the host.
+
+## Diagnosis
+
+1. Inspect both utilization ratios to find the direction, and
+   `windows:net_received_bytes:rate` and `windows:net_sent_bytes:rate` for the
+   absolute volume.
+2. Check `windows_net_current_bandwidth_bytes`. A link that negotiated a lower
+   speed than expected, for example 100 Mbit/s instead of 1 Gbit/s, shows high
+   utilization at normal traffic; compare it with `Get-NetAdapter`.
+3. Correlate with `windows:net_received_discarded:rate`,
+   `windows:net_outbound_discarded:rate` and
+   `windows_net_output_queue_length_packets` to see whether the saturation
+   already drops packets.
+4. Identify the traffic with Resource Monitor, `Get-NetTCPConnection`, or
+   switch-port statistics: backups, replication, file copies and monitoring
+   bursts are common causes.
+
+## Mitigation
+
+Move or schedule the bulk transfer, apply QoS, add bandwidth, or fix a link that
+negotiated the wrong speed. Confirm utilization returns below the threshold and
+discards stop. Tune the threshold for interfaces that run near line rate by
+design, such as storage replication links.

docs/runbooks/windowspagefilealmostfull.md

@@ -0,0 +1,41 @@
+---
+title: WindowsPagefileAlmostFull
+---
+
+## Meaning
+
+The page file `file` has been more than the configured share in use for 15
+minutes. The default `pagefileUsageThreshold` is 80 percent of
+`windows_pagefile_limit_bytes`, the current size of the file. The alert needs
+the `pagefile` collector. A system-managed page file grows up to its maximum
+when it fills, so a high usage of the current size on such a system shows that
+the file has already grown.
+
+## Impact
+
+When the page file is full, Windows cannot commit more memory. Allocations
+fail, processes crash with out-of-memory errors, and the system can become
+unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/)
+usually fires at the same time.
+
+## Diagnosis
+
+1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes`
+   with `windows:memory_committed:ratio` and `windows_memory_available_bytes`
+   to see whether physical memory is exhausted too.
+2. Find the processes that hold the commit charge:
+
+   ```powershell
+   Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}}
+   Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
+   ```
+
+3. Check for a recent change: a new workload, a memory leak that grows over
+   days, or a page file that was made smaller or fixed in size.
+
+## Mitigation
+
+Stop or restart the process that leaks memory, or move workload off the host.
+Add physical memory or enlarge the page file through the normal change
+process; a fixed page file needs enough room for the peak commit charge of the
+host. Confirm usage falls below the threshold and stays there.

docs/runbooks/windowsscheduledtaskfailed.md

@@ -0,0 +1,51 @@
+---
+title: WindowsScheduledTaskFailed
+---
+
+## Meaning
+
+The last run of the scheduled task `task` ended with the result `unknown` or
+`terminated`, and the task has not completed successfully since for one hour.
+The alert needs the `scheduled_task` collector. `terminated` means the run was
+stopped by a user or by the scheduler's time limit. `unknown` is every result
+code without a named status, which includes the exit code of the task's
+program and the error codes of the Task Scheduler; the documentation of the
+collector lists the named codes. A program that exits with a code other than
+zero on success raises this alert on every run; exclude such tasks with
+`taskSelector` or with the include and exclude flags of the collector. A
+disabled task keeps its last result but does not raise the alert.
+
+## Impact
+
+The work of the task did not happen: backups, cleanups, reports, certificate
+renewals or synchronizations that are scheduled on this host. The impact
+depends on the task.
+
+## Diagnosis
+
+1. Inspect `windows_scheduled_task_last_result_status` for the task and
+   `windows_scheduled_task_state` to see whether the task is still enabled
+   and queued.
+2. On the host, read the last result and the history:
+
+   ```powershell
+   Get-ScheduledTaskInfo -TaskName '<name>' -TaskPath '<folder>'
+   Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '*<name>*'
+   ```
+
+   The history log is off by default; enable it with
+   `wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`.
+3. Translate the result code: `0x1` is a generic failure of the program, `0x2`
+   a missing file, `0x80070005` an access denied, and codes of the form
+   `0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other
+   codes between `0x41300` and `0x41325` are named statuses, such as `ready`
+   or `running`, and do not raise this alert.
+4. Run the program by hand under the task's account to reproduce the failure.
+
+## Mitigation
+
+Fix the program, its arguments, its working directory or the permissions of
+the account the task runs as. Start the task once with `Start-ScheduledTask`
+and confirm the result turns to `success`. Disable or delete tasks that are
+not needed anymore instead of leaving them failing; disabling a task clears
+the alert.

docs/runbooks/windowsscrapequeuetimeout.md

@@ -0,0 +1,40 @@
+---
+title: WindowsScrapeQueueTimeout
+---
+
+## Meaning
+
+`windows_exporter_scrape_queue_timeout == 1` for ten minutes. Scrapes run one
+at a time; a scrape whose timeout ends while it waits for an earlier scrape to
+finish runs no collector and reports this metric as `1`. On such a scrape
+`windows_exporter_collector_success` is `0` for every collector.
+
+## Impact
+
+The affected scrapes return no collector metrics, so the host has gaps in all
+of its series and `WindowsCollectorFailed` can fire for every collector at
+once. The exporter itself is reachable.
+
+## Diagnosis
+
+1. Confirm the queue timeout and find the scrape that holds the slot:
+
+   ```promql
+   windows_exporter_scrape_queue_timeout{job="windows_exporter"} == 1
+   windows_exporter_collector_timeout{job="windows_exporter"} == 1
+   scrape_duration_seconds{job="windows_exporter"}
+   ```
+
+2. Check whether more than one Prometheus server or job scrapes the same
+   exporter, or whether a job scrapes it more often than its scrape duration
+   allows. Compare `scrape_duration_seconds` with the `scrape_interval`.
+3. Identify the slow collector with `windows_exporter_collector_duration_seconds`
+   and follow `WindowsCollectorTimeout` for it.
+
+## Mitigation
+
+Scrape each exporter from one job, or give the jobs a `scrape_interval` longer
+than a full scrape takes. Shorten the slow collector or raise the
+`scrape_timeout` so a scrape finishes before the next one starts. Confirm
+`windows_exporter_scrape_queue_timeout` stays `0` and collector success returns
+to `1`.

docs/runbooks/windowsservicecrashlooping.md

@@ -0,0 +1,51 @@
+---
+title: WindowsServiceCrashLooping
+---
+
+## Meaning
+
+The service `service` has run as more processes in the last 15 minutes than
+the configured number of restarts allows. The default is two restarts. Every
+service process has its own `windows_service_process` series, identified by
+`process_id`, so the series seen within 15 minutes count the restarts.
+
+Three limits apply. Windows reuses process IDs, so a restart that lands on the
+same ID is not counted. Services that share a process, such as `svchost.exe`
+groups, share a `process_id`: one crash of the shared process alerts for every
+service in it. Services without a `windows_service_process` series, for example
+stopped services or services the exporter cannot read the process of, are
+invisible to this alert.
+
+## Impact
+
+The service crashes or is stopped and started repeatedly. It is unavailable
+during every restart, and the recovery options of the Service Control Manager
+give up after a configurable number of failures, after which
+`WindowsServiceNotRunning` follows.
+
+## Diagnosis
+
+1. List the processes the service ran as:
+
+   ```promql
+   count_over_time(windows_service_process{job="windows_exporter", service="<service>"}[15m])
+   ```
+
+   The value of each series is the start time of the process. If several
+   services report the same `process_id`, the shared host process crashed and
+   the faulty service is one of that group.
+2. Check the Application log for crashes of the service executable (events
+   1000 and 1026, .NET Runtime errors) and the System log for Service Control
+   Manager events 7031 and 7034, which also show the recovery action.
+3. Check whether a deployment, configuration push, scheduled task or another
+   automation restarts the service on purpose.
+4. Check for resources the service fails to acquire at start, such as a port
+   that is in use, a certificate, a database or a license.
+
+## Mitigation
+
+Fix the crash or the start failure, roll back the deployment that caused it, or
+stop the automation that restarts the service. Keep the service stopped until
+the cause is known if every restart has side effects. Confirm the service keeps
+one process and stays running. Raise the threshold for services that restart by
+design, for example worker services that recycle themselves.

docs/runbooks/windowsservicenotrunning.md

@@ -0,0 +1,45 @@
+---
+title: WindowsServiceNotRunning
+---
+
+## Meaning
+
+The service `service` has the start mode `auto` but has not been in the
+`running` state for 15 minutes. Trigger-start services, which Windows stops by
+design when their trigger condition ends, are left out through the
+`trigger_start` label of `windows_service_info`. Delayed-start services are
+included; the 15 minute duration covers their later start after a reboot.
+
+## Impact
+
+A service that should always run is stopped, so the function it provides is
+unavailable, for example printing, time synchronization, a site or a
+database instance. Windows restarts a crashed service only when its recovery
+options say so.
+
+## Diagnosis
+
+1. Inspect the current state of the service and when it changed:
+
+   ```promql
+   windows_service_state{job="windows_exporter", service="<service>"} == 1
+   ```
+
+2. Check the System event log for Service Control Manager events 7000, 7009,
+   7031 and 7034, and the Application log for crashes of the service process.
+3. Check `windows_service_info` for the account in `run_as`; an expired
+   password or a removed right to log on as a service stops the service at the
+   next start. `windows_service_process` shows whether a process is running.
+4. Check whether the service was stopped on purpose for maintenance or
+   disabled by policy, and whether it depends on another service that is also
+   stopped.
+
+## Mitigation
+
+Start the service after fixing the cause: the account, a missing dependency, a
+configuration error or a crash. Set recovery options so Windows restarts the
+service itself. Change the start mode to `manual` or `disabled` for services
+that should not run, or exclude them with the `serviceSelector` when the mixin
+should not watch them. For example
+`serviceSelector: 'service!~"sppsvc|gupdate|edgeupdate"'` ignores services that
+exit on their own by design. Confirm the service stays running.

docs/runbooks/windowsstoragepoolunhealthy.md

@@ -0,0 +1,48 @@
+---
+title: WindowsStoragePoolUnhealthy
+---
+
+## Meaning
+
+The Storage Spaces pool `pool` reports a health status of warning or unhealthy
+for ten minutes. The critical alert fires for unhealthy, the warning for
+warning. The alert needs the `storage_spaces` collector and joins
+`windows_storage_spaces_pool_info` so the primordial pool, which only groups
+the disks that are not in a pool yet, does not raise it. The pool's
+operational status, exposed as `windows_storage_spaces_pool_operational_status`,
+tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every
+node reports the clustered pool, so the alert fires once per node; the
+`instance` label names the reporter.
+
+## Impact
+
+A pool in warning state has lost a physical disk or redundancy; the virtual
+disks on it still work but have no tolerance left for another failure, or
+cannot allocate more space. An unhealthy pool has lost more disks than its
+resiliency tolerates, so virtual disks on it are detached or read-only.
+
+## Diagnosis
+
+1. Inspect `windows_storage_spaces_pool_health_status`,
+   `windows_storage_spaces_pool_operational_status` and
+   `windows_storage_spaces_virtual_disk_health_status` for the pool and its
+   virtual disks.
+2. On the host, find the failed disk:
+
+   ```powershell
+   Get-StoragePool -FriendlyName '<pool>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly
+   Get-StoragePool -FriendlyName '<pool>' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage
+   Get-StoragePool -FriendlyName '<pool>' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-StorageJob
+   ```
+
+3. Check the System log for disk and storage events, and whether a repair job
+   is already running.
+
+## Mitigation
+
+Replace the failed physical disk: retire it with
+`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`,
+repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk
+with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the
+health returns to healthy once the repair jobs finish.

docs/runbooks/windowsstoragevirtualdiskunhealthy.md

@@ -0,0 +1,43 @@
+---
+title: WindowsStorageVirtualDiskUnhealthy
+---
+
+## Meaning
+
+The Storage Spaces virtual disk `virtual_disk` reports a health status of
+warning or unhealthy for ten minutes. The critical alert fires for unhealthy,
+the warning for warning. The alert needs the `storage_spaces` collector. A
+virtual disk is in warning state when it is degraded or incomplete and still
+accessible, and unhealthy when it is detached or has lost data. On Storage
+Spaces Direct every node reports the clustered virtual disks, so the alert
+fires once per node; the `instance` label names the reporter.
+
+## Impact
+
+A degraded virtual disk still serves data but has no redundancy left; a
+further disk failure loses the data. A detached virtual disk takes its volumes
+offline, and everything stored on them is unavailable.
+
+## Diagnosis
+
+1. Inspect `windows_storage_spaces_virtual_disk_health_status` and
+   `windows_storage_spaces_pool_health_status` to see whether the pool is
+   affected too; [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
+   then names the pool.
+2. On the host, read the virtual disk, its footprint and the repair jobs:
+
+   ```powershell
+   Get-VirtualDisk -FriendlyName '<disk>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, ResiliencySettingName, DetachedReason
+   Get-VirtualDisk -FriendlyName '<disk>' | Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-StorageJob
+   ```
+
+3. Check for a failed physical disk, a pool that ran out of space for
+   allocation, or a disk that was detached by a policy after an error.
+
+## Mitigation
+
+Replace the failed physical disk as in [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
+and repair the virtual disk with `Repair-VirtualDisk`. Attach a detached disk
+with `Connect-VirtualDisk` once the cause is fixed. Confirm the health returns
+to healthy once the repair jobs finish.

docs/runbooks/windowsupdatespending.md

@@ -0,0 +1,50 @@
+---
+title: WindowsUpdatesPending
+---
+
+## Meaning
+
+The host has updates waiting for installation for longer than
+`updatePendingDuration`, seven days by default. The alert is an info alert, its
+value is the number of pending updates, and it needs the `update` collector,
+which queries the Windows Update agent in the background at the interval of
+its scrape setting. Updates waiting for uninstallation do not count. Prometheus counts the
+duration and restores it after a restart shorter than its
+`--rules.alert.for-outage-tolerance`, one hour by default; a longer outage or
+a new server starts the wait again.
+
+## Impact
+
+Security updates that wait longer than the patch window leave known
+vulnerabilities open. Other updates carry fixes for reliability problems. The
+host may also be waiting for a restart to finish an installation.
+
+## Diagnosis
+
+1. List the pending updates with their category and severity:
+
+   ```promql
+   windows_update_pending_info{job="windows_exporter", instance="<instance>", deployment_action="installation"}
+   ```
+
+   `windows_update_pending_published_timestamp` shows how old each update is.
+2. On the host, check the update client and the last installation attempts:
+
+   ```powershell
+   Get-WindowsUpdateLog
+   Get-WinEvent -LogName Microsoft-Windows-WindowsUpdateClient/Operational -MaxEvents 50
+   Get-CimInstance -Namespace root/cimv2 -ClassName Win32_QuickFixEngineering | Sort-Object InstalledOn -Descending | Select-Object -First 5
+   ```
+
+3. Check whether the host is in a maintenance window, a deferral period or a
+   WSUS or Configuration Manager deployment that has not approved the updates
+   yet, and whether a pending restart blocks the installation.
+
+## Mitigation
+
+Install the updates in the next maintenance window, or approve them in the
+update management tool. Restart the host if an installation waits for it.
+Raise `updatePendingDuration` to the patch window of the environment if seven
+days is shorter than the policy, or turn the alert off with
+`alerts+: { WindowsUpdatesPending: false }` where another system tracks patch
+compliance.