mixin: add alerts for the opt-in collectors

Every optional collector of the mixin now has alerts, generated only with
that collector enabled in _config.collectors: clock source, disk drive
status, page file usage, license status, scheduled task results, pending
updates, Hyper-V WMI and replication health, failover cluster nodes,
resources, groups and shared volumes, Storage Spaces pool and virtual disk
health, DHCP scope exhaustion and DNS zone transfer failures. The alerts
of the pagefile, diskdrive and license collectors join the memory, disk
and system groups; the other collectors get a group of their own.

New knobs: taskSelector, sharedVolumeSelector, pagefileUsageThreshold,
dhcpScopeFreeWarningThreshold, dhcpScopeFreeCriticalThreshold and
updatePendingDuration. The test configuration enables every optional
collector and tests/optional.test.yaml covers the new alerts.

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

docs/runbooks/_index.md

@@ -11,4 +11,6 @@ The default thresholds are starting points. Check the generated alert rules for
 your configured thresholds and durations. The rules are grouped by collector
 area, and each alert is generated only when the collectors it reads are enabled
 in the mixin configuration. Active Directory and NTP clock alerts need the `ad`
-and `time` collectors, which are off by default.
+and `time` collectors, which are off by default, and the alerts for page files,
+disk drives, licensing, scheduled tasks, updates, Hyper-V, failover clusters,
+Storage Spaces, DHCP and DNS need the collector of that role.

docs/runbooks/windowsclocksourcelocal.md

@@ -0,0 +1,56 @@
+---
+title: WindowsClockSourceLocal
+---
+
+## Meaning
+
+Windows Time reports `Local CMOS Clock` or `NoSync` as the source of the system
+clock for ten minutes. The alert carries the source in its `type` label. It
+needs the `time` collector with its `clock_source` subcollector and covers
+domain members too, whose source is `NT5DS` when they synchronize from the
+domain hierarchy; the NTP source count of `WindowsClockNotSynchronising` does
+not see those hosts. A host that is meant to be the free-running reference of
+an isolated network reports the local clock by design; exclude it with
+`alerts+: { WindowsClockSourceLocal: false }` on that configuration.
+
+## Impact
+
+The clock drifts without an external source. Clock errors break Kerberos
+authentication beyond the five minute tolerance, TLS validation, scheduled
+work, log correlation and distributed systems that depend on time order.
+
+## Diagnosis
+
+1. Inspect `windows_time_clock_sync_source` for the host, and
+   `windows_time_computed_time_offset_seconds` to see how far the clock has
+   already drifted.
+2. On the host, check the configured and the effective source:
+
+   ```powershell
+   w32tm /query /source
+   w32tm /query /status
+   w32tm /query /configuration
+   Get-Service W32Time
+   ```
+
+3. `Local CMOS Clock` after a reboot often means the Windows Time service
+   started before the network was up or the configured peers were unreachable.
+   `NoSync` means the time provider is configured not to synchronize.
+4. For domain members, check that a domain controller is reachable and that
+   the PDC emulator synchronizes from an external source. For virtual machines,
+   check whether hypervisor time integration overrides the configured source.
+   A Hyper-V guest outside a domain that is kept in time by the integration
+   service reports `NoSync` while `w32tm /query /source` shows the
+   `VM IC Time Synchronization Provider`; that is by design, so turn the alert
+   off for such guests with `alerts+: { WindowsClockSourceLocal: false }` or
+   configure an NTP source in the guest.
+
+## Mitigation
+
+Configure the intended source and restart synchronization with
+`w32tm /config /syncfromflags:domhier /update` on domain members or
+`w32tm /config /manualpeerlist:"<servers>" /syncfromflags:manual /update` on
+stand-alone hosts, then `w32tm /resync /rediscover`. Fix network access to the
+peers over UDP port 123. Confirm the source changes to `NT5DS` or `NTP` and the
+offset returns to normal; follow [WindowsClockSkewDetected](../windowsclockskewdetected/)
+if it does not.

docs/runbooks/windowsclusternodedown.md

@@ -0,0 +1,46 @@
+---
+title: WindowsClusterNodeDown
+---
+
+## Meaning
+
+The failover cluster reports the node `node` as down for five minutes. The
+alert needs the `mscluster` collector. Every node that runs the exporter
+reports the state of all nodes, so the alert fires once per reporting node;
+the `instance` label names the reporter, and the `node` label the node that is
+down. A paused node, for example during cluster-aware updating, has a
+different state and does not raise this alert.
+
+## Impact
+
+The roles that the node hosted have failed over to the remaining nodes, with a
+short outage during the move. The cluster has less capacity and less
+redundancy; with too few nodes for quorum, the whole cluster stops.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_node_state` for all nodes and
+   `windows_mscluster_cluster_quorum_type` and the dynamic quorum metrics to
+   see how far the cluster is from losing quorum. Check whether the node's own
+   exporter target is up.
+2. On a working node, read the cluster state:
+
+   ```powershell
+   Get-ClusterNode
+   Get-ClusterGroup
+   Get-ClusterLog -Node '<node>' -TimeSpan 60 -Destination .
+   Get-WinEvent -LogName Microsoft-Windows-FailoverClustering/Operational -MaxEvents 50
+   ```
+
+3. Check whether the node is powered on and reachable, whether the Cluster
+   Service runs on it, and whether the cluster networks between the nodes are
+   working. Events 1135 and 1177 in the System log name the lost node and the
+   quorum state.
+
+## Mitigation
+
+Bring the node back: power it on, repair its network, or start the Cluster
+Service with `Start-ClusterNode`. Evict a node that is removed for good with
+`Remove-ClusterNode` so the quorum configuration matches the remaining nodes.
+Move the roles back to their preferred owners once the node is up and confirm
+the node state returns to up.

docs/runbooks/windowsclusterresourcefailed.md

@@ -0,0 +1,45 @@
+---
+title: WindowsClusterResourceFailed
+---
+
+## Meaning
+
+The failover cluster reports the resource `resource` of type `type` in the
+group `resource_group` as failed for five minutes. The alert needs the
+`mscluster` collector and fires once per reporting node, named by the
+`instance` label. A failed resource has exhausted its restart attempts on the
+current node, or its group has reached its failover threshold.
+
+## Impact
+
+The role that depends on the resource is down: a clustered disk, IP address,
+network name, file share, SQL Server instance or virtual machine. Other
+resources in the same group that depend on it are offline too.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_resource_state` for the group and
+   `windows_mscluster_resourcegroup_state` and `windows_mscluster_resource_owner_node`
+   to see where the group is and whether the whole group failed.
+2. On a cluster node, read the resource and the reason:
+
+   ```powershell
+   Get-ClusterResource -Name '<resource>' | Format-List *
+   Get-ClusterResource -Name '<resource>' | Get-ClusterParameter
+   Get-ClusterLog -TimeSpan 30 -Destination .
+   Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205)]]" -MaxEvents 20
+   ```
+
+   Event 1069 names the failed resource and its error code.
+3. Check the dependency of the resource: a disk that is not reachable from the
+   node, an IP address whose network is down, a network name whose computer
+   object in Active Directory is missing or disabled, or an application whose
+   service fails to start.
+
+## Mitigation
+
+Fix the dependency, then bring the resource online with
+`Start-ClusterResource`, or move the group to another node with
+`Move-ClusterGroup`. Review the restart and failover thresholds of the
+resource if it failed because of a transient problem. Confirm the resource
+state returns to online and the group is online.

docs/runbooks/windowsclusterresourcegroupfailed.md

@@ -0,0 +1,41 @@
+---
+title: WindowsClusterResourceGroupFailed
+---
+
+## Meaning
+
+The failover cluster reports the resource group `resource_group` as failed for
+five minutes. The alert needs the `mscluster` collector and fires once per
+reporting node, named by the `instance` label. A group fails when a resource in
+it fails and the group cannot fail over anymore: it reached its failover
+threshold within the failover period, or no other node can host it.
+
+## Impact
+
+The whole clustered role is down and stays down until an operator intervenes.
+
+## Diagnosis
+
+1. Inspect `windows_mscluster_resource_state` for the resources of the group
+   to find the failed resource, and `windows_mscluster_resourcegroup_owner_node`
+   to see on which node the group stopped.
+2. On a cluster node, read the group and its history:
+
+   ```powershell
+   Get-ClusterGroup -Name '<group>' | Format-List *
+   Get-ClusterGroup -Name '<group>' | Get-ClusterResource
+   Get-ClusterLog -TimeSpan 60 -Destination .
+   Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20
+   ```
+
+   Event 1254 says the group exceeded its failover threshold.
+3. Follow [WindowsClusterResourceFailed](../windowsclusterresourcefailed/) for
+   the failed resource, and check the possible owners of the group.
+
+## Mitigation
+
+Fix the failed resource, then bring the group online with `Start-ClusterGroup`
+on a node that can host it. A group that failed over repeatedly may need a
+higher failover threshold or a longer failover period, but a resource that
+fails on every node needs its cause fixed first. Confirm the group state
+returns to online.

docs/runbooks/windowsclustersharedvolumealmostfull.md

@@ -0,0 +1,44 @@
+---
+title: WindowsClusterSharedVolumeAlmostFull
+---
+
+## Meaning
+
+The Cluster Shared Volume `shared_volume` has less free space than the
+configured threshold for 15 minutes. The warning fires below
+`diskFreeWarningThreshold` (10 percent by default) and the critical alert
+below `diskFreeCriticalThreshold` (5 percent); both fire below the critical
+threshold, so inhibit the warning in Alertmanager. The alert needs the
+`mscluster` collector and fires once per reporting node, named by the
+`instance` label. `sharedVolumeSelector` excludes the
+`ClusterPerformanceHistory` volume of the cluster's own health history.
+
+## Impact
+
+Virtual machines and databases on the volume cannot grow their files. Dynamic
+virtual hard disks pause the virtual machine when the volume is full, Hyper-V
+checkpoints and backups fail, and SQL Server stops writing.
+
+## Diagnosis
+
+1. Compare `windows_mscluster_shared_volumes_free_bytes` and
+   `windows_mscluster_shared_volumes_total_bytes` over the last days to see
+   whether the volume fills steadily or filled suddenly.
+2. On a cluster node, find the large files on the volume:
+
+   ```powershell
+   Get-ClusterSharedVolume | Select-Object Name, @{n='FreeGB';e={[int]($_.SharedVolumeInfo.Partition.FreeSpace/1GB)}}, @{n='SizeGB';e={[int]($_.SharedVolumeInfo.Partition.Size/1GB)}}
+   Get-ChildItem 'C:\ClusterStorage\<volume>' -Recurse -File | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='GB';e={[int]($_.Length/1GB)}}
+   ```
+
+3. Look for leftover checkpoints (`.avhdx` files), orphaned virtual hard
+   disks, backup snapshots that were not cleaned up, and dynamic disks that
+   grew.
+
+## Mitigation
+
+Delete or merge checkpoints, remove orphaned files, move virtual machines to
+another volume with storage migration, or extend the volume on the storage
+array and then in the cluster with `Resize-Partition`. Confirm free space is
+above the threshold. Raise the thresholds only for volumes whose growth is
+known and planned.

docs/runbooks/windowsdhcpscopealmostexhausted.md

@@ -0,0 +1,48 @@
+---
+title: WindowsDHCPScopeAlmostExhausted
+---
+
+## Meaning
+
+The enabled DHCP scope `scope` has fewer free addresses than the configured
+share of its address pool for 15 minutes. The warning fires below
+`dhcpScopeFreeWarningThreshold` (10 percent by default) and the critical alert
+below `dhcpScopeFreeCriticalThreshold` (2 percent); both fire below the
+critical threshold, so inhibit the warning in Alertmanager. The share divides
+`windows_dhcp_scope_addresses_free` by the sum of free and in-use addresses,
+and the alert only counts scopes whose `windows_dhcp_scope_state` is enabled.
+On a failover pair the free and in-use counts are the totals of both servers.
+The alert needs the `dhcp` collector.
+
+## Impact
+
+Once the scope is exhausted, new clients receive no address and cannot use
+the network. Existing clients keep their lease until it expires and then lose
+their address too.
+
+## Diagnosis
+
+1. Compare the free and in-use addresses of the scope over the last days to
+   see whether the exhaustion is a trend or a sudden burst, and inspect
+   `windows_dhcp_scope_reserved_address` and
+   `windows_dhcp_scope_pending_offers` for reservations and offers in flight.
+2. On the server, read the scope and its leases:
+
+   ```powershell
+   Get-DhcpServerv4ScopeStatistics -ScopeId <scope>
+   Get-DhcpServerv4Scope -ScopeId <scope> | Select-Object ScopeId, SubnetMask, StartRange, EndRange, LeaseDuration, State
+   Get-DhcpServerv4Lease -ScopeId <scope> | Group-Object AddressState
+   Get-DhcpServerv4ExclusionRange -ScopeId <scope>
+   ```
+
+3. Look for a long lease duration on a network with many short-lived clients,
+   large exclusion ranges, a flood of leases from one device that changes its
+   MAC address, or a failover partner that holds the free addresses.
+
+## Mitigation
+
+Shorten the lease duration for networks with transient clients, remove
+exclusion ranges that are not needed, add a range through a superscope or a
+larger subnet, or reclaim bad leases. On a failover pair, check the split of
+the addresses between the partners. Confirm the free share stays above the
+threshold.

docs/runbooks/windowsdiskdrivenotok.md

@@ -0,0 +1,46 @@
+---
+title: WindowsDiskDriveNotOK
+---
+
+## Meaning
+
+The physical drive `device_id` reports an operational status other than `OK`
+for five minutes. The alert carries the status, such as `Pred Fail`, `Degraded`
+or `Error`, in its `status` label. It needs the `diskdrive` collector, which
+reads the status that the drive and its driver report through WMI. The
+`Unknown` status does not raise the alert: some USB and virtual drives report
+it permanently. Transient statuses such as `Starting` or `Stopping` raise it
+only when they last longer than five minutes.
+
+## Impact
+
+`Pred Fail` is the drive's own prediction that it is about to fail. Any other
+status means the drive is not fully working. Data on a single drive is at risk;
+a drive in a mirror or parity set leaves the set without redundancy until it is
+replaced.
+
+## Diagnosis
+
+1. Inspect `windows_diskdrive_status` for the host and check whether the
+   volumes on the drive still report normal latency with
+   `windows:logical_disk_read_latency:seconds` and
+   `windows:logical_disk_write_latency:seconds`.
+2. On the host, read the drive and its reliability counters:
+
+   ```powershell
+   Get-CimInstance Win32_DiskDrive | Select-Object DeviceID, Model, Status
+   Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, Temperature, Wear
+   ```
+
+3. Check the System event log for disk, storahci, stornvme or RAID controller
+   events. On a virtual machine, check the hypervisor and the datastore of the
+   virtual disk.
+
+## Mitigation
+
+Back up the data on the drive first. Replace a drive that predicts its failure
+or reports errors through the hardware or virtualization team. For a drive in
+a storage pool or RAID set, follow the vendor's replacement and rebuild
+procedure and confirm the set is redundant again. The alert resolves when the
+status returns to `OK`.

docs/runbooks/windowsdnszonetransferfailures.md

@@ -0,0 +1,54 @@
+---
+title: WindowsDNSZoneTransferFailures
+---
+
+## Meaning
+
+The DNS server has failed zone transfers for 15 minutes, measured as the rate
+of `windows_dns_zone_transfer_failures_total` over `rateInterval` above
+`dnsZoneTransferFailureRateThreshold`, zero by default. The counter counts
+transfers this server sends as the primary to its secondaries, including
+requests it refuses, and it has no zone label. The alert needs the `dns`
+collector and keeps firing for `keepFiringFor` after the failures stop. Raise
+the threshold on servers that refuse transfer requests from scanners all the
+time.
+
+## Impact
+
+The secondary servers of the zone serve stale records. When the zone on a
+secondary expires, it stops answering for the zone altogether. Active
+Directory integrated zones replicate through Active Directory and are not
+affected unless they also have secondaries.
+
+## Diagnosis
+
+1. Inspect `windows_dns_zone_transfer_failures_total` next to
+   `windows_dns_zone_transfer_requests_received_total` and
+   `windows_dns_zone_transfer_success_sent_total` to see what share of the
+   transfers fail, and whether the failures started with a change.
+2. On this primary, find the refused or failed requests and the zone
+   settings; the DNS Server log records a refused transfer request with the
+   address of the requester:
+
+   ```powershell
+   Get-WinEvent -LogName 'DNS Server' -MaxEvents 100 | Where-Object Message -match 'zone transfer'
+   Get-DnsServerZone | Where-Object { $_.ZoneType -eq 'Primary' } | Get-DnsServerZoneTransferPolicy
+   Get-DnsServerZone -Name '<zone>' | Select-Object ZoneName, SecureSecondaries, SecondaryServers, NotifyServers
+   ```
+
+   On the secondary, events 6522 to 6527 of the DNS Server log say why its
+   transfer of the zone failed: refused by the primary, invalid response or
+   an expired zone.
+
+3. Check that the requesting secondary is in the zone transfer list, that TCP
+   port 53 is open between the servers, and that the zone is not paused or
+   broken. Failures from unknown addresses are refused transfers and may be
+   expected.
+
+## Mitigation
+
+Add the secondary to the allowed servers with `Set-DnsServerPrimaryZone -SecureSecondaries`
+and the server list, fix the network path, or remove a decommissioned
+secondary from the zone's notify list so it stops requesting transfers.
+Confirm the failure counter stops rising and the secondaries show a current
+serial number.

docs/runbooks/windowshypervreplicationcritical.md

@@ -0,0 +1,47 @@
+---
+title: WindowsHyperVReplicationCritical
+---
+
+## Meaning
+
+Hyper-V Replica reports critical health for the virtual machine `vm` for five
+minutes. The alert carries the `relationship` label, `primary` for the normal
+replication and `extended` for an extended replica, and needs the `hyperv`
+collector with its `replica_vm` subcollector. Critical health means that
+replication has not succeeded for long enough that the replica misses the
+recovery point objective, or that replication is paused because of an error.
+
+## Impact
+
+The replica of the virtual machine is stale. A failover to it loses the
+changes since the last successful replication, and the recovery point
+objective of the service is not met. If the primary host fails now, data is
+lost.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_replica_vm_health` and
+   `windows_hyperv_replica_vm_state` for the virtual machine; the state tells
+   whether replication is suspended, resynchronizing or in error, and
+   `windows_hyperv_replica_vm_replication_latency_seconds` how long the last
+   cycle took.
+2. On the primary host, read the replication details:
+
+   ```powershell
+   Measure-VMReplication -VMName '<vm>' | Format-List *
+   Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, ReplicationMode
+   Get-WinEvent -LogName Microsoft-Windows-Hyper-V-VMMS-Admin -MaxEvents 50 | Where-Object Message -like '*<vm>*'
+   ```
+
+3. Check the replica server: free space on the replica storage, the
+   Hyper-V Replica HTTP or HTTPS listener, the certificate for authentication,
+   and the network path between the hosts. Check whether a snapshot merge or a
+   storage migration on the primary blocks replication.
+
+## Mitigation
+
+Fix the cause on the replica server or the network, then resume replication
+with `Resume-VMReplication -VMName '<vm>'`; add `-Resynchronize` when the
+state asks for a resynchronization. Confirm the health returns to normal and
+the last replication time advances. If the replica cannot be recovered, remove
+and re-enable replication, which sends a full initial copy.

docs/runbooks/windowshypervreplicationwarning.md

@@ -0,0 +1,43 @@
+---
+title: WindowsHyperVReplicationWarning
+---
+
+## Meaning
+
+Hyper-V Replica reports warning health for the virtual machine `vm` for 15
+minutes. Warning health means that some replication cycles missed their
+interval, that more than a fifth of the cycles in the last hour failed, or
+that the replication is waiting for an initial copy or a resynchronization.
+The alert carries the `relationship` label and needs the `hyperv` collector
+with its `replica_vm` subcollector. [WindowsHyperVReplicationCritical](../windowshypervreplicationcritical/)
+follows when replication stops working.
+
+## Impact
+
+The replica lags behind the primary more than planned. The recovery point
+objective is at risk, and a failover would lose more changes than expected.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_replica_vm_health`, `windows_hyperv_replica_vm_state`
+   and `windows_hyperv_replica_vm_replication_latency_seconds` for the
+   virtual machine, and `windows_hyperv_replica_vm_average_replication_size_bytes`
+   to see whether the change rate of the virtual machine has grown.
+2. On the primary host, read the statistics of the last hour:
+
+   ```powershell
+   Measure-VMReplication -VMName '<vm>' | Format-List *
+   Get-VMReplication -VMName '<vm>' | Format-List State, Health, LastReplicationTime, FrequencySec
+   ```
+
+3. Check the bandwidth between the hosts, the load on the replica storage,
+   and whether the replication interval is too short for the change rate of
+   the virtual machine.
+
+## Mitigation
+
+Relieve the bottleneck: more bandwidth, less load on the replica storage, or a
+longer replication interval through `Set-VMReplication -ReplicationFrequencySec`.
+Resume a waiting replication with `Resume-VMReplication`. Reset the statistics
+with `Reset-VMReplicationStatistics` after the fix; the health is computed
+over the last hour and takes up to an hour to return to normal.

docs/runbooks/windowshypervwmiunhealthy.md

@@ -0,0 +1,43 @@
+---
+title: WindowsHyperVWMIUnhealthy
+---
+
+## Meaning
+
+The Hyper-V WMI namespace `root/virtualization/v2` on the host has not
+answered a minimal query for ten minutes. The alert needs the `hyperv`
+collector with its `wmi_health` subcollector. While the namespace is broken,
+the virtual machine, replication and virtual hard disk metrics that come from
+WMI are missing, so the other Hyper-V alerts cannot fire.
+
+## Impact
+
+Management tools such as Hyper-V Manager, Failover Cluster Manager and System
+Center cannot manage the virtual machines on the host. Backups that use the
+Hyper-V VSS writer fail. The virtual machines themselves usually keep running.
+
+## Diagnosis
+
+1. Inspect `windows_hyperv_wmi_health` and `windows_exporter_collector_success{collector="hyperv"}`
+   for the host.
+2. On the host, test the namespace and the management service:
+
+   ```powershell
+   Get-CimInstance -Namespace root/virtualization/v2 -ClassName Msvm_ComputerSystem | Select-Object -First 1
+   Get-Service vmms, vmcompute, winmgmt
+   winmgmt /verifyrepository
+   ```
+
+3. Check the System and the Microsoft-Windows-Hyper-V-VMMS logs for errors of
+   the Virtual Machine Management service, and the WMI-Activity log for
+   provider failures. A stuck `vmms` process or a corrupt WMI repository are
+   the usual causes.
+
+## Mitigation
+
+Restart the Virtual Machine Management service with `Restart-Service vmms`;
+running virtual machines are not affected. If the repository is inconsistent,
+run `winmgmt /salvagerepository`, and register the Hyper-V provider again with
+`mofcomp` of the Hyper-V MOF files only when Microsoft support advises it. Plan a
+restart of the host if the service cannot be recovered. Confirm the health
+metric returns to 1 and the Hyper-V metrics appear again.

docs/runbooks/windowslicensenotgenuine.md

@@ -0,0 +1,45 @@
+---
+title: WindowsLicenseNotGenuine
+---
+
+## Meaning
+
+Windows reports a license status other than `genuine` for one hour. The alert
+carries the status in its `state` label and needs the `license` collector. The
+status comes from the Software Licensing service and changes when an
+activation expires, a KMS host is unreachable for more than the grace period,
+the hardware changed, or the key was blocked. `offline` means the genuine
+validation could not reach Microsoft; on hosts without internet access that
+is the permanent state, so turn the alert off there with
+`alerts+: { WindowsLicenseNotGenuine: false }` or scope it with the exporter
+selector.
+
+## Impact
+
+An unlicensed Windows shows activation notices, may limit personalization and,
+on some editions, restarts on a schedule. A Windows Server that leaves its
+activation grace period is out of compliance.
+
+## Diagnosis
+
+1. The `state` label of the alert names the status; `windows_license_status`
+   shows all states of the host.
+2. On the host, read the license details:
+
+   ```powershell
+   slmgr.vbs /dlv
+   slmgr.vbs /xpr
+   Get-CimInstance SoftwareLicensingProduct -Filter "PartialProductKey IS NOT NULL" | Select-Object Name, LicenseStatus, GracePeriodRemaining
+   ```
+
+3. For KMS clients, check that the KMS host resolves through the `_vlmcs._tcp`
+   DNS record and answers on TCP port 1688. For Active Directory based
+   activation, check the activation object in the forest. For MAK or retail
+   keys, check whether the activation count is exhausted.
+
+## Mitigation
+
+Restore reachability to the KMS host or the activation object, then run
+`slmgr.vbs /ato`. Install the correct key with `slmgr.vbs /ipk <key>` if the
+key was wrong, or reactivate after a hardware change. Confirm the status
+returns to `genuine`.

docs/runbooks/windowspagefilealmostfull.md

@@ -0,0 +1,41 @@
+---
+title: WindowsPagefileAlmostFull
+---
+
+## Meaning
+
+The page file `file` has been more than the configured share in use for 15
+minutes. The default `pagefileUsageThreshold` is 80 percent of
+`windows_pagefile_limit_bytes`, the current size of the file. The alert needs
+the `pagefile` collector. A system-managed page file grows up to its maximum
+when it fills, so a high usage of the current size on such a system shows that
+the file has already grown.
+
+## Impact
+
+When the page file is full, Windows cannot commit more memory. Allocations
+fail, processes crash with out-of-memory errors, and the system can become
+unresponsive. The commit limit alert [WindowsMemoryCommitHighUsage](../windowsmemorycommithighusage/)
+usually fires at the same time.
+
+## Diagnosis
+
+1. Compare `windows_pagefile_free_bytes` and `windows_pagefile_limit_bytes`
+   with `windows:memory_committed:ratio` and `windows_memory_available_bytes`
+   to see whether physical memory is exhausted too.
+2. Find the processes that hold the commit charge:
+
+   ```powershell
+   Get-Process | Sort-Object PagedMemorySize64 -Descending | Select-Object -First 10 Name, Id, @{n='CommitMB';e={[int]($_.PagedMemorySize64/1MB)}}
+   Get-CimInstance Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
+   ```
+
+3. Check for a recent change: a new workload, a memory leak that grows over
+   days, or a page file that was made smaller or fixed in size.
+
+## Mitigation
+
+Stop or restart the process that leaks memory, or move workload off the host.
+Add physical memory or enlarge the page file through the normal change
+process; a fixed page file needs enough room for the peak commit charge of the
+host. Confirm usage falls below the threshold and stays there.

docs/runbooks/windowsscheduledtaskfailed.md

@@ -0,0 +1,51 @@
+---
+title: WindowsScheduledTaskFailed
+---
+
+## Meaning
+
+The last run of the scheduled task `task` ended with the result `unknown` or
+`terminated`, and the task has not completed successfully since for one hour.
+The alert needs the `scheduled_task` collector. `terminated` means the run was
+stopped by a user or by the scheduler's time limit. `unknown` is every result
+code without a named status, which includes the exit code of the task's
+program and the error codes of the Task Scheduler; the documentation of the
+collector lists the named codes. A program that exits with a code other than
+zero on success raises this alert on every run; exclude such tasks with
+`taskSelector` or with the include and exclude flags of the collector. A
+disabled task keeps its last result but does not raise the alert.
+
+## Impact
+
+The work of the task did not happen: backups, cleanups, reports, certificate
+renewals or synchronizations that are scheduled on this host. The impact
+depends on the task.
+
+## Diagnosis
+
+1. Inspect `windows_scheduled_task_last_result_status` for the task and
+   `windows_scheduled_task_state` to see whether the task is still enabled
+   and queued.
+2. On the host, read the last result and the history:
+
+   ```powershell
+   Get-ScheduledTaskInfo -TaskName '<name>' -TaskPath '<folder>'
+   Get-WinEvent -LogName Microsoft-Windows-TaskScheduler/Operational -MaxEvents 50 | Where-Object Message -like '*<name>*'
+   ```
+
+   The history log is off by default; enable it with
+   `wevtutil set-log Microsoft-Windows-TaskScheduler/Operational /enabled:true`.
+3. Translate the result code: `0x1` is a generic failure of the program, `0x2`
+   a missing file, `0x80070005` an access denied, and codes of the form
+   `0x8xxxxxxx` are HRESULTs of the scheduler or of the program. The other
+   codes between `0x41300` and `0x41325` are named statuses, such as `ready`
+   or `running`, and do not raise this alert.
+4. Run the program by hand under the task's account to reproduce the failure.
+
+## Mitigation
+
+Fix the program, its arguments, its working directory or the permissions of
+the account the task runs as. Start the task once with `Start-ScheduledTask`
+and confirm the result turns to `success`. Disable or delete tasks that are
+not needed anymore instead of leaving them failing; disabling a task clears
+the alert.

docs/runbooks/windowsstoragepoolunhealthy.md

@@ -0,0 +1,48 @@
+---
+title: WindowsStoragePoolUnhealthy
+---
+
+## Meaning
+
+The Storage Spaces pool `pool` reports a health status of warning or unhealthy
+for ten minutes. The critical alert fires for unhealthy, the warning for
+warning. The alert needs the `storage_spaces` collector and joins
+`windows_storage_spaces_pool_info` so the primordial pool, which only groups
+the disks that are not in a pool yet, does not raise it. The pool's
+operational status, exposed as `windows_storage_spaces_pool_operational_status`,
+tells why: degraded, read-only, or incomplete. On Storage Spaces Direct every
+node reports the clustered pool, so the alert fires once per node; the
+`instance` label names the reporter.
+
+## Impact
+
+A pool in warning state has lost a physical disk or redundancy; the virtual
+disks on it still work but have no tolerance left for another failure, or
+cannot allocate more space. An unhealthy pool has lost more disks than its
+resiliency tolerates, so virtual disks on it are detached or read-only.
+
+## Diagnosis
+
+1. Inspect `windows_storage_spaces_pool_health_status`,
+   `windows_storage_spaces_pool_operational_status` and
+   `windows_storage_spaces_virtual_disk_health_status` for the pool and its
+   virtual disks.
+2. On the host, find the failed disk:
+
+   ```powershell
+   Get-StoragePool -FriendlyName '<pool>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, IsReadOnly
+   Get-StoragePool -FriendlyName '<pool>' | Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, HealthStatus, OperationalStatus, Usage
+   Get-StoragePool -FriendlyName '<pool>' | Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-StorageJob
+   ```
+
+3. Check the System log for disk and storage events, and whether a repair job
+   is already running.
+
+## Mitigation
+
+Replace the failed physical disk: retire it with
+`Set-PhysicalDisk -Usage Retired`, add the new disk with `Add-PhysicalDisk`,
+repair the virtual disks with `Repair-VirtualDisk`, and remove the old disk
+with `Remove-PhysicalDisk`. Add capacity when the pool is full. Confirm the
+health returns to healthy once the repair jobs finish.

docs/runbooks/windowsstoragevirtualdiskunhealthy.md

@@ -0,0 +1,43 @@
+---
+title: WindowsStorageVirtualDiskUnhealthy
+---
+
+## Meaning
+
+The Storage Spaces virtual disk `virtual_disk` reports a health status of
+warning or unhealthy for ten minutes. The critical alert fires for unhealthy,
+the warning for warning. The alert needs the `storage_spaces` collector. A
+virtual disk is in warning state when it is degraded or incomplete and still
+accessible, and unhealthy when it is detached or has lost data. On Storage
+Spaces Direct every node reports the clustered virtual disks, so the alert
+fires once per node; the `instance` label names the reporter.
+
+## Impact
+
+A degraded virtual disk still serves data but has no redundancy left; a
+further disk failure loses the data. A detached virtual disk takes its volumes
+offline, and everything stored on them is unavailable.
+
+## Diagnosis
+
+1. Inspect `windows_storage_spaces_virtual_disk_health_status` and
+   `windows_storage_spaces_pool_health_status` to see whether the pool is
+   affected too; [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
+   then names the pool.
+2. On the host, read the virtual disk, its footprint and the repair jobs:
+
+   ```powershell
+   Get-VirtualDisk -FriendlyName '<disk>' | Select-Object FriendlyName, HealthStatus, OperationalStatus, ResiliencySettingName, DetachedReason
+   Get-VirtualDisk -FriendlyName '<disk>' | Get-PhysicalDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus
+   Get-StorageJob
+   ```
+
+3. Check for a failed physical disk, a pool that ran out of space for
+   allocation, or a disk that was detached by a policy after an error.
+
+## Mitigation
+
+Replace the failed physical disk as in [WindowsStoragePoolUnhealthy](../windowsstoragepoolunhealthy/)
+and repair the virtual disk with `Repair-VirtualDisk`. Attach a detached disk
+with `Connect-VirtualDisk` once the cause is fixed. Confirm the health returns
+to healthy once the repair jobs finish.

docs/runbooks/windowsupdatespending.md

@@ -0,0 +1,50 @@
+---
+title: WindowsUpdatesPending
+---
+
+## Meaning
+
+The host has updates waiting for installation for longer than
+`updatePendingDuration`, seven days by default. The alert is an info alert, its
+value is the number of pending updates, and it needs the `update` collector,
+which queries the Windows Update agent in the background at the interval of
+its scrape setting. Updates waiting for uninstallation do not count. Prometheus counts the
+duration and restores it after a restart shorter than its
+`--rules.alert.for-outage-tolerance`, one hour by default; a longer outage or
+a new server starts the wait again.
+
+## Impact
+
+Security updates that wait longer than the patch window leave known
+vulnerabilities open. Other updates carry fixes for reliability problems. The
+host may also be waiting for a restart to finish an installation.
+
+## Diagnosis
+
+1. List the pending updates with their category and severity:
+
+   ```promql
+   windows_update_pending_info{job="windows_exporter", instance="<instance>", deployment_action="installation"}
+   ```
+
+   `windows_update_pending_published_timestamp` shows how old each update is.
+2. On the host, check the update client and the last installation attempts:
+
+   ```powershell
+   Get-WindowsUpdateLog
+   Get-WinEvent -LogName Microsoft-Windows-WindowsUpdateClient/Operational -MaxEvents 50
+   Get-CimInstance -Namespace root/cimv2 -ClassName Win32_QuickFixEngineering | Sort-Object InstalledOn -Descending | Select-Object -First 5
+   ```
+
+3. Check whether the host is in a maintenance window, a deferral period or a
+   WSUS or Configuration Manager deployment that has not approved the updates
+   yet, and whether a pending restart blocks the installation.
+
+## Mitigation
+
+Install the updates in the next maintenance window, or approve them in the
+update management tool. Restart the host if an installation waits for it.
+Raise `updatePendingDuration` to the patch window of the environment if seven
+days is shorter than the policy, or turn the alert off with
+`alerts+: { WindowsUpdatesPending: false }` where another system tracks patch
+compliance.