Jan-Otto Kröpke
d819362150
Every optional collector of the mixin now has alerts, generated only with that collector enabled in _config.collectors: clock source, disk drive status, page file usage, license status, scheduled task results, pending updates, Hyper-V WMI and replication health, failover cluster nodes, resources, groups and shared volumes, Storage Spaces pool and virtual disk health, DHCP scope exhaustion and DNS zone transfer failures. The alerts of the pagefile, diskdrive and license collectors join the memory, disk and system groups; the other collectors get a group of their own. New knobs: taskSelector, sharedVolumeSelector, pagefileUsageThreshold, dhcpScopeFreeWarningThreshold, dhcpScopeFreeCriticalThreshold and updatePendingDuration. The test configuration enables every optional collector and tests/optional.test.yaml covers the new alerts. Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
1.7 KiB
title
| title |
|---|
| WindowsClusterResourceGroupFailed |
Meaning
The failover cluster reports the resource group resource_group as failed for
five minutes. The alert needs the mscluster collector and fires once per
reporting node, named by the instance label. A group fails when a resource in
it fails and the group cannot fail over anymore: it reached its failover
threshold within the failover period, or no other node can host it.
Impact
The whole clustered role is down and stays down until an operator intervenes.
Diagnosis
-
Inspect
windows_mscluster_resource_statefor the resources of the group to find the failed resource, andwindows_mscluster_resourcegroup_owner_nodeto see on which node the group stopped. -
On a cluster node, read the group and its history:
Get-ClusterGroup -Name '<group>' | Format-List * Get-ClusterGroup -Name '<group>' | Get-ClusterResource Get-ClusterLog -TimeSpan 60 -Destination . Get-WinEvent -LogName System -FilterXPath "*[System[Provider[@Name='Microsoft-Windows-FailoverClustering'] and (EventID=1069 or EventID=1205 or EventID=1254)]]" -MaxEvents 20Event 1254 says the group exceeded its failover threshold.
-
Follow WindowsClusterResourceFailed for the failed resource, and check the possible owners of the group.
Mitigation
Fix the failed resource, then bring the group online with Start-ClusterGroup
on a node that can host it. A group that failed over repeatedly may need a
higher failover threshold or a longer failover period, but a resource that
fails on every node needs its cause fixed first. Confirm the group state
returns to online.