From 89c5df9d8b3157b70b5567afef43ca22ed49176e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan-Otto=20Kr=C3=B6pke?= Date: Mon, 12 Oct 2026 16:59:59 +0200 Subject: [PATCH] iis: document the 250 ms Web Service counter refresh wait (#2701) Co-authored-by: Claude Signed-off-by: Claude --- docs/collector.iis.md | 16 ++++++++++++++++ internal/collector/iis/AGENTS.md | 27 +++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 internal/collector/iis/AGENTS.md diff --git a/docs/collector.iis.md b/docs/collector.iis.md index 3acc4b95..c5cc0db2 100644 --- a/docs/collector.iis.md +++ b/docs/collector.iis.md @@ -26,6 +26,22 @@ If given, an application needs to match the include regular expression in order If given, an application needs to *not* match the exclude regular expression in order for the corresponding metrics to be reported. +## Collection duration + +`windows_exporter_collector_duration_seconds{collector="iis"}` can jump to about 250 ms on some scrapes, while it normally stays in the low milliseconds. At the same moments, some other collectors that read performance counters, such as `system`, `logical_disk` or `net`, can show the same duration. + +The wait happens inside Windows, not in the exporter. The `Web Service` counters are served by the IIS counter DLL `w3ctrs.dll`. When its counter snapshot is older than 15 seconds, it asks the World Wide Web Publishing Service (W3SVC) to refresh it and sleeps 250 ms before it checks the snapshot again. It repeats this up to 4 times, so a single wait can take up to about 1 s; the spikes observed so far were one 250 ms step. Counter reads of other collectors that start at the same moment can wait for the same time. In the observed cases the scrape still completed; only the reported duration was longer. + +The DLL uses built-in defaults, which can be overridden with nonzero `REG_DWORD` values below `HKLM\SYSTEM\CurrentControlSet\Services\W3SVC\Performance` (a missing value or 0 keeps the default). Microsoft does not document these values, and changing them has not been tested as a mitigation, so try any change on a non-production system first. `w3ctrs.dll` reads them when it is loaded, so restart windows_exporter after a change. `CheckCountersEveryNMiliseconds` is spelled this way in Windows. + +| Value | Default | +|----------------------------------|----------| +| `FreshTimeForCounters` | 15000 ms | +| `CheckCountersEveryNMiliseconds` | 250 ms | +| `NumberOfTimesToCheckCounters` | 4 | + +See [#2698](https://github.com/prometheus-community/windows_exporter/issues/2698) for the analysis. + ## Metrics | Name | Description | Type | Labels | diff --git a/internal/collector/iis/AGENTS.md b/internal/collector/iis/AGENTS.md new file mode 100644 index 00000000..c2f28286 --- /dev/null +++ b/internal/collector/iis/AGENTS.md @@ -0,0 +1,27 @@ +# iis collector + +- The `Web Service` and `Web Service Cache` objects are served by the perflib + V1 provider `w3ctrs.dll`, which reads a counter snapshot that W3SVC writes to + shared memory. When the snapshot is older than `FreshTimeForCounters` + (default 15 s), the provider signals W3SVC, calls `Sleep` for + `CheckCountersEveryNMiliseconds` (default 250 ms) and checks the snapshot + again, up to `NumberOfTimesToCheckCounters` (default 4) times. The defaults + are built into the DLL; nonzero `REG_DWORD` values below + `HKLM\SYSTEM\CurrentControlSet\Services\W3SVC\Performance` override them. + Microsoft does not document these values. +- So a scrape that hits a stale snapshot waits 250 ms, up to 4 × 250 ms, + inside `PdhCollectQueryData` of the `Web Service` query. Go execution traces + from a production server showed 259 to 261 ms there, and PDH reads of + `system`, `logical_disk` and `net` that started at the same moment blocked + for the same time. `cpu`, `memory` and the other IIS objects, including + `Web Service Cache`, were never slow, and there were no IIS event log + entries. The exact lock inside Windows that makes the other reads wait is + not known + ([#2698](https://github.com/prometheus-community/windows_exporter/issues/2698)). +- This is Windows behavior, not an exporter lock or timeout. Don't look for an + exporter-side cause of flat ~250 ms `iis` durations, and don't add + workarounds that hide them. A repro on a GitHub runner with IIS did not show + it in 40 minutes, so don't expect to reproduce it in CI. +- To find which Windows call a collector blocks in, capture a Go execution + trace with `--debug.enabled` and `/debug/pprof/trace?seconds=600`. Collectors + waiting in a Windows call use no CPU, so a CPU profile does not show it.