From 4cd3e5411d1efdeb20eb49eb452f3604d916e28a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan-Otto=20Kr=C3=B6pke?= Date: Fri, 16 Oct 2026 10:32:34 +0200 Subject: [PATCH] docs: document memory and leak investigations in AGENTS.md (#2749) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jan-Otto Kröpke --- AGENTS.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 806bfd70..4a676d49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -253,6 +253,72 @@ available on development machines (AD DS, AD CS, AD FS, DFSR, Hyper-V, IIS, SQL Server, NPS, RemoteFX sessions). Check a type or unit question there first. +## Memory and leak investigations + +Reports of growing memory are usually warm-up or bounded high-water marks, so +measure before changing code: + +- `process_virtual_memory_bytes` is the private (committed) bytes on Windows; + `process_resident_memory_bytes` is the working set, which Windows trims, so + it drops and recovers without any change in the process. Compare the daily + minimum of the private bytes over several days. Go holds about + `go_memstats_sys_bytes - go_memstats_heap_released_bytes` of it; the rest is + native memory plus the executable's data and BSS sections, which include a + 32 MiB buffer in builds without the release overlay (golang/go#81956). +- Go memory levels off within a few hours. The Go runtime keeps idle OS + threads (Ms) parked instead of destroying them; threads it adds for blocking + calls and MI callbacks stay, each with about six handles, so `go_threads` + and `process_open_fds` rise in steps and then stay flat. A thread only exits + when its goroutine ends while locked with `runtime.LockOSThread`, for example + a closed `ole.Worker`. Investigate handles only if they grow while threads + stay flat. `go_sched_goroutines_not_in_go_goroutines` rises on Windows on its + own (golang/go#82136); it does not indicate a goroutine leak. +- `--debug.enabled` exposes `/debug/pprof/`. Diff two `heap` profiles taken + hours apart for live Go memory and `allocs` profiles for allocation churn. +- Handle counts by object type need no admin rights: + `NtQuerySystemInformation(SystemExtendedHandleInformation)` plus + `NtQueryObject(ObjectTypesInformation)`, for example from a PowerShell + `Add-Type` script. `handle.exe` needs admin rights. + +For native heap growth, use UMDH from the Debugging Tools for Windows in an +elevated shell. It records an allocation stack for every heap allocation of +processes started after the flag is set: + +```powershell +$env:_NT_SYMBOL_PATH = 'srv*C:\symbols*https://msdl.microsoft.com/download/symbols' +gflags /i windows_exporter.exe +ust +Restart-Service windows_exporter +# After warm-up, and again hours later: +umdh -p: -f:a.txt +umdh -p: -f:b.txt +umdh a.txt b.txt > diff.txt +# Afterwards: +gflags /i windows_exporter.exe -ust +Restart-Service windows_exporter +``` + +Snapshots from different processes cannot be compared. Stack recording cost +about 24 MB for its database plus heap overhead on a development machine. +Expected entries that are not leaks: single 1 MB data buffers of the `perfos` +and `perfts` counter providers inside `PdhCollectQueryData`, and PDH counter +path strings whose count grows until every instance has been seen. A leak adds +allocations under the same stack in every snapshot. + +UMDH does not see memory a library reserves with `VirtualAlloc` directly. A +full dump shows it. `cdb -pv` attaches without changing the process and pauses +it only while it writes the dump: + +```powershell +cdb -pv -p -c ".dump /ma C:\dumps\exporter.dmp; q" +cdb -z C:\dumps\exporter.dmp -c "!address -summary; !heap -s; q" +``` + +If the debugger cannot download a PDB, fetch it with cURL from +`https://msdl.microsoft.com/download/symbols/.pdb//.pdb` into +`\.pdb\\.pdb` and add `` to the symbol path. +`` is the RSDS GUID of the module as 32 uppercase hex digits without +dashes, followed by its age in hex. + ## Dashboards, rules and runbooks Read [`contrib/mixin/README.md`](contrib/mixin/README.md) and