cmd: disable console QuickEdit mode while the exporter runs (#2782)

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
This commit is contained in:
Jan-Otto Kröpke
2026-10-18 19:12:11 +00:00
committed by GitHub
parent 7cc97de7f8
commit 3e8a2c6fe7
3 changed files with 144 additions and 0 deletions
+55
View File
@@ -354,6 +354,61 @@ If the debugger cannot download a PDB, fetch it with cURL from
`<id>` is the RSDS GUID of the module as 32 uppercase hex digits without
dashes, followed by its age in hex.
## Console mode and hangs
The Windows console pauses all output of the attached processes while a text
selection is active. In a conhost window (cmd.exe, PowerShell, RDP sessions)
QuickEdit mode starts one on a click into the window, mark mode (`Ctrl+M` or
the window menu) on request; Windows Terminal selects without pausing. A
paused console blocks `WriteConsole`, so a `slog` handler writing to stdout or
stderr blocks while holding its mutex, and every scrape that logs queues
behind it: `/metrics` stops answering while the process stays alive. Ctrl+C
copies the selection instead of sending the break signal, so it looks like a
crash without a log line (#2024). `--log.level=debug` logs on every scrape and
hits this within minutes; the default level needs a warning or error first.
`cmd/windows_exporter/console.go` turns QuickEdit off while the exporter logs
to a console and restores the previous mode when `run` returns. Rules for that
code, from the conhost source (`src/host/getset.cpp`):
- The flags live on the console input buffer. Open it as `CONIN$` instead of
using stdin, which may be redirected while the output still is a console.
- `ENABLE_EXTENDED_FLAGS` must be set in the mode passed to `SetConsoleMode`
to change `ENABLE_QUICK_EDIT_MODE`, `ENABLE_INSERT_MODE` or
`ENABLE_AUTO_POSITION`; a mode without any of these four bits leaves them
alone and only stops `GetConsoleMode` from reporting them.
- conhost sets `CONSOLE_USE_PRIVATE_FLAGS` when it creates the console
(`src/host/output.cpp`), so `GetConsoleMode` reports those four bits with
their real state from the start. Restore the mode exactly as read; do not
guess defaults. The mode is shared with the shell that started the exporter,
so restore it on every exit path after flag parsing (`--help`, usage errors
and startup failures included). A hard kill skips the restore.
- Run it only when the log destination is stdout or stderr. A service has no
console, and a file or the event log is not affected by selections.
When `/metrics` stops answering while the process stays alive, take
`/debug/pprof/goroutine?debug=2` (needs `--debug.enabled`), twice a few seconds
apart. One goroutine in `syscall.WriteConsole` under `log/slog` with the
collectors waiting on a mutex is the paused console above. Without goroutines
in Go code, or when pprof doesn't answer either, ask for a full process dump
and the loaded module list (`tasklist /m`); EDR hooks have caused such hangs
(#2024). A hang of the service cannot be QuickEdit.
## Service startup
The service control manager gives a service 30 seconds by default (45 in some
reports) to connect, and a host busy after a Windows update can spend that on
loading the executable alone (#551). `cmd/windows_exporter/0_service.go`
therefore calls `svc.Run` from the package-level `IsService` declaration, which
runs before every `init` function and `main`; the `0_` prefix keeps it the first
file of the package, and the file must not import packages with slow
initialization. `Execute` reports `StartPending` and `Running` at once and
leaves the actual startup to `run`, so a slow collector `Build` cannot time
out the service start. The MSI adds delayed automatic start, a dependency on
`wmiApSrv` and restart on failure after 60 seconds (`installer/files.wxs`).
A startup failure before the logger exists goes to the event log and to
`%TEMP%` through `logStartupError`.
## Dashboards, rules and runbooks
Read [`contrib/mixin/README.md`](contrib/mixin/README.md) and
+76
View File
@@ -0,0 +1,76 @@
// SPDX-License-Identifier: Apache-2.0
//
// Copyright The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build windows
package main
import (
"golang.org/x/sys/windows"
)
// disableConsoleQuickEdit turns off the QuickEdit mode of the console the
// process is attached to. It returns whether the mode was changed and a
// function that restores the previous input mode. Without an attached console,
// for example as a service, it changes nothing and the restore is a no-op.
//
// With QuickEdit on, a click into a conhost window starts a text selection and
// the console stops accepting output until the selection ends. A log write
// then blocks in WriteConsole while holding the slog handler lock, so every
// scrape that logs waits behind it and /metrics stops answering. Ctrl+C copies
// the selection instead of interrupting the process, which is why such a hang
// looks like a crash without a log line (#2024).
//
// The console input buffer is opened as CONIN$, because the hang comes from
// writing to the console and happens with a redirected stdin as well.
func disableConsoleQuickEdit() (bool, func()) {
noop := func() {}
conin, err := windows.CreateFile(
windows.StringToUTF16Ptr("CONIN$"),
windows.GENERIC_READ|windows.GENERIC_WRITE,
windows.FILE_SHARE_READ|windows.FILE_SHARE_WRITE,
nil,
windows.OPEN_EXISTING,
0,
0,
)
if err != nil {
return false, noop
}
var mode uint32
if err := windows.GetConsoleMode(conin, &mode); err != nil {
_ = windows.CloseHandle(conin)
return false, noop
}
// ENABLE_EXTENDED_FLAGS is required to change ENABLE_QUICK_EDIT_MODE. conhost
// reports it, together with the QuickEdit and insert mode flags, from the
// creation of the console on, so the mode as read is the complete state.
newMode := (mode | windows.ENABLE_EXTENDED_FLAGS) &^ windows.ENABLE_QUICK_EDIT_MODE
if err := windows.SetConsoleMode(conin, newMode); err != nil {
_ = windows.CloseHandle(conin)
return false, noop
}
return true, func() {
_ = windows.SetConsoleMode(conin, mode)
_ = windows.CloseHandle(conin)
}
}
+13
View File
@@ -216,6 +216,19 @@ func run(ctx context.Context, args []string) int {
logger.LogAttrs(ctx, slog.LevelDebug, "logging has Started")
// A console that pauses its output, for example for a QuickEdit text
// selection, blocks the logger and with it every scrape. The console mode
// is shared with the shell that started the exporter, so the previous mode
// is restored when run returns.
if dest := logFile.String(); dest == "stdout" || dest == "stderr" {
disabled, restoreConsole := disableConsoleQuickEdit()
defer restoreConsole()
if disabled {
logger.LogAttrs(ctx, slog.LevelDebug, "disabled the QuickEdit mode of the console")
}
}
if configFile != nil && *configFile != "" {
logger.LogAttrs(ctx, slog.LevelInfo, "using configuration file",
slog.String("file", *configFile),