Warm Model Management

The warm-model manager controls Ollama model residency without putting model-management I/O on the inference hot path.

Classes

hot: keep the model resident on the requested number of eligible workers. A hot model is proactively preloaded.
warm: optionally preload the model and unload it after idle_timeout.
cold: do not preload and unload aggressively after idle_timeout.

Rules support exact names and one trailing * wildcard. The most specific matching rule wins.

{
  "warm_models": {
    "enabled": true,
    "reconcile_interval": "30s",
    "operation_timeout": "2m",
    "policies": {
      "qwen3:8b": {
        "class": "hot",
        "workers": ["rtx-4090"],
        "replicas": 1,
        "preload": true,
        "idle_timeout": "30m"
      },
      "gemma4:*": {
        "class": "warm",
        "replicas": 1,
        "preload": false,
        "idle_timeout": "20m"
      }
    }
  }
}

Safety rules

Warm-model operations are subordinate to normal routing policy:

worker must be healthy and in active maintenance mode;
preload must be allowed by Model Placement;
when worker inventory is known, preload requires the model to be installed;
workers with an open/half-open circuit are not used for proactive preload;
a preload/unload reserves a per-worker/per-model maintenance slot before calling Ollama;
inference acquisition for the same worker/model is blocked during that short operation;
unload is refused while the model has active inference jobs.

The maintenance reservation closes the race where an idle unload could otherwise begin at the same moment a new request is routed to the model.

Persistence

Runtime rules edited in Admin → Warm Models are stored in warm-models.json. The static config remains the baseline and can be restored with Auf Config zurücksetzen.

Action history is retained with the same state file. Last-use timestamps are deliberately runtime state; after restart, currently loaded models receive a fresh grace period before idle eviction.

VRAM pressure

The manager does not automatically evict arbitrary models under memory pressure. It exposes eviction suggestions for inactive, non-hot models. This avoids surprise unloads and lets the operator decide whether to act.

Metrics

ollama_gateway_warm_actions_running
ollama_gateway_warm_eviction_suggestions

2.4 KiB Raw Permalink Blame History