66 lines
2.4 KiB
Markdown
66 lines
2.4 KiB
Markdown
# Warm Model Management
|
|
|
|
The warm-model manager controls Ollama model residency without putting model-management I/O on the inference hot path.
|
|
|
|
## Classes
|
|
|
|
- `hot`: keep the model resident on the requested number of eligible workers. A hot model is proactively preloaded.
|
|
- `warm`: optionally preload the model and unload it after `idle_timeout`.
|
|
- `cold`: do not preload and unload aggressively after `idle_timeout`.
|
|
|
|
Rules support exact names and one trailing `*` wildcard. The most specific matching rule wins.
|
|
|
|
```json
|
|
{
|
|
"warm_models": {
|
|
"enabled": true,
|
|
"reconcile_interval": "30s",
|
|
"operation_timeout": "2m",
|
|
"policies": {
|
|
"qwen3:8b": {
|
|
"class": "hot",
|
|
"workers": ["rtx-4090"],
|
|
"replicas": 1,
|
|
"preload": true,
|
|
"idle_timeout": "30m"
|
|
},
|
|
"gemma4:*": {
|
|
"class": "warm",
|
|
"replicas": 1,
|
|
"preload": false,
|
|
"idle_timeout": "20m"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Safety rules
|
|
|
|
Warm-model operations are subordinate to normal routing policy:
|
|
|
|
1. worker must be healthy and in `active` maintenance mode;
|
|
2. preload must be allowed by Model Placement;
|
|
3. when worker inventory is known, preload requires the model to be installed;
|
|
4. workers with an open/half-open circuit are not used for proactive preload;
|
|
5. a preload/unload reserves a per-worker/per-model maintenance slot before calling Ollama;
|
|
6. inference acquisition for the same worker/model is blocked during that short operation;
|
|
7. unload is refused while the model has active inference jobs.
|
|
|
|
The maintenance reservation closes the race where an idle unload could otherwise begin at the same moment a new request is routed to the model.
|
|
|
|
## Persistence
|
|
|
|
Runtime rules edited in **Admin → Warm Models** are stored in `warm-models.json`. The static config remains the baseline and can be restored with **Auf Config zurücksetzen**.
|
|
|
|
Action history is retained with the same state file. Last-use timestamps are deliberately runtime state; after restart, currently loaded models receive a fresh grace period before idle eviction.
|
|
|
|
## VRAM pressure
|
|
|
|
The manager does not automatically evict arbitrary models under memory pressure. It exposes eviction suggestions for inactive, non-hot models. This avoids surprise unloads and lets the operator decide whether to act.
|
|
|
|
## Metrics
|
|
|
|
- `ollama_gateway_warm_actions_running`
|
|
- `ollama_gateway_warm_eviction_suggestions`
|