Model Placement

Model Placement is the gateway's hard model-to-worker routing policy. It is evaluated before adaptive worker scoring, so GPU load, VRAM pressure, model affinity or learned throughput can never override a placement prohibition.

Why it is separate from tenant policies

Tenant/Fairness policies answer:

How much compute may a tenant or actor consume?

Model Placement answers:

On which Ollama workers may a model execute?

Keeping those concerns separate makes both the configuration and the UI predictable.

UI workflow

Open Admin -> Model Placement.

The matrix has one row per discovered model and one column per worker:

✓ installed — allowed and installed on this worker.
○ allowed — policy allows it, but the model is not currently installed there.
● loaded — allowed, installed and currently resident according to /api/ps.
⛔ blocked — the effective placement rule denies the model.
↳ — the cell is resolved by an exact model rule.

Click a cell to create an exact allow/deny exception. Clicking an exact UI exception again removes that exact rule so the broader prefix/default rule becomes effective again.

The worker cards open a ruleset editor with:

Allow all: allow by default and optionally deny selected models/prefixes.
Whitelist: deny by default and only allow selected models/prefixes.
Only installed preset: builds a whitelist from the worker's current /api/tags inventory.
Block all preset: an empty whitelist.
Reset to config default: deletes the persistent UI override and immediately restores the worker's bootstrap/persistent-config baseline.

All UI changes are written to storage.model_placement_file (default model-placement.json) using the gateway's atomic state writer and take effect for new requests immediately.

Pattern semantics

Patterns support:

qwen3:8b       exact model
gemma4:*       prefix wildcard
orcarouter/*   prefix wildcard
*              all models

Wildcards are only supported as a single trailing *.

Resolution uses specificity:

exact model rule;
longest matching prefix rule;
worker mode (allow_all or whitelist).

At equal specificity, deny wins.

This means the following is valid and useful:

{
  "mode": "allow_all",
  "allowed_models": ["gemma4:latest"],
  "denied_models": ["gemma4:*"]
}

gemma4:latest is allowed because the exact rule is more specific, while other gemma4:* variants are blocked.

Example: model A on node 1, model B on node 1 + 2

"workers": [
  {
    "name": "node-1",
    "url": "http://10.10.11.10:11434",
    "max_concurrent": 4,
    "model_placement": {
      "mode": "whitelist",
      "allowed_models": ["model-a:*", "model-b:*"],
      "denied_models": []
    }
  },
  {
    "name": "node-2",
    "url": "http://10.10.11.20:11434",
    "max_concurrent": 2,
    "model_placement": {
      "mode": "whitelist",
      "allowed_models": ["model-b:*"],
      "denied_models": []
    }
  }
]

The adaptive router then chooses between node 1 and node 2 for model-b:*, but model-a:* can only run on node 1.

Inventory behavior

Each worker's /api/tags inventory is tracked separately from /api/ps loaded state.

Routing behavior is:

remove unhealthy workers;
remove workers blocked by Model Placement;
if inventory is known, require the model to be installed;
enforce global and per-model concurrency;
score the remaining workers by loaded affinity, load, learned tok/s, GPU utilization and VRAM pressure.

If at least one eligible worker has a known installed copy, only those installed workers are considered. If all eligible inventories are known and none contains the model, the gateway returns a descriptive model ... is not installed on any eligible worker error instead of sending the request to an arbitrary node.

A worker whose inventory has never been successfully retrieved is treated as inventory unknown, not as "model absent". That provides a controlled fail-open path for transient /api/tags discovery failures while still respecting hard placement rules.

Client model discovery

/api/tags and /v1/models are gateway-native aggregated endpoints. Models that are installed only on placement-blocked workers are excluded from client discovery. This keeps OpenWebUI and other clients aligned with the gateway's actual routing policy.

The admin model inventory remains unfiltered so operators can still see and manage models that are installed but deliberately blocked from inference.

Persistence precedence

For a worker:

persistent UI placement override
        ↓ (if absent)
config.json / persistent config baseline
        ↓
effective routing rule

Deleting the UI override restores the baseline immediately. The placement file is included in gateway ZIP backups and is shown on Admin -> Persistenz.

4.8 KiB Raw Blame History