48 lines
2.2 KiB
Markdown
48 lines
2.2 KiB
Markdown
# Jobs and cancellation
|
|
|
|
The gateway distinguishes between **model lifecycle operations**, **transient inference jobs**, and **durable batch jobs**.
|
|
|
|
## Model Stop
|
|
|
|
The Models page uses **Stop** to unload a resident Ollama model. Ollama does not expose `/api/stop`; the gateway sends:
|
|
|
|
```http
|
|
POST /api/generate
|
|
Content-Type: application/json
|
|
|
|
{"model":"<model>","keep_alive":0,"stream":false}
|
|
```
|
|
|
|
This asks Ollama to unload the model immediately. It does not cancel a specific request.
|
|
|
|
## Inference jobs
|
|
|
|
Every compute request receives an `X-Request-ID` and is registered as an in-memory job while it is queued, routing, running, or streaming.
|
|
|
|
Admin endpoints:
|
|
|
|
```text
|
|
GET /gateway/ui-api/jobs
|
|
POST /gateway/ui-api/jobs/<request-id>/cancel
|
|
```
|
|
|
|
The web UI exposes these under **Jobs** and also places an **Abbrechen** button next to active requests in Live Flow.
|
|
|
|
### Cancellation semantics
|
|
|
|
- **queued**: remove the ticket from the WFQ heap immediately and release the credit reservation;
|
|
- **waiting for worker**: cancel worker acquisition and release the scheduler lease;
|
|
- **running/streaming**: cancel the upstream HTTP context so the Ollama request terminates, then release worker/scheduler slots;
|
|
- reconcile any usage already observable from the response;
|
|
- store terminal live state `cancelled` with internal status `499`.
|
|
|
|
For a streaming response whose upstream HTTP headers were already forwarded, the downstream client may still have HTTP status `200`; cancellation is observable as an early stream termination. Internal gateway telemetry records the request as `499`.
|
|
|
|
Transient inference-job state is process-local and disappears on gateway restart, consistent with the in-memory hot-path design.
|
|
|
|
## Durable batch jobs
|
|
|
|
P3.1 adds a separate durable job type under `/gateway/v1/batches`. Batch definitions, state, input references and output references survive restart; an active execution attempt itself is cancelled on shutdown and converted back to a restart-safe queued state.
|
|
|
|
The Admin UI exposes durable work on the dedicated **Batch Jobs** page rather than mixing it into the transient **Jobs** page. See `docs/BATCH-JOBS.md` for API, state-machine, persistence, privacy, retention and accounting semantics.
|