og/docs/JOBS.md

Jobs and cancellation

The gateway distinguishes between model lifecycle operations, transient inference jobs, and durable batch jobs.

Model Stop

The Models page uses Stop to unload a resident Ollama model. Ollama does not expose /api/stop; the gateway sends:

POST /api/generate
Content-Type: application/json

{"model":"<model>","keep_alive":0,"stream":false}

This asks Ollama to unload the model immediately. It does not cancel a specific request.

Inference jobs

Every compute request receives an X-Request-ID and is registered as an in-memory job while it is queued, routing, running, or streaming.

Admin endpoints:

GET  /gateway/ui-api/jobs
POST /gateway/ui-api/jobs/<request-id>/cancel

The web UI exposes these under Jobs and also places an Abbrechen button next to active requests in Live Flow.

Cancellation semantics

• queued: remove the ticket from the WFQ heap immediately and release the credit reservation;
• waiting for worker: cancel worker acquisition and release the scheduler lease;
• running/streaming: cancel the upstream HTTP context so the Ollama request terminates, then release worker/scheduler slots;
• reconcile any usage already observable from the response;
• store terminal live state cancelled with internal status 499.

For a streaming response whose upstream HTTP headers were already forwarded, the downstream client may still have HTTP status 200; cancellation is observable as an early stream termination. Internal gateway telemetry records the request as 499.

Transient inference-job state is process-local and disappears on gateway restart, consistent with the in-memory hot-path design.

Durable batch jobs

P3.1 adds a separate durable job type under /gateway/v1/batches. Batch definitions, state, input references and output references survive restart; an active execution attempt itself is cancelled on shutdown and converted back to a restart-safe queued state.

The Admin UI exposes durable work on the dedicated Batch Jobs page rather than mixing it into the transient Jobs page. See docs/BATCH-JOBS.md for API, state-machine, persistence, privacy, retention and accounting semantics.