# Neural Hunt Service Controller V4.3 adds a distributed worker-host control plane. The existing Customer Service remains the **Master** for customers, prepaid billing, worker leases, task assignment and reward delegation. A Service Controller is a small process that runs on each worker host, owns that host's Docker access and registers itself with the Master. ## Architecture ```text Game Server https://play.example ^ | Worker traffic | Customer Portal/Master | customer-service :8090 | admin :8091 | internal :8092 | | | | authenticated private control +------------+-------------------+ | | Controller A Controller B :8102 API :8102 API :8101 UI :8101 UI | | docker.sock docker.sock | | Worker A1,A2... Worker B1,B2... ``` The public Customer Portal API does **not** change. Browser, CLI, game server and worker image remain compatible. Only worker runtime operations are routed through the controller assigned to that worker. ## Modes on Customer Service `CS_WORKER_ORCHESTRATION_MODE` supports: - `direct` โ€” legacy mode; Customer Service talks to its local Docker Engine directly. - `controller` โ€” Customer Service does not need `docker.sock`; new workers must run on registered Service Controllers. - `hybrid` โ€” new workers prefer Service Controllers, while local Docker remains a fallback and preserves pre-controller workers. For a safe upgrade from an existing installation, start with `hybrid`. Existing workers that already have a local container remain sticky to `local`; they are never silently moved because their named identity volume lives on that host. New workers are assigned to the least-loaded online controller. ## Master configuration Generate a **new independent secret**: ```env CS_WORKER_ORCHESTRATION_MODE=controller SERVICE_CONTROLLER_SHARED_SECRET=<32+ random characters> CS_CONTROLLER_OFFLINE_AFTER=45s CS_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest CS_WORKER_AUTO_PULL=true # These two addresses must be usable by workers on remote hosts unless a # controller-specific override is supplied. CS_GAME_PUBLIC_URL=https://play.example.com CS_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register ``` In `controller` mode the Customer Service container no longer needs `/var/run/docker.sock`. Keep `:8092` reachable only over the private/VPN network because it handles controller registration and worker leases. ## Controller configuration Every worker host needs a stable unique `SC_ID`: ```env SC_ID=worker-host-01 SC_NAME=Worker Host 01 SC_MASTER_URL=http://10.20.0.10:8092 SC_ADVERTISE_URL=http://10.20.0.21:8102 SERVICE_CONTROLLER_SHARED_SECRET= SC_CONTROL_ADDR=:8102 SC_ADMIN_ADDR=:8101 SC_ADMIN_USER=admin SC_ADMIN_PASSWORD= SC_ADMIN_COOKIE_SECURE=auto SC_MAX_WORKERS=500 SC_MAX_RUNNING_WORKERS=100 SC_HEARTBEAT_INTERVAL=15s # Host-specific Docker topology overrides. `bridge` is usually sufficient when # the game is public HTTPS and the Master VPN IP is routed from Docker. SC_WORKER_NETWORK=bridge SC_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register SC_GAME_PUBLIC_URL=https://play.example.com SC_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest SC_WORKER_REGISTRY_SERVER=git.example SC_WORKER_REGISTRY_USERNAME= SC_WORKER_REGISTRY_PASSWORD= DOCKER_HOST=unix:///var/run/docker.sock ``` `SC_WORKER_NETWORK`, `SC_WORKER_REGISTER_URL` and `SC_GAME_PUBLIC_URL` override the values sent by the Master only on that particular worker host. This is useful when remote Docker networks differ from the central host. ## Registration and placement A controller posts a shared-secret-authenticated heartbeat to: ```text POST /internal/controllers/register ``` The Master stores its ID, private control URL, capacities, reported worker counts and last heartbeat. A controller becomes unavailable for **new** assignments when: - its heartbeat is older than `CS_CONTROLLER_OFFLINE_AFTER`, - it is disabled by the Master admin, - it is in `DRAIN` mode, - or its worker inventory capacity is full. Placement is sticky. After a worker is assigned to `worker-host-02`, all start/stop/image/identity/delete operations for that worker are sent to that same controller. The Master never silently creates an empty identity volume on another host. ## Customer Admin integration The private Customer Service Admin now shows all registered controllers with: - online/offline state, - heartbeat, - configured capacity, - reported host counts, - Master-assigned worker/running counts, - `DRAIN` / `DRAIN AUFHEBEN`, - `AKTIVIEREN` / `DEAKTIVIEREN`. The existing per-worker **IMAGE UPDATE** and **IMAGE LADEN ยท ALLE WORKER AKTUALISIEREN** actions are distributed automatically. A bulk image update pulls the configured worker image once on every host involved and recreates workers on their existing identity host. ## Emergency UI Each controller has a separate private emergency UI on `SC_ADMIN_ADDR` (default `:8101`). It intentionally exposes only containers labeled as Neural-Hunt-managed. It can: - view local worker/container state, - start a local container, - stop a local container, - restart a local container, - stop all local Neural-Hunt workers, - pull the configured worker image, - see whether the last registration/heartbeat to the Master succeeded and the last error if it did not. Use this UI only for host-level emergencies. The Master remains authoritative for billing, customer status and leases. If an emergency action starts a worker whose Master lease is revoked, the worker cannot resume hunting. ## Billing and failure behavior Worker leases are still issued by Customer Service. V4.3 records `last_lease_at`, which gives the Master a controller-independent liveness signal. Billing therefore does not depend solely on a successful Docker API query to a remote host. If a controller goes offline, existing workers are not automatically migrated because their private identity volume is host-local. The Master keeps the assignment and rejects destructive operations that would risk orphaning a volume. Use `hybrid` during migration of old local workers, or explicitly export/import an identity when moving an existing worker to another host. ## Security - Never publish `:8102` to the Internet. It can create/start/stop Docker workloads. - Never publish `:8101` publicly either; keep it behind VPN/private routing. - Use a separate `SERVICE_CONTROLLER_SHARED_SECRET`; do not reuse JWT, Customer-Service or admin secrets. - The controller gets Docker access; treat it as privileged infrastructure. - Prefer a Docker Socket Proxy with only the required endpoints if practical. - Registry credentials remain server-side. Normal Master operations send the Docker `X-Registry-Auth` value only to the authenticated controller; workers never receive registry credentials. - `SC_ID` must remain stable for a host. ## Image build A fourth independent pipeline image is now available: ```text Dockerfile.server Dockerfile.customer-service Dockerfile.worker Dockerfile.service-controller ``` Example registry tag: ```text git.example/neuralhunt/service-controller:v4.3 ``` The sample `docker-compose.service-controller.yml` is intended for a Linux worker host and uses host networking for the controller itself plus `docker.sock` for orchestration. Adapt the private/VPN routing to your environment.