neural-hunt/SERVICE_CONTROLLER.md

Neural Hunt Service Controller

V4.3 adds a distributed worker-host control plane. The existing Customer Service remains the Master for customers, prepaid billing, worker leases, task assignment and reward delegation. A Service Controller is a small process that runs on each worker host, owns that host's Docker access and registers itself with the Master.

Architecture

                         Game Server
                     https://play.example
                              ^
                              |
                         Worker traffic
                              |
Customer Portal/Master        |
customer-service :8090        |
admin :8091                   |
internal :8092                |
       |                      |
       | authenticated private control
       +------------+-------------------+
                    |                   |
            Controller A           Controller B
             :8102 API              :8102 API
             :8101 UI               :8101 UI
                |                       |
           docker.sock              docker.sock
                |                       |
          Worker A1,A2...          Worker B1,B2...

The public Customer Portal API does not change. Browser, CLI, game server and worker image remain compatible. Only worker runtime operations are routed through the controller assigned to that worker.

Modes on Customer Service

CS_WORKER_ORCHESTRATION_MODE supports:

• direct — legacy mode; Customer Service talks to its local Docker Engine directly.
• controller — Customer Service does not need docker.sock; new workers must run on registered Service Controllers.
• hybrid — new workers prefer Service Controllers, while local Docker remains a fallback and preserves pre-controller workers.

For a safe upgrade from an existing installation, start with hybrid. Existing workers that already have a local container remain sticky to local; they are never silently moved because their named identity volume lives on that host. New workers are assigned to the least-loaded online controller.

Master configuration

Generate a new independent secret:

CS_WORKER_ORCHESTRATION_MODE=controller
SERVICE_CONTROLLER_SHARED_SECRET=<32+ random characters>
CS_CONTROLLER_OFFLINE_AFTER=45s

CS_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest
CS_WORKER_AUTO_PULL=true

# These two addresses must be usable by workers on remote hosts unless a
# controller-specific override is supplied.
CS_GAME_PUBLIC_URL=https://play.example.com
CS_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register

In controller mode the Customer Service container no longer needs /var/run/docker.sock. Keep :8092 reachable only over the private/VPN network because it handles controller registration and worker leases.

Controller configuration

Every worker host needs a stable unique SC_ID:

SC_ID=worker-host-01
SC_NAME=Worker Host 01

SC_MASTER_URL=http://10.20.0.10:8092
SC_ADVERTISE_URL=http://10.20.0.21:8102
SERVICE_CONTROLLER_SHARED_SECRET=<same controller secret as Master>

SC_CONTROL_ADDR=:8102
SC_ADMIN_ADDR=:8101
SC_ADMIN_USER=admin
SC_ADMIN_PASSWORD=<strong unique password>
SC_ADMIN_COOKIE_SECURE=auto

SC_MAX_WORKERS=500
SC_MAX_RUNNING_WORKERS=100
SC_HEARTBEAT_INTERVAL=15s

# Host-specific Docker topology overrides. `bridge` is usually sufficient when
# the game is public HTTPS and the Master VPN IP is routed from Docker.
SC_WORKER_NETWORK=bridge
SC_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register
SC_GAME_PUBLIC_URL=https://play.example.com

SC_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest
SC_WORKER_REGISTRY_SERVER=git.example
SC_WORKER_REGISTRY_USERNAME=<read-only deploy token user>
SC_WORKER_REGISTRY_PASSWORD=<read-only deploy token>

DOCKER_HOST=unix:///var/run/docker.sock

SC_WORKER_NETWORK, SC_WORKER_REGISTER_URL and SC_GAME_PUBLIC_URL override the values sent by the Master only on that particular worker host. This is useful when remote Docker networks differ from the central host.

Registration and placement

A controller posts a shared-secret-authenticated heartbeat to:

POST /internal/controllers/register

The Master stores its ID, private control URL, capacities, reported worker counts and last heartbeat. A controller becomes unavailable for new assignments when:

• its heartbeat is older than CS_CONTROLLER_OFFLINE_AFTER,
• it is disabled by the Master admin,
• it is in DRAIN mode,
• or its worker inventory capacity is full.

Placement is sticky. After a worker is assigned to worker-host-02, all start/stop/image/identity/delete operations for that worker are sent to that same controller. The Master never silently creates an empty identity volume on another host.

Customer Admin integration

The private Customer Service Admin now shows all registered controllers with:

• online/offline state,
• heartbeat,
• configured capacity,
• reported host counts,
• Master-assigned worker/running counts,
• DRAIN / DRAIN AUFHEBEN,
• AKTIVIEREN / DEAKTIVIEREN.

The existing per-worker IMAGE UPDATE and IMAGE LADEN · ALLE WORKER AKTUALISIEREN actions are distributed automatically. A bulk image update pulls the configured worker image once on every host involved and recreates workers on their existing identity host.

Emergency UI

Each controller has a separate private emergency UI on SC_ADMIN_ADDR (default :8101). It intentionally exposes only containers labeled as Neural-Hunt-managed. It can:

• view local worker/container state,
• start a local container,
• stop a local container,
• restart a local container,
• stop all local Neural-Hunt workers,
• pull the configured worker image,
• see whether the last registration/heartbeat to the Master succeeded and the last error if it did not.

Use this UI only for host-level emergencies. The Master remains authoritative for billing, customer status and leases. If an emergency action starts a worker whose Master lease is revoked, the worker cannot resume hunting.

Billing and failure behavior

Worker leases are still issued by Customer Service. V4.3 records last_lease_at, which gives the Master a controller-independent liveness signal. Billing therefore does not depend solely on a successful Docker API query to a remote host.

If a controller goes offline, existing workers are not automatically migrated because their private identity volume is host-local. The Master keeps the assignment and rejects destructive operations that would risk orphaning a volume. Use hybrid during migration of old local workers, or explicitly export/import an identity when moving an existing worker to another host.

Security

• Never publish :8102 to the Internet. It can create/start/stop Docker workloads.
• Never publish :8101 publicly either; keep it behind VPN/private routing.
• Use a separate SERVICE_CONTROLLER_SHARED_SECRET; do not reuse JWT, Customer-Service or admin secrets.
• The controller gets Docker access; treat it as privileged infrastructure.
• Prefer a Docker Socket Proxy with only the required endpoints if practical.
• Registry credentials remain server-side. Normal Master operations send the Docker X-Registry-Auth value only to the authenticated controller; workers never receive registry credentials.
• SC_ID must remain stable for a host.

Image build

A fourth independent pipeline image is now available:

Dockerfile.server
Dockerfile.customer-service
Dockerfile.worker
Dockerfile.service-controller

Example registry tag:

git.example/neuralhunt/service-controller:v4.3

The sample docker-compose.service-controller.yml is intended for a Linux worker host and uses host networking for the controller itself plus docker.sock for orchestration. Adapt the private/VPN routing to your environment.