180 lines
7.7 KiB
Markdown
180 lines
7.7 KiB
Markdown
# Neural Hunt Service Controller
|
|
|
|
V4.3 adds a distributed worker-host control plane. The existing Customer Service remains the **Master** for customers, prepaid billing, worker leases, task assignment and reward delegation. A Service Controller is a small process that runs on each worker host, owns that host's Docker access and registers itself with the Master.
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
Game Server
|
|
https://play.example
|
|
^
|
|
|
|
|
Worker traffic
|
|
|
|
|
Customer Portal/Master |
|
|
customer-service :8090 |
|
|
admin :8091 |
|
|
internal :8092 |
|
|
| |
|
|
| authenticated private control
|
|
+------------+-------------------+
|
|
| |
|
|
Controller A Controller B
|
|
:8102 API :8102 API
|
|
:8101 UI :8101 UI
|
|
| |
|
|
docker.sock docker.sock
|
|
| |
|
|
Worker A1,A2... Worker B1,B2...
|
|
```
|
|
|
|
The public Customer Portal API does **not** change. Browser, CLI, game server and worker image remain compatible. Only worker runtime operations are routed through the controller assigned to that worker.
|
|
|
|
## Modes on Customer Service
|
|
|
|
`CS_WORKER_ORCHESTRATION_MODE` supports:
|
|
|
|
- `direct` — legacy mode; Customer Service talks to its local Docker Engine directly.
|
|
- `controller` — Customer Service does not need `docker.sock`; new workers must run on registered Service Controllers.
|
|
- `hybrid` — new workers prefer Service Controllers, while local Docker remains a fallback and preserves pre-controller workers.
|
|
|
|
For a safe upgrade from an existing installation, start with `hybrid`. Existing workers that already have a local container remain sticky to `local`; they are never silently moved because their named identity volume lives on that host. New workers are assigned to the least-loaded online controller.
|
|
|
|
## Master configuration
|
|
|
|
Generate a **new independent secret**:
|
|
|
|
```env
|
|
CS_WORKER_ORCHESTRATION_MODE=controller
|
|
SERVICE_CONTROLLER_SHARED_SECRET=<32+ random characters>
|
|
CS_CONTROLLER_OFFLINE_AFTER=45s
|
|
|
|
CS_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest
|
|
CS_WORKER_AUTO_PULL=true
|
|
|
|
# These two addresses must be usable by workers on remote hosts unless a
|
|
# controller-specific override is supplied.
|
|
CS_GAME_PUBLIC_URL=https://play.example.com
|
|
CS_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register
|
|
```
|
|
|
|
In `controller` mode the Customer Service container no longer needs `/var/run/docker.sock`. Keep `:8092` reachable only over the private/VPN network because it handles controller registration and worker leases.
|
|
|
|
## Controller configuration
|
|
|
|
Every worker host needs a stable unique `SC_ID`:
|
|
|
|
```env
|
|
SC_ID=worker-host-01
|
|
SC_NAME=Worker Host 01
|
|
|
|
SC_MASTER_URL=http://10.20.0.10:8092
|
|
SC_ADVERTISE_URL=http://10.20.0.21:8102
|
|
SERVICE_CONTROLLER_SHARED_SECRET=<same controller secret as Master>
|
|
|
|
SC_CONTROL_ADDR=:8102
|
|
SC_ADMIN_ADDR=:8101
|
|
SC_ADMIN_USER=admin
|
|
SC_ADMIN_PASSWORD=<strong unique password>
|
|
SC_ADMIN_COOKIE_SECURE=auto
|
|
|
|
SC_MAX_WORKERS=500
|
|
SC_MAX_RUNNING_WORKERS=100
|
|
SC_HEARTBEAT_INTERVAL=15s
|
|
|
|
# Host-specific Docker topology overrides. `bridge` is usually sufficient when
|
|
# the game is public HTTPS and the Master VPN IP is routed from Docker.
|
|
SC_WORKER_NETWORK=bridge
|
|
SC_WORKER_REGISTER_URL=http://10.20.0.10:8092/internal/workers/register
|
|
SC_GAME_PUBLIC_URL=https://play.example.com
|
|
|
|
SC_WORKER_IMAGE=git.example/neuralhunt/worker:worker_latest
|
|
SC_WORKER_REGISTRY_SERVER=git.example
|
|
SC_WORKER_REGISTRY_USERNAME=<read-only deploy token user>
|
|
SC_WORKER_REGISTRY_PASSWORD=<read-only deploy token>
|
|
|
|
DOCKER_HOST=unix:///var/run/docker.sock
|
|
```
|
|
|
|
`SC_WORKER_NETWORK`, `SC_WORKER_REGISTER_URL` and `SC_GAME_PUBLIC_URL` override the values sent by the Master only on that particular worker host. This is useful when remote Docker networks differ from the central host.
|
|
|
|
## Registration and placement
|
|
|
|
A controller posts a shared-secret-authenticated heartbeat to:
|
|
|
|
```text
|
|
POST /internal/controllers/register
|
|
```
|
|
|
|
The Master stores its ID, private control URL, capacities, reported worker counts and last heartbeat. A controller becomes unavailable for **new** assignments when:
|
|
|
|
- its heartbeat is older than `CS_CONTROLLER_OFFLINE_AFTER`,
|
|
- it is disabled by the Master admin,
|
|
- it is in `DRAIN` mode,
|
|
- or its worker inventory capacity is full.
|
|
|
|
Placement is sticky. After a worker is assigned to `worker-host-02`, all start/stop/image/identity/delete operations for that worker are sent to that same controller. The Master never silently creates an empty identity volume on another host.
|
|
|
|
## Customer Admin integration
|
|
|
|
The private Customer Service Admin now shows all registered controllers with:
|
|
|
|
- online/offline state,
|
|
- heartbeat,
|
|
- configured capacity,
|
|
- reported host counts,
|
|
- Master-assigned worker/running counts,
|
|
- `DRAIN` / `DRAIN AUFHEBEN`,
|
|
- `AKTIVIEREN` / `DEAKTIVIEREN`.
|
|
|
|
The existing per-worker **IMAGE UPDATE** and **IMAGE LADEN · ALLE WORKER AKTUALISIEREN** actions are distributed automatically. A bulk image update pulls the configured worker image once on every host involved and recreates workers on their existing identity host.
|
|
|
|
## Emergency UI
|
|
|
|
Each controller has a separate private emergency UI on `SC_ADMIN_ADDR` (default `:8101`). It intentionally exposes only containers labeled as Neural-Hunt-managed. It can:
|
|
|
|
- view local worker/container state,
|
|
- start a local container,
|
|
- stop a local container,
|
|
- restart a local container,
|
|
- stop all local Neural-Hunt workers,
|
|
- pull the configured worker image,
|
|
- see whether the last registration/heartbeat to the Master succeeded and the last error if it did not.
|
|
|
|
Use this UI only for host-level emergencies. The Master remains authoritative for billing, customer status and leases. If an emergency action starts a worker whose Master lease is revoked, the worker cannot resume hunting.
|
|
|
|
## Billing and failure behavior
|
|
|
|
Worker leases are still issued by Customer Service. V4.3 records `last_lease_at`, which gives the Master a controller-independent liveness signal. Billing therefore does not depend solely on a successful Docker API query to a remote host.
|
|
|
|
If a controller goes offline, existing workers are not automatically migrated because their private identity volume is host-local. The Master keeps the assignment and rejects destructive operations that would risk orphaning a volume. Use `hybrid` during migration of old local workers, or explicitly export/import an identity when moving an existing worker to another host.
|
|
|
|
## Security
|
|
|
|
- Never publish `:8102` to the Internet. It can create/start/stop Docker workloads.
|
|
- Never publish `:8101` publicly either; keep it behind VPN/private routing.
|
|
- Use a separate `SERVICE_CONTROLLER_SHARED_SECRET`; do not reuse JWT, Customer-Service or admin secrets.
|
|
- The controller gets Docker access; treat it as privileged infrastructure.
|
|
- Prefer a Docker Socket Proxy with only the required endpoints if practical.
|
|
- Registry credentials remain server-side. Normal Master operations send the Docker `X-Registry-Auth` value only to the authenticated controller; workers never receive registry credentials.
|
|
- `SC_ID` must remain stable for a host.
|
|
|
|
## Image build
|
|
|
|
A fourth independent pipeline image is now available:
|
|
|
|
```text
|
|
Dockerfile.server
|
|
Dockerfile.customer-service
|
|
Dockerfile.worker
|
|
Dockerfile.service-controller
|
|
```
|
|
|
|
Example registry tag:
|
|
|
|
```text
|
|
git.example/neuralhunt/service-controller:v4.3
|
|
```
|
|
|
|
The sample `docker-compose.service-controller.yml` is intended for a Linux worker host and uses host networking for the controller itself plus `docker.sock` for orchestration. Adapt the private/VPN routing to your environment.
|