neural-hunt/HOSTED_SERVICE.md

# Neural Hunt Hosted Service (optional)

V4.1 splits the deployment into three dedicated Docker images. V4.0 added an optional commercial control plane without changing the normal game
server. The default `docker compose up` still starts only `app`. For a local source build,
build all three images first and then start the hosted layer:

```bash
docker compose --profile images build app customer-service worker-image
docker compose --profile hosted up -d
```

With Docker Buildx the three images can also be built/tagged in one operation:

```bash
docker build -f Dockerfile.server -t neuralhunt-server:local .
docker build -f Dockerfile.customer-service -t neuralhunt-customer-service:local .
docker build -f Dockerfile.worker -t neuralhunt-worker:local .
```


## Three Docker images

V4.1 no longer ships one monolithic runtime image. `Dockerfile` has three final
targets:

- `server` -> `neuralhunt-server:local` (game/API + private game admin)
- `customer-service` -> `neuralhunt-customer-service:local` (portal/billing/control plane)
- `worker` -> `neuralhunt-worker:local` (CLI hunting agent only)

The tags are configurable through `.env` / the build environment:

```env
NEURALHUNT_SERVER_IMAGE=registry.example.com/neuralhunt/server:v4.1
NEURALHUNT_CUSTOMER_IMAGE=registry.example.com/neuralhunt/customer-service:v4.1
CS_WORKER_IMAGE=registry.example.com/neuralhunt/worker:v4.1
```

`CS_WORKER_IMAGE` is both the worker build tag and the exact image reference the
Customer Service passes to Docker Engine. With `CS_WORKER_AUTO_PULL=true` (the
default), a missing worker image is pulled lazily when a customer first needs a
worker. For a private registry, configure a read-only deploy/robot credential:

```env
CS_WORKER_REGISTRY_SERVER=registry.example.com
CS_WORKER_REGISTRY_USERNAME=neuralhunt-worker-pull
CS_WORKER_REGISTRY_PASSWORD=<read-only-token>
```

Those credentials are converted to Docker's `X-Registry-Auth` header only for
the image-pull request and are never passed to managed worker containers. Set
`CS_WORKER_AUTO_PULL=false` if production policy requires images to be
pre-pulled instead.

The dedicated worker image already declares `/app/neuralhunt-client` as its
ENTRYPOINT. `CS_WORKER_ENTRYPOINT` should therefore stay empty. It exists only
as a compatibility override for older monolithic images.

For registry publishing with Buildx:

```bash
export NEURALHUNT_SERVER_IMAGE=registry.example.com/neuralhunt/server:v4.1
export NEURALHUNT_CUSTOMER_IMAGE=registry.example.com/neuralhunt/customer-service:v4.1
export CS_WORKER_IMAGE=registry.example.com/neuralhunt/worker:v4.1
# CI/CD: Dockerfile.server, Dockerfile.customer-service und Dockerfile.worker getrennt bauen/pushen.
```

## Ports and trust zones

- `app:8080` — public Neural Hunt game
- `app:8081` — private game admin + private Customer-Service delegation API
- `customer-service:8090` — customer portal (put behind HTTPS)
- `customer-service:8091` — private/VPN Customer-Service admin
- `customer-service:8092` — Docker-network-only worker registration

Do not route 8081, 8091 or 8092 to the public Internet.

### Local `go run` test

When both processes run directly on the same development machine, use:

```env
CS_GAME_PUBLIC_URL=http://127.0.0.1:8080
CS_GAME_ADMIN_URL=http://127.0.0.1:8081
```

The Docker hostname `app` only exists inside the Compose network. `docker-compose.yml` overrides the two values to `http://app:8080` and `http://app:8081` automatically. The Customer Service now probes the authenticated game control plane at startup and prints a concrete warning if the URL points to the public listener, DNS is wrong, or `CUSTOMER_SERVICE_SHARED_SECRET` differs.

## PrePaid billing model

The reference implementation bills **running worker time**, not individual
submitted guesses. This makes the commercial unit explicit and predictable, but
it is only a billing design; it does not by itself determine the legal/payment-
provider classification of a paid prize/chance product. `CS_WORKER_CREDITS_PER_MINUTE=1.0` means one paid minute of
one worker consumes one credit. The next minute is debited atomically before it
is allowed to continue; when there is insufficient balance the service stops
the worker. A 2-second reconciliation loop also catches containers that stop or
exit outside the portal. If Docker fails before a newly purchased start minute
actually begins, that start debit is automatically refunded to the ledger.

Credit packages are configured server-side:

```env
CS_CREDIT_PACKAGES=starter:499:EUR:60;plus:1999:EUR:300;power:4999:EUR:900
```

The four fields are `id:price_cents:CURRENCY:credits`. These are examples, not a
pricing recommendation.

## PayPal

PayPal is disabled by default and should first be configured in sandbox:

```env
PAYPAL_ENABLED=true
PAYPAL_ENVIRONMENT=sandbox
PAYPAL_CLIENT_ID=...
PAYPAL_CLIENT_SECRET=...
PAYPAL_WEBHOOK_ID=...
```

The browser never receives the PayPal secret. Customer Service creates Orders
server-side, redirects the user to the returned approval URL, captures the
Order server-side after return and reconciles the captured amount/currency
against the package stored before crediting the ledger. Webhooks are verified
through PayPal's webhook-signature verification API. Ledger references make a
completed Order idempotent.

Live PayPal is additionally locked until:

```env
PAYPAL_LIVE_APPROVAL_ACK=I_HAVE_PAYPAL_APPROVAL
```

is set. This is intentional: products involving payment plus prizes/chance can
require provider approval and legal review. Do not use that flag as a substitute
for the approval/review itself.

## Manual test/support credits

Manual top-ups are a separate protected bypass and are **off by default**:

```env
CS_ALLOW_MANUAL_CREDITS=1
```

They are available only from the private Customer-Service admin listener on
8091 after its own HttpOnly admin session. Each grant is written to the same
immutable-style credit ledger with a unique `manual:*` reference. Keep the flag
off on a normal public production deployment unless support operations require
it.

## Multiple workers and reward ownership

Each managed worker gets:

- its own P-256 Neural Hunt identity in a dedicated Docker named volume;
- its own normal one-identity/one-presence lease;
- its own task assignment and Beacon path (`auto`, `PULSE`, `FLUX`, `ORBIT`);
- an authenticated private registration token known only to Customer Service.

The customer configures one **main reward identity** in the portal, but the
portal does not trust a pasted public Client-ID. While logged into the game with
the intended owner identity, the customer requests a **HOSTED CODE** (or runs
`hosted-code` in the CLI). The game stores only the SHA-256 hash of this random
one-time code; it expires after 10 minutes. Customer Service redeems the code
over the private game control plane and receives the proven Client-ID. The
P-256 private key never leaves the browser/CLI.

When a worker starts, it registers its own Client-ID with Customer Service,
which installs a delegation on the private game control plane:

```text
worker identity -> main reward identity
```

A winning task records both values:

- `winner_client_id` = durable reward owner
- `winner_worker_client_id` = cryptographic identity that actually submitted the winning guess

Thus several workers can run concurrently without disabling the game's normal
single-presence rule. The original NFT is retrieved later by logging into the
browser/CLI with the main reward identity.

Worker raw identity files can be downloaded or replaced from the portal. They
contain a private key and must be treated as secrets. Replacing an identity
stops the worker and forces private delegation registration again on next start.
Deleting a worker also deletes its identity volume, so back it up first if it
must remain portable.

## Worker containment

Managed worker containers receive no Docker socket and no published ports. The
reference Docker Engine request additionally uses:

- read-only root filesystem
- all Linux capabilities dropped
- `no-new-privileges`
- 256 MiB memory limit
- 1 CPU limit
- PID limit 128
- only the private identity volume writable

The Customer Service itself needs Docker Engine control. Directly mounting
`/var/run/docker.sock` effectively gives this service host-level Docker control.
The provided Compose setup supports it for a simple self-hosted deployment, but
a production setup should preferably put a narrowly permissioned Docker Socket
Proxy in front and set `DOCKER_HOST` to that proxy instead.

Capacity is also bounded independently of reverse-proxy rate limiting. Stopped worker records do not allocate a Docker volume until the worker is actually started (or an identity is explicitly uploaded):

```env
CS_MAX_WORKERS_PER_CUSTOMER=20
CS_MAX_WORKERS_GLOBAL=1000
CS_MAX_RUNNING_WORKERS_PER_CUSTOMER=10
CS_MAX_RUNNING_WORKERS_GLOBAL=100
```

## Optional Beacon Hunt

Enable from the game Admin Runtime settings or defaults:

```env
DEFAULT_BEACON_HUNT_ENABLED=1
DEFAULT_BEACON_BONUS_WEIGHT=2
```

Players/agents choose PULSE, FLUX or ORBIT before the current lottery window
ends. Neural Hunt plans the **first drand round strictly after that boundary**.
When it becomes available, `SHA-256(signature)` is used to derive both the
boosted path and the weighted selection. A matching path receives the configured
weight; a non-matching path still has weight 1.

Successful draws are stored in `beacon_draws` with beacon ID, round, signature,
derived randomness, boosted path, ticket count and selected count, and the latest
record is available from `/api/public/beacon/<task-id>/latest` for independent
audit.

The current implementation relies on HTTPS to the configured drand endpoint and
stores the returned signature for external verification. It does **not** yet
perform local BLS signature verification against the drand chain public key.
If cryptographic self-verification is a product requirement, add a vetted drand
client/verifier before making that claim in customer-facing material.


## V4.2.6 engagement credits and account controls

The Hosted Service has two optional customer-funded-by-game credit rewards. Both default to zero and are administered persistently from the private Customer Admin:

```env
CS_NEW_CUSTOMER_CREDITS=0
CS_POSITIVE_TIP_CREDITS=0
```

`CS_NEW_CUSTOMER_CREDITS` is booked atomically with account creation. `CS_POSITIVE_TIP_CREDITS` is booked only for a delegated Hosted Worker when an accepted/evaluated guess creates a new personal best score. A ticket that is not selected by the lottery is not scored and earns no reward. Normal browser/CLI identities do not create these Customer-Service reward events.

The game persists each positive-tip event to an outbox and retries delivery to the private Customer Service (`CUSTOMER_SERVICE_INTERNAL_URL`, normally `http://customer-service:8092` in Docker). Customer Service applies its current configured reward amount and uses an idempotent ledger reference, so temporary outages and retries neither lose nor duplicate credits.

When PayPal is disabled the public Customer Portal hides the purchase panel entirely. Its visible credit history is aggregated by UTC day/reason for readability; the raw ledger remains unchanged in the database.

The private Customer Admin can also block/unblock accounts, stop individual/all workers, disable new logins, disable new registrations, configure registration Proof-of-Work and optionally enforce time-limited one-shot invite codes. Blocking an account deletes its active sessions and revokes Worker leases immediately.

## Worker Recovery und Image-Rollout (V4.2.9)

Hosted Worker erkennen stille/halb-offene Game-WebSockets nach spätestens etwa 90 Sekunden und verbinden sich mit Backoff neu. Ein abgelaufener Game-JWT wird mit derselben persistenten Worker-Identity automatisch erneuert. Auch ein noch nicht sofort verfügbarer Folge-Task wird wiederholt angefragt.

Neue Worker-Container werden mit Docker `RestartPolicy=unless-stopped` erzeugt. Bewusstes Stoppen über das Portal/Admin bleibt gestoppt. Die interne Worker-Registrierung akzeptiert nur Worker mit aktivem `running`/`starting`-Lease, sodass die Restart-Policy keine Sperre umgeht.

Im privaten Customer Admin kann das konfigurierte `CS_WORKER_IMAGE` explizit neu gepullt werden. Ein einzelner Worker oder alle Worker können anschließend mit dem neuen Image neu erzeugt werden. Named Identity Volumes werden dabei nicht entfernt; laufende Worker werden nach dem Update wieder gestartet, gestoppte bleiben gestoppt.

## Distributed Service Controllers (V4.3)

Worker orchestration can now be separated from the commercial Customer Service. In `controller` mode, Customer Service keeps customer accounts, prepaid billing, PayPal, reward delegation, worker configuration and leases, while Docker lifecycle operations are sent to authenticated Service Controllers on remote worker hosts.

This means the Customer Service no longer needs `docker.sock` in controller-only deployments. Existing public Customer Portal endpoints and worker semantics remain unchanged. `hybrid` mode is provided as a migration path for already-created local worker identity volumes.

See `SERVICE_CONTROLLER.md` for the full topology and deployment settings.