glpi-neuroforge-mega/platform/neuroforge/IMPLEMENTATION-NOTES-v0.6.0-dev.md

NeuroForge v0.6.0-dev – Sequential PQ segment scan

Status: implemented, intentionally not built/tested/benchmarked per request.

Change

The v0.5 -> v0.6 disk-PQ migration fallback no longer performs one random ReadAt/segment lookup per memory during the full encode pass.

Segment scanner

internal/store/segment.go now contains a forward-only live-record scanner:

• captures a point-in-time snapshot of the latest live segment locations;
• groups/sorts locations by segment and physical offset;
• opens each segment once;
• consumes bytes strictly forward through a 1 MiB buffered reader;
• reuses the payload buffer across records;
• remains safe with concurrent appends because segment files are append-only and the captured offsets do not move.

Vector-only decode path

IterateLiveVectorsSequential(dim, callback) decodes only the fields needed by disk ANN:

• top-level record ID/op;
• memory status;
• vector dimension;
• vector values.

Memory text, tags, session metadata, timestamps and other unrelated fields are skipped by encoding/json instead of being materialized into a full core.Memory.

PQ builder integration

internal/store/diskann.go uses the new vector-only sequential iterator for both:

1. training-sample collection on a v0.5 migration store; and
2. the complete IVF-PQ encode pass.

While the first pass seeds the compact vector journal, only ID + Vector are buffered, not the full Memory body.

Durability / authority

The memory segments remain the authoritative source. The vector journal remains a rebuildable acceleration cache. This change does not alter the memory segment format.

Validation status

The sequential PQ migration path remains intentionally unbenchmarked and was not directly exercised after its implementation. For the later observability addition only the targeted internal/httpapi tests plus dashboard JavaScript/OpenAPI syntax validation were run; no PQ benchmark or migration benchmark was started.

Admin Dashboard / Prometheus

• /admin serves the embedded administration UI; / remains compatible.
• The Observability tab renders client-side rolling charts for heap, request rate/p95 and memory count.
• GET /metrics exports Prometheus text exposition 0.0.4 and requires Authorization: Bearer <metrics_token> (admin token accepted as bearer fallback).
• metrics_token is generated at first startup and can be overridden with NEUROFORGE_METRICS_TOKEN.
• HTTP metrics use the matched Go ServeMux pattern as the route label, preventing dynamic memory/goal IDs from exploding cardinality.
• Store observability is collected through ObservabilitySnapshot, which does not iterate over the complete memory catalog.
• The periodic dashboard refresh no longer performs remote provider health checks; those remain an explicit admin action.

Admin model routing

• The internal routing.chat_provider / routing.embedding_provider and per-Ollama chat_model / embedding_model settings are now exposed in a dedicated Modelle & Routing admin view.
• GET/PUT /admin/api/model-routing supports partial updates; a body containing only routing and ollama is sufficient.
• routing.chat_node_id and routing.embedding_node_id optionally pin core routes to one Ollama node.
• routing.critic, routing.consolidator, and routing.goal are role routes with provider/model/node_id.
• Explicit Ollama node pins are strict; unpinned routes retain weighted multi-Ollama failover.
• Provider health now parses /api/tags and reports model names without generating text or embeddings.