jbergner
a6bc71fb3a
release-tag / Resolve release metadata (push) Successful in 30s
release-tag / Build knowledge (push) Failing after 4m51s
release-tag / Build control (push) Failing after 5m0s
release-tag / Build agent (push) Failing after 5m0s
release-tag / Build agent-data-init (push) Failing after 5m5s
release-tag / Build neuroforge-worker (push) Failing after 5m7s
release-tag / Build neuroforge (push) Failing after 5m9s
67 lines
3.5 KiB
Markdown
67 lines
3.5 KiB
Markdown
# NeuroForge v0.6.0-dev – Sequential PQ segment scan
|
||
|
||
Status: **implemented, intentionally not built/tested/benchmarked** per request.
|
||
|
||
## Change
|
||
|
||
The v0.5 -> v0.6 disk-PQ migration fallback no longer performs one random `ReadAt`/segment lookup per memory during the full encode pass.
|
||
|
||
### Segment scanner
|
||
|
||
`internal/store/segment.go` now contains a forward-only live-record scanner:
|
||
|
||
- captures a point-in-time snapshot of the latest live segment locations;
|
||
- groups/sorts locations by segment and physical offset;
|
||
- opens each segment once;
|
||
- consumes bytes strictly forward through a 1 MiB buffered reader;
|
||
- reuses the payload buffer across records;
|
||
- remains safe with concurrent appends because segment files are append-only and the captured offsets do not move.
|
||
|
||
### Vector-only decode path
|
||
|
||
`IterateLiveVectorsSequential(dim, callback)` decodes only the fields needed by disk ANN:
|
||
|
||
- top-level record ID/op;
|
||
- memory status;
|
||
- vector dimension;
|
||
- vector values.
|
||
|
||
Memory text, tags, session metadata, timestamps and other unrelated fields are skipped by `encoding/json` instead of being materialized into a full `core.Memory`.
|
||
|
||
### PQ builder integration
|
||
|
||
`internal/store/diskann.go` uses the new vector-only sequential iterator for both:
|
||
|
||
1. training-sample collection on a v0.5 migration store; and
|
||
2. the complete IVF-PQ encode pass.
|
||
|
||
While the first pass seeds the compact vector journal, only `ID + Vector` are buffered, not the full Memory body.
|
||
|
||
## Durability / authority
|
||
|
||
The memory segments remain the authoritative source. The vector journal remains a rebuildable acceleration cache. This change does not alter the memory segment format.
|
||
|
||
## Validation status
|
||
|
||
The sequential PQ migration path remains intentionally unbenchmarked and was not directly exercised after its implementation. For the later observability addition only the targeted `internal/httpapi` tests plus dashboard JavaScript/OpenAPI syntax validation were run; no PQ benchmark or migration benchmark was started.
|
||
|
||
## Admin Dashboard / Prometheus
|
||
|
||
- `/admin` serves the embedded administration UI; `/` remains compatible.
|
||
- The Observability tab renders client-side rolling charts for heap, request rate/p95 and memory count.
|
||
- `GET /metrics` exports Prometheus text exposition `0.0.4` and requires `Authorization: Bearer <metrics_token>` (admin token accepted as bearer fallback).
|
||
- `metrics_token` is generated at first startup and can be overridden with `NEUROFORGE_METRICS_TOKEN`.
|
||
- HTTP metrics use the matched Go `ServeMux` pattern as the route label, preventing dynamic memory/goal IDs from exploding cardinality.
|
||
- Store observability is collected through `ObservabilitySnapshot`, which does not iterate over the complete memory catalog.
|
||
- The periodic dashboard refresh no longer performs remote provider health checks; those remain an explicit admin action.
|
||
|
||
|
||
## Admin model routing
|
||
|
||
- The internal `routing.chat_provider` / `routing.embedding_provider` and per-Ollama `chat_model` / `embedding_model` settings are now exposed in a dedicated **Modelle & Routing** admin view.
|
||
- `GET/PUT /admin/api/model-routing` supports partial updates; a body containing only `routing` and `ollama` is sufficient.
|
||
- `routing.chat_node_id` and `routing.embedding_node_id` optionally pin core routes to one Ollama node.
|
||
- `routing.critic`, `routing.consolidator`, and `routing.goal` are role routes with provider/model/node_id.
|
||
- Explicit Ollama node pins are strict; unpinned routes retain weighted multi-Ollama failover.
|
||
- Provider health now parses `/api/tags` and reports model names without generating text or embeddings.
|