111 lines
5.2 KiB
Markdown
111 lines
5.2 KiB
Markdown
# Experimental Mathematical Vector Graph
|
||
|
||
This option builds a sparse Knowledge↔Knowledge neighbourhood layer from embeddings that are already stored in `graph.db`.
|
||
|
||
It is deliberately **not** an AI relation classifier. It performs no chat call and no new embedding call. If a node already has its normal search embedding, edge calculation and optional layout are CPU-only arithmetic.
|
||
|
||
## Why a separate relation type?
|
||
|
||
The generated edge type is `semantic_neighbor`, origin `vector-math`.
|
||
|
||
Vector proximity is useful for candidate discovery, navigation and graph structure, but it does not prove that two articles are factually `same_topic`, that one `depends_on` another, or that two statements are logically equivalent. Those stronger relations remain AI/review decisions.
|
||
|
||
## Algorithm
|
||
|
||
1. Use production Knowledge vectors with a common embedding dimension.
|
||
2. Generate deterministic sparse random-projection signatures (LSH) to avoid all-pairs Cosine evaluation.
|
||
3. Keep a bounded coarse shortlist per node.
|
||
4. Calculate exact Cosine similarity only on that shortlist.
|
||
5. Keep the local `k` nearest neighbours.
|
||
6. Convert global distance into a locally scaled affinity using the node-specific k-neighbour distance.
|
||
7. Emit an edge when the neighbourhood is reciprocal, or when a one-sided neighbour is exceptionally strong.
|
||
8. Optional layout: deterministically project the stored vectors to 3D and smooth them over the accepted neighbour graph. This changes only visualization coordinates.
|
||
|
||
The local scaling is important for this corpus because different knowledge regions have very different similarity density. A single raw Cosine threshold otherwise over-links highly templated areas and under-links sparse areas.
|
||
|
||
## Test against the supplied graph
|
||
|
||
The production graph contained 21,289 production Knowledge vectors with 768 dimensions.
|
||
|
||
A deterministic 500-node exact-nearest-neighbour sample produced these raw Cosine medians:
|
||
|
||
- nearest neighbour: 0.9030
|
||
- 2nd neighbour: 0.8518
|
||
- 3rd neighbour: 0.8297
|
||
- 5th neighbour: 0.8120
|
||
- 10th neighbour: 0.7915
|
||
|
||
This confirms that the corpus has a high and non-uniform similarity baseline; raw `cosine >= 0.80` alone is not a sufficient thematic relation rule.
|
||
|
||
With the experimental defaults over the full 21,289-vector corpus:
|
||
|
||
- `k`: 4
|
||
- exact shortlist: 96
|
||
- minimum raw Cosine: 0.80
|
||
- minimum local affinity: 0.35
|
||
- accepted `semantic_neighbor` edges: 16,940
|
||
- reciprocal edges: 16,007
|
||
- Knowledge nodes touched by at least one accepted edge: 16,069 (75.5%)
|
||
- exact Cosine comparisons: 2,043,744
|
||
- full directed all-pairs comparisons avoided: >99.5%
|
||
- CPU runtime in the supplied execution environment: about 6–7 seconds for the mathematical build itself
|
||
|
||
These numbers are a benchmark for this snapshot, not a universal quality guarantee. The option therefore stays disabled by default.
|
||
|
||
## Configuration
|
||
|
||
```env
|
||
BRAIN_VECTOR_GRAPH_ENABLED=false
|
||
BRAIN_VECTOR_GRAPH_NEIGHBORS=4
|
||
BRAIN_VECTOR_GRAPH_CANDIDATES=96
|
||
BRAIN_VECTOR_GRAPH_MIN_SIMILARITY=0.80
|
||
BRAIN_VECTOR_GRAPH_MIN_AFFINITY=0.35
|
||
BRAIN_VECTOR_GRAPH_ORPHAN_PASS=false
|
||
BRAIN_VECTOR_GRAPH_ORPHAN_NEIGHBORS=2
|
||
BRAIN_VECTOR_GRAPH_ORPHAN_CANDIDATES=256
|
||
BRAIN_VECTOR_GRAPH_ORPHAN_MIN_SIMILARITY=0.80
|
||
BRAIN_VECTOR_GRAPH_ORPHAN_MIN_AFFINITY=0.30
|
||
BRAIN_VECTOR_GRAPH_AGENT_OFFLOAD=false
|
||
BRAIN_VECTOR_GRAPH_AGENT_REQUIRED=false
|
||
BRAIN_VECTOR_GRAPH_AGENT_WAIT=2m
|
||
BRAIN_THINKING_VECTOR_GUIDED=true
|
||
BRAIN_VECTOR_GRAPH_LAYOUT=false
|
||
```
|
||
|
||
Recommended A/B sequence:
|
||
|
||
1. enable `BRAIN_VECTOR_GRAPH_ENABLED=true` with layout still `false`;
|
||
2. compare orphan coverage, neighbour examples and false-positive rate;
|
||
3. tune thresholds if necessary;
|
||
4. only then test `BRAIN_VECTOR_GRAPH_LAYOUT=true`.
|
||
|
||
## Agent CPU offload
|
||
|
||
The `internal/vectorgraph` package has no Brain model, Ollama, network or persistence dependency. v3 adds an authenticated pull-job protocol so an integrated `BRAIN_MODE=agent` worker can calculate the vector graph without any Chat or Embed call.
|
||
|
||
The Brain still owns the vector index and graph state. For a rebuild it streams the current float32 vectors in a compact binary format, the Agent returns only mathematical links/positions/statistics, and the Brain validates graph version, endpoints and numeric ranges before applying `semantic_neighbor` edges.
|
||
|
||
Because a full 21k×768 matrix is roughly 62 MiB before protocol overhead, offload is intentionally optional rather than automatic. On a single host the local 3.7-second rebuild may remain cheaper than transferring the whole matrix. Offload is more attractive when the Agent has spare CPU on another machine.
|
||
|
||
```env
|
||
BRAIN_VECTOR_GRAPH_AGENT_OFFLOAD=true
|
||
BRAIN_VECTOR_GRAPH_AGENT_REQUIRED=false
|
||
```
|
||
|
||
Agent side:
|
||
|
||
```env
|
||
BRAIN_AGENT_COMPUTE_ENABLED=true
|
||
BRAIN_AGENT_COMPUTE_POLL_INTERVAL=5s
|
||
BRAIN_AGENT_COMPUTE_MAX_BYTES=134217728
|
||
```
|
||
|
||
`AGENT_REQUIRED=false` retains local CPU fallback when no compute Agent is online or when a returned job is stale. See `VECTOR-GRAPH-AGENT-OFFLOAD-V3.md`.
|
||
|
||
## Safety / interpretation
|
||
|
||
- The layer does not overwrite AI `same_topic`, `related_to` or `depends_on` edges.
|
||
- Vector edges are replaceable derived data (`origin=vector-math`).
|
||
- Layout is independently switchable and disabled by default.
|
||
- The Brain only runs the experiment when real Ollama embeddings are healthy; deterministic fallback embeddings are not used for this layer.
|