Experimental Mathematical Vector Graph

This option builds a sparse Knowledge↔Knowledge neighbourhood layer from embeddings that are already stored in graph.db.

It is deliberately not an AI relation classifier. It performs no chat call and no new embedding call. If a node already has its normal search embedding, edge calculation and optional layout are CPU-only arithmetic.

Why a separate relation type?

The generated edge type is semantic_neighbor, origin vector-math.

Vector proximity is useful for candidate discovery, navigation and graph structure, but it does not prove that two articles are factually same_topic, that one depends_on another, or that two statements are logically equivalent. Those stronger relations remain AI/review decisions.

Algorithm

Use production Knowledge vectors with a common embedding dimension.
Generate deterministic sparse random-projection signatures (LSH) to avoid all-pairs Cosine evaluation.
Keep a bounded coarse shortlist per node.
Calculate exact Cosine similarity only on that shortlist.
Keep the local k nearest neighbours.
Convert global distance into a locally scaled affinity using the node-specific k-neighbour distance.
Emit an edge when the neighbourhood is reciprocal, or when a one-sided neighbour is exceptionally strong.
Optional layout: deterministically project the stored vectors to 3D and smooth them over the accepted neighbour graph. This changes only visualization coordinates.

The local scaling is important for this corpus because different knowledge regions have very different similarity density. A single raw Cosine threshold otherwise over-links highly templated areas and under-links sparse areas.

Test against the supplied graph

The production graph contained 21,289 production Knowledge vectors with 768 dimensions.

A deterministic 500-node exact-nearest-neighbour sample produced these raw Cosine medians:

nearest neighbour: 0.9030
2nd neighbour: 0.8518
3rd neighbour: 0.8297
5th neighbour: 0.8120
10th neighbour: 0.7915

This confirms that the corpus has a high and non-uniform similarity baseline; raw cosine >= 0.80 alone is not a sufficient thematic relation rule.

With the experimental defaults over the full 21,289-vector corpus:

k: 4
exact shortlist: 96
minimum raw Cosine: 0.80
minimum local affinity: 0.35
accepted semantic_neighbor edges: 16,940
reciprocal edges: 16,007
Knowledge nodes touched by at least one accepted edge: 16,069 (75.5%)
exact Cosine comparisons: 2,043,744
full directed all-pairs comparisons avoided: >99.5%
CPU runtime in the supplied execution environment: about 6–7 seconds for the mathematical build itself

These numbers are a benchmark for this snapshot, not a universal quality guarantee. The option therefore stays disabled by default.

Configuration

BRAIN_VECTOR_GRAPH_ENABLED=false
BRAIN_VECTOR_GRAPH_NEIGHBORS=4
BRAIN_VECTOR_GRAPH_CANDIDATES=96
BRAIN_VECTOR_GRAPH_MIN_SIMILARITY=0.80
BRAIN_VECTOR_GRAPH_MIN_AFFINITY=0.35
BRAIN_VECTOR_GRAPH_ORPHAN_PASS=false
BRAIN_VECTOR_GRAPH_ORPHAN_NEIGHBORS=2
BRAIN_VECTOR_GRAPH_ORPHAN_CANDIDATES=256
BRAIN_VECTOR_GRAPH_ORPHAN_MIN_SIMILARITY=0.80
BRAIN_VECTOR_GRAPH_ORPHAN_MIN_AFFINITY=0.30
BRAIN_VECTOR_GRAPH_AGENT_OFFLOAD=false
BRAIN_VECTOR_GRAPH_AGENT_REQUIRED=false
BRAIN_VECTOR_GRAPH_AGENT_WAIT=2m
BRAIN_THINKING_VECTOR_GUIDED=true
BRAIN_VECTOR_GRAPH_LAYOUT=false

Recommended A/B sequence:

enable BRAIN_VECTOR_GRAPH_ENABLED=true with layout still false;
compare orphan coverage, neighbour examples and false-positive rate;
tune thresholds if necessary;
only then test BRAIN_VECTOR_GRAPH_LAYOUT=true.

Agent CPU offload

The internal/vectorgraph package has no Brain model, Ollama, network or persistence dependency. v3 adds an authenticated pull-job protocol so an integrated BRAIN_MODE=agent worker can calculate the vector graph without any Chat or Embed call.

The Brain still owns the vector index and graph state. For a rebuild it streams the current float32 vectors in a compact binary format, the Agent returns only mathematical links/positions/statistics, and the Brain validates graph version, endpoints and numeric ranges before applying semantic_neighbor edges.

Because a full 21k×768 matrix is roughly 62 MiB before protocol overhead, offload is intentionally optional rather than automatic. On a single host the local 3.7-second rebuild may remain cheaper than transferring the whole matrix. Offload is more attractive when the Agent has spare CPU on another machine.

BRAIN_VECTOR_GRAPH_AGENT_OFFLOAD=true
BRAIN_VECTOR_GRAPH_AGENT_REQUIRED=false

Agent side:

BRAIN_AGENT_COMPUTE_ENABLED=true
BRAIN_AGENT_COMPUTE_POLL_INTERVAL=5s
BRAIN_AGENT_COMPUTE_MAX_BYTES=134217728

AGENT_REQUIRED=false retains local CPU fallback when no compute Agent is online or when a returned job is stale. See VECTOR-GRAPH-AGENT-OFFLOAD-V3.md.

Safety / interpretation

The layer does not overwrite AI same_topic, related_to or depends_on edges.
Vector edges are replaceable derived data (origin=vector-math).
Layout is independently switchable and disabled by default.
The Brain only runs the experiment when real Ollama embeddings are healthy; deterministic fallback embeddings are not used for this layer.

5.2 KiB Raw Permalink Blame History Unescape Escape