jbergner
a6bc71fb3a
release-tag / Resolve release metadata (push) Successful in 30s
release-tag / Build knowledge (push) Failing after 4m51s
release-tag / Build control (push) Failing after 5m0s
release-tag / Build agent (push) Failing after 5m0s
release-tag / Build agent-data-init (push) Failing after 5m5s
release-tag / Build neuroforge-worker (push) Failing after 5m7s
release-tag / Build neuroforge (push) Failing after 5m9s
36 lines
1.4 KiB
Markdown
36 lines
1.4 KiB
Markdown
# Retrieval & Learning Replay Benchmark
|
|
|
|
v1.3.0 adds a read-only benchmark endpoint: `POST /api/quality/replay`.
|
|
It does **not** write to GLPI, does not learn and does not call the answer LLM. It replays
|
|
historical ticket text through the current Knowledge retrieval and the human-validated
|
|
Outcome retrieval so quality changes can be measured before a rollout.
|
|
|
|
Each case may specify:
|
|
|
|
- `query`: historical ticket subject/body snapshot.
|
|
- `expected_knowledge_id`: the KB article known to be correct at that time.
|
|
- `expected_solution_terms`: terms expected in a technician-validated outcome.
|
|
- `k`: evaluation depth (default 10, max 50).
|
|
|
|
Reported KPIs:
|
|
|
|
- `knowledge_recall_at_k`
|
|
- `knowledge_mrr`
|
|
- `outcome_recall_at_k`
|
|
- `outcome_mrr`
|
|
- `experience_rescued_cases`: cases where the expected KB was not retrieved in K but a
|
|
matching human-validated experience was retrieved. This is a conservative proxy for
|
|
learning lift; it is not counted as auto-reply authority.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
./scripts/quality-replay.py docs/QUALITY-REPLAY-example.json \
|
|
--url http://127.0.0.1:8080 --user "$WEB_BASIC_USER" --password "$WEB_BASIC_PASSWORD" \
|
|
--output ./data/quality-replay-$(date +%F).json
|
|
```
|
|
|
|
For production acceptance, build a versioned set of historical tickets and require fixed
|
|
minimum thresholds before changing retrieval weights, embedding models, HNSW settings or
|
|
Outcome retrieval thresholds.
|