From 31d3efb8a2278377df4b93150018471d3a95b294 Mon Sep 17 00:00:00 2001 From: groot Date: Wed, 5 Aug 2026 10:15:56 +0200 Subject: [PATCH] Bugfixes --- CHANGELOG-FILTER-SCOPES-SOURCES.md | 26 +++ FILTER-SCOPES-SOURCES.md | 137 ++++++++++++ README.md | 26 ++- RUNTIME-CONTROLS-HONEYCOMB.md | 12 +- SHA256SUMS.txt | 44 ++-- VALIDATION-FILTER-SCOPES-SOURCES.md | 55 +++++ deployment/docker-compose.full.yml | 3 + docker-compose.yml | 3 + internal/config/config.go | 6 + internal/config/config_test.go | 6 + internal/engine/article.go | 45 ++-- internal/engine/article_research.go | 11 + internal/engine/article_research_cache.go | 5 +- internal/engine/engine.go | 33 +-- internal/engine/engine_test.go | 5 +- internal/engine/runtime.go | 259 ++++++++++++++++++---- internal/engine/runtime_filter_test.go | 45 ++++ internal/graph/filter.go | 116 ++++++++++ internal/graph/filter_test.go | 76 +++++++ internal/graph/store.go | 49 ++-- internal/web/server.go | 11 +- internal/web/server_test.go | 21 +- internal/web/static/app.css | 1 + internal/web/static/app.js | 202 +++++++++++++---- internal/web/static/index.html | 21 +- 25 files changed, 1034 insertions(+), 184 deletions(-) create mode 100644 CHANGELOG-FILTER-SCOPES-SOURCES.md create mode 100644 FILTER-SCOPES-SOURCES.md create mode 100644 VALIDATION-FILTER-SCOPES-SOURCES.md create mode 100644 internal/engine/runtime_filter_test.go create mode 100644 internal/graph/filter.go create mode 100644 internal/graph/filter_test.go diff --git a/CHANGELOG-FILTER-SCOPES-SOURCES.md b/CHANGELOG-FILTER-SCOPES-SOURCES.md new file mode 100644 index 0000000..c748c25 --- /dev/null +++ b/CHANGELOG-FILTER-SCOPES-SOURCES.md @@ -0,0 +1,26 @@ +# Changelog: Filter Scopes und Sources + +## Korrigiert + +- Environment-Filter werden nicht mehr von `runtime-settings.json` ersetzt. +- Environment und WebUI werden als administrative Obergrenze und zusätzliche Laufzeiteinschränkung geschnitten. +- Disjunkte Filter ergeben zuverlässig „keine Treffer“ statt versehentlich „alle“. +- Alte Runtime-Dateien werden feldweise zusammengeführt. +- Anzeige-Kanten dürfen keine fremden Wissens-Nodes mehr nachladen. +- Semantische Abfragen berücksichtigen den wirksamen Lernfilter. +- Relations-, Artikel- und Research-Auswahl berücksichtigen den wirksamen Thinking-Filter. + +## Ergänzt + +- `BRAIN_LEARNING_SOURCES` +- `BRAIN_DISPLAY_SOURCES` +- `BRAIN_THINKING_SOURCES` +- WebUI-Quellenlisten für Lernen, Anzeige und Thinking +- `GET /api/sources` +- effektive und administrative Filterbereiche in `/api/runtime-settings` und `/api/status` +- Domain-basierte Quellenbezeichnung für SearXNG-/Web-Nodes +- `__unsourced__` als virtueller Filterwert + +## Speicherverhalten + +Bestehende Vektoren außerhalb des Lernfilters werden nicht gelöscht. Sie bleiben in `graph.db`, sind aber für lerngefiltertes Retrieval inaktiv. Thinking besitzt weiterhin seinen eigenen wirksamen Kategorie- und Quellenfilter. Dadurch ist ein späteres Erweitern des Lernfilters ohne erneutes Embedding möglich. diff --git a/FILTER-SCOPES-SOURCES.md b/FILTER-SCOPES-SOURCES.md new file mode 100644 index 0000000..9526700 --- /dev/null +++ b/FILTER-SCOPES-SOURCES.md @@ -0,0 +1,137 @@ +# Wirksame Kategorie- und Quellenfilter + +Diese Version korrigiert die Filtersemantik und ergänzt Quellenfilter für Lernen, Anzeige und Thinking. + +## Zwei Ebenen + +Die Environment-Variablen sind administrative Obergrenzen: + +```env +BRAIN_LEARNING_CATEGORIES= +BRAIN_DISPLAY_CATEGORIES= +BRAIN_THINKING_CATEGORIES= +BRAIN_LEARNING_SOURCES= +BRAIN_DISPLAY_SOURCES= +BRAIN_THINKING_SOURCES= +``` + +Die WebUI speichert zusätzliche Laufzeiteinschränkungen in `runtime-settings.json`. + +```text +wirksamer Filter = Environment ∩ WebUI +``` + +Eine leere Environment-Liste oder `*` ist unbegrenzt. Eine leere WebUI-Liste bedeutet „alle durch das Environment erlaubten Werte“. Sind Environment und WebUI disjunkt, ist der wirksame Filter absichtlich leer und nicht versehentlich „alle“. + +Die API liefert sowohl die veränderbaren Werte als auch die effektiven und administrativen Bereiche: + +```json +{ + "learning_categories": [], + "learning_sources": ["internal-kb"], + "effective_learning": { + "categories": ["IT-Security"], + "sources": ["internal-kb"], + "categories_restricted": true, + "sources_restricted": true, + "matches_none": false + }, + "admin_learning": { + "categories": ["IT-Security", "Netzwerk"], + "sources": ["GLPI Knowledge Base", "internal-kb"], + "categories_restricted": true, + "sources_restricted": true, + "matches_none": false + } +} +``` + +Ältere `runtime-settings.json`-Dateien werden feldweise eingelesen. Fehlende neue Felder setzen vorhandene Schalter, Ansicht oder Performancewerte nicht mehr auf Nullwerte zurück. + +## Kombinationsregeln + +Innerhalb einer Dimension gilt ODER: + +```text +Kategorie = IT-Security ODER Netzwerk +``` + +Zwischen Kategorie und Quelle gilt UND: + +```text +(Kategorie = IT-Security ODER Netzwerk) +UND +(Quelle = GLPI Knowledge Base ODER internal-kb) +``` + +Verglichen wird exakt und ohne Beachtung der Groß-/Kleinschreibung. Die virtuellen Werte sind: + +```text +__uncategorized__ = ohne Kategorie +__unsourced__ = ohne erkennbare Quelle +``` + +## Welche Quelle wird verwendet? + +Die logische Quelle eines Wissensknotens wird in dieser Reihenfolge bestimmt: + +1. `metadata.source` beziehungsweise das `source`-Feld des KB-Dokuments; +2. bei Webrecherche die Domain der URL, beispielsweise `learn.microsoft.com`; +3. als Rückfall der technische `origin`-Wert. + +Damit können produktive GLPI-Beiträge, lokale KB-Dateien, AI-Synthesis-Beiträge und einzelne Webdomains getrennt gesteuert werden. + +## Lernen + +Der wirksame Lernfilter gilt jetzt für: + +- neue oder geänderte Embeddings; +- lokale Fallback-Embeddings; +- semantische Abfragen über `/api/query`; +- sofortiges Lernen neuer AI-Synthesis-Artikel; +- sofortiges Lernen akzeptierter Research-Evidence. + +Bereits vorhandene Vektoren außerhalb des Filters bleiben in SQLite gespeichert, werden aber bei Retrieval und neuem Lernen nicht verwendet. Eine spätere Filtererweiterung kann sie ohne erneute Berechnung wieder aktivieren. + +## Anzeige + +Die Anzeige filtert zunächst ausschließlich `knowledge`, `ai-think` und `external`. Danach werden nur direkt angeschlossene Taxonomie-Nodes ergänzt: + +- `category`; +- `source`; +- `concept`. + +Semantische Kanten dürfen keine anders kategorisierten oder aus einer anderen Quelle stammenden Wissensartikel mehr in die Ansicht hineinziehen. Das gilt für Neural, Honeycomb und Constellation. + +Der Anzeige-Filter ist weiterhin eine Rendersteuerung und keine Zugriffskontrolle; `/api/graph` liefert den vollständigen Graphen. + +## Thinking + +Der wirksame Thinking-Filter gilt jetzt für: + +- beide Nodes eines Relationskandidaten; +- zusätzliche Quellen des Themenverbunds; +- AI-Synthesis-Quellenauswahl; +- wiederverwendete Research-Evidence; +- SearXNG-Treffer der Relationsrecherche; +- neu akzeptierte Volltextbelege der iterativen Artikelrecherche. + +Eine externe Quelle außerhalb des Thinking-Quellenfilters wird nicht als Beleg akzeptiert und nicht in die Artikelsynthese übernommen. + +## WebUI + +Unter **FILTER** stehen nun sechs Listen zur Verfügung: + +- Kategorien: Lernen, Anzeige, Thinking; +- Quellen: Lernen, Anzeige, Thinking. + +Die WebUI zeigt die wirksame Schnittmenge an. Optionen außerhalb der Environment-Obergrenze werden deaktiviert und als `durch ENV ausgeschlossen` markiert. + +## APIs + +```http +GET /api/runtime-settings +PUT /api/runtime-settings +GET /api/categories +GET /api/sources +``` diff --git a/README.md b/README.md index bb57b98..6baffaf 100644 --- a/README.md +++ b/README.md @@ -138,25 +138,41 @@ Mehr Details: [`PERSISTENCE.md`](PERSISTENCE.md) und [`SQLITE-STORAGE.md`](SQLIT Die untere Steuerleiste enthält direkte Schalter für **LEARNING**, **THINKING**, **NEURAL**, **HONEYCOMB**, **CONSTELLATION** und **ECO**. Sind Learning und Thinking deaktiviert, bleibt das System im Living-Modus; eingehende Agent- oder KB-Anfragen können weiterhin die tatsächlich verwendeten Notes aktivieren. -Über **FILTER** lassen sich drei unabhängige Kategorienlisten pflegen: +Über **FILTER** lassen sich für **Lernen**, **Anzeige** und **Thinking** jeweils Kategorien und Quellen auswählen. -- **Lernen**: neue Embeddings nur für passende Kategorien; -- **Anzeige**: Browser-Rendering nur für passende Notes; -- **Thinking**: neue AI-THINK-Kandidaten nur innerhalb der gewählten Kategorien. +- **Lernen** steuert neue Embeddings **und** die Wissensknoten, die bei Abfragen verwendet werden dürfen. +- **Anzeige** rendert nur passende Wissens-Nodes; lediglich direkt verbundene Kategorie-, Quellen- und Konzept-Nodes werden ergänzend angezeigt. +- **Thinking** begrenzt Relationskandidaten, Artikelquellen, wiederverwendete Recherchebelege und neu akzeptierte Webquellen. -Leere Listen bedeuten „alle Kategorien“. Die Werte werden über die vorhandene Persistenzqueue in `runtime-settings.json` geschrieben. +Kategorie und Quelle werden mit **UND** kombiniert; mehrere Werte innerhalb einer Liste verwenden **ODER**. Leere WebUI-Listen bedeuten „alle administrativ erlaubten Werte“. + +Die Environment-Listen sind feste Obergrenzen. Die WebUI kann sie nur weiter einschränken: + +```text +wirksamer Filter = Environment ∩ WebUI +``` ```env BRAIN_LEARNING_ENABLED=true BRAIN_THINKING_ENABLED=true + +# Administrative Kategoriegrenzen; leer oder * = unbegrenzt BRAIN_LEARNING_CATEGORIES= BRAIN_DISPLAY_CATEGORIES= BRAIN_THINKING_CATEGORIES= + +# Administrative Quellengrenzen; z. B. GLPI Knowledge Base, internal-kb, docs.example.org +BRAIN_LEARNING_SOURCES= +BRAIN_DISPLAY_SOURCES= +BRAIN_THINKING_SOURCES= + BRAIN_DEFAULT_VIEW=neural # neural | honeycomb | constellation BRAIN_MAX_DISPLAY_NODES=0 # 0 = unbegrenzt; alternativ z. B. 5000 BRAIN_LOW_POWER_MODE=false ``` +Die WebUI zeigt die wirksame Schnittmenge an und kennzeichnet Optionen, die durch das Environment ausgeschlossen sind. Gespeichert werden nur die veränderbaren WebUI-Auswahlen in `runtime-settings.json`. + Die Ansichten erfüllen unterschiedliche Aufgaben: - **Neural** zeigt das lebende semantische Netz mit LOD, Cortex, Synapsen und Partikeln. diff --git a/RUNTIME-CONTROLS-HONEYCOMB.md b/RUNTIME-CONTROLS-HONEYCOMB.md index 4cd644c..e890d98 100644 --- a/RUNTIME-CONTROLS-HONEYCOMB.md +++ b/RUNTIME-CONTROLS-HONEYCOMB.md @@ -68,9 +68,11 @@ Der vollständige Graph bleibt serverseitig erhalten. Der Filter ist keine Lösc ### Thinking -`thinking_categories` begrenzt beide Nodes eines Kandidatenpaares für neue AI-THINK-Beziehungen. Bereits vorhandene Edges und AI-THINK-Beiträge bleiben erhalten. +`thinking_categories` und `thinking_sources` begrenzen beide Nodes eines Kandidatenpaares, die Artikelquellenauswahl und externe Belege. Bereits vorhandene Edges und AI-THINK-Beiträge bleiben gespeichert, werden außerhalb des aktuellen Filters aber nicht für neue Thinking-Schritte verwendet. -Die virtuelle Kategorie `__uncategorized__` steht im Web als **Ohne Kategorie** zur Verfügung. +`learning_categories` und `learning_sources` gelten sowohl für neue Embeddings als auch für semantische Abfragen. Bereits gespeicherte Vektoren außerhalb des Filters werden nicht gelöscht, sondern bleiben inaktiv und können nach einer späteren Erweiterung ohne Neuberechnung wiederverwendet werden. + +Die virtuellen Werte `__uncategorized__` und `__unsourced__` erscheinen im Web als **Ohne Kategorie** beziehungsweise **Ohne Quelle**. ## Konfiguration @@ -82,6 +84,9 @@ BRAIN_THINKING_ENABLED=true BRAIN_LEARNING_CATEGORIES= BRAIN_DISPLAY_CATEGORIES= BRAIN_THINKING_CATEGORIES= +BRAIN_LEARNING_SOURCES= +BRAIN_DISPLAY_SOURCES= +BRAIN_THINKING_SOURCES= BRAIN_DEFAULT_VIEW=neural BRAIN_MAX_DISPLAY_NODES=0 BRAIN_LOW_POWER_MODE=false @@ -89,7 +94,7 @@ BRAIN_LOW_POWER_MODE=false Zulässige Werte für `BRAIN_DEFAULT_VIEW` sind `neural`, `honeycomb` und `constellation`. -Nach der ersten Änderung im Web haben die in `runtime-settings.json` gespeicherten Laufzeitwerte Vorrang vor den Startwerten. Zum Zurücksetzen kann die Datei bei gestopptem Dienst entfernt werden. +Die Environment-Kategorie- und Quellenlisten bleiben immer administrative Obergrenzen. `runtime-settings.json` speichert nur zusätzliche WebUI-Einschränkungen sowie Schalter, Ansicht und Performancewerte. Fehlende Felder in älteren Dateien werden feldweise mit den aktuellen Defaults ergänzt. ## API @@ -97,6 +102,7 @@ Nach der ersten Änderung im Web haben die in `runtime-settings.json` gespeicher GET /api/runtime-settings PUT /api/runtime-settings GET /api/categories +GET /api/sources ``` Beispiel: diff --git a/SHA256SUMS.txt b/SHA256SUMS.txt index 67d120a..abf413b 100644 --- a/SHA256SUMS.txt +++ b/SHA256SUMS.txt @@ -1,6 +1,7 @@ -6a1332aec3461a9852258f428366903f9266369e474c27083a43d273c2ff1609 ./.env.example +8f8861c8f91c787238b019804f05f5ce1a33878480c8de3239de97c98f492f21 ./.env.example 8955cfbeff229e73f0cad664863ff21711c270a4233c3225680c89aa6268a901 ./ARCHITECTURE.md 2bc149241c2d25f755e4a0470dc517f646527ee3e98d3a6c2e7798639bd70866 ./CHANGELOG-CONSTELLATION-ECO-TRANSITIONS.md +176f089da30ddea637d7a4c81ef45dfa889e0b36a9180a17145ef0931b523a69 ./CHANGELOG-FILTER-SCOPES-SOURCES.md 213ac897cd415bbeb9764a847843b9eff366983cf25bb3cbce4efa4c04b9e5f5 ./CHANGELOG-GLPI-POOL-PERSISTENCE.md e2f1f0400999cb59b8be09bc2743e06e0a0b2ecdc44a583af7c9a08b70d8509e ./CHANGELOG-GPU-NODE-LIMIT.md a317376127be1e47b9ec9f0781fcfebdcbf7fccfaeb7840d9241f9586048fadb ./CHANGELOG-GROUNDED-KNOWLEDGE-SYNTHESIS.md @@ -12,17 +13,19 @@ e167a9d64f63c3933ad40c5078684bc019db303043c34ea3965f3b089f3d32c7 ./CHANGELOG-KN 5433a7c2e67ab35fb320bc872e9024fa5f3e765736184e9f878340b8b45407aa ./CHANGELOG-SQLITE-STARTUP-FIX.md 5b9deeab0cd59b3c649fd73f129361a1e773ed3955cded0048b8cb280bb32e88 ./CHANGELOG-SQLITE-STORAGE.md 42c1cacdc1f792773622017c3528dc626674456f478c41aaf44ff0a32e5f87bd ./Dockerfile +a1aed7c198bc1ffc7af4a8f69ccf137541e4887ce5d59a0d67f2be7216a37dcd ./FILTER-SCOPES-SOURCES.md 5534536965bf0479455f97324c242160202650ca1256f1ba0420b4ad67125e49 ./GLPI-KB.md c546524a3add0b1beb8d1ca675fd602e2ea7c7bddfc04759519c21a02667eb61 ./ITERATIVE-GROUNDED-RESEARCH.md e3ae87108607ca494668a9974c467a5c529b9599bf88d3d2a79ddc15b64e4a38 ./KNOWLEDGE-SYNTHESIS.md 696d2da2338cd8190b9614707e4059d78ce291e7334f273633aad815c3b6a6df ./Makefile 381d7d6ac9e3c2e63c9ecdaa42ed4c73058f5d78e57c7532bb75a9663c919530 ./OLLAMA-POOL.md 2c0062941ef3edbd40d46b823934a7d0a3a9da7581b83d0b9360e8aaa7694b1b ./PERSISTENCE.md -892649393e1b5b0e427feb20410652c4094b8903ec3a97015a4af286d39e8834 ./README.md -dfa1c9fa47f53aa4fbe6eb8ee82f65c2a6213538af7b71a4842e40cdc271bab6 ./RUNTIME-CONTROLS-HONEYCOMB.md +1d5ef2ee13147d582d0a1fccea15884912a3b7742d3c847460bc3f660eb108f9 ./README.md +24ebc1a996f9792a9b38ba1f5dca55b217d61378af9fe9a751c1478c31fb9ea1 ./RUNTIME-CONTROLS-HONEYCOMB.md c3da43b33e550901d55789f2ee526c2e50f61ee028a3f59e0a40e77e1057fde7 ./SEARXNG-VISUALIZATION.md ae7bc1f1959071f79b76ca8a4ba103064ec0d5c5752af346175731ef4f636d9d ./SQLITE-STORAGE.md e05449bab6250e585a6c0ac0735008cd533baf07dbf7149ddae609ae252d4425 ./VALIDATION-CONSTELLATION-ECO-TRANSITIONS.md +3830269b6584e63b4d5cd3627ac5c713c0aedc80e189b35e920d1228de25ff5e ./VALIDATION-FILTER-SCOPES-SOURCES.md e7ab2cddec372db906c7883a6e0861b71999043cfb9b94e527c3b2aee4532035 ./VALIDATION-ITERATIVE-GROUNDED-RESEARCH.md e08b0eaab827fe03d5a72327e8fdfe9ce4025097e28208714246969e7d059561 ./VALIDATION-SEARXNG-DIAGNOSTICS.md 4cd120496664388717fe422a8c54380708df723fea26b35f81799665dfaf2c1c ./VALIDATION-SQLITE.md @@ -30,8 +33,8 @@ f6699e4cdbaadc720e4b8a22c557d02319325283775a2b8a87ff78ca202e3386 ./VISUALIZATIO 8970f2abf17bddcd0d84387e8a4975588df2776f03a7a54c5a283470769ca0c8 ./cmd/brain/main.go e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ./data/.gitkeep 21b51d0e1b7ed07c20f7f3a5da76dedab8df44a94a51724b67b0c3411599fe15 ./deployment/README.md -e3403fd3a7ea9e467b09f7ced5ab63c8368fa4a75370ed62aa58b1cc7006177b ./deployment/docker-compose.full.yml -ad6e2f4d47004232a09620eb83b59b582833a9901db3b2eae756febbb1abcc67 ./docker-compose.yml +70deb54b2c756cd53bd5606b9a1abc0bab8ad9194e96b6d86df79cfa8cac2903 ./deployment/docker-compose.full.yml +d7902167be92bd1d2fc2872cc69814e17f569e605ddb817cbd36b12768d2d818 ./docker-compose.yml 1542b064707bef0c4488f558f77c02db5f4cfc8d4da228f615ebdc3ee9ace8fd ./go.mod 864c3376212497b070feca13d26cfc28e96876078ce7a0b5b0c0470e2dd4fbf8 ./go.sum bf239391e61040b00d2f49d805b0d49a44bd3679f3c53849dfbc5e3b5a3fcc7a ./integrations/agent/README.md @@ -39,25 +42,28 @@ a0105475dc054977223fac36618b8cd8137c55be1d11fddcd24e9a4d3074c170 ./integrations 73ab7c49600cdaa4795e76c50e66ce919dec171b3a07f2d16ff6f45ce5f73365 ./integrations/knowledgebase/README.md 36666043ebf4139e13610e5fcd0b0c6f45e4e6a53fed33ec47d03a66878e7b08 ./integrations/knowledgebase/glpi-ai-knowledgebase-neural-brain.patch 50d05fa2a183f5f3eaab0545cb48d3abb64be62eb5d84dc2c6d99c7125f7344b ./internal/activity/broker.go -ccca255316e838042261da0329e44a4a36dd57fd44aa2a0ba5cccee6944f35a2 ./internal/config/config.go -f7d8c7f9051fa27ceeb2b23f92af2883ae659ad1dfa681c2723d010a08e977b8 ./internal/config/config_test.go -b81e53fc1e80a2d4c677e7145097473e772a7d963ffaddc7fa1eae2b2c95ebda ./internal/engine/article.go +3b23bf7119463350fc1a8454bc8e7d3713b6ee03457618634d154912c41de787 ./internal/config/config.go +c619fb8ae9c68d43ce3db2588b41de3db4407c1402220499fa55d34755fd374e ./internal/config/config_test.go +0560ae79ab5b100de402d6c25026fbacdd36d1d4cf9a17f464eaa4586a61aa52 ./internal/engine/article.go d5d18a26010fe5f308c05b79e3a3d7501c8490bb33098357cde4373affd7e60d ./internal/engine/article_format_test.go -688c71c14f1112673bac63627738cfbd713272aae748db15204e9de04bf8d10b ./internal/engine/article_research.go -1e0fc5d81f2238927842d854eb062cef08e5c39b9970f68d4a181cba67d56d47 ./internal/engine/article_research_cache.go +0563c5455889f0747c1c2dd3ec5e7b639f7102d2f8966b1ed863e401ee537450 ./internal/engine/article_research.go +501597b1b9340726e13c200099fdc10dc2a661410c76dce16f250b4dc6233871 ./internal/engine/article_research_cache.go 3ca7037231935329d49b6b80553fbfef206c079a6aed4c2379dadd46d39ebc0d ./internal/engine/article_research_cache_test.go 289d7dca4f0294072521daa229ff2dcae7f159af4631d6567853f54b920506ce ./internal/engine/article_research_test.go -fc9f71e4bb8d400ec78bdc6e81d29722ffff17ddd9026ff97889ca7f306be92a ./internal/engine/engine.go -d2c5f2b64704656a4afe856b60872a29c1d491b16289144a5052b23c2900d08c ./internal/engine/engine_test.go +94ce4c2cab802e0bc6870c58eb95bf47d699acbe233a0851ddc289520a238e41 ./internal/engine/engine.go +ecc5960311c88b4fb179dec3c2fa7662130f2163c14a258b886e682ce93eb933 ./internal/engine/engine_test.go e3595489273b2e5f6b8686d7912e1d56c968fb8d105eea58973c25496e1cf412 ./internal/engine/research_diagnostics.go 0eb3f00e2ab73d2dc6a4cbc1a4038533a4bb20fbbbb6190abd39feff6b34b8e1 ./internal/engine/research_diagnostics_test.go 716d42138db9eb64480c1bbaf4cb3b70bce1edf72c17c3d85a8d84f866fd8700 ./internal/engine/research_events.go -7eb1c0cec5e4135d8e4988093f093d4ada7d1047db7b30243a7b5122c2af2a0f ./internal/engine/runtime.go +62de2df4f031f46921f2cb7a27d29e9dbb9e8ad344a6ea78c213e98f77ff377d ./internal/engine/runtime.go +459a86b2fcbb3be6233310dd3bb6f0c3700d661251eb582f9f50f2d13f3dfcc3 ./internal/engine/runtime_filter_test.go b82980a646a92751bdd27a866ba1ffc6d34a3ba81d537f7b6e5a78e1432ee6fa ./internal/glpi/client.go 525102be56bc51ce8a08655b1b2bb53b67f4a1828903585fd664ed6a5133f617 ./internal/glpi/client_test.go +7f5028a6cb24ede68316dbe1708b5482fed44bbc3aee3347d1d5d615c9b7a826 ./internal/graph/filter.go +ee3668d71652630a3e5778f9840618e5e1a599a654abf7c679a4b735e34d2491 ./internal/graph/filter_test.go 6b3df883bb30e4d969a5aa7e4cb74ca98119ec1155f6b566555d06c270842c93 ./internal/graph/sqlite_backend.go 744d306b9c7151543e91421575775ff6eb1b7e0af187ba7d92d8a298c51704de ./internal/graph/sqlite_backend_test.go -1eff087ab94d6b94b5ae2205df0f1fcfcc3a35965808a3f504d2f96cb45459fe ./internal/graph/store.go +abe9818db121f96ea0f73437182df13ed8a51259c21dcdbab199032b5a62c3c6 ./internal/graph/store.go 11daaf15e175818ddd62d70ac415d3493d6a473a2dbb1b65a59be33a249242d3 ./internal/graph/store_test.go 4476351d388d11c6becd78b4c918fc8d47dfd70500d8b3e2ddf81f7ff61a2cea ./internal/ingest/agent.go 05bdcd8f82756af028807a9ba37e64232e2d93b6a803468d0a29665bbdb3067f ./internal/ingest/glpikb.go @@ -73,9 +79,9 @@ e7138877303bcc07c429323c4792fed112d8c16d35fd44222fe144ab0b69344f ./internal/res 6a9f269783a7c41d5b63f9bd5022f415ed1b5572041ca5ccae1512d6dc2a0b80 ./internal/research/fetch_test.go a46b953c22fb2aef112b11ba65c61bda1e1582903e39489c2d081d17519ebd9d ./internal/research/searxng.go 2f82e80ad91590f419b1bbb19647b5191596e97c4d765e8736c1867ad899d457 ./internal/research/searxng_test.go -4e55706a88d1a8c9f58678a9087add98618654fd2697f990f0f6fade187a9576 ./internal/web/server.go -cd371ec2ffc178b986e7b7cb840f9d764e3d05e7f2547b1193acc8d3c8a82276 ./internal/web/server_test.go -4a0e7f3170b0a5cc682cc3e2a4337601f7e29e603b61127e5a61c674ba52bf34 ./internal/web/static/app.css -266a7243d034e92e5956b288e3d2b8fefb69bddb7ce52024971cf98486242b9c ./internal/web/static/app.js -bab60b900385858773148b939988dfd2e45ed34f8041f8cb689281e63669aefd ./internal/web/static/index.html +53a6808587288a1396bc069518814e361c08b88034cf2715e6d14663bed424b3 ./internal/web/server.go +680fa1cc3c15fce43920f75e4592370609a0e6cc2b9ebf2a208f700612ad3d43 ./internal/web/server_test.go +48e7fed4f954232ced9cd05f90a78bcb8793df582ab21faaf3cd243c9cb7b7ea ./internal/web/static/app.css +8bc0391f849b77e6a08da13d0696f1bd122ba07aa3c58600911d922cf7871c10 ./internal/web/static/app.js +a9d35b1de4a98f76e4851fc2e0c083f62c262468e9c357fd59851daa2c84d9c7 ./internal/web/static/index.html 83aded814b6225395935e61fe957963c3c470f368fc9089f505b6de23e959115 ./preview.png diff --git a/VALIDATION-FILTER-SCOPES-SOURCES.md b/VALIDATION-FILTER-SCOPES-SOURCES.md new file mode 100644 index 0000000..87c43f3 --- /dev/null +++ b/VALIDATION-FILTER-SCOPES-SOURCES.md @@ -0,0 +1,55 @@ +# Validierung: Filter Scopes und Sources + +Stand: 2026-08-05 + +## Automatisch geprüft + +```text +node --check internal/web/static/app.js +YAML-Parsing: docker-compose.yml +YAML-Parsing: deployment/docker-compose.full.yml +go test -run '^$' ./... +go vet ./... +go test ./internal/config +go test ./internal/graph -run 'TestNodeFilter|TestScoped' +go test ./internal/engine -run 'TestEffectiveFilter|TestRuntimeJSONFieldMerge' +``` + +Die vollständige Go-Codebasis wurde typgeprüft. Für die reine Kompilierungsprüfung wurde ausschließlich außerhalb des Projekts ein minimaler `database/sql`-Treiberstub verwendet, weil die Ausführungsumgebung `modernc.org/sqlite` nicht von `proxy.golang.org` herunterladen konnte. `go.mod` und `go.sum` im ausgelieferten Projekt verwenden unverändert den echten Treiber `modernc.org/sqlite v1.37.1`. + +## Abgedeckte Backend-Fälle + +- Kategorien und Quellen werden zwischen den Dimensionen mit UND kombiniert. +- Mehrere Werte innerhalb einer Dimension verwenden ODER-Semantik. +- Leere Listen und `*` sind uneingeschränkt. +- `__uncategorized__` und `__unsourced__` treffen Nodes ohne entsprechenden Wert. +- Disjunkte Environment- und WebUI-Listen ergeben `MatchNone`. +- Environment-Grenzen bleiben erhalten, wenn eine ältere `runtime-settings.json` neue Felder noch nicht enthält. +- Embedding-Auswahl, Retrieval und AI-THINK-Kandidaten respektieren den jeweils wirksamen Scope. +- Externe Nodes werden über die Domain ihrer URL als Quelle erkannt. +- `GET /api/sources` und die erweiterten Runtime-Einstellungen sind typgeprüft. + +## Browserprüfung + +Die Weboberfläche wurde mit einem simulierten Graph-Snapshot geprüft: + +```text +passender Wissens-Node → sichtbar +passende Category-Taxonomie → sichtbar +passende Source-Taxonomie → sichtbar +fremder Wissens-Node, nur über semantische Edge verbunden → unsichtbar +``` + +Damit ist die frühere Display-Leckage über Kanten behoben. Die drei Visualisierungen verwenden denselben strikt gefilterten Snapshot. + +Zusätzlich wurden die sechs WebUI-Listen für Kategorien und Quellen, die ENV-Sperrmarkierung und die Anzeige der wirksamen Schnittmenge geprüft. + +## Nicht in dieser Umgebung ausführbar + +Ein echter SQLite-Lauf mit `modernc.org/sqlite` war wegen blockierter DNS-/Modulauflösung nicht möglich. Auf dem Zielsystem sollte deshalb zusätzlich ausgeführt werden: + +```powershell +go test ./... +go vet ./... +docker compose up -d --build --force-recreate brain +``` diff --git a/deployment/docker-compose.full.yml b/deployment/docker-compose.full.yml index ab975f6..f9c100c 100644 --- a/deployment/docker-compose.full.yml +++ b/deployment/docker-compose.full.yml @@ -135,6 +135,9 @@ services: BRAIN_LEARNING_CATEGORIES: ${BRAIN_LEARNING_CATEGORIES:-} BRAIN_DISPLAY_CATEGORIES: ${BRAIN_DISPLAY_CATEGORIES:-} BRAIN_THINKING_CATEGORIES: ${BRAIN_THINKING_CATEGORIES:-} + BRAIN_LEARNING_SOURCES: ${BRAIN_LEARNING_SOURCES:-} + BRAIN_DISPLAY_SOURCES: ${BRAIN_DISPLAY_SOURCES:-} + BRAIN_THINKING_SOURCES: ${BRAIN_THINKING_SOURCES:-} BRAIN_DEFAULT_VIEW: ${BRAIN_DEFAULT_VIEW:-neural} BRAIN_MAX_DISPLAY_NODES: ${BRAIN_MAX_DISPLAY_NODES:-0} BRAIN_LOW_POWER_MODE: ${BRAIN_LOW_POWER_MODE:-false} diff --git a/docker-compose.yml b/docker-compose.yml index e577f64..232eb20 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -34,6 +34,9 @@ services: BRAIN_LEARNING_CATEGORIES: ${BRAIN_LEARNING_CATEGORIES:-} BRAIN_DISPLAY_CATEGORIES: ${BRAIN_DISPLAY_CATEGORIES:-} BRAIN_THINKING_CATEGORIES: ${BRAIN_THINKING_CATEGORIES:-} + BRAIN_LEARNING_SOURCES: ${BRAIN_LEARNING_SOURCES:-} + BRAIN_DISPLAY_SOURCES: ${BRAIN_DISPLAY_SOURCES:-} + BRAIN_THINKING_SOURCES: ${BRAIN_THINKING_SOURCES:-} BRAIN_DEFAULT_VIEW: ${BRAIN_DEFAULT_VIEW:-neural} BRAIN_MAX_DISPLAY_NODES: ${BRAIN_MAX_DISPLAY_NODES:-0} BRAIN_LOW_POWER_MODE: ${BRAIN_LOW_POWER_MODE:-false} diff --git a/internal/config/config.go b/internal/config/config.go index fd854fc..ff26a69 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -68,6 +68,9 @@ type Config struct { LearningCategories []string DisplayCategories []string ThinkingCategories []string + LearningSources []string + DisplaySources []string + ThinkingSources []string DefaultView string MaxDisplayNodes int LowPowerMode bool @@ -161,6 +164,9 @@ func Load() (Config, error) { LearningCategories: stringList("BRAIN_LEARNING_CATEGORIES"), DisplayCategories: stringList("BRAIN_DISPLAY_CATEGORIES"), ThinkingCategories: stringList("BRAIN_THINKING_CATEGORIES"), + LearningSources: stringList("BRAIN_LEARNING_SOURCES"), + DisplaySources: stringList("BRAIN_DISPLAY_SOURCES"), + ThinkingSources: stringList("BRAIN_THINKING_SOURCES"), DefaultView: strings.ToLower(env("BRAIN_DEFAULT_VIEW", "neural")), MaxDisplayNodes: integer("BRAIN_MAX_DISPLAY_NODES", 0), LowPowerMode: boolean("BRAIN_LOW_POWER_MODE", false), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 866d81f..c9e5385 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -45,6 +45,9 @@ func TestLoadRuntimeControlDefaultsAndFilters(t *testing.T) { t.Setenv("BRAIN_LEARNING_CATEGORIES", "Netzwerk,GLPI KB") t.Setenv("BRAIN_DISPLAY_CATEGORIES", "GLPI KB") t.Setenv("BRAIN_THINKING_CATEGORIES", "Netzwerk") + t.Setenv("BRAIN_LEARNING_SOURCES", "GLPI Knowledge Base,internal-kb") + t.Setenv("BRAIN_DISPLAY_SOURCES", "GLPI Knowledge Base") + t.Setenv("BRAIN_THINKING_SOURCES", "internal-kb") t.Setenv("BRAIN_DEFAULT_VIEW", "constellation") t.Setenv("BRAIN_MAX_DISPLAY_NODES", "12000") t.Setenv("BRAIN_LOW_POWER_MODE", "true") @@ -58,6 +61,9 @@ func TestLoadRuntimeControlDefaultsAndFilters(t *testing.T) { if len(cfg.LearningCategories) != 2 || len(cfg.DisplayCategories) != 1 || len(cfg.ThinkingCategories) != 1 { t.Fatalf("unexpected category defaults: %+v", cfg) } + if len(cfg.LearningSources) != 2 || len(cfg.DisplaySources) != 1 || len(cfg.ThinkingSources) != 1 { + t.Fatalf("unexpected source defaults: %+v", cfg) + } } func TestLoadRejectsInvalidMaxDisplayNodes(t *testing.T) { diff --git a/internal/engine/article.go b/internal/engine/article.go index 7d35e70..ab8aa5e 100644 --- a/internal/engine/article.go +++ b/internal/engine/article.go @@ -55,7 +55,8 @@ func (e *Engine) synthesizeKnowledgeArticle(ctx context.Context, trigger string, // Previously accepted full-text evidence is already learned knowledge. It must // influence the create/update/merge decision, otherwise a later cycle could // skip before it ever sees the external facts it learned in an earlier cycle. - planningResearch := uniqueResearchEvidence(append(filterUsableResearchEvidence(initialResearch), e.researchEvidenceForSources(sources)...)) + initialResearch = e.filterResearchEvidenceForThinking(filterUsableResearchEvidence(initialResearch), categoriesFromArticleSources(sources)) + planningResearch := uniqueResearchEvidence(append(initialResearch, e.researchEvidenceForSources(sources)...)) e.Broker.Publish(model.Activity{Type: "article.plan.started", Source: "brain", Phase: "knowledge-planning", NodeIDs: nodeIDsFromArticleSources(sources), Message: fmt.Sprintf("%d Quellen werden auf einen echten Wissensmehrwert geprüft", len(sources)), Strength: .84, Metadata: map[string]any{"trigger": trigger, "productive_sources": productionCount, "ai_sources": aiCount, "production_ratio": productionRatio, "generation_depth": generationDepth, "model": e.Cfg.ChatModel, "learned_research_sources": len(planningResearch)}}) var plan model.ArticlePlanDecision @@ -85,7 +86,7 @@ func (e *Engine) synthesizeKnowledgeArticle(ctx context.Context, trigger string, return articleSynthesisOutcome{Skipped: true, Reason: "equivalent_staging_draft", Action: plan.Action}, nil } - newResearchResults := filterUsableResearchEvidence(initialResearch) + newResearchResults := initialResearch if len(newResearchResults) > 0 { e.addResearchToSources(selected, newResearchResults) e.learnResearchEvidence(ctx, newResearchResults) @@ -184,13 +185,13 @@ func (e *Engine) selectArticleSources(seeds []model.Node) []articleSource { direct[edge.Source] += math.Max(.2, math.Max(edge.Confidence, edge.Weight)) } } - filters := e.thinkingCategories() + filter := e.effectiveThinkingFilter() var production, ai []articleSource for _, node := range snapshot.Nodes { if node.Kind != "knowledge" && node.Kind != "ai-think" { continue } - if !matchesCategories(node, filters) { + if !filter.Matches(node) { continue } depth := nodeGenerationDepth(node) @@ -1093,7 +1094,7 @@ func (e *Engine) addRuntimeArticleNode(articleID string, sources []articleSource ID: nodeID, Kind: "ai-think", Label: draft.Title, Summary: clamp(strings.TrimSpace(draft.Text)+"\n\n"+formatArticleAnswer(draft), 1400), Status: "staging", Origin: "knowledge-staging", ExternalID: articleID, URI: "brain://article/" + articleID, Categories: unique(append([]string{"AI-THINK", "AI-Staging", "AI-Synthesis"}, draft.Categories...)), Keywords: unique(draft.Keywords), Weight: 1.45, - Metadata: map[string]any{"subtype": "knowledge_synthesis", "action": plan.Action, "target_node_id": plan.TargetArticleID, "generation_depth": generationDepth, "confidence": draft.Confidence, "source_node_ids": nodeIDsFromArticleSources(sources), "productive_source_count": productionCount, "ai_source_count": aiCount, "production_ratio": productionRatio}, UpdatedAt: now, + Metadata: map[string]any{"subtype": "knowledge_synthesis", "action": plan.Action, "target_node_id": plan.TargetArticleID, "generation_depth": generationDepth, "confidence": draft.Confidence, "source_node_ids": nodeIDsFromArticleSources(sources), "productive_source_count": productionCount, "ai_source_count": aiCount, "production_ratio": productionRatio, "source": "Neural Brain / " + e.Cfg.ChatModel + " (Knowledge Synthesis)"}, UpdatedAt: now, } e.Graph.UpsertNode(node) for _, source := range sources { @@ -1116,7 +1117,7 @@ func (e *Engine) learnRuntimeArticle(ctx context.Context, articleID string) { } nodeID := graph.ID("knowledge", articleID) node, ok := e.Graph.GetNode(nodeID) - if !ok || !matchesCategories(node, e.learningCategories()) { + if !ok || !e.effectiveLearningFilter().Matches(node) { return } text := embeddingText(node) @@ -1208,6 +1209,18 @@ func researchResultNode(id string, result model.ResearchResult, evidencePath, co return model.Node{ID: id, Kind: "external", Label: result.Title, Summary: clamp(content, 1800), Status: "research", Origin: "research", ExternalID: result.URL, URI: result.URL, Categories: unique(categories), Weight: weight, Metadata: metadata, UpdatedAt: time.Now().UTC()} } +func (e *Engine) filterResearchEvidenceForThinking(results []model.ResearchResult, categories []string) []model.ResearchResult { + filter := e.effectiveThinkingFilter() + out := make([]model.ResearchResult, 0, len(results)) + for _, result := range results { + node := model.Node{Kind: "external", Origin: "research", URI: result.URL, ExternalID: result.URL, Categories: categories} + if filter.Matches(node) { + out = append(out, result) + } + } + return out +} + func formatArticleAnswer(draft model.KnowledgeArticleDraft) string { var b strings.Builder b.WriteString(strings.TrimSpace(draft.Answer)) @@ -1432,25 +1445,7 @@ func categoryAffinity(node model.Node, seeds []model.Node) float64 { } func matchesCategories(node model.Node, filters []string) bool { - if len(filters) == 0 { - return true - } - wanted := map[string]bool{} - for _, filter := range filters { - wanted[strings.ToLower(strings.TrimSpace(filter))] = true - } - if wanted["*"] { - return true - } - if len(node.Categories) == 0 { - return wanted["__uncategorized__"] - } - for _, category := range node.Categories { - if wanted[strings.ToLower(strings.TrimSpace(category))] { - return true - } - } - return false + return (graph.NodeFilter{Categories: filters}).Matches(node) } func metadataString(metadata map[string]any, key string) string { diff --git a/internal/engine/article_research.go b/internal/engine/article_research.go index 9f89834..bc8fc76 100644 --- a/internal/engine/article_research.go +++ b/internal/engine/article_research.go @@ -479,6 +479,8 @@ func (e *Engine) executeArticleResearchQuery(ctx context.Context, trigger string assessed := e.rankResearchCandidates(ctx, question, fetched, true) accepted := make([]model.ResearchResult, 0, len(assessed)) + thinkingFilter := e.effectiveThinkingFilter() + researchCategories := e.categoriesForNodeIDs(nodeIDs) for _, candidate := range assessed { item := candidate.Result assessment := candidate.Assessment @@ -493,6 +495,15 @@ func (e *Engine) executeArticleResearchQuery(ctx context.Context, trigger string if question.ExpectActionable && !assessment.Actionable { acceptedByGate = false } + researchNode := model.Node{Kind: "external", Origin: "research", URI: item.URL, ExternalID: item.URL, Categories: researchCategories} + if !thinkingFilter.Matches(researchNode) { + acceptedByGate = false + if strings.TrimSpace(item.AssessmentReason) == "" { + item.AssessmentReason = "Die Quelle liegt außerhalb des wirksamen Thinking-Quellenfilters." + } else { + item.AssessmentReason += " · außerhalb des wirksamen Thinking-Quellenfilters" + } + } metadata := mergeResearchMetadata(startMetadata, map[string]any{"result_url": item.URL, "result_title": item.Title, "relevance": item.Relevance, "source_quality": item.SourceQuality, "source_quality_score": item.SourceQualityScore, "actionable": item.Actionable, "covered_gap_ids": item.CoveredGapIDs, "assessment_reason": item.AssessmentReason}) if !acceptedByGate { stats.Rejected++ diff --git a/internal/engine/article_research_cache.go b/internal/engine/article_research_cache.go index 502f978..7f85f92 100644 --- a/internal/engine/article_research_cache.go +++ b/internal/engine/article_research_cache.go @@ -86,8 +86,9 @@ func (e *Engine) researchEvidenceForSources(sources []articleSource) []model.Res return nil } nodes := make([]model.Node, 0, len(externalIDs)) + filter := e.effectiveThinkingFilter() for _, node := range snapshot.Nodes { - if externalIDs[node.ID] && node.Kind == "external" { + if externalIDs[node.ID] && node.Kind == "external" && filter.Matches(node) { nodes = append(nodes, node) } } @@ -199,7 +200,7 @@ func (e *Engine) learnResearchEvidence(ctx context.Context, results []model.Rese for _, result := range results { id := graph.ID("external", result.URL) node, ok := e.Graph.GetNode(id) - if !ok || !matchesCategories(node, e.learningCategories()) { + if !ok || !e.effectiveLearningFilter().Matches(node) { continue } content := strings.TrimSpace(result.Content) diff --git a/internal/engine/engine.go b/internal/engine/engine.go index 27d05ee..9840cad 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -425,7 +425,7 @@ func (e *Engine) Scan(ctx context.Context) error { if err != nil { return err } - pendingEmbeddings := len(e.Graph.NodesForEmbeddingFiltered(e.learningCategories())) + pendingEmbeddings := len(e.Graph.NodesForEmbeddingScoped(e.effectiveLearningFilter())) if e.Graph.Version() != beforeVersion || pendingEmbeddings > 0 { e.Broker.Publish(model.Activity{Type: "scan.started", Source: "brain", Phase: "ingest", Message: "Neue oder geänderte Wissenselemente werden verarbeitet", Strength: .45, Metadata: map[string]any{"pending_embeddings": pendingEmbeddings}}) } @@ -461,7 +461,7 @@ func (e *Engine) Scan(ctx context.Context) error { return nil } func (e *Engine) ensureEmbeddings(ctx context.Context) error { - pending := e.Graph.NodesForEmbeddingFiltered(e.learningCategories()) + pending := e.Graph.NodesForEmbeddingScoped(e.effectiveLearningFilter()) if len(pending) == 0 { return nil } @@ -492,7 +492,7 @@ func (e *Engine) ensureEmbeddings(ctx context.Context) error { return nil } func (e *Engine) ensureFallbackEmbeddings() { - for _, n := range e.Graph.NodesForEmbeddingFiltered(e.learningCategories()) { + for _, n := range e.Graph.NodesForEmbeddingScoped(e.effectiveLearningFilter()) { e.Graph.SetVector(n.ID, hashEmbedding(embeddingText(n), 256)) } } @@ -538,7 +538,7 @@ func (e *Engine) Query(ctx context.Context, q string) (model.QueryResponse, erro if err != nil || len(vecs) == 0 { vecs = [][]float64{hashEmbedding(q, 256)} } - hits := e.Graph.Similar(vecs[0], e.Cfg.TopK) + hits := e.Graph.SimilarFiltered(vecs[0], e.Cfg.TopK, e.effectiveLearningFilter()) nodeIDs := make([]string, 0, len(hits)) for i, h := range hits { nodeIDs = append(nodeIDs, h.NodeID) @@ -648,7 +648,7 @@ func (e *Engine) enrichOne(ctx context.Context, trigger string) (EnrichOutcome, e.setOllamaOK(true) } - a, b, sim, ok, comparisons := e.Graph.NextPairFilteredDepth(e.Cfg.SimilarityThreshold, e.Cfg.EnrichAnchors, e.thinkingCategories(), e.Cfg.ArticleMaxGenerationDepth) + a, b, sim, ok, comparisons := e.Graph.NextPairScopedDepth(e.Cfg.SimilarityThreshold, e.Cfg.EnrichAnchors, e.effectiveThinkingFilter(), e.Cfg.ArticleMaxGenerationDepth) if !ok { e.stateMu.Lock() e.lastAttempt = time.Now().UTC() @@ -685,20 +685,23 @@ func (e *Engine) enrichOne(ctx context.Context, trigger string) (EnrichOutcome, slog.Warn("research failed", "query", decision.ResearchQuery, "base_url", diagnostic.BaseURL, "kind", diagnostic.ErrorKind, "http_status", diagnostic.HTTPStatus, "duration_ms", diagnostic.DurationMS, "error", err) e.Broker.Publish(model.Activity{Type: "research.failed", Source: "searxng", Phase: "research", NodeIDs: []string{a.ID, b.ID}, Message: "SearXNG-Recherche ist fehlgeschlagen", Strength: .35, Metadata: metadata}) } else { - resultMetadata := mergeResearchMetadata(researchEventMetadata(trigger, researchID, decision.ResearchQuery, results, time.Since(researchStarted)), researchDiagnosticMetadata(diagnostic)) - message := fmt.Sprintf("SearXNG hat %d verwertbare Webquellen geliefert", len(results)) - if len(results) == 0 { - message = "SearXNG hat keine verwertbaren Webquellen geliefert" + allowedResults := e.filterResearchEvidenceForThinking(results, unique(append(append([]string{}, a.Categories...), b.Categories...))) + resultMetadata := mergeResearchMetadata(researchEventMetadata(trigger, researchID, decision.ResearchQuery, allowedResults, time.Since(researchStarted)), researchDiagnosticMetadata(diagnostic)) + resultMetadata["unfiltered_result_count"] = len(results) + resultMetadata["source_filter_rejected_count"] = len(results) - len(allowedResults) + message := fmt.Sprintf("SearXNG hat %d durch den Thinking-Filter erlaubte Webquellen geliefert", len(allowedResults)) + if len(allowedResults) == 0 { + message = "SearXNG-Treffer lagen außerhalb des wirksamen Thinking-Quellenfilters" } e.Broker.Publish(model.Activity{Type: "research.results", Source: "searxng", Phase: "research-results", NodeIDs: []string{a.ID, b.ID}, Message: message, Strength: .92, Metadata: resultMetadata}) - if len(results) > 0 { - researchResults = results - refs := e.addResearch(a, b, results) + if len(allowedResults) > 0 { + researchResults = allowedResults + refs := e.addResearch(a, b, allowedResults) ingestMetadata := mergeResearchMetadata(resultMetadata, map[string]any{"result_node_ids": refs.NodeIDs, "result_edge_ids": refs.EdgeIDs}) e.Broker.Publish(model.Activity{Type: "research.ingested", Source: "searxng", Phase: "research-ingest", NodeIDs: append([]string{a.ID, b.ID}, refs.NodeIDs...), EdgeIDs: refs.EdgeIDs, Message: fmt.Sprintf("%d Webquellen wurden als neue Forschungs-Nodes in den Graphen übernommen", len(refs.NodeIDs)), Strength: 1, Metadata: ingestMetadata}) var reviewed model.RelationDecision reviewSystem := "Bewerte die Beziehung erneut anhand der zwei internen Wissenseinträge und der beigefügten Web-Suchergebnisse. Suchtreffer sind Hinweise, keine garantierten Fakten. Erfinde nichts, kennzeichne verbleibende Unsicherheit und gib ausschließlich JSON nach Schema zurück." - if err := e.Ollama.ChatJSON(ctx, reviewSystem, relationContextWithResearch(a, b, sim, results), relationSchema(), &reviewed); err != nil { + if err := e.Ollama.ChatJSON(ctx, reviewSystem, relationContextWithResearch(a, b, sim, allowedResults), relationSchema(), &reviewed); err != nil { slog.Warn("research review failed; keeping pre-research decision", "error", err) } else { decision = reviewed @@ -743,7 +746,7 @@ func (e *Engine) addResearch(a, b model.Node, results []model.ResearchResult) re refs := researchGraphRefs{} for _, r := range results { id := graph.ID("external", r.URL) - n := model.Node{ID: id, Kind: "external", Label: r.Title, Summary: clamp(r.Content, 700), Status: "research", Origin: "research", ExternalID: r.URL, URI: r.URL, Weight: .8, Metadata: map[string]any{"query_pair": []string{a.ID, b.ID}}, UpdatedAt: time.Now().UTC()} + n := model.Node{ID: id, Kind: "external", Label: r.Title, Summary: clamp(r.Content, 700), Status: "research", Origin: "research", ExternalID: r.URL, URI: r.URL, Categories: unique(append(append([]string{}, a.Categories...), b.Categories...)), Weight: .8, Metadata: map[string]any{"query_pair": []string{a.ID, b.ID}}, UpdatedAt: time.Now().UTC()} e.Graph.UpsertNode(n) refs.NodeIDs = append(refs.NodeIDs, id) for _, targetID := range []string{a.ID, b.ID} { @@ -777,7 +780,7 @@ func (e *Engine) Status() map[string]any { "research_enabled": e.Cfg.ResearchEnabled, "chat_model": e.Cfg.ChatModel, "embedding_model": e.Cfg.EmbeddingModel, "searxng": e.ResearchStatus(), "ollama_pool": e.Ollama.PoolStatus(), "persistence": e.Persistence.Status(), "graph_storage": e.Graph.StorageStatus(), - "runtime_settings": e.RuntimeSettings(), + "runtime_settings": e.RuntimeSettingsView(), } if e.GLPIKB != nil { status["glpi_kb"] = e.GLPIKB.Status() diff --git a/internal/engine/engine_test.go b/internal/engine/engine_test.go index 9bc411c..7a93f85 100644 --- a/internal/engine/engine_test.go +++ b/internal/engine/engine_test.go @@ -392,6 +392,9 @@ func TestRuntimeSettingsDisableThinkingAndPersist(t *testing.T) { LearningCategories: []string{"Netzwerk"}, DisplayCategories: []string{"GLPI KB"}, ThinkingCategories: []string{"Netzwerk"}, + LearningSources: []string{"internal-kb"}, + DisplaySources: []string{"GLPI Knowledge Base"}, + ThinkingSources: []string{"internal-kb"}, ViewMode: "constellation", MaxDisplayNodes: 4321, LowPowerMode: true, @@ -418,7 +421,7 @@ func TestRuntimeSettingsDisableThinkingAndPersist(t *testing.T) { } e2 := New(cfg, g2, activity.New(20)) loaded := e2.RuntimeSettings() - if loaded.LearningEnabled || loaded.ThinkingEnabled || loaded.ViewMode != "constellation" || loaded.MaxDisplayNodes != 4321 || !loaded.LowPowerMode || len(loaded.DisplayCategories) != 1 { + if loaded.LearningEnabled || loaded.ThinkingEnabled || loaded.ViewMode != "constellation" || loaded.MaxDisplayNodes != 4321 || !loaded.LowPowerMode || len(loaded.DisplayCategories) != 1 || len(loaded.DisplaySources) != 1 { t.Fatalf("runtime settings were not restored: %+v", loaded) } } diff --git a/internal/engine/runtime.go b/internal/engine/runtime.go index 62d2e73..2284150 100644 --- a/internal/engine/runtime.go +++ b/internal/engine/runtime.go @@ -7,10 +7,12 @@ import ( "sort" "strings" + "github.com/local/glpi-neural-brain/internal/graph" "github.com/local/glpi-neural-brain/internal/model" ) -const uncategorizedFilter = "__uncategorized__" +const uncategorizedFilter = graph.UncategorizedFilter +const unsourcedFilter = graph.UnsourcedFilter type RuntimeSettings struct { LearningEnabled bool `json:"learning_enabled"` @@ -18,33 +20,59 @@ type RuntimeSettings struct { LearningCategories []string `json:"learning_categories"` DisplayCategories []string `json:"display_categories"` ThinkingCategories []string `json:"thinking_categories"` + LearningSources []string `json:"learning_sources"` + DisplaySources []string `json:"display_sources"` + ThinkingSources []string `json:"thinking_sources"` ViewMode string `json:"view_mode"` MaxDisplayNodes int `json:"max_display_nodes"` LowPowerMode bool `json:"low_power_mode"` } +type RuntimeFilterInfo struct { + Categories []string `json:"categories"` + Sources []string `json:"sources"` + CategoriesRestricted bool `json:"categories_restricted"` + SourcesRestricted bool `json:"sources_restricted"` + MatchesNone bool `json:"matches_none"` +} + +type RuntimeSettingsView struct { + RuntimeSettings + EffectiveLearning RuntimeFilterInfo `json:"effective_learning"` + EffectiveDisplay RuntimeFilterInfo `json:"effective_display"` + EffectiveThinking RuntimeFilterInfo `json:"effective_thinking"` + AdminLearning RuntimeFilterInfo `json:"admin_learning"` + AdminDisplay RuntimeFilterInfo `json:"admin_display"` + AdminThinking RuntimeFilterInfo `json:"admin_thinking"` +} + type CategoryInfo struct { Name string `json:"name"` Count int `json:"count"` } +type SourceInfo = CategoryInfo + func (e *Engine) defaultRuntimeSettings() RuntimeSettings { + // Category/source values in Config are administrative ceilings. The WebUI + // starts at "all allowed" (empty selection), not by copying the ceiling into + // mutable runtime state. return normalizeRuntimeSettings(RuntimeSettings{ - LearningEnabled: e.Cfg.LearningEnabled, - ThinkingEnabled: e.Cfg.ThinkingEnabled, - LearningCategories: append([]string(nil), e.Cfg.LearningCategories...), - DisplayCategories: append([]string(nil), e.Cfg.DisplayCategories...), - ThinkingCategories: append([]string(nil), e.Cfg.ThinkingCategories...), - ViewMode: e.Cfg.DefaultView, - MaxDisplayNodes: e.Cfg.MaxDisplayNodes, - LowPowerMode: e.Cfg.LowPowerMode, + LearningEnabled: e.Cfg.LearningEnabled, + ThinkingEnabled: e.Cfg.ThinkingEnabled, + ViewMode: e.Cfg.DefaultView, + MaxDisplayNodes: e.Cfg.MaxDisplayNodes, + LowPowerMode: e.Cfg.LowPowerMode, }) } func normalizeRuntimeSettings(in RuntimeSettings) RuntimeSettings { - in.LearningCategories = normalizeCategories(in.LearningCategories) - in.DisplayCategories = normalizeCategories(in.DisplayCategories) - in.ThinkingCategories = normalizeCategories(in.ThinkingCategories) + in.LearningCategories = normalizeValues(in.LearningCategories) + in.DisplayCategories = normalizeValues(in.DisplayCategories) + in.ThinkingCategories = normalizeValues(in.ThinkingCategories) + in.LearningSources = normalizeValues(in.LearningSources) + in.DisplaySources = normalizeValues(in.DisplaySources) + in.ThinkingSources = normalizeValues(in.ThinkingSources) in.ViewMode = strings.ToLower(strings.TrimSpace(in.ViewMode)) if in.ViewMode == "" { in.ViewMode = "neural" @@ -61,7 +89,7 @@ func normalizeRuntimeSettings(in RuntimeSettings) RuntimeSettings { return in } -func normalizeCategories(values []string) []string { +func normalizeValues(values []string) []string { seen := map[string]string{} for _, value := range values { value = strings.TrimSpace(value) @@ -81,21 +109,47 @@ func normalizeCategories(values []string) []string { return out } +func normalizeCategories(values []string) []string { return normalizeValues(values) } + +// loadRuntimeSettings merges individual persisted fields. Older files that do +// not contain newly introduced fields inherit defaults instead of silently +// zeroing unrelated settings. func (e *Engine) loadRuntimeSettings() { settings := e.defaultRuntimeSettings() if strings.TrimSpace(e.runtimePath) != "" { if data, err := os.ReadFile(e.runtimePath); err == nil { - var stored RuntimeSettings - if json.Unmarshal(data, &stored) == nil { - settings = normalizeRuntimeSettings(stored) - } + mergeRuntimeSettingsJSON(&settings, data) } } + settings = normalizeRuntimeSettings(settings) e.runtimeMu.Lock() e.runtime = settings e.runtimeMu.Unlock() } +func mergeRuntimeSettingsJSON(settings *RuntimeSettings, data []byte) { + var raw map[string]json.RawMessage + if json.Unmarshal(data, &raw) != nil { + return + } + decode := func(key string, target any) { + if value, ok := raw[key]; ok { + _ = json.Unmarshal(value, target) + } + } + decode("learning_enabled", &settings.LearningEnabled) + decode("thinking_enabled", &settings.ThinkingEnabled) + decode("learning_categories", &settings.LearningCategories) + decode("display_categories", &settings.DisplayCategories) + decode("thinking_categories", &settings.ThinkingCategories) + decode("learning_sources", &settings.LearningSources) + decode("display_sources", &settings.DisplaySources) + decode("thinking_sources", &settings.ThinkingSources) + decode("view_mode", &settings.ViewMode) + decode("max_display_nodes", &settings.MaxDisplayNodes) + decode("low_power_mode", &settings.LowPowerMode) +} + func (e *Engine) RuntimeSettings() RuntimeSettings { e.runtimeMu.RLock() settings := e.runtime @@ -103,9 +157,58 @@ func (e *Engine) RuntimeSettings() RuntimeSettings { settings.LearningCategories = append([]string{}, settings.LearningCategories...) settings.DisplayCategories = append([]string{}, settings.DisplayCategories...) settings.ThinkingCategories = append([]string{}, settings.ThinkingCategories...) + settings.LearningSources = append([]string{}, settings.LearningSources...) + settings.DisplaySources = append([]string{}, settings.DisplaySources...) + settings.ThinkingSources = append([]string{}, settings.ThinkingSources...) return settings } +func (e *Engine) RuntimeSettingsView() RuntimeSettingsView { + settings := e.RuntimeSettings() + return RuntimeSettingsView{ + RuntimeSettings: settings, + EffectiveLearning: filterInfo(e.effectiveLearningFilter()), + EffectiveDisplay: filterInfo(e.effectiveDisplayFilter()), + EffectiveThinking: filterInfo(e.effectiveThinkingFilter()), + AdminLearning: adminFilterInfo(e.Cfg.LearningCategories, e.Cfg.LearningSources), + AdminDisplay: adminFilterInfo(e.Cfg.DisplayCategories, e.Cfg.DisplaySources), + AdminThinking: adminFilterInfo(e.Cfg.ThinkingCategories, e.Cfg.ThinkingSources), + } +} + +func filterInfo(filter graph.NodeFilter) RuntimeFilterInfo { + return RuntimeFilterInfo{ + Categories: append([]string{}, filter.Categories...), + Sources: append([]string{}, filter.Sources...), + CategoriesRestricted: dimensionRestricted(filter.Categories) || filter.MatchNone, + SourcesRestricted: dimensionRestricted(filter.Sources) || filter.MatchNone, + MatchesNone: filter.MatchNone, + } +} + +func adminFilterInfo(categories, sources []string) RuntimeFilterInfo { + categories = normalizeValues(categories) + sources = normalizeValues(sources) + return RuntimeFilterInfo{ + Categories: categories, + Sources: sources, + CategoriesRestricted: dimensionRestricted(categories), + SourcesRestricted: dimensionRestricted(sources), + } +} + +func dimensionRestricted(values []string) bool { + if len(values) == 0 { + return false + } + for _, value := range values { + if strings.TrimSpace(value) == "*" { + return false + } + } + return true +} + func (e *Engine) SetRuntimeSettings(settings RuntimeSettings) (RuntimeSettings, error) { if settings.MaxDisplayNodes < 0 || settings.MaxDisplayNodes > 500000 { return e.RuntimeSettings(), fmt.Errorf("max_display_nodes must be between 0 and 500000") @@ -137,21 +240,28 @@ func (e *Engine) SetRuntimeSettings(settings RuntimeSettings) (RuntimeSettings, } } + view := e.RuntimeSettingsView() e.Broker.Publish(model.Activity{ Type: "runtime.settings.updated", Source: "ui", Phase: "control", - Message: "Lern-, Anzeige- und Thinking-Einstellungen wurden aktualisiert", + Message: "Lern-, Anzeige-, Quellen- und Thinking-Einstellungen wurden aktualisiert", Strength: .32, Metadata: map[string]any{ - "learning_enabled": settings.LearningEnabled, - "thinking_enabled": settings.ThinkingEnabled, - "learning_categories": len(settings.LearningCategories), - "display_categories": len(settings.DisplayCategories), - "thinking_categories": len(settings.ThinkingCategories), - "view_mode": settings.ViewMode, - "max_display_nodes": settings.MaxDisplayNodes, - "low_power_mode": settings.LowPowerMode, + "learning_enabled": settings.LearningEnabled, + "thinking_enabled": settings.ThinkingEnabled, + "learning_categories": len(settings.LearningCategories), + "display_categories": len(settings.DisplayCategories), + "thinking_categories": len(settings.ThinkingCategories), + "learning_sources": len(settings.LearningSources), + "display_sources": len(settings.DisplaySources), + "thinking_sources": len(settings.ThinkingSources), + "learning_filter_matches_none": view.EffectiveLearning.MatchesNone, + "display_filter_matches_none": view.EffectiveDisplay.MatchesNone, + "thinking_filter_matches_none": view.EffectiveThinking.MatchesNone, + "view_mode": settings.ViewMode, + "max_display_nodes": settings.MaxDisplayNodes, + "low_power_mode": settings.LowPowerMode, }, }) return settings, nil @@ -171,20 +281,60 @@ func (e *Engine) ThinkingEnabled() bool { return enabled } -func (e *Engine) learningCategories() []string { - e.runtimeMu.RLock() - out := append([]string(nil), e.runtime.LearningCategories...) - e.runtimeMu.RUnlock() - return out +func (e *Engine) effectiveLearningFilter() graph.NodeFilter { + settings := e.RuntimeSettings() + return effectiveNodeFilter(e.Cfg.LearningCategories, settings.LearningCategories, e.Cfg.LearningSources, settings.LearningSources) } -func (e *Engine) thinkingCategories() []string { - e.runtimeMu.RLock() - out := append([]string(nil), e.runtime.ThinkingCategories...) - e.runtimeMu.RUnlock() - return out +func (e *Engine) effectiveDisplayFilter() graph.NodeFilter { + settings := e.RuntimeSettings() + return effectiveNodeFilter(e.Cfg.DisplayCategories, settings.DisplayCategories, e.Cfg.DisplaySources, settings.DisplaySources) } +func (e *Engine) effectiveThinkingFilter() graph.NodeFilter { + settings := e.RuntimeSettings() + return effectiveNodeFilter(e.Cfg.ThinkingCategories, settings.ThinkingCategories, e.Cfg.ThinkingSources, settings.ThinkingSources) +} + +func effectiveNodeFilter(adminCategories, runtimeCategories, adminSources, runtimeSources []string) graph.NodeFilter { + categories, categoryNone := intersectFilterValues(adminCategories, runtimeCategories) + sources, sourceNone := intersectFilterValues(adminSources, runtimeSources) + return graph.NodeFilter{Categories: categories, Sources: sources, MatchNone: categoryNone || sourceNone} +} + +func intersectFilterValues(admin, runtime []string) ([]string, bool) { + admin = normalizeValues(admin) + runtime = normalizeValues(runtime) + adminRestricted := dimensionRestricted(admin) + runtimeRestricted := dimensionRestricted(runtime) + if !adminRestricted && !runtimeRestricted { + return nil, false + } + if adminRestricted && !runtimeRestricted { + return admin, false + } + if !adminRestricted && runtimeRestricted { + return runtime, false + } + allowed := make(map[string]string, len(admin)) + for _, value := range admin { + allowed[strings.ToLower(strings.TrimSpace(value))] = value + } + intersection := make([]string, 0) + for _, value := range runtime { + if canonical, ok := allowed[strings.ToLower(strings.TrimSpace(value))]; ok { + intersection = append(intersection, canonical) + } + } + intersection = normalizeValues(intersection) + return intersection, len(intersection) == 0 +} + +// Compatibility helpers retained for tests and internal callers that only need +// the category projection. New code should use the scoped filters above. +func (e *Engine) learningCategories() []string { return e.effectiveLearningFilter().Categories } +func (e *Engine) thinkingCategories() []string { return e.effectiveThinkingFilter().Categories } + func (e *Engine) Categories() []CategoryInfo { snapshot := e.Graph.Snapshot() counts := map[string]int{} @@ -222,11 +372,46 @@ func (e *Engine) Categories() []CategoryInfo { if uncategorized > 0 { out = append(out, CategoryInfo{Name: uncategorizedFilter, Count: uncategorized}) } + sortFilterInfo(out) + return out +} + +func (e *Engine) Sources() []SourceInfo { + snapshot := e.Graph.Snapshot() + counts := map[string]int{} + names := map[string]string{} + unsourced := 0 + for _, node := range snapshot.Nodes { + if node.Kind != "knowledge" && node.Kind != "ai-think" && node.Kind != "external" { + continue + } + source := strings.TrimSpace(graph.NodeSource(node)) + if source == "" { + unsourced++ + continue + } + key := strings.ToLower(source) + if _, ok := names[key]; !ok { + names[key] = source + } + counts[key]++ + } + result := make([]SourceInfo, 0, len(counts)+1) + for key, count := range counts { + result = append(result, SourceInfo{Name: names[key], Count: count}) + } + if unsourced > 0 { + result = append(result, SourceInfo{Name: unsourcedFilter, Count: unsourced}) + } + sortFilterInfo(result) + return result +} + +func sortFilterInfo(out []CategoryInfo) { sort.Slice(out, func(i, j int) bool { if out[i].Count == out[j].Count { return strings.ToLower(out[i].Name) < strings.ToLower(out[j].Name) } return out[i].Count > out[j].Count }) - return out } diff --git a/internal/engine/runtime_filter_test.go b/internal/engine/runtime_filter_test.go new file mode 100644 index 0000000..1a23b57 --- /dev/null +++ b/internal/engine/runtime_filter_test.go @@ -0,0 +1,45 @@ +package engine + +import ( + "testing" + + "github.com/local/glpi-neural-brain/internal/model" +) + +func TestEffectiveFilterIsEnvironmentIntersectionRuntime(t *testing.T) { + filter := effectiveNodeFilter( + []string{"IT-Security", "Netzwerk"}, []string{"Netzwerk", "Backup"}, + []string{"GLPI Knowledge Base", "internal-kb"}, []string{"internal-kb"}, + ) + if filter.MatchNone { + t.Fatal("intersection should not be empty") + } + if len(filter.Categories) != 1 || filter.Categories[0] != "Netzwerk" { + t.Fatalf("unexpected category intersection: %#v", filter.Categories) + } + if len(filter.Sources) != 1 || filter.Sources[0] != "internal-kb" { + t.Fatalf("unexpected source intersection: %#v", filter.Sources) + } +} + +func TestEffectiveFilterDisjointSelectionMatchesNothing(t *testing.T) { + filter := effectiveNodeFilter([]string{"IT-Security"}, []string{"Backup"}, nil, nil) + if !filter.MatchNone { + t.Fatalf("disjoint administrative/runtime filters must match none: %+v", filter) + } + node := model.Node{Kind: "knowledge", Categories: []string{"IT-Security"}, Metadata: map[string]any{"source": "internal-kb"}} + if filter.Matches(node) { + t.Fatal("empty intersection must not fall back to all") + } +} + +func TestRuntimeJSONFieldMergePreservesDefaults(t *testing.T) { + settings := RuntimeSettings{LearningEnabled: true, ThinkingEnabled: true, ViewMode: "constellation", MaxDisplayNodes: 5000, LowPowerMode: true} + mergeRuntimeSettingsJSON(&settings, []byte(`{"display_categories":["Cloud"],"display_sources":["internal-kb"]}`)) + if !settings.LearningEnabled || !settings.ThinkingEnabled || settings.ViewMode != "constellation" || settings.MaxDisplayNodes != 5000 || !settings.LowPowerMode { + t.Fatalf("missing JSON fields overwrote defaults: %+v", settings) + } + if len(settings.DisplayCategories) != 1 || len(settings.DisplaySources) != 1 { + t.Fatalf("present JSON fields were not merged: %+v", settings) + } +} diff --git a/internal/graph/filter.go b/internal/graph/filter.go new file mode 100644 index 0000000..d7800e5 --- /dev/null +++ b/internal/graph/filter.go @@ -0,0 +1,116 @@ +package graph + +import ( + "fmt" + "net/url" + "strings" + + "github.com/local/glpi-neural-brain/internal/model" +) + +const ( + UncategorizedFilter = "__uncategorized__" + UnsourcedFilter = "__unsourced__" +) + +// NodeFilter limits knowledge-bearing nodes by category and logical source. +// Categories and Sources are AND-combined; values inside each dimension use +// OR semantics. Empty dimensions and "*" mean unrestricted. MatchNone is used +// for an empty administrative/runtime intersection and deliberately matches no +// node. +type NodeFilter struct { + Categories []string + Sources []string + MatchNone bool +} + +func (f NodeFilter) Matches(node model.Node) bool { + if f.MatchNone { + return false + } + return matchesDimension(node.Categories, f.Categories, UncategorizedFilter) && + matchesDimension(nodeSources(node), f.Sources, UnsourcedFilter) +} + +func matchesDimension(values, filters []string, emptyToken string) bool { + if len(filters) == 0 { + return true + } + wanted := make(map[string]struct{}, len(filters)) + for _, filter := range filters { + filter = strings.ToLower(strings.TrimSpace(filter)) + if filter != "" { + wanted[filter] = struct{}{} + } + } + if len(wanted) == 0 { + return true + } + if _, ok := wanted["*"]; ok { + return true + } + clean := make([]string, 0, len(values)) + for _, value := range values { + value = strings.TrimSpace(value) + if value != "" { + clean = append(clean, value) + } + } + if len(clean) == 0 { + _, ok := wanted[emptyToken] + return ok + } + for _, value := range clean { + if _, ok := wanted[strings.ToLower(value)]; ok { + return true + } + } + return false +} + +// NodeSource returns the logical source shown to users. Explicit source +// metadata wins. Research nodes fall back to their host name; all other nodes +// fall back to their technical origin. +func NodeSource(node model.Node) string { + values := nodeSources(node) + if len(values) == 0 { + return "" + } + return values[0] +} + +func nodeSources(node model.Node) []string { + if source := metadataText(node.Metadata, "source"); source != "" { + return []string{source} + } + if node.Kind == "source" && strings.TrimSpace(node.Label) != "" { + return []string{strings.TrimSpace(node.Label)} + } + if node.Kind == "external" { + for _, raw := range []string{node.URI, node.ExternalID} { + parsed, err := url.Parse(strings.TrimSpace(raw)) + if err == nil && strings.TrimSpace(parsed.Hostname()) != "" { + return []string{strings.ToLower(strings.TrimSpace(parsed.Hostname()))} + } + } + } + if strings.TrimSpace(node.Origin) != "" { + return []string{strings.TrimSpace(node.Origin)} + } + return nil +} + +func metadataText(metadata map[string]any, key string) string { + if metadata == nil { + return "" + } + value, ok := metadata[key] + if !ok || value == nil { + return "" + } + text := strings.TrimSpace(fmt.Sprint(value)) + if text == "" || strings.EqualFold(text, "") || strings.EqualFold(text, "null") { + return "" + } + return text +} diff --git a/internal/graph/filter_test.go b/internal/graph/filter_test.go new file mode 100644 index 0000000..6edf7b4 --- /dev/null +++ b/internal/graph/filter_test.go @@ -0,0 +1,76 @@ +package graph + +import ( + "testing" + + "github.com/local/glpi-neural-brain/internal/model" +) + +func TestNodeFilterCombinesCategoryAndSource(t *testing.T) { + node := model.Node{ + Kind: "knowledge", + Categories: []string{"IT-Security", "Cloud"}, + Origin: "knowledge-production", + Metadata: map[string]any{"source": "GLPI Knowledge Base"}, + } + if !(NodeFilter{Categories: []string{"Cloud"}, Sources: []string{"glpi knowledge base"}}).Matches(node) { + t.Fatal("matching category and source should be accepted case-insensitively") + } + if (NodeFilter{Categories: []string{"Cloud"}, Sources: []string{"internal-kb"}}).Matches(node) { + t.Fatal("source mismatch must reject even when category matches") + } + if (NodeFilter{Categories: []string{"Backup"}, Sources: []string{"GLPI Knowledge Base"}}).Matches(node) { + t.Fatal("category mismatch must reject even when source matches") + } + if (NodeFilter{MatchNone: true}).Matches(node) { + t.Fatal("MatchNone must reject every node") + } +} + +func TestNodeFilterVirtualEmptyValuesAndResearchHost(t *testing.T) { + unsourced := model.Node{Kind: "knowledge"} + if !(NodeFilter{Categories: []string{UncategorizedFilter}, Sources: []string{UnsourcedFilter}}).Matches(unsourced) { + t.Fatal("virtual empty category/source filters should match") + } + research := model.Node{Kind: "external", Origin: "research", URI: "https://docs.example.org/guide", Categories: []string{"Cloud"}} + if got := NodeSource(research); got != "docs.example.org" { + t.Fatalf("unexpected research source %q", got) + } + if !(NodeFilter{Sources: []string{"DOCS.EXAMPLE.ORG"}}).Matches(research) { + t.Fatal("research host should be source-filterable") + } +} + +func TestScopedEmbeddingRetrievalAndThinking(t *testing.T) { + s := &Store{ + nodes: map[string]model.Node{ + "a": {ID: "a", Kind: "knowledge", Label: "A", Categories: []string{"Cloud"}, Metadata: map[string]any{"source": "internal-kb"}}, + "b": {ID: "b", Kind: "knowledge", Label: "B", Categories: []string{"Cloud"}, Metadata: map[string]any{"source": "internal-kb"}}, + "c": {ID: "c", Kind: "knowledge", Label: "C", Categories: []string{"Cloud"}, Metadata: map[string]any{"source": "GLPI Knowledge Base"}}, + }, + edges: map[string]model.Edge{}, + vectors: map[string][]float32{"a": {1, 0}, "b": {.99, .01}, "c": {1, 0}}, + dirtyNodes: map[string]uint64{}, + dirtyEdges: map[string]uint64{}, + dirtyVectors: map[string]uint64{}, + deletedNodes: map[string]uint64{}, + deletedEdges: map[string]uint64{}, + deletedVectors: map[string]uint64{}, + } + filter := NodeFilter{Categories: []string{"Cloud"}, Sources: []string{"internal-kb"}} + hits := s.SimilarFiltered([]float64{1, 0}, 10, filter) + if len(hits) != 2 { + t.Fatalf("retrieval escaped source filter: %+v", hits) + } + a, b, _, ok, _ := s.NextPairScopedDepth(.5, 8, filter, 0) + if !ok || NodeSource(a) != "internal-kb" || NodeSource(b) != "internal-kb" { + t.Fatalf("thinking escaped scoped filter: ok=%v a=%+v b=%+v", ok, a, b) + } + if pending := s.NodesForEmbeddingScoped(NodeFilter{Sources: []string{"GLPI Knowledge Base"}}); len(pending) != 0 { + t.Fatalf("nodes with existing vectors should not be pending: %+v", pending) + } + delete(s.vectors, "c") + if pending := s.NodesForEmbeddingScoped(NodeFilter{Sources: []string{"GLPI Knowledge Base"}}); len(pending) != 1 || pending[0].ID != "c" { + t.Fatalf("embedding scope escaped source filter: %+v", pending) + } +} diff --git a/internal/graph/store.go b/internal/graph/store.go index 12e9ed7..a5863cc 100644 --- a/internal/graph/store.go +++ b/internal/graph/store.go @@ -232,10 +232,14 @@ func (s *Store) ClearVectorsByDimension(dim int) int { } func (s *Store) NodesForEmbedding() []model.Node { - return s.NodesForEmbeddingFiltered(nil) + return s.NodesForEmbeddingScoped(NodeFilter{}) } func (s *Store) NodesForEmbeddingFiltered(categories []string) []model.Node { + return s.NodesForEmbeddingScoped(NodeFilter{Categories: categories}) +} + +func (s *Store) NodesForEmbeddingScoped(filter NodeFilter) []model.Node { s.mu.RLock() defer s.mu.RUnlock() out := []model.Node{} @@ -243,7 +247,7 @@ func (s *Store) NodesForEmbeddingFiltered(categories []string) []model.Node { if n.Kind != "knowledge" && n.Kind != "ai-think" && n.Kind != "external" { continue } - if !nodeMatchesCategories(n, categories) { + if !filter.Matches(n) { continue } if _, ok := s.vectors[n.ID]; !ok { @@ -513,12 +517,16 @@ func (s *Store) Snapshot() model.Snapshot { return model.Snapshot{Version: s.version, Nodes: n, Edges: e, UpdatedAt: time.Now().UTC()} } func (s *Store) Similar(query []float64, limit int) []model.Hit { + return s.SimilarFiltered(query, limit, NodeFilter{}) +} + +func (s *Store) SimilarFiltered(query []float64, limit int, filter NodeFilter) []model.Hit { s.mu.RLock() defer s.mu.RUnlock() hits := []model.Hit{} for id, v := range s.vectors { n, ok := s.nodes[id] - if !ok || (n.Kind != "knowledge" && n.Kind != "ai-think" && n.Kind != "external") { + if !ok || (n.Kind != "knowledge" && n.Kind != "ai-think" && n.Kind != "external") || !filter.Matches(n) { continue } score := cosineMixed(query, v) @@ -544,6 +552,10 @@ func (s *Store) NextPairFiltered(min float64, anchorLimit int, categories []stri } func (s *Store) NextPairFilteredDepth(min float64, anchorLimit int, categories []string, maxAIDepth int) (model.Node, model.Node, float64, bool, int) { + return s.NextPairScopedDepth(min, anchorLimit, NodeFilter{Categories: categories}, maxAIDepth) +} + +func (s *Store) NextPairScopedDepth(min float64, anchorLimit int, filter NodeFilter, maxAIDepth int) (model.Node, model.Node, float64, bool, int) { s.mu.Lock() defer s.mu.Unlock() @@ -552,7 +564,7 @@ func (s *Store) NextPairFilteredDepth(min float64, anchorLimit int, categories [ if n.Kind != "knowledge" && n.Kind != "ai-think" { continue } - if !nodeMatchesCategories(n, categories) { + if !filter.Matches(n) { continue } if n.Kind == "ai-think" && maxAIDepth > 0 && graphNodeGenerationDepth(n) >= maxAIDepth { @@ -660,35 +672,6 @@ func (s *Store) ConnectingEdges(ids []string) []string { } return out } -func nodeMatchesCategories(n model.Node, filters []string) bool { - if len(filters) == 0 { - return true - } - wanted := make(map[string]struct{}, len(filters)) - for _, filter := range filters { - filter = strings.ToLower(strings.TrimSpace(filter)) - if filter != "" { - wanted[filter] = struct{}{} - } - } - if len(wanted) == 0 { - return true - } - if _, ok := wanted["*"]; ok { - return true - } - if len(n.Categories) == 0 { - _, ok := wanted["__uncategorized__"] - return ok - } - for _, category := range n.Categories { - if _, ok := wanted[strings.ToLower(strings.TrimSpace(category))]; ok { - return true - } - } - return false -} - func graphNodeGenerationDepth(n model.Node) int { if n.Kind != "ai-think" { return 0 diff --git a/internal/web/server.go b/internal/web/server.go index c133a95..c633d53 100644 --- a/internal/web/server.go +++ b/internal/web/server.go @@ -37,6 +37,7 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /api/runtime-settings", s.handleGetRuntimeSettings) mux.HandleFunc("PUT /api/runtime-settings", s.handleSetRuntimeSettings) mux.HandleFunc("GET /api/categories", s.handleCategories) + mux.HandleFunc("GET /api/sources", s.handleSources) mux.HandleFunc("GET /api/research/status", s.handleResearchStatus) mux.HandleFunc("POST /api/research/test", s.handleResearchTest) mux.HandleFunc("GET /api/stream", s.Broker.ServeSSE) @@ -71,7 +72,7 @@ func (s *Server) handleAnalysis(w http.ResponseWriter, r *http.Request) { } func (s *Server) handleGetRuntimeSettings(w http.ResponseWriter, r *http.Request) { - writeJSON(w, http.StatusOK, s.Engine.RuntimeSettings()) + writeJSON(w, http.StatusOK, s.Engine.RuntimeSettingsView()) } func (s *Server) handleSetRuntimeSettings(w http.ResponseWriter, r *http.Request) { @@ -84,18 +85,22 @@ func (s *Server) handleSetRuntimeSettings(w http.ResponseWriter, r *http.Request writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()}) return } - updated, err := s.Engine.SetRuntimeSettings(settings) + _, err := s.Engine.SetRuntimeSettings(settings) if err != nil { writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()}) return } - writeJSON(w, http.StatusOK, updated) + writeJSON(w, http.StatusOK, s.Engine.RuntimeSettingsView()) } func (s *Server) handleCategories(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, map[string]any{"categories": s.Engine.Categories()}) } +func (s *Server) handleSources(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, map[string]any{"sources": s.Engine.Sources()}) +} + func (s *Server) handleResearchStatus(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, s.Engine.ResearchStatus()) } diff --git a/internal/web/server_test.go b/internal/web/server_test.go index 243d6a0..d3017f8 100644 --- a/internal/web/server_test.go +++ b/internal/web/server_test.go @@ -22,8 +22,8 @@ func TestRuntimeSettingsAndCategoriesAPI(t *testing.T) { if err != nil { t.Fatal(err) } - g.UpsertNode(model.Node{ID: "n1", Kind: "knowledge", Label: "GLPI", Origin: "test", Categories: []string{"GLPI"}}) - g.UpsertNode(model.Node{ID: "n2", Kind: "knowledge", Label: "Ollama", Origin: "test", Categories: []string{"Ollama"}}) + g.UpsertNode(model.Node{ID: "n1", Kind: "knowledge", Label: "GLPI", Origin: "test", Categories: []string{"GLPI"}, Metadata: map[string]any{"source": "GLPI Knowledge Base"}}) + g.UpsertNode(model.Node{ID: "n2", Kind: "knowledge", Label: "Ollama", Origin: "test", Categories: []string{"Ollama"}, Metadata: map[string]any{"source": "internal-kb"}}) g.UpsertNode(model.Node{ID: "n3", Kind: "knowledge", Label: "Ohne Kategorie", Origin: "test"}) broker := activity.New(20) @@ -37,7 +37,7 @@ func TestRuntimeSettingsAndCategoriesAPI(t *testing.T) { }, g, broker) h := (&Server{Engine: eng, Graph: g, Broker: broker}).Handler() - body := `{"learning_enabled":false,"thinking_enabled":false,"learning_categories":["GLPI"],"display_categories":["Ollama"],"thinking_categories":["GLPI"],"view_mode":"constellation","max_display_nodes":1500,"low_power_mode":true}` + body := `{"learning_enabled":false,"thinking_enabled":false,"learning_categories":["GLPI"],"display_categories":["Ollama"],"thinking_categories":["GLPI"],"learning_sources":["GLPI Knowledge Base"],"display_sources":["internal-kb"],"thinking_sources":["GLPI Knowledge Base"],"view_mode":"constellation","max_display_nodes":1500,"low_power_mode":true}` req := httptest.NewRequest(http.MethodPut, "/api/runtime-settings", strings.NewReader(body)) req.Header.Set("Content-Type", "application/json") res := httptest.NewRecorder() @@ -74,6 +74,21 @@ func TestRuntimeSettingsAndCategoriesAPI(t *testing.T) { t.Fatalf("uncategorized virtual category missing: %+v", categories.Categories) } + res = httptest.NewRecorder() + h.ServeHTTP(res, httptest.NewRequest(http.MethodGet, "/api/sources", nil)) + if res.Code != http.StatusOK { + t.Fatalf("GET sources returned %d", res.Code) + } + var sources struct { + Sources []engine.SourceInfo `json:"sources"` + } + if err := json.NewDecoder(res.Body).Decode(&sources); err != nil { + t.Fatal(err) + } + if len(sources.Sources) < 2 { + t.Fatalf("source options missing: %+v", sources.Sources) + } + res = httptest.NewRecorder() h.ServeHTTP(res, httptest.NewRequest(http.MethodPost, "/api/enrich?async=1", strings.NewReader(`{}`))) if res.Code != http.StatusConflict { diff --git a/internal/web/static/app.css b/internal/web/static/app.css index 71618f8..4d907c3 100644 --- a/internal/web/static/app.css +++ b/internal/web/static/app.css @@ -65,3 +65,4 @@ body.low-power .glass{backdrop-filter:blur(12px)} .eco-switch input:checked{background:rgba(255,180,82,.2)} .eco-switch input:checked:after{background:var(--amber);box-shadow:0 0 10px rgba(255,180,82,.72)} @media(max-width:620px){.view-selector{grid-template-columns:1fr}.metrics span[title]{display:none}} +.filter-scope-summary{margin:2px 0 10px;line-height:1.45}.category-option.disabled{opacity:.38}.category-option.disabled span{cursor:not-allowed;border-style:dashed}.category-option span small{display:block;margin-top:2px;font-size:7px;letter-spacing:.04em;color:#8a6f79}.filter-scope-summary.warn{color:#ffb37f}.filter-scope-summary.ok{color:#7898aa} diff --git a/internal/web/static/app.js b/internal/web/static/app.js index ba2ab87..e644fbe 100644 --- a/internal/web/static/app.js +++ b/internal/web/static/app.js @@ -51,8 +51,8 @@ lodOpenUntil: new Map(), lodHotUntil: new Map(), lodDirty: true, lodLastBuild: 0, lodNextExpiry: 0, lodZoomBand: 2, renderNodes: [], renderEdges: [], renderIdleEdges: [], renderNodeById: new Map(), renderEdgeById: new Map(), visibleForNode: new Map(), edgeRenderMap: new Map(), renderActive: new Map(), renderEdgeActive: new Map(), renderStats: {nodes: 0, edges: 0, hiddenNodes: 0, hiddenEdges: 0}, - fullSnapshot: null, fullNodeById: new Map(), runtimeSettings: {learning_enabled: true, thinking_enabled: true, learning_categories: [], display_categories: [], thinking_categories: [], view_mode: 'neural', max_display_nodes: 0, low_power_mode: false}, - availableCategories: [], viewMode: 'neural', honeycombNodes: [], honeycombSpacing: 0, honeySlotByID: new Map(), honeyPointPool: [], honeyFreeSlots: [], + fullSnapshot: null, fullNodeById: new Map(), runtimeSettings: {learning_enabled: true, thinking_enabled: true, learning_categories: [], display_categories: [], thinking_categories: [], learning_sources: [], display_sources: [], thinking_sources: [], effective_learning: {categories: [], sources: [], matches_none: false}, effective_display: {categories: [], sources: [], matches_none: false}, effective_thinking: {categories: [], sources: [], matches_none: false}, admin_learning: {categories: [], sources: []}, admin_display: {categories: [], sources: []}, admin_thinking: {categories: [], sources: []}, view_mode: 'neural', max_display_nodes: 0, low_power_mode: false}, + availableCategories: [], availableSources: [], viewMode: 'neural', honeycombNodes: [], honeycombSpacing: 0, honeySlotByID: new Map(), honeyPointPool: [], honeyFreeSlots: [], constellationNodes: [], constellationLinks: [], settingsOpen: false, settingsDraft: null, forcedDisplayUntil: new Map(), nextDisplayLimitExpiry: 0, graphVersion: null, displaySignature: '', displayLimitStats: {limit: 0, eligible: 0, shown: 0}, researchAnimations: new Map(), researchSequence: 0, @@ -141,13 +141,46 @@ return data; } - function categoryFilterMatches(node, categories) { - if (!categories || categories.length === 0) return true; - const wanted = new Set(categories.map(value => String(value).trim().toLowerCase()).filter(Boolean)); - if (wanted.has('*')) return true; - const nodeCategories = (node.categories || []).map(value => String(value).trim().toLowerCase()).filter(Boolean); - if (nodeCategories.length === 0) return wanted.has('__uncategorized__'); - return nodeCategories.some(category => wanted.has(category)); + function normalizedFilterValues(values) { + return [...new Set((values || []).map(value => String(value).trim().toLowerCase()).filter(Boolean))]; + } + + function nodeSource(node) { + const explicit = node?.metadata?.source; + if (explicit !== undefined && explicit !== null) { + const value = String(explicit).trim(); + if (value && value.toLowerCase() !== '' && value.toLowerCase() !== 'null') return value; + } + if (node?.kind === 'source' && String(node.label || '').trim()) return String(node.label).trim(); + if (node?.kind === 'external') { + for (const raw of [node.uri, node.external_id]) { + try { + const host = new URL(String(raw || '')).hostname.trim().toLowerCase(); + if (host) return host; + } catch {} + } + } + return String(node?.origin || '').trim(); + } + + function filterDimensionMatches(values, filters, emptyToken) { + const wanted = new Set(normalizedFilterValues(filters)); + if (!wanted.size || wanted.has('*')) return true; + const clean = (values || []).map(value => String(value).trim().toLowerCase()).filter(Boolean); + if (!clean.length) return wanted.has(emptyToken); + return clean.some(value => wanted.has(value)); + } + + function nodeFilterMatches(node, filter) { + if (filter?.matches_none) return false; + return filterDimensionMatches(node.categories || [], filter?.categories || [], '__uncategorized__') && + filterDimensionMatches(nodeSource(node) ? [nodeSource(node)] : [], filter?.sources || [], '__unsourced__'); + } + + function effectiveRuntimeFilter(key, settings = state.runtimeSettings) { + const effective = settings?.[`effective_${key}`]; + if (effective) return effective; + return {categories: settings?.[`${key}_categories`] || [], sources: settings?.[`${key}_sources`] || [], matches_none: false}; } function activeForcedDisplayIDs(now = Date.now()) { @@ -232,18 +265,24 @@ } function filteredSnapshot(snapshot) { - const filters = state.runtimeSettings.display_categories || []; + const filter = effectiveRuntimeFilter('display'); + const restricted = Boolean(filter?.matches_none || (filter?.categories || []).length || (filter?.sources || []).length); let filtered = snapshot; - if (filters.length) { + if (restricted) { const visible = new Set(); const noteKinds = new Set(['knowledge', 'ai-think', 'external']); + const taxonomyKinds = new Set(['category', 'source', 'concept']); + const nodeMap = new Map(snapshot.nodes.map(node => [node.id, node])); for (const node of snapshot.nodes) { - if (noteKinds.has(node.kind) && categoryFilterMatches(node, filters)) visible.add(node.id); - if (node.kind === 'category' && filters.some(value => String(value).toLowerCase() === String(node.label || '').toLowerCase())) visible.add(node.id); + if (noteKinds.has(node.kind) && nodeFilterMatches(node, filter)) visible.add(node.id); } + // Add only the taxonomy directly attached to an already matching note. + // Never pull another knowledge/external note through a semantic edge. for (const edge of snapshot.edges) { - if (visible.has(edge.source)) visible.add(edge.target); - if (visible.has(edge.target)) visible.add(edge.source); + const source = nodeMap.get(edge.source); + const target = nodeMap.get(edge.target); + if (visible.has(edge.source) && target && taxonomyKinds.has(target.kind)) visible.add(edge.target); + if (visible.has(edge.target) && source && taxonomyKinds.has(source.kind)) visible.add(edge.source); } filtered = { ...snapshot, @@ -254,20 +293,28 @@ return limitSnapshot(filtered, state.runtimeSettings.max_display_nodes); } + function displaySignatureFor(settings = state.runtimeSettings) { + const filter = effectiveRuntimeFilter('display', settings); + return {categories: filter.categories || [], sources: filter.sources || [], matchesNone: Boolean(filter.matches_none), limit: Number(settings.max_display_nodes || 0)}; + } + function currentDisplaySignature() { const forced = [...activeForcedDisplayIDs()].sort(); - return JSON.stringify({categories: state.runtimeSettings.display_categories || [], limit: Number(state.runtimeSettings.max_display_nodes || 0), forced}); + return JSON.stringify({...displaySignatureFor(), forced}); } async function loadRuntimeConfiguration() { try { - const [settings, categories] = await Promise.all([api('/api/runtime-settings'), api('/api/categories')]); + const [settings, categories, sources] = await Promise.all([api('/api/runtime-settings'), api('/api/categories'), api('/api/sources')]); state.runtimeSettings = {...state.runtimeSettings, ...settings}; state.availableCategories = categories.categories || []; + state.availableSources = sources.sources || []; state.viewMode = ['neural', 'honeycomb', 'constellation'].includes(state.runtimeSettings.view_mode) ? state.runtimeSettings.view_mode : 'neural'; applyPerformanceMode(Boolean(state.runtimeSettings.low_power_mode), true); syncRuntimeControls(); renderCategoryFilters(); + renderSourceFilters(); + renderFilterScopeSummary(); } catch { syncRuntimeControls(); } @@ -312,32 +359,97 @@ return name === '__uncategorized__' ? 'Ohne Kategorie' : name; } - function filterSet(key) { - const source = state.settingsDraft || state.runtimeSettings; - return new Set((source[`${key}_categories`] || []).map(value => String(value).toLowerCase())); + function sourceLabel(name) { + return name === '__unsourced__' ? 'Ohne Quelle' : name; } - function renderCategoryFilters(search = '') { + function filterSet(key, dimension) { + const source = state.settingsDraft || state.runtimeSettings; + return new Set((source[`${key}_${dimension}`] || []).map(value => String(value).toLowerCase())); + } + + function adminAllows(key, dimension, value) { + const admin = state.runtimeSettings?.[`admin_${key}`] || {}; + const restricted = Boolean(admin[`${dimension}_restricted`]); + if (!restricted) return true; + const allowed = new Set(normalizedFilterValues(admin[dimension] || [])); + return allowed.has(String(value).trim().toLowerCase()); + } + + function renderFilterOptions({dimension, search = '', options, targets, labeler, dataAttribute}) { const query = String(search || '').trim().toLowerCase(); - const targets = {learning: $('learningCategoryList'), display: $('displayCategoryList'), thinking: $('thinkingCategoryList')}; for (const [key, target] of Object.entries(targets)) { if (!target) continue; - const selected = filterSet(key); + const selected = filterSet(key, dimension); target.innerHTML = ''; - const categories = state.availableCategories.filter(category => !query || categoryLabel(category.name).toLowerCase().includes(query)); - for (const category of categories) { + const visibleOptions = options.filter(option => !query || labeler(option.name).toLowerCase().includes(query)); + for (const option of visibleOptions) { + const allowed = adminAllows(key, dimension, option.name); const label = document.createElement('label'); - label.className = 'category-option'; - const checked = selected.has(String(category.name).toLowerCase()); - label.innerHTML = `${escapeHTML(categoryLabel(category.name))}${Number(category.count || 0).toLocaleString('de-DE')}`; + label.className = `category-option${allowed ? '' : ' disabled'}`; + const checked = selected.has(String(option.name).toLowerCase()); + const disabled = allowed ? '' : ' disabled'; + const notice = allowed ? '' : 'durch ENV ausgeschlossen'; + label.innerHTML = `${escapeHTML(labeler(option.name))}${Number(option.count || 0).toLocaleString('de-DE')}${notice}`; target.appendChild(label); } - if (!categories.length) target.innerHTML = 'Keine passende Kategorie'; + if (!visibleOptions.length) target.innerHTML = `Keine passende ${dimension === 'categories' ? 'Kategorie' : 'Quelle'}`; } } - function collectCategoryFilter(key) { - return [...document.querySelectorAll(`[data-category-filter="${key}"]:checked`)].map(input => input.value); + function renderCategoryFilters(search = '') { + renderFilterOptions({ + dimension: 'categories', search, options: state.availableCategories, + targets: {learning: $('learningCategoryList'), display: $('displayCategoryList'), thinking: $('thinkingCategoryList')}, + labeler: categoryLabel, dataAttribute: 'data-category-filter' + }); + } + + function renderSourceFilters(search = '') { + renderFilterOptions({ + dimension: 'sources', search, options: state.availableSources, + targets: {learning: $('learningSourceList'), display: $('displaySourceList'), thinking: $('thinkingSourceList')}, + labeler: sourceLabel, dataAttribute: 'data-source-filter' + }); + } + + function intersectClientValues(adminValues, runtimeValues, restrictedFlag) { + const runtime = normalizedFilterValues(runtimeValues); + if (!restrictedFlag) return {values: runtime, restricted: runtime.length > 0, none: false}; + const admin = normalizedFilterValues(adminValues); + if (!runtime.length) return {values: admin, restricted: true, none: false}; + const allowed = new Set(admin); + const values = runtime.filter(value => allowed.has(value)); + return {values, restricted: true, none: values.length === 0}; + } + + function previewEffectiveFilter(key) { + const settings = state.settingsDraft || state.runtimeSettings; + const admin = state.runtimeSettings?.[`admin_${key}`] || {}; + const categories = intersectClientValues(admin.categories || [], settings?.[`${key}_categories`] || [], Boolean(admin.categories_restricted)); + const sources = intersectClientValues(admin.sources || [], settings?.[`${key}_sources`] || [], Boolean(admin.sources_restricted)); + return {categories: categories.values, sources: sources.values, matches_none: categories.none || sources.none}; + } + + function compactFilterValues(values, labeler) { + if (!values?.length) return 'alle'; + const labels = values.slice(0, 3).map(labeler); + return labels.join(', ') + (values.length > 3 ? ` +${values.length - 3}` : ''); + } + + function renderFilterScopeSummary() { + const target = $('filterScopeSummary'); + if (!target) return; + const parts = []; + let hasEmpty = false; + for (const [key, label] of [['learning', 'Lernen'], ['display', 'Anzeige'], ['thinking', 'Thinking']]) { + const effective = previewEffectiveFilter(key); + hasEmpty ||= effective.matches_none; + parts.push(`${label}: ${effective.matches_none ? 'keine Übereinstimmung' : `${compactFilterValues(effective.categories, categoryLabel)} · ${compactFilterValues(effective.sources, sourceLabel)}`}`); + } + target.textContent = `Wirksam (ENV ∩ WebUI) — ${parts.join(' | ')}`; + target.classList.toggle('warn', hasEmpty); + target.classList.toggle('ok', !hasEmpty); } async function persistRuntimeSettings(settings = state.runtimeSettings) { @@ -347,18 +459,22 @@ learning_categories: [...(settings.learning_categories || [])], display_categories: [...(settings.display_categories || [])], thinking_categories: [...(settings.thinking_categories || [])], + learning_sources: [...(settings.learning_sources || [])], + display_sources: [...(settings.display_sources || [])], + thinking_sources: [...(settings.thinking_sources || [])], view_mode: ['neural', 'honeycomb', 'constellation'].includes(settings.view_mode) ? settings.view_mode : 'neural', max_display_nodes: Math.max(0, Math.min(500000, Math.trunc(Number(settings.max_display_nodes) || 0))), low_power_mode: Boolean(settings.low_power_mode) }; - const previousDisplay = JSON.stringify({categories: state.runtimeSettings.display_categories || [], limit: Number(state.runtimeSettings.max_display_nodes || 0)}); + const previousDisplay = JSON.stringify(displaySignatureFor(state.runtimeSettings)); const updated = await api('/api/runtime-settings', {method: 'PUT', body: JSON.stringify(normalized)}); state.runtimeSettings = {...normalized, ...updated}; const nextViewMode = state.runtimeSettings.view_mode; applyPerformanceMode(Boolean(state.runtimeSettings.low_power_mode), true); syncRuntimeControls(); + renderFilterScopeSummary(); applyViewMode(nextViewMode, false); - if (previousDisplay !== JSON.stringify({categories: state.runtimeSettings.display_categories || [], limit: Number(state.runtimeSettings.max_display_nodes || 0)})) { + if (previousDisplay !== JSON.stringify(displaySignatureFor(state.runtimeSettings))) { state.displaySignature = ''; await loadGraph(); } @@ -373,6 +489,8 @@ $('settingsBackdrop')?.classList.remove('hidden'); syncRuntimeControls(); renderCategoryFilters($('categorySearch')?.value || ''); + renderSourceFilters($('sourceSearch')?.value || ''); + renderFilterScopeSummary(); } function closeSettingsPanel() { @@ -2882,6 +3000,7 @@ $('closeSettings').addEventListener('click', closeSettingsPanel); $('settingsBackdrop').addEventListener('click', closeSettingsPanel); $('categorySearch').addEventListener('input', e => renderCategoryFilters(e.currentTarget.value)); + $('sourceSearch').addEventListener('input', e => renderSourceFilters(e.currentTarget.value)); $('settingsLearning').addEventListener('change', e => { if (state.settingsDraft) state.settingsDraft.learning_enabled = e.currentTarget.checked; }); $('settingsThinking').addEventListener('change', e => { if (state.settingsDraft) state.settingsDraft.thinking_enabled = e.currentTarget.checked; }); $('settingsLowPower').addEventListener('change', e => { if (state.settingsDraft) state.settingsDraft.low_power_mode = e.currentTarget.checked; }); @@ -2900,19 +3019,28 @@ $('settingsViewHoneycomb').addEventListener('click', () => selectSettingsView('honeycomb')); $('settingsViewConstellation').addEventListener('click', () => selectSettingsView('constellation')); $('settingsPanel').addEventListener('change', e => { - const input = e.target.closest('[data-category-filter]'); - if (!input || !state.settingsDraft) return; - const key = input.dataset.categoryFilter; - const property = `${key}_categories`; + const input = e.target.closest('[data-category-filter], [data-source-filter]'); + if (!input || !state.settingsDraft || input.disabled) return; + const dimension = input.dataset.categoryFilter !== undefined ? 'categories' : 'sources'; + const key = input.dataset.categoryFilter ?? input.dataset.sourceFilter; + const property = `${key}_${dimension}`; const selected = new Map((state.settingsDraft[property] || []).map(value => [String(value).toLowerCase(), value])); const normalized = String(input.value).toLowerCase(); if (input.checked) selected.set(normalized, input.value); else selected.delete(normalized); state.settingsDraft[property] = [...selected.values()]; + renderFilterScopeSummary(); }); document.querySelectorAll('[data-clear-filter]').forEach(button => button.addEventListener('click', () => { if (!state.settingsDraft) return; state.settingsDraft[`${button.dataset.clearFilter}_categories`] = []; renderCategoryFilters($('categorySearch').value); + renderFilterScopeSummary(); + })); + document.querySelectorAll('[data-clear-source-filter]').forEach(button => button.addEventListener('click', () => { + if (!state.settingsDraft) return; + state.settingsDraft[`${button.dataset.clearSourceFilter}_sources`] = []; + renderSourceFilters($('sourceSearch').value); + renderFilterScopeSummary(); })); document.querySelectorAll('[data-node-limit]').forEach(button => button.addEventListener('click', () => { if (!state.settingsDraft) return; diff --git a/internal/web/static/index.html b/internal/web/static/index.html index 251469b..91170ec 100644 --- a/internal/web/static/index.html +++ b/internal/web/static/index.html @@ -131,7 +131,8 @@
-

Kategorie-Filter

Leer = alle Kategorien
+

Kategorie-Filter

Leer = alle erlaubten Kategorien
+

Environment-Grenzen werden geladen …

@@ -148,6 +149,24 @@
+
+

Quellen-Filter

Quelle und Kategorie gelten gemeinsam
+ + +
+
LernenNur Wissen aus diesen Quellen wird neu eingebettet und bei Abfragen verwendet.
+
+
+
+
AnzeigeNur passende Quellen werden gerendert; Taxonomie bleibt sichtbar.
+
+
+
+
ThinkingNur diese Quellen dürfen neue Relationen und Artikel speisen.
+
+
+
+