diff --git a/ANALYSE-COMPACT-STORAGE.md b/ANALYSE-COMPACT-STORAGE.md deleted file mode 100644 index 32c47cf..0000000 --- a/ANALYSE-COMPACT-STORAGE.md +++ /dev/null @@ -1,59 +0,0 @@ -# Analyse: Warum 19,2 Mio. Zeilen nach zwei Tagen entstanden - -## Befund - -Die vorige Version war noch nicht wirklich „Metadata-first“: - -1. `insertBatch()` führte für **jedes** eingehende Event weiterhin ein `INSERT INTO event_logs` aus. -2. Danach wurden zusätzlich `event_count_buckets`, `event_catalog` und `event_occurrences` geschrieben. -3. `event_occurrences` verwendete zu viele Dimensionen für die Eindeutigkeit. Provider, Benutzer, IP, Workstation, LogonType, Status und FailureReason konnten die Aggregation stark fragmentieren. -4. Mehrere UEBA-/Detection-Regeln lasen weiterhin aus `event_logs`. -5. `/ui` las die letzten Events und den 24h-Eventzähler aus `event_occurrences`. Bei hoher Kardinalität wurde damit bereits die Startseite teuer. - -19,2 Mio. Events in 48 Stunden entsprechen im Mittel ungefähr 111 Events/s. Diese Rate ist für kleine aggregierte Upserts nicht das Kernproblem. Eine breite Einzelzeile mit mehreren Sekundärindizes pro Event dagegen erzeugt sehr viel Daten- und Index-I/O. - -## Korrektur - -### Universal: `event_count_buckets` - -Für jedes Event wird nur noch ein Counter auf - -`5-Minuten-Bucket × Host × Channel × Event-ID` - -hochgezählt. Ein Host mit 200 verschiedenen Event-IDs erzeugt damit theoretisch höchstens etwa 57.600 Counter-Zeilen pro Tag, selbst wenn dahinter Millionen reale Events liegen (200 IDs × 288 Buckets), praktisch meist deutlich weniger. - -### Kontext nur bei Bedarf - -`event_occurrences` wird nur für eine konfigurierbare Allowlist von Security-IDs geschrieben. Der Dimensionsschlüssel enthält nur stabile, detection-relevante Werte. Beispiel 4740: Zieluser und CallerComputer/Workstation; nicht FailureReason, Status, IP oder Subject-Machine-Account. - -### Keine Einzelzeilen im Normalbetrieb - -`STORE_EVENT_ROWS=false` ist der neue Standard. `event_logs` wird dann überhaupt nicht beschrieben. `STORE_RAW_XML=true` aktiviert implizit wieder Full-Event-Zeilen und ist deshalb nur für kurze, gezielte Forensik gedacht. - -### UI - -`/ui` verwendet nur `event_count_buckets`. Die Eventliste hat ohne explizites `from` automatisch ein 24h-Fenster. User-/IP-Suchen wechseln gezielt auf die kleine Kontexttabelle. - -## Sofortmaßnahme für die bestehende Datenbank - -Nach Deployment des neuen Backends: - -```sql -SOURCE deploy/mariadb/migrations/004-compact-storage.sql; -SOURCE deploy/mariadb/emergency-compact-cleanup.sql; -SOURCE deploy/mariadb/cardinality-diagnostics.sql; -``` - -Der Emergency-Cleanup löscht absichtlich die redundante alte Full-Event-/Kontext-Historie, nicht jedoch `event_count_buckets`, `event_catalog`, Detections oder Baselines. - -## Retention - -Die detaillierten Zeit-Buckets werden standardmäßig 30 Tage gehalten. `event_catalog` bleibt dauerhaft und behält First/Last-Seen plus Gesamtzähler. Damit wächst die Zeitreihenhistorie nicht unbegrenzt. - -## Erwartetes Monitoring - -Der wichtigste Wert ist künftig nicht `COUNT(*)` allein, sondern das Verhältnis: - -`SUM(cnt) / COUNT(*)` - -in den Aggregattabellen. Je höher dieser Wert, desto mehr reale Events werden durch eine Datenbankzeile repräsentiert. `cardinality-diagnostics.sql` zeigt diesen Wert für die letzten 24 Stunden. diff --git a/ANALYSE-METADATA-FIRST.md b/ANALYSE-METADATA-FIRST.md deleted file mode 100644 index 22b11bb..0000000 --- a/ANALYSE-METADATA-FIRST.md +++ /dev/null @@ -1,104 +0,0 @@ -> **Überholt:** Dieses Dokument beschreibt die erste Metadata-first-Iteration. -> Für den produktiven Stand nach dem 19,2-Mio.-Zeilen-Befund gilt -> `ANALYSE-COMPACT-STORAGE.md`. Insbesondere werden `event_logs` jetzt -> standardmäßig gar nicht mehr geschrieben und `event_occurrences` ist nur noch -> eine selektive Security-Kontexttabelle. - -# Untersuchung und Anpassung: Metadata-first - -## Befund - -Die Performanceprobleme kommen nicht nur von „zu viel XML“, sondern aus mehreren sich verstärkenden Ursachen: - -1. Jedes Event wurde breit in `event_logs` und zusätzlich vollständig in `event_log_raw` gespeichert. -2. Die Partitionswartung verwendete `event_logs_raw`, die Tabelle heißt jedoch `event_log_raw`. Der Lauf brach damit ab; alte Partitionen wurden nicht gelöscht. -3. `event_count_buckets` und `ueba_context_buckets` waren nicht partitioniert und wuchsen unbegrenzt. -4. `target_user_norm` und `subject_user_norm` waren im Schema und in Indizes vorgesehen, wurden beim Insert aber nicht befüllt. -5. Dashboard und Event-UI lasen direkt aus der größten Einzeltabelle. Die Event-Filter enthielten zusätzlich eine korrelierte `EXISTS`-Abfrage auf Detections. -6. Die „neue Event-ID“-Regel verwendete einen teuren `NOT EXISTS`-Vergleich gegen die gesamte Eventhistorie. -7. Zählerregeln wie Failed-Logon-Spike, Reboot-Spike und Password-Spray zählten Einzelevents, obwohl dafür Aggregate ausreichen. - -Ohne Zugriff auf die produktive Datenbank konnte ich keine realen `EXPLAIN ANALYZE`-Pläne oder Größenverteilungen messen. Die oben genannten Punkte ergeben sich direkt aus Schema und Abfragepfaden im Projekt. - -## Umgesetztes Zielmodell - -### Hot Data - -`event_logs` enthält weiterhin kompakte Einzelevent-Metadaten, weil einige Regeln die genaue Reihenfolge benötigen, etwa „erfolgreicher Login nach Fehlversuchen“. Die Standardaufbewahrung beträgt 72 Stunden. - -### Langfristige Metadaten - -`event_occurrences` speichert pro Zeit-Bucket und Dimensionskombination: - -- Host, Channel, Event-ID und Provider -- Target-/Subject-User -- Source-IP und Workstation/Gerät -- Logon-Type, Status und Fehlergrund -- Anzahl sowie erstes und letztes Auftreten - -Die UI liest Bucket-Zeilen direkt mit `ORDER BY bucket_start DESC LIMIT ...`; dadurch bleibt die Abfrage index- und partitionsfreundlich und benötigt keinen großen `GROUP BY`. - -### Event-Katalog - -`event_catalog` enthält pro Host/Channel/Event-ID nur erstes Auftreten, letztes Auftreten und Gesamtzahl. Die „neue Event-ID“-Regel benötigt dadurch keinen Anti-Join mehr. - -### Raw XML - -Raw-XML ist mit `STORE_RAW_XML=false` standardmäßig deaktiviert. Bei temporärem forensischem Bedarf kann es aktiviert und mit eigener kurzer `RAW_RETENTION` betrieben werden. - -### Direkter Metadaten-Ingest - -Collector können weiterhin XML in `msg` senden. Neu ist ein `meta`-Objekt, das ohne XML auskommt. Für Event 4740 wird `CallerComputerName` als Workstation/Gerät verarbeitet. - -## Erwartete Wirkung - -- Wegfall der dauerhaften XML-Doppelhaltung. -- Garantierte Aufbewahrungsgrenzen durch korrigierte und erweiterte Partitionswartung. -- Deutlich weniger Zeilen in langfristigen Abfragen durch Zeit-Buckets. -- Schnellere Dashboard-/Event-Seiten, da sie nicht mehr `event_logs` durchsuchen. -- Schnellere Zählerregeln durch `SUM(cnt)` auf `event_occurrences`. -- Kleine, konstante Abfrage für neue Event-IDs über `event_catalog`. -- Nutzbare User-Indizes, da normalisierte User-Spalten nun tatsächlich befüllt und in wichtigen Regeln verwendet werden. - -Die tatsächliche Kompressionswirkung hängt von der Kardinalität ab. Viele identische Events innerhalb einer Minute werden stark verdichtet. Einzigartige Kombinationen aus User, IP und Workstation reduzieren sich weniger, bleiben aber deutlich kleiner als Voll-XML plus breite Einzeleventzeile. - -## Lockout-Abfrage - -```sql -SELECT - first_event_ts, - last_event_ts, - cnt, - hostname AS reporting_host, - target_user, - workstation AS caller_device, - status_text, - failure_reason -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4740 - AND bucket_start >= UTC_TIMESTAMP() - INTERVAL 7 DAY -ORDER BY bucket_start DESC; -``` - -Bei einem Domain-Lockout ist `hostname` typischerweise der meldende Domain Controller; `workstation` enthält – sofern im Event vorhanden – `CallerComputerName` und damit das verursachende Gerät. - -## Empfohlener Rollout - -1. Datenbank sichern und Ingest stoppen. -2. Tabellen- und Partitionsgrößen mit `deploy/mariadb/diagnostics.sql` erfassen. -3. `deploy/mariadb/migrations/002-metadata-first.sql` in einem Wartungsfenster ausführen. -4. `.env.example` nach `.env` kopieren beziehungsweise bestehende Secrets übernehmen und die neuen Variablen ergänzen. -5. Backend deployen, zunächst `STORE_RAW_XML=false`, `EVENT_RETENTION=72h`, `METADATA_BUCKET=1m`. -6. Partition-Maintenance-Logs und Datenbankgrößen kontrollieren. -7. Nach 3–7 Tagen Schwellenwerte und Bucketgröße anhand der realen Eventrate nachjustieren. - -## Wann ein Wechsel der Datenbank sinnvoll wäre - -MariaDB ist für dieses Metadatenmodell weiterhin geeignet. Ein Wechsel zu ClickHouse wäre interessant, wenn dauerhaft sehr hohe Raten und große analytische Zeitreihen über Jahre benötigt werden. OpenSearch wäre sinnvoll, wenn Volltextsuche im Rohinhalt eine Kernanforderung bleibt. Für „welches Event, wann, auf welchem Gerät, für welchen User und wie oft“ ist ein Engine-Wechsel derzeit nicht erforderlich. - -## Ergänzung: `new_event_id` ist Inventar, nicht automatisch Incident - -Die bisherige Regel hat jedes erste Auftreten pauschal als offenen Medium-Alarm behandelt. Das überschätzt insbesondere diagnostische und operative Channels. Die neue Standardklassifikation ist `inventory`: nach Lernphase und Mindestanzahl entsteht höchstens eine plausible Info-Meldung. `Microsoft-Windows-WMI-Activity/Operational` wird standardmäßig ignoriert. Nur explizit konfigurierte High-Risk-IDs oder der Betriebsmodus `alert` können eine offene Detection erzeugen. - -Für reproduzierbare Lasttests liegt unter `cmd/siem-stress-agent` ein begrenzter Metadaten-Agent. Er markiert sämtliche Daten eindeutig, benötigt `--confirm-load-test` und besitzt harte Maximalwerte für Rate, Laufzeit, Worker und Eventzahl. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..3983797 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,42 @@ +# Architekturentscheidungen + +## 1. MariaDB ist kein Event Store mehr + +PostgreSQL ist ausschließlich Control Plane. ClickHouse ist ausschließlich Event-/Analytics-Plane. Dadurch konkurrieren Agent-Updates, Incident-Status und Benutzeraktionen nicht mit Milliarden append-only Events. + +## 2. Queue vor Datenbank + +Der Ingress quittiert erst, nachdem Redpanda den Batch bestätigt hat. ClickHouse-Ausfälle führen damit nicht zu HTTP-Timeout-Kaskaden bei den Collectoren. Der Processor kann nachholen, solange die Queue-Retention nicht überschritten wird. + +## 3. Ein kanonisches Event + +Es gibt keine parallelen `event_logs`, `event_occurrences`, `event_count_buckets`, `raw_event`- und Rule-Helper-Kopien mehr. Die einzige langfristige Eventtabelle ist `siem.events`. Das Dashboard-Rollup ist eine explizite, kleine Materialized View. + +## 4. Raw getrennt von Analytics + +Raw-Payloads sind für Parserfehler und Forensik wertvoll, aber ungeeignet als primäre Analytics-Zeile. Sie werden gzip-komprimiert in S3-kompatiblen Object Storage verschoben. ClickHouse speichert nur den Object-Key und den Index im Batch. + +## 5. Kein Graph als Primärspeicher + +Graph-Sichten können später aus ClickHouse abgeleitet werden, z. B. `(User)-[:LOGGED_ON_TO]->(Host)` mit `first_seen`, `last_seen`, `count`. Jedes einzelne Event als Graph-Knoten würde die gleiche Explosion nur in einer anderen Engine wiederholen. + +## 6. Keine TSDB für Security-Events + +Prometheus überwacht die Pipeline, speichert aber keine Benutzer-/Host-/IP-Eventdimensionen. Hochkardinale Security-Attribute gehören nach ClickHouse. + +## 7. Idempotenz + +Redpanda/Kafka-Consumer liefern at-least-once und auch ein Collector kann nach einem HTTP-Timeout denselben Batch erneut senden. Deshalb berechnet der Ingress `batch_uid = SHA-256(agent_id + Batch-Inhalt)` und der Processor verwendet `event_uid = batch_uid:index`. ReplacingMergeTree und `uniqExact`-States verhindern damit Zählfehler bei Queue- und identischen HTTP-Retries. + +## 8. UI ist kein Compute-Job + +Die Startseite liest `events_5m` und PostgreSQL-Findings. Sie startet keine Detection-Regeln. Die Event-Timeline hat immer ein begrenztes Zeitfenster und Limit. + +## 9. Retention nach Datenklasse + +- normalisierte Events: 90 Tage +- Raw: 30 Tage +- Rollups: 730 Tage +- Queue: 24 Stunden + +Diese Werte sind Defaults, keine Compliance-Aussage. Rechtliche und organisatorische Anforderungen haben Vorrang. diff --git a/Dockerfile b/Dockerfile index a693a48..c8dac4f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,32 +1,15 @@ -FROM golang:1.26 AS builder - +FROM golang:1.26.5-alpine AS build +RUN apk add --no-cache ca-certificates WORKDIR /src - -COPY go.mod go.sum ./ +COPY go.mod ./ RUN go mod download - COPY . . +RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/siem ./cmd/siem -RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ - go build -trimpath -ldflags="-s -w" -o /out/eventcollector . - -FROM debian:trixie-slim - -ENV LISTEN_ADDR=:8080 -ENV TZ=Europe/Berlin - -RUN apt-get update \ - && apt-get install -y --no-install-recommends ca-certificates tzdata wget \ - && rm -rf /var/lib/apt/lists/* \ - && groupadd --system --gid 10001 app \ - && useradd --system --uid 10001 --gid 10001 --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin app - +FROM scratch WORKDIR /app - -COPY --from=builder /out/eventcollector /app/eventcollector - -USER 10001:10001 - -EXPOSE 8080 - -ENTRYPOINT ["/app/eventcollector"] \ No newline at end of file +COPY --from=build /out/siem /siem +COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt +COPY web /app/web +USER 65532:65532 +ENTRYPOINT ["/siem"] diff --git a/Dockerfile.stress b/Dockerfile.stress deleted file mode 100644 index c29d056..0000000 --- a/Dockerfile.stress +++ /dev/null @@ -1,10 +0,0 @@ -FROM golang:1.26 AS builder -WORKDIR /src -COPY go.mod go.sum ./ -RUN go mod download -COPY cmd/siem-stress-agent ./cmd/siem-stress-agent -RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o /out/siem-stress-agent ./cmd/siem-stress-agent - -FROM gcr.io/distroless/static-debian12:nonroot -COPY --from=builder /out/siem-stress-agent /siem-stress-agent -ENTRYPOINT ["/siem-stress-agent"] diff --git a/README.md b/README.md index f04f12c..072dfb8 100644 --- a/README.md +++ b/README.md @@ -1,4701 +1,335 @@ -# SIEM Backend – Compact Metadata Variante +# Greenfield SIEM -## Ursache des erneuten Datenbankwachstums +Kompletter Neuaufbau des bisherigen Projekts. Vom Altprojekt bleibt absichtlich nur der HTTP-Ingress-Vertrag erhalten. -Die vorherige „Metadata-first“-Variante hatte noch einen entscheidenden Fehler: `insertBatch()` schrieb weiterhin **jedes einzelne Event** in `event_logs`. Zusätzlich war der Schlüssel von `event_occurrences` zu fein (u. a. Benutzer, IP, Workstation, Status und FailureReason), sodass diese Tabelle je nach Eventquelle fast wieder Event-Kardinalität erreichen konnte. +## Was dieses Projekt löst -Bei 19,2 Mio. Events in zwei Tagen entspricht der Eingang im Mittel rund 9,6 Mio. Events/Tag bzw. 111 Events/Sekunde. Für eine MariaDB ist diese Eventrate als reine Aggregatlast unkritisch; problematisch war die Speicherung und spätere Abfrage nahezu jeder Einzelzeile. +Der Eventpfad ist append-only, asynchron und auf große Datenmengen ausgelegt: -## Neues Speichermodell +```text +Windows Agents + │ POST /ingest + ▼ +Ingress API ──► PostgreSQL (nur Agent-Authentifizierung / Control Plane) + │ + ▼ +Redpanda (persistente Queue, 24h Recovery-Puffer) + │ + ▼ +Processor + ├──► ClickHouse (normalisierte, durchsuchbare Events, 90 Tage) + └──► gzip Spool ──► Garage/S3 (Raw-Batches, 30 Tage) + │ + └── rclone übernimmt Upload + Retention -- `event_count_buckets`: **universeller Primärspeicher für alle Events**. Eine Zeile pro 5-Minuten-Bucket, Host, Channel und Event-ID mit `cnt`, erstem und letztem Auftreten. -- `event_catalog`: dauerhafte kleine Landkarte pro Host/Channel/Event-ID. -- `event_occurrences`: nur noch **ausgewählte Security-Kontexte** (z. B. 4624, 4625, 4672, 4740). Benutzer/IP/Workstation werden nur dort gespeichert, wo sie für Detection/Forensik benötigt werden. -- `event_logs`: Full-Event-Hotstore ist jetzt standardmäßig **aus** (`STORE_EVENT_ROWS=false`). -- `event_log_raw`: Raw-XML ist standardmäßig **aus** (`STORE_RAW_XML=false`). Raw-XML impliziert technisch Full-Event-Zeilen und sollte nur für kurze gezielte Forensik aktiviert werden. - -Für einen Lockout (Security 4740) bleiben damit beispielsweise Event-ID, Host, Zielbenutzer, CallerComputer/Workstation, Anzahl sowie erstes/letztes Auftreten erhalten. Status-/Failure-Text, Source-IP und Subject-Machine-Account werden bei 4740 bewusst nicht zur Dimensionsbildung verwendet. - -## Produktionskonfiguration - -```env -STORE_EVENT_ROWS=false -STORE_RAW_XML=false -METADATA_CONTEXT_EVENT_IDS=4624,4625,4648,4672,4720,4726,4728,4732,4740,4756,4768,4769,4771,4776 -BASELINE_WINDOW=5m -METADATA_BUCKET=5m -EVENT_RETENTION=6h -RAW_RETENTION=6h -METADATA_RETENTION=720h -PARTITION_MAINTENANCE_ENABLED=true +ClickHouse ──► Detector ──► PostgreSQL (Detections / Status) +ClickHouse + PostgreSQL ──► API/UI ``` -`EVENT_RETENTION` und `RAW_RETENTION` spielen im empfohlenen Modus keine Rolle, weil diese Tabellen nicht mehr beschrieben werden. Sie begrenzen lediglich die Datenmenge, falls Full-Event-/Raw-Speicherung bewusst aktiviert wird. +**Keine Event-Zeilen in PostgreSQL. Keine MariaDB. Keine synchronen Detection-Abfragen im Ingress. Keine Raw-XML-Duplikate in ClickHouse.** -## UI und Detection +## One-click Deployment -`/ui` liest die 24h-Eventzahl und die letzten Events jetzt ausschließlich aus `event_count_buckets`. Ungefilterte Eventseiten erhalten außerdem automatisch ein 24h-Zeitfenster. Damit berührt die Startseite weder `event_logs` noch die detaillierte Kontexttabelle. +Voraussetzung: Docker Engine / Docker Desktop mit Docker Compose Plugin. -Failed-Logon-Spikes und Reboot-Spikes verwenden die universellen Counter. Password-Spray, Success-after-Failures, neue Source-IP und UEBA verwenden die kompakte Security-Kontexttabelle. Dynamische Regeln auf `hostname`, `channel`, `event_id`, `target_user`, `subject_user`, `src_ip` und `workstation` funktionieren ohne Full-Event-Store. Regeln auf `msg` oder `process_name` benötigen weiterhin `STORE_EVENT_ROWS=true` und werden ansonsten bewusst übersprungen und geloggt. +### Linux / macOS / WSL -## Bestehende Installation retten +```bash +chmod +x deploy.sh +./deploy.sh +``` -1. Datenbank-Snapshot/Backup erstellen und Ingest kurz stoppen. -2. Neues Backend deployen und sicherstellen, dass `STORE_EVENT_ROWS=false` sowie `STORE_RAW_XML=false` gesetzt sind. -3. `deploy/mariadb/migrations/004-compact-storage.sql` ausführen. -4. Backend starten und im Log die Zeile `storage mode: store_event_rows=false store_raw_xml=false ...` prüfen. -5. Erst danach `deploy/mariadb/emergency-compact-cleanup.sql` ausführen. Das Script leert `event_logs`, `event_log_raw` und die alte hochkardinale `event_occurrences`-Historie. `event_count_buckets`, `event_catalog`, Detections und Baselines bleiben erhalten. -6. Mit `deploy/mariadb/cardinality-diagnostics.sql` prüfen, wie viele reale Events eine Aggregatzeile repräsentiert. +### Windows PowerShell -Der Cleanup ist bewusst nicht Bestandteil der normalen Migration, da er historische Full-Event-/Kontextdaten löscht. +```powershell +./deploy.ps1 +``` -## Bevorzugtes Metadaten-Ingest +Das Deployment erzeugt beim ersten Start kryptographisch zufällige Secrets in `.env`, erzeugt die Garage-Konfiguration, baut die SIEM-Binaries und startet alle Dienste. -Collector können weiterhin XML senden, besser ist aber das strukturierte `meta`-Objekt. Das Backend extrahiert daraus nur die für die jeweilige Event-ID benötigten Dimensionen. +Danach: + +- UI: `http://SERVER:8080/ui` +- Ingress: `http://SERVER:8090/ingest` +- Redpanda Console: lokal `http://127.0.0.1:8081` +- Prometheus: lokal `http://127.0.0.1:9090` + +Beim ersten Start werden `UI_USERNAME=admin` und ein zufälliges `UI_PASSWORD` erzeugt. Browser fragen diese Zugangsdaten beim Öffnen von `/ui` ab. + +UI-Benutzer/Passwort und Enrollment-Key werden am Ende des Deployments ausgegeben und stehen außerdem in der geschützten `.env`. Die UI/API ist standardmäßig per HTTP Basic Auth geschützt; für Zugriff über ein fremdes Netz gehört zusätzlich TLS davor. + +> Bestehende Collector können den bisherigen Enrollment-Key weiterverwenden, ohne eine `.env` von Hand anzulegen. Linux/macOS/WSL: `ENROLLMENT_KEY="ALTER_KEY" ./deploy.sh`. PowerShell: `$env:ENROLLMENT_KEY="ALTER_KEY"; ./deploy.ps1`. Alle anderen Secrets werden trotzdem zufällig erzeugt. + +## Bestehender Ingress bleibt kompatibel + +```http +POST /ingest +X-API-Key: +X-Enrollment-Key: +Content-Type: application/json +``` + +Beispiel: ```json [ { - "host": "DC01.example.local", + "host": "CLIENT-01", "channel": "Security", "id": 4740, - "source": "Microsoft-Windows-Security-Auditing", - "ts": "2026-07-18T12:34:56Z", + "source": "windows-agent", + "ts": "2026-07-23T12:00:00Z", "meta": { + "provider": "Microsoft-Windows-Security-Auditing", "target_user": "alice", - "device": "CLIENT-42", - "provider": "Microsoft-Windows-Security-Auditing" + "workstation": "CLIENT-42", + "src_ip": "10.10.20.15" } } ] ``` -## Datenbankwahl +Alternativ darf `msg` weiterhin das bisherige Windows-XML enthalten. Der Processor extrahiert daraus die relevanten Felder. Das XML wird **nicht** als ClickHouse-Spalte gespeichert, sondern nur im komprimierten Raw-Archiv. -MariaDB bleibt für dieses Modell sinnvoll. Der entscheidende Punkt ist, dass die Datenbank nicht mehr als Roh-Event-Store verwendet wird. Die langfristige Last besteht nun überwiegend aus Upserts auf Zeit-Buckets und kleinen relationalen Dimensionen. Ein Wechsel zu ClickHouse wäre erst dann sinnvoll, wenn sehr große Rohdatenmengen oder breite Ad-hoc-Analysen ausdrücklich wieder Teil des Ziels werden. +Erfolgreiche Aufnahme: ---- - -# SIEM-lite Admin-Handbuch: Anlernphase, manuelle Bewertung und Lerneffekt - -> **Architekturhinweis:** Das folgende ältere Handbuch enthält an einigen Stellen -> noch SQL-/Tabellenbeispiele aus der Full-Event-Architektur. Für Speichermodell, -> Performance und aktuelle Detection-Datenquellen ist ausschließlich der Abschnitt -> „Compact Metadata Variante“ oben maßgeblich. - -Stand: 2026-04-27 -Zielgruppe: Administratoren ohne SIEM-/SOC-Vorerfahrung -Projekt: `siem-backend` / `SIEM-lite Security Operations` - ---- - -## 1. Ziel dieses Dokuments - -Dieses Dokument erklärt, wie die Software lernt, wie die Anlernphase funktioniert und wie man Detections manuell bewertet, ohne den Lerneffekt zu verfälschen. - -Es richtet sich ausdrücklich an Administratoren, die die Software sicher im Alltag betreiben sollen. Der Fokus liegt nicht auf Go-Code, sondern auf dem praktischen Umgang mit der Oberfläche, den Bewertungen und den Auswirkungen auf Baseline, UEBA und Host-Risiko. - -Nach dem Lesen sollte ein Admin verstehen: - -- warum am Anfang viele Meldungen entstehen, -- welche Meldungen normal sind, -- welche Meldungen nicht einfach weggeklickt werden dürfen, -- wie die Status `plausible`, `legitimate`, `false_positive`, `suppressed`, `resolved` und `confirmed_incident` zu verwenden sind, -- wann eine Baseline lernen darf, -- wann eine Baseline nicht lernen darf, -- wie man Wartungsfenster, Patchtage und bekannte Sonderfälle korrekt bewertet, -- warum eine falsche Bewertung gefährlich sein kann, -- wie der Host Risk Score beeinflusst wird. - ---- - -## 2. Grundprinzip der Software - -SIEM-lite sammelt Windows-Events von Agents, speichert diese zentral und wertet sie regelmäßig über Regeln aus. - -Es gibt grob vier Arten von Erkennung: - -1. **Feste Regeln** - - z. B. viele fehlgeschlagene Logons, Reboot-Spikes, Password-Spray. - -2. **Dynamische Regeln** - - Regeln, die ein Admin in der Weboberfläche selbst anlegt. - -3. **Baseline-Regeln** - - statistische Erkennung von ungewöhnlich vielen Events im Vergleich zum gelernten Normalzustand. - -4. **UEBA-Regeln** - - verhaltensbasierte Regeln, z. B. neuer Login-Kontext für Benutzer oder privilegierter Benutzer auf neuem Host. - -Die Software ist kein fertiges „Wahrheitsorakel“. Sie liefert Hinweise. Ein Mensch muss bewerten, ob ein Hinweis legitim, plausibel, ein Fehlalarm oder ein echter Vorfall ist. - ---- - -## 3. Was bedeutet „Anlernphase“? - -Die Anlernphase ist der Zeitraum, in dem SIEM-lite sammelt, was in der Umgebung normal ist. - -Beispiele: - -- Welche EventIDs sendet ein bestimmter Host normalerweise? -- Wie viele Security-Events erzeugt ein Domain Controller typischerweise pro 5-Minuten-Fenster? -- Welche Benutzer melden sich normalerweise an welchen Hosts an? -- Von welchen Quell-IPs kommt ein Benutzer normalerweise? -- Welche Workstations sind für einen Benutzer bekannt? -- Welche privilegierten Benutzer arbeiten auf welchen Systemen? - -Die Software baut daraus Baselines und bekannte Kontexte auf. - ---- - -## 4. Warum ist die Anlernphase kritisch? - -Während der Anlernphase entscheidet sich, was später als „normal“ gilt. - -Wenn während der Anlernphase ein Sicherheitsvorfall passiert und dieser unkritisch eingelernt wird, kann die Software dieses Verhalten später als normal ansehen. - -Beispiel: - -Ein Angreifer nutzt nachts einen Admin-Account auf einem ungewöhnlichen Client. Wenn dieses Verhalten als normal eingelernt wird, kann später ein ähnlicher Zugriff weniger auffallen. - -Deshalb ist wichtig: - -- Die Anlernphase sollte bewusst begleitet werden. -- Auffällige Detections sollten nicht wahllos als legitim markiert werden. -- Confirmed Incidents dürfen nicht in die Baseline einfließen. -- Wartungsfenster sollten dokumentiert und korrekt bewertet werden. - ---- - -## 5. Welche Lernmechanismen gibt es? - -SIEM-lite hat mehrere Lernbereiche. Diese müssen getrennt verstanden werden. - ---- - -# 6. Lernmechanismus 1: Event-Rate-Baseline - -## 6.1 Zweck - -Die Event-Rate-Baseline lernt, wie viele Events eines bestimmten Typs normalerweise auftreten. - -Sie betrachtet Kombinationen aus: - -```text -Host -Channel -EventID -Wochentag -Stunde +```json +{"accepted":1} ``` -Beispiel: +HTTP `202` bedeutet: Der Batch wurde authentifiziert und dauerhaft in die Queue geschrieben. ClickHouse muss dafür nicht synchron antworten. + +## Speicherdesign + +### ClickHouse `siem.events` + +Eine kanonische Tabelle enthält u. a.: + +- Event- und Ingest-Zeit +- Agent / Host / Channel / Provider / Event-Code +- normalisierte `category`, `action`, `outcome`, `severity` +- User / Subject / Target User +- Source-/Destination-IP und Ports +- Workstation +- Logon-Type +- Prozess / Parent-Prozess / Commandline +- seltene XML-Felder in `attributes Map(String,String)` +- Raw-Object-Key + Index im archivierten Originalbatch +- Queue-Partition / Offset / stabile Event-UID +- Ingest-Verzögerung zur Erkennung von Agent-/Queue-Zeitproblemen +- häufige Authentifizierungsdetails (Domain, Status/Substatus, FailureReason, Auth-Package) + +Häufig genutzte Felder werden **nicht** zusätzlich in `attributes` dupliziert. Die Map enthält nur unbekannte Restfelder, maximal 32 Einträge pro Event mit begrenzter Länge. Das vollständige Original bleibt im Raw-Archiv. + +Die Tabelle verwendet `ReplacingMergeTree`, Monats-Partitionen und einen zeitbasierten Sortierschlüssel. Die Standard-TTL beträgt 90 Tage. + +### Deduplication + +Die Pipeline ist absichtlich at-least-once. Ein Crash nach ClickHouse-Insert aber vor dem Redpanda-Commit kann einen Batch erneut liefern. + +Daher erhält schon der Ingress-Batch eine stabile ID aus `SHA-256(agent_id + Batch-Inhalt)`. Die Event-ID lautet: ```text -DC01 -Security -4625 -Montag -08:00 Uhr +: ``` -Die Software lernt also nicht nur: „DC01 hat normalerweise 4625-Events“, sondern genauer: „DC01 hat montags um 08:00 Uhr normalerweise eine bestimmte Menge 4625-Events“. +Damit werden sowohl ein Redpanda-Retry **als auch ein identischer HTTP-Retry des Collectors** als dasselbe Event erkannt. Queue-Partition und Offset werden zusätzlich nur zur Provenienz gespeichert. ---- +- Eventtabelle: `ReplacingMergeTree` +- Detection-Zähler: `uniqExact(event_uid)` +- Dashboard-Rollups: `uniqExactState(event_uid)` / `uniqExactMerge` -## 6.2 Warum nach Wochentag und Stunde? +Wiederholte Verarbeitung verfälscht damit keine Zähler. -Viele Systeme haben zeitabhängige Muster. +### Dashboard-Rollup -Beispiele: +`siem.events_5m` wird über eine Materialized View fortlaufend gepflegt. Die Startseite zählt nicht Millionen Einzelereignisse, sondern arbeitet auf dem 5-Minuten-Rollup. -- morgens viele Logons, -- nachts Backup-Jobs, -- montags mehr Fehlversuche nach Passwortänderungen, -- Patchfenster am Mittwochabend, -- GPO-/Softwareverteilung morgens nach dem Einschalten vieler Clients, -- Monitoring-Jobs alle paar Minuten. +Standard-Retention: -Ohne Zeitbezug würde die Baseline zu ungenau. - ---- - -## 6.3 Was wird gespeichert? - -In `baseline_event_stats` werden statistische Werte gespeichert: - -| Wert | Bedeutung | -|---|---| -| `avg_count` | durchschnittliche Event-Anzahl | -| `stddev_count` | Standardabweichung | -| `sample_count` | Anzahl gelernter Samples | -| `m2_count` | Hilfswert für laufende Varianzberechnung | -| `last_updated` | letzter Lernzeitpunkt | - ---- - -## 6.4 Wann lernt die Baseline? - -Die Baseline wird regelmäßig durch diese Regel aktualisiert: - -```text -baseline_update -``` - -Sie betrachtet das aktuelle Zeitfenster, z. B. 5 Minuten, zählt Events und aktualisiert die Statistik. - -Default: - -```text -BASELINE_WINDOW=5m -``` - ---- - -## 6.5 Wann erzeugt die Baseline einen Alarm? - -Die Detection-Regel heißt: - -```text -baseline_event_rate_anomaly -``` - -Sie erzeugt einen Alarm, wenn die aktuelle Event-Anzahl deutlich vom gelernten Normalwert abweicht. - -Vereinfacht: - -```text -aktueller Wert deutlich größer als gelernter Durchschnitt -``` - -Technisch wird ein Z-Score berechnet: - -```text -z = (aktueller Wert - Durchschnitt) / Standardabweichung -``` - ---- - -## 6.6 Mindestvoraussetzungen für Baseline-Alarme - -Ein Baseline-Alarm entsteht nicht sofort am ersten Tag. Die Software benötigt zuerst Samples. - -Default-Werte: - -| Parameter | Default | Bedeutung | -|---|---:|---| -| `BASELINE_MIN_SAMPLES` | `24` | mindestens 24 Vergleichswerte | -| `BASELINE_MIN_COUNT` | `10` | mindestens 10 Events im aktuellen Fenster | -| `BASELINE_MEDIUM_Z` | `2.5` | ab hier Medium | -| `BASELINE_HIGH_Z` | `4.0` | ab hier High | - -Das bedeutet: Ein Eventtyp muss erst mehrfach beobachtet worden sein, bevor eine zuverlässige Abweichung erkannt werden kann. - ---- - -## 6.7 Typisches Verhalten während der Anlernphase - -In den ersten Stunden oder Tagen können folgende Effekte auftreten: - -- noch keine Baseline-Alerts, weil `sample_count` zu niedrig ist, -- später plötzlich Baseline-Alerts, weil genug Samples vorhanden sind, -- einzelne Systeme wirken auffällig, obwohl sie nur besondere Aufgaben haben, -- Domain Controller erzeugen viel mehr Events als Clients, -- Patch-/Wartungstage erzeugen Sondermuster, -- neue Software erzeugt neue EventIDs oder neue Mengen. - -Das ist normal. - ---- - -## 6.8 Wie lange sollte die Baseline lernen? - -Eine sinnvolle Mindestdauer hängt von der Umgebung ab. - -Empfehlung: - -| Umgebung | Mindest-Anlernzeit | +| Datenklasse | Retention | |---|---:| -| kleine Testumgebung | 2 bis 3 Tage | -| normale Büro-IT | 7 bis 14 Tage | -| größere Umgebung mit Wochenmustern | 14 bis 30 Tage | -| Umgebung mit Monatsjobs | mindestens 30 Tage | +| Vollständig durchsuchbare normalisierte Events | 90 Tage | +| 5-Minuten-Rollups | 730 Tage | +| Raw-Batches in Garage/S3 | 30 Tage | +| Redpanda Recovery-Queue | 24 Stunden | +| Detections / Agent-Metadaten | dauerhaft, bis organisatorisch bereinigt | +| Prometheus Betriebsmetriken | 30 Tage | -Für produktive Sicherheitsbewertung sind 7 Tage oft das Minimum, weil damit jeder Wochentag einmal gesehen wurde. +Anpassbar vor dem ersten Start in `.env`: -Besser sind 14 bis 30 Tage. - ---- - -# 7. Lernmechanismus 2: UEBA-Baseline - -## 7.1 Zweck - -UEBA steht für User and Entity Behavior Analytics. - -SIEM-lite lernt Benutzerkontexte, also welche Benutzer sich normalerweise wie und wo anmelden. - -Ein Kontext besteht aus: - -```text -Benutzer -Host -Quell-IP -Workstation -``` - -Beispiel: - -```text -max.mustermann -CLIENT042 -10.2.40.15 -CLIENT042 -``` - ---- - -## 7.2 Welche Events nutzt UEBA? - -UEBA nutzt erfolgreiche Logons: - -```text -Security EventID 4624 -``` - -Die Software ignoriert dabei leere Benutzer, `-` und Maschinenkonten. - -Empfohlen ist zusätzlich, technische Accounts wie `SYSTEM`, `LOCAL SERVICE`, `NETWORK SERVICE` und ähnliche zu ignorieren. - ---- - -## 7.3 Welche Tabelle wird genutzt? - -UEBA schreibt bekannte Kontexte in: - -```text -ueba_user_baseline -``` - -Wichtige Felder: - -| Feld | Bedeutung | -|---|---| -| `username` | Benutzer | -| `hostname` | Zielhost | -| `src_ip` | Quell-IP | -| `workstation` | Workstation | -| `first_seen` | erster bekannter Zeitpunkt | -| `last_seen` | letzter bekannter Zeitpunkt | -| `seen_count` | Häufigkeit | - ---- - -## 7.4 Wie funktioniert UEBA-Lernen? - -Die Regel: - -```text -ueba_update -``` - -übernimmt erfolgreiche Logons in die UEBA-Baseline. - -Wenn ein Kontext neu ist: - -```text -first_seen = jetzt -last_seen = jetzt -seen_count = Anzahl -``` - -Wenn ein Kontext bekannt ist: - -```text -last_seen wird aktualisiert -seen_count wird erhöht -``` - ---- - -## 7.5 Warum ist die Reihenfolge wichtig? - -Im Detection-Loop sollte erst erkannt und danach gelernt werden. - -Empfohlene Reihenfolge: - -```text -ueba_admin_new_host -ueba_offhours_login -ueba_first_privileged_use -ueba_new_user_context -ueba_update -``` - -Wenn `ueba_update` zu früh läuft, würde ein neuer Kontext zuerst gelernt und danach nicht mehr als neu erkannt. - ---- - -## 7.6 Was erkennt `ueba_new_user_context`? - -Diese Regel erkennt, wenn ein Benutzer sich in einem neuen Kontext anmeldet. - -Beispiel: - -Benutzer `max.mustermann` arbeitet normalerweise auf `CLIENT042`, meldet sich aber plötzlich per RDP von einer unbekannten Workstation auf `SERVER17` an. - -Das kann legitim sein, aber es ist erklärungsbedürftig. - ---- - -## 7.7 Was erkennt `ueba_admin_new_host`? - -Diese Regel ist besonders wichtig. - -Sie erkennt, wenn ein privilegierter Benutzer sich erstmals auf einem Host anmeldet. - -Privilegierte Benutzer werden erkannt über: - -- Eintrag in `privileged_users`, -- Namensmuster wie `adm-*`, `*-adm`, `*.adm`. - -Beispiel: - -```text -adm.max meldet sich erstmals auf CLIENT099 an -``` - -Das ist oft relevanter als ein normaler Benutzerkontext. - ---- - -## 7.8 Was erkennt `ueba_offhours_login`? - -Diese Regel erkennt Anmeldungen außerhalb der Arbeitszeit. - -Empfohlene Arbeitszeit im aktuellen Konzept: - -```text -06:00 bis 20:00 Uhr -``` - -Problematisch sind insbesondere: - -- Admin-Logins nachts, -- RDP-Logins nach Feierabend, -- neue Quell-IP plus Off-Hours, -- vorherige Fehlversuche plus erfolgreicher Login, -- Logins auf Domain Controllern oder Servern. - -Wichtig: Technische Accounts wie `SYSTEM` müssen ausgeschlossen werden, sonst flutet diese Regel. - ---- - -## 7.9 Was erkennt `ueba_first_privileged_use`? - -Diese Regel erkennt, wenn ein Benutzer erstmals privilegierte Rechte nutzt. - -Basis ist: - -```text -Security EventID 4672 -``` - -Event 4672 bedeutet, dass einem neuen Logon besondere Rechte zugewiesen wurden. - -Diese Regel braucht eine eigene Baseline: - -```text -user_privilege_baseline -``` - -Ohne diese Baseline würde derselbe Benutzer immer wieder als „erstmals privilegiert“ auffallen. - ---- - -# 8. Lernmechanismus 3: Dynamic Rules - -## 8.1 Lernen Dynamic Rules? - -Dynamische Regeln lernen nicht im statistischen Sinne. - -Sie erkennen nach festen Bedingungen: - -- bestimmte EventIDs, -- bestimmte Channels, -- bestimmte Felder, -- bestimmte Schwellenwerte. - -Beispiel: - -```text -Wenn Security 1102 kommt, melde critical. -``` - -Dynamic Rules können aber durch Bewertung und Suppression indirekt beeinflusst werden. - ---- - -## 8.2 Wie beeinflusst Bewertung Dynamic Rules? - -Wenn eine Detection aus einer dynamischen Regel als `false_positive`, `legitimate`, `plausible`, `resolved` oder `suppressed` bewertet wird, beeinflusst das vor allem: - -- Host Risk Score, -- SOC-Übersicht, -- Arbeitsliste der Analysten. - -Die Regel selbst wird dadurch nicht geändert. - -Wenn eine dynamische Regel dauerhaft zu viele Meldungen erzeugt, muss die Regel angepasst werden: - -- EventID einschränken, -- Channel korrigieren, -- Match Field setzen, -- Threshold erhöhen, -- Suppress-Zeit setzen, -- Regel deaktivieren. - ---- - -# 9. Was bedeutet manuelle Bewertung? - -Manuelle Bewertung bedeutet: Ein Admin oder Analyst entscheidet, wie eine Detection fachlich einzuordnen ist. - -Eine Detection ist zunächst nur ein Hinweis. - -Beispiel: - -```text -UEBA: Privilegierter Benutzer adm.max meldet sich erstmals auf Host CLIENT099 an -``` - -Mögliche Bewertungen: - -- echter Vorfall, -- legitime Admin-Arbeit, -- plausibel wegen Rollout, -- Fehlalarm, -- bereits erledigt, -- künftig unterdrücken. - ---- - -## 10. Warum ist manuelle Bewertung wichtig? - -Die Bewertung beeinflusst: - -1. die SOC-Arbeitsliste, -2. den Host Risk Score, -3. die Qualität der Baseline, -4. die zukünftige Alarmflut, -5. die Nachvollziehbarkeit für andere Admins, -6. die Entscheidung, ob Verhalten gelernt oder ausgeschlossen wird. - -Eine falsche Bewertung kann gefährlich sein. - -Beispiele: - -| Falsche Bewertung | Risiko | -|---|---| -| Angriff als `legitimate` markiert | Verhalten verschwindet aus Risk Score | -| Incident nicht als `confirmed_incident` markiert | Baseline lernt möglicherweise bösartiges Verhalten | -| Wartungsfenster nicht dokumentiert | künftige Admins verstehen Detections nicht | -| Alles als `false_positive` markiert | echte Auffälligkeiten werden übersehen | -| Zu viele Suppressions | blinde Flecken entstehen | - ---- - -# 11. Statusmodell - -SIEM-lite verwendet mehrere Status. Diese sollten konsequent verwendet werden. - ---- - -## 11.1 `open` - -## Bedeutung - -Die Detection ist neu und noch nicht bewertet. - -## Wann verwenden? - -Normalerweise automatisch beim Erzeugen einer Detection. - -## Admin-Aktion - -Prüfen: - -- Welche Regel hat ausgelöst? -- Welcher Host ist betroffen? -- Welcher Benutzer ist betroffen? -- Welche Events liegen zugrunde? -- Gibt es zeitgleichen Kontext? -- Ist es ein Wartungsfenster? -- Gibt es mehrere ähnliche Detections? - -## Wirkung - -- zählt im Host Risk Score, -- erscheint in offenen Listen, -- bleibt prüfbedürftig. - ---- - -## 11.2 `acknowledged` - -## Bedeutung - -Die Detection wurde gesehen, aber noch nicht abschließend bewertet. - -## Wann verwenden? - -Wenn ein Admin signalisiert: - -```text -Ich habe es gesehen, prüfe aber später weiter. -``` - -## Beispiel - -Ein Server meldet viele fehlgeschlagene Logons. Der Admin hat gesehen, dass es relevant sein könnte, muss aber erst Logs auf dem Quellsystem prüfen. - -## Wirkung - -- zählt weiterhin im Host Risk Score, -- zählt schwächer als `investigating`, -- bleibt offen genug für Nachverfolgung. - ---- - -## 11.3 `investigating` - -## Bedeutung - -Die Detection wird aktiv untersucht. - -## Wann verwenden? - -Wenn konkrete Prüfung läuft: - -- Loganalyse, -- Rückfrage beim Benutzer, -- Prüfung von Quell-IP, -- Prüfung von RDP-/VPN-/Firewall-Logs, -- Prüfung auf kompromittierten Account. - -## Wirkung - -- zählt stärker im Host Risk Score, -- zeigt im SOC-Dashboard erhöhte Relevanz, -- signalisiert anderen Admins: Nicht ignorieren. - ---- - -## 11.4 `plausible` - -## Bedeutung - -Die Detection sieht nachvollziehbar aus, ist aber nicht zwingend dauerhaft legitim. - -`plausible` bedeutet: - -```text -Das passt wahrscheinlich zum Kontext, soll aber nicht als dauerhaft normaler Zustand betrachtet werden. -``` - -## Typische Fälle - -- Patchday, -- Software-Rollout, -- einmalige Migration, -- einmalige Admin-Aktion, -- Inventarisierung, -- Testlauf, -- neue GPO-Verteilung, -- einmaliges Wartungsfenster. - -## Beispiel - -```text -reboot_spike auf 200 Clients während Patchfenster -``` - -Das ist plausibel, aber nicht automatisch dauerhaft normal. - -## Wirkung - -- wird aus Host Risk Score ausgeschlossen, -- bleibt historisch nachvollziehbar, -- sollte mit Notiz versehen werden, -- sollte nicht blind als dauerhafte Legitimität gelten. - -## Wichtig - -`plausible` ist oft die beste Bewertung für temporäre Sonderlagen. - ---- - -## 11.5 `legitimate` - -## Bedeutung - -Die Detection ist fachlich korrekt, aber das Verhalten ist dauerhaft oder regelmäßig legitim. - -`legitimate` bedeutet: - -```text -Dieses Verhalten ist bekannt, erwartet und akzeptiert. -``` - -## Typische Fälle - -- bekannter Admin-Jump-Host, -- legitimer Service-Account, -- regelmäßiger Backup-Job, -- bekannter Monitoring-Login, -- geplanter nächtlicher Wartungsprozess, -- Admin meldet sich regelmäßig auf einem bestimmten Server an. - -## Wirkung - -- wird aus Host Risk Score ausgeschlossen, -- setzt `is_legitimate = true`, -- kann zusammen mit Suppression oder Baseline-Exclusion genutzt werden. - -## Vorsicht - -Nicht zu schnell `legitimate` verwenden. - -Eine einmalige plausible Aktion ist nicht automatisch dauerhaft legitim. - ---- - -## 11.6 `false_positive` - -## Bedeutung - -Die Detection ist ein Fehlalarm. - -Das heißt: - -```text -Die Regel hat fachlich falsch oder irrelevant ausgelöst. -``` - -## Typische Fälle - -- Parser erkennt falsches Feld, -- Event wird falsch interpretiert, -- Regel ist zu breit, -- Testdaten, -- bekannter technischer Account wurde nicht ausgeschlossen, -- LogonType wurde nicht gefiltert, -- Event stammt nicht aus realem Benutzerverhalten. - -## Wirkung - -- wird aus Host Risk Score ausgeschlossen, -- setzt `is_false_positive = true`, -- sollte Anlass sein, die Regel zu verbessern. - -## Wichtig - -`false_positive` sollte nicht für echte, aber harmlose Ereignisse verwendet werden. - -Für echte, aber harmlose Ereignisse besser: - -```text -plausible -oder -legitimate -``` - ---- - -## 11.7 `resolved` - -## Bedeutung - -Die Detection wurde bearbeitet und abgeschlossen. - -## Wann verwenden? - -Wenn: - -- die Ursache gefunden wurde, -- keine weitere Aktion nötig ist, -- ein Incident abgearbeitet wurde, -- eine legitime Ursache dokumentiert wurde, -- technische Korrektur erfolgt ist. - -## Wirkung - -- wird aus Host Risk Score ausgeschlossen, -- bleibt historisch sichtbar, -- eignet sich für abgeschlossene Einzelfälle. - ---- - -## 11.8 `suppressed` - -## Bedeutung - -Die Detection wird bewusst unterdrückt oder soll nicht mehr aktiv betrachtet werden. - -## Wann verwenden? - -Wenn das Verhalten bekannt ist und künftig nicht mehr alarmieren soll. - -Beispiele: - -- bekannter technischer Prozess, -- bewusst akzeptiertes Rauschen, -- System außerhalb Überwachungsumfang, -- Testhost, -- bestimmte Regel auf bestimmtem Host irrelevant. - -## Wirkung - -- wird aus Host Risk Score ausgeschlossen, -- kann zusätzlich eine Detection-Suppression erzeugen, -- reduziert künftige Meldungen. - -## Vorsicht - -Suppression erzeugt blinde Flecken. - -Vor Suppression immer prüfen: - -- Ist die Regel auf diesem Host wirklich irrelevant? -- Soll nur diese Regel unterdrückt werden? -- Soll nur ein EventID-Typ unterdrückt werden? -- Soll die Suppression zeitlich begrenzt sein? -- Könnte ein Angreifer dieses Verhalten ausnutzen? - ---- - -## 11.9 `confirmed_incident` - -## Bedeutung - -Die Detection ist ein bestätigter Sicherheitsvorfall. - -## Wann verwenden? - -Wenn es Hinweise oder Belege gibt für: - -- kompromittierten Account, -- erfolgreiche unberechtigte Anmeldung, -- Malware, -- unautorisierte Service-Installation, -- gelöschte Security Logs, -- laterale Bewegung, -- verdächtige Admin-Aktivität, -- echter Password-Spray mit Treffer. - -## Wirkung - -- erhöht Host Risk Score stark, -- zählt als bestätigter Incident, -- verhindert Baseline-Lernen für passende Events im Fenster, -- erzeugt automatisch Baseline-Exclusion für 7 Tage, sofern keine andere Baseline-Aktion gewählt wird. - -## Wichtig - -`confirmed_incident` ist der wichtigste Status, um gefährliches Verhalten nicht als normal einzulernen. - ---- - -# 12. Welche Bewertung beeinflusst den Lerneffekt? - -Nicht jeder Status beeinflusst Lernen gleich. - -## 12.1 Einfluss auf Host Risk Score - -Diese Status werden aus dem Host Risk Score ausgeschlossen: - -```text -false_positive -suppressed -legitimate -resolved -plausible -``` - -Diese Status bleiben relevant: - -```text -open -acknowledged -investigating -confirmed_incident -``` - -`confirmed_incident` erhöht den Risk Score besonders stark. - ---- - -## 12.2 Einfluss auf Baseline-Lernen - -Baseline-Lernen wird beeinflusst durch: - -1. `baseline_exclusions` -2. `confirmed_incident` -3. manuelle Baseline-Aktionen im UI - -Wichtig: - -Eine Bewertung als `legitimate` oder `plausible` verhindert nicht automatisch immer das zukünftige Lernen aller ähnlichen Events, außer es wird zusätzlich eine Baseline-Exclusion gesetzt. - ---- - -## 12.3 Einfluss auf UEBA-Lernen - -UEBA-Lernen passiert über: - -```text -ueba_update -``` - -Dieses lernt erfolgreiche Logon-Kontexte. - -Die aktuelle Bewertung einer Detection ändert nicht automatisch rückwirkend die UEBA-Baseline. - -Das bedeutet: - -Wenn ein verdächtiger Login bereits in `ueba_user_baseline` eingelernt wurde, muss man ihn gegebenenfalls manuell aus der Baseline entfernen oder die Logik erweitern. - ---- - -## 12.4 Einfluss auf Dynamic Rules - -Manuelle Bewertung ändert eine Dynamic Rule nicht automatisch. - -Wenn eine Dynamic Rule zu breit ist, muss sie angepasst werden. - ---- - -# 13. Baseline-Aktionen im UI - -In der Detection-Bewertung gibt es eine Baseline-Auswahl: - -```text -keine Änderung -nicht lernen: 24h -nicht lernen: 7 Tage -nicht lernen: 30 Tage -nicht lernen: dauerhaft -``` - -Diese Aktionen erzeugen Einträge in: - -```text -baseline_exclusions -``` - ---- - -## 13.1 `keine Änderung` - -## Bedeutung - -Die Bewertung beeinflusst die Baseline nicht direkt. - -## Wann verwenden? - -Wenn: - -- die Detection nicht aus der Baseline-Regel stammt, -- die Baseline weiter normal lernen soll, -- das Verhalten künftig durchaus als normal gelten darf, -- keine Gefahr besteht, dass ein Incident eingelernt wird. - ---- - -## 13.2 `nicht lernen: 24h` - -## Bedeutung - -Für passende Host/Channel/EventID-Kombination wird 24 Stunden nicht gelernt. - -## Wann verwenden? - -- kurzer Test, -- kleines Wartungsfenster, -- einmaliger Rollout, -- temporärer Zustand, -- kurze Störung. - -## Beispiel - -```text -Ein Client erzeugt wegen Testkonfiguration ungewöhnlich viele Events. -``` - ---- - -## 13.3 `nicht lernen: 7 Tage` - -## Bedeutung - -Die Baseline lernt eine Woche lang nicht für diese Kombination. - -## Wann verwenden? - -- Patchwoche, -- Migrationsphase, -- bekannter Vorfall, -- größere Umstellung, -- temporäre Fehlkonfiguration. - -## Beispiel - -```text -Ein neues Login-Script erzeugt eine Woche lang erhöhte Event-Mengen, bis es korrigiert wird. -``` - ---- - -## 13.4 `nicht lernen: 30 Tage` - -## Bedeutung - -Ein Monat lang wird nicht gelernt. - -## Wann verwenden? - -- längerer Parallelbetrieb, -- größere Migration, -- unsicherer Zustand, -- neue Infrastrukturphase, deren Normalität noch nicht bewertet ist. - -## Vorsicht - -30 Tage sind lang. Nicht leichtfertig verwenden. - ---- - -## 13.5 `nicht lernen: dauerhaft` - -## Bedeutung - -Die Kombination wird dauerhaft nicht in die Baseline übernommen. - -## Wann verwenden? - -Nur wenn klar ist: - -- diese Eventklasse ist für Baseline-Lernen ungeeignet, -- der Host erzeugt absichtlich untypisches Verhalten, -- das Ereignis soll nie als Normalzustand betrachtet werden, -- es handelt sich um dauerhaftes Rauschen. - -## Vorsicht - -Dauerhafte Ausschlüsse können Erkennungsqualität reduzieren. - ---- - -# 14. Detection-Suppression im UI - -Neben Baseline-Aktionen gibt es: - -```text -künftig unterdrücken -24h -7 Tage -30 Tage -dauerhaft -``` - -Das betrifft nicht die Baseline, sondern künftige Detections. - ---- - -## 14.1 Unterschied zwischen Suppression und Baseline-Exclusion - -| Mechanismus | Wirkung | -|---|---| -| Suppression | verhindert künftige Detections | -| Baseline-Exclusion | verhindert Lernen in der Baseline | -| Status `plausible` | bewertet aktuelle Detection | -| Status `legitimate` | bewertet aktuelle Detection als legitim | -| Status `false_positive` | bewertet aktuelle Detection als Fehlalarm | - -Diese Dinge sind nicht identisch. - ---- - -## 14.2 Beispiel - -Ein Patchday erzeugt viele `reboot_spike`-Meldungen. - -Mögliche Bewertung: - -```text -Status: plausible -Notiz: Patchday April 2026 -Baseline: nicht lernen: 24h -Suppression: optional 24h -``` - -Damit wird: - -- die aktuelle Detection aus dem Risiko genommen, -- dokumentiert, warum sie auftrat, -- verhindert, dass das Patchday-Verhalten als Normalzustand gelernt wird, -- optional weitere identische Meldungen für 24h reduziert. - ---- - -# 15. Empfohlener Ablauf während der Anlernphase - -## 15.1 Phase 0: Vorbereitung - -Vor dem produktiven Start prüfen: - -- Agents liefern Events? -- Zeiten stimmen? -- Zeitzone ist korrekt? -- Domain Controller, Server und Clients melden? -- Security Channel wird korrekt übertragen? -- Event XML wird korrekt normalisiert? -- `target_user`, `subject_user`, `src_ip`, `logon_type` werden gefüllt? -- technische Accounts werden ausgeschlossen? -- Privileged Users sind gepflegt? -- bekannte Service Accounts sind bekannt? -- Patchfenster sind dokumentiert? - ---- - -## 15.2 Phase 1: Erste 24 Stunden - -Ziel: - -```text -Datenqualität prüfen, nicht sofort hart alarmieren. -``` - -Aufgaben: - -- `/ui/events` prüfen. -- Sind EventIDs plausibel? -- Sind Hostnamen korrekt? -- Werden Domain Controller anders sichtbar als Clients? -- Gibt es Encoding-Probleme? -- Werden `SYSTEM`-Logons ausgeschlossen? -- Werden Maschinenkonten ignoriert? -- Sind `4624`, `4625`, `4672` sinnvoll befüllt? -- Erste Dynamic Rules vorsichtig aktivieren. - -Bewertung: - -- technische Fehlalarme als `false_positive`, -- normale Start-/Patch-/Rollout-Effekte als `plausible`, -- echte Auffälligkeiten als `investigating`, -- bestätigte Vorfälle als `confirmed_incident`. - ---- - -## 15.3 Phase 2: Tag 2 bis Tag 7 - -Ziel: - -```text -Normale Tagesmuster verstehen. -``` - -Aufgaben: - -- täglich offene Detections prüfen, -- wiederkehrende False Positives identifizieren, -- Dynamic Rules nachschärfen, -- technische Accounts ergänzen, -- bekannte Wartungsjobs dokumentieren, -- Privileged Users vervollständigen, -- Host Risk Scores beobachten. - -Bewertung: - -- nicht alles unterdrücken, -- lieber Notizen setzen, -- bei Wartung `plausible`, -- bei dauerhaft bekannten Mustern `legitimate`, -- bei Regelproblemen `false_positive`. - ---- - -## 15.4 Phase 3: Woche 2 bis Woche 4 - -Ziel: - -```text -Wochenmuster stabilisieren. -``` - -Aufgaben: - -- Baseline-Anomalien ernsthaft bewerten, -- wiederkehrende Muster prüfen, -- dauerhafte Ausnahmen bewusst setzen, -- Suppressions aufräumen, -- echte Admin-Jump-Hosts dokumentieren, -- Service Accounts von echten Benutzern trennen, -- kritische Dynamic Rules finalisieren. - -Bewertung: - -- wiederkehrende legitime Jobs: `legitimate`, -- einmalige Wartung: `plausible`, -- bestätigte Incidents: `confirmed_incident`, -- schlechte Regel: `false_positive` und Regel korrigieren. - ---- - -## 15.5 Phase 4: Regelbetrieb - -Ziel: - -```text -Tägliche Sicherheitsüberwachung. -``` - -Aufgaben: - -- offene High/Critical Detections prüfen, -- Host Risk Score prüfen, -- confirmed incidents nachverfolgen, -- neue Dynamic Rules bei Bedarf erstellen, -- Baseline-Exclusions regelmäßig prüfen, -- Suppressions regelmäßig prüfen. - ---- - -# 16. Praktische Bewertungsentscheidung - -## 16.1 Entscheidungsbaum - -Bei jeder Detection fragen: - -```text -1. Ist das Ereignis technisch korrekt erkannt? -``` - -Wenn nein: - -```text -false_positive -Regel/Parser korrigieren -``` - -Wenn ja: - -```text -2. Ist das Verhalten erwartet? -``` - -Wenn nein oder unklar: - -```text -investigating -``` - -Wenn ja: - -```text -3. Ist es einmalig/temporär oder dauerhaft normal? -``` - -Temporär: - -```text -plausible -Notiz setzen -ggf. Baseline nicht lernen 24h/7d -``` - -Dauerhaft normal: - -```text -legitimate -ggf. Suppression oder Regelanpassung -``` - -Wenn bestätigt bösartig: - -```text -confirmed_incident -Baseline nicht lernen -Incident-Prozess starten -``` - ---- - -## 16.2 Kurzmatrix - -| Situation | Status | Baseline-Aktion | Suppression | -|---|---|---|---| -| echter Angriff | `confirmed_incident` | nicht lernen 7d/30d | nein | -| unklar, wird geprüft | `investigating` | keine oder nicht lernen 24h | nein | -| Patchday | `plausible` | nicht lernen 24h | optional 24h | -| geplanter Rollout | `plausible` | nicht lernen 24h/7d | optional | -| dauerhafter Backup-Job | `legitimate` | ggf. dauerhaft nicht lernen | ggf. dauerhaft | -| Parserfehler | `false_positive` | keine | nein, Regel fixen | -| Regel zu breit | `false_positive` | keine | nein, Regel anpassen | -| Testsystem rauscht | `suppressed` | optional | ja | -| Vorfall abgeschlossen | `resolved` | abhängig vom Fall | abhängig vom Fall | - ---- - -# 17. Beispiele aus dem Alltag - ---- - -## 17.1 Beispiel: Patchday erzeugt Reboot-Spikes - -Detection: - -```text -reboot_spike auf vielen Clients -``` - -Analyse: - -- Zeitpunkt passt zum Patchfenster. -- Viele Clients betroffen. -- Keine weiteren verdächtigen Events. -- Reboots sind erwartet. - -Bewertung: - -```text -Status: plausible -Notiz: Patchday April 2026, WSUS-Rollout -Baseline: nicht lernen: 24h -Suppression: optional 24h -``` - -Nicht verwenden: - -```text -false_positive -``` - -Warum? - -Die Detection war fachlich korrekt. Es war kein Fehlalarm, sondern erwartetes Verhalten. - ---- - -## 17.2 Beispiel: SYSTEM flutet Off-Hours-Login - -Detection: - -```text -ueba_offhours_login: Benutzer SYSTEM auf SERVER01 -``` - -Analyse: - -- SYSTEM ist technischer Account. -- Solche Logons sind normal. -- Die Regel ist zu breit. - -Bewertung: - -```text -Status: false_positive -Notiz: Technischer Account SYSTEM, Regel um Noise-Account-Filter erweitern -Baseline: keine Änderung -Suppression: nur kurzfristig, besser Regel fixen -``` - -Zusätzliche Maßnahme: - -- `SYSTEM` in `isNoiseAccount()` ausschließen. -- LogonType auf `2,7,10,11` begrenzen. - ---- - -## 17.3 Beispiel: Admin meldet sich erstmals auf Client an - -Detection: - -```text -ueba_admin_new_host: adm.max auf CLIENT099 -``` - -Analysefragen: - -- Hat der Admin dort gearbeitet? -- Gab es ein Ticket? -- War es Remotehilfe? -- Ist CLIENT099 ein normaler Client? -- Gab es vorher Fehlversuche? -- War es außerhalb Arbeitszeit? -- Von welcher Quell-IP kam der Login? - -Mögliche Bewertungen: - -### Legitime Remotehilfe - -```text -Status: plausible -Notiz: Remotehilfe Ticket #12345 durch adm.max -Baseline: keine Änderung oder nicht lernen 24h -``` - -### Admin nutzt Client regelmäßig als Admin-Station - -```text -Status: legitimate -Notiz: CLIENT099 ist Admin-Workstation von adm.max -Baseline: lernen erlauben -``` - -### Nicht erklärbar - -```text -Status: investigating -Notiz: Admin-Login auf ungewohntem Client, Rückfrage offen -``` - -### Kompromittierung bestätigt - -```text -Status: confirmed_incident -Notiz: adm.max kompromittiert, unautorisierter Login -Baseline: nicht lernen 7d/30d -``` - ---- - -## 17.4 Beispiel: Password-Spray - -Detection: - -```text -password_spray von 10.2.50.77 gegen 18 Benutzer -``` - -Analyse: - -- Quell-IP identifizieren. -- Gehört sie zu VPN, Proxy, Server, Scanner? -- Gab es erfolgreiche Logons danach? -- Sind privilegierte Benutzer betroffen? -- Gibt es Account Lockouts? - -Bewertung bei echtem Angriff: - -```text -Status: confirmed_incident -Notiz: Password-Spray von 10.2.50.77, mehrere Benutzer betroffen -Baseline: nicht lernen 7d -Suppression: nein -``` - -Bewertung bei bekanntem Scanner mit falschen Credentials: - -```text -Status: legitimate oder plausible -Notiz: Scanner X mit falschem Credential, Ticket zur Korrektur erstellt -Baseline: nicht lernen 24h oder 7d -Suppression: optional kurzzeitig -``` - ---- - -## 17.5 Beispiel: Baseline-Anomalie auf Domain Controller - -Detection: - -```text -baseline_event_rate_anomaly -DC01 Security 4625 kam 200-mal in 5 Minuten -``` - -Analyse: - -- Gibt es zeitgleich `password_spray`? -- Gibt es Account Lockouts? -- Welche Quell-IP? -- Welche Benutzer? -- Patch-/Wartungsfenster? -- Neue Anwendung? -- VPN/RADIUS/LDAP-Fehler? - -Bewertung: - -Bei unbekannter Ursache: - -```text -Status: investigating -Baseline: nicht lernen 24h -``` - -Bei bestätigtem Angriff: - -```text -Status: confirmed_incident -Baseline: nicht lernen 7d/30d -``` - -Bei bekannter Fehlkonfiguration: - -```text -Status: plausible -Notiz: Dienst XY nutzt altes Passwort, Korrektur geplant -Baseline: nicht lernen 7d -``` - ---- - -# 18. Was darf während der Anlernphase nicht passieren? - -## 18.1 Nicht alles als legitim markieren - -Falsch: - -```text -Alle Detections der ersten Woche sind bestimmt normal. -``` - -Warum gefährlich? - -- Angriffe können eingelernt werden. -- Host Risk Score wird künstlich reduziert. -- Auffällige Admin-Logins verschwinden aus der Priorisierung. - ---- - -## 18.2 Nicht alles dauerhaft unterdrücken - -Falsch: - -```text -Diese Regel nervt, also Suppression dauerhaft. -``` - -Warum gefährlich? - -- Künftige echte Vorfälle werden nicht mehr gemeldet. -- Angreifer können Verhalten nutzen, das blindgeschaltet wurde. -- SOC-Übersicht wird scheinbar ruhig, aber weniger sicher. - ---- - -## 18.3 Confirmed Incidents nicht als plausible markieren - -Falsch: - -```text -Der Angriff ist ja vorbei, also plausible. -``` - -Richtig: - -```text -confirmed_incident -``` - -Warum? - -Nur so wird klar: - -- es war ein echter Vorfall, -- der Host Risk Score wird korrekt erhöht, -- Baseline-Lernen wird verhindert, -- spätere Auswertungen bleiben korrekt. - ---- - -## 18.4 Parserfehler nicht als legitimate markieren - -Falsch: - -```text -SYSTEM-Noise ist normal, also legitimate. -``` - -Besser: - -```text -false_positive -Regel korrigieren -``` - -Warum? - -Wenn die Regel falsch filtert, ist das kein legitimer Sicherheitsfall, sondern ein Regelproblem. - ---- - -# 19. Wie Notizen geschrieben werden sollten - -Jede manuelle Bewertung sollte eine brauchbare Notiz enthalten. - -Gute Notizen beantworten: - -- Was war die Ursache? -- Wer hat es geprüft? -- Gibt es ein Ticket? -- Ist es einmalig oder dauerhaft? -- Welche Maßnahme wurde getroffen? -- Soll das Verhalten künftig gelernt werden? - ---- - -## 19.1 Gute Beispiele - -```text -Patchday April 2026, WSUS-Rollout auf Clients, Reboots erwartet. -``` - -```text -Remotehilfe Ticket #4711, adm.max hat sich auf CLIENT099 angemeldet. -``` - -```text -Dienstkonto svc-backup nach Passwortwechsel mit altem Credential, Ticket #4812 zur Korrektur. -``` - -```text -Bestätigter Password-Spray von 10.2.50.77, Firewall-Block gesetzt, betroffene Benutzer geprüft. -``` - ---- - -## 19.2 Schlechte Beispiele - -```text -ok -``` - -```text -normal -``` - -```text -passt -``` - -```text -weg -``` - -```text -keine Ahnung -``` - -Solche Notizen helfen später niemandem. - ---- - -# 20. Auswirkungen auf den Host Risk Score - -Der Host Risk Score aggregiert relevante Detections pro Host. - -## 20.1 Diese Status zählen weiter - -```text -open -acknowledged -investigating -confirmed_incident -``` - -## 20.2 Diese Status zählen nicht weiter - -```text -false_positive -suppressed -legitimate -resolved -plausible -``` - -## 20.3 Warum ist das wichtig? - -Wenn zu viel als `plausible`, `legitimate` oder `false_positive` markiert wird, fällt der Host Risk Score künstlich ab. - -Wenn echte Vorfälle als `confirmed_incident` markiert werden, steigt der Host Risk Score deutlich. - ---- - -## 20.4 Beispiel - -Host `CLIENT099` hat: - -- 1x `ueba_admin_new_host` high open, -- 1x `success_after_failures` high investigating, -- 1x `password_spray` medium open. - -Der Host Risk Score ist erhöht. - -Wenn alle drei Detections als `plausible` markiert werden, fällt der Host aus der Priorisierung. - -Das ist nur korrekt, wenn wirklich alle drei plausibel sind. - ---- - -# 21. Auswirkungen auf Baseline-Qualität - -## 21.1 Gute Baseline - -Eine gute Baseline entsteht, wenn normale Ereignisse gelernt werden und unnormale Ereignisse nicht gelernt werden. - -Normales Verhalten: - -- tägliche Anmeldemuster, -- übliche Eventmengen, -- regelmäßige Systemjobs, -- bekannte Wartungsfenster, wenn sie regelmäßig sind. - -Nicht lernen: - -- Angriffe, -- Fehlkonfigurationen, -- Testfluten, -- einmalige Rollouts, -- Massenfehler, -- außergewöhnliche Störungen. - ---- - -## 21.2 Schlechte Baseline - -Eine schlechte Baseline entsteht, wenn außergewöhnliches Verhalten als normal gelernt wird. - -Beispiele: - -- ein kompromittierter Account während Anlernphase, -- ein falsch konfigurierter Dienst mit tausenden Fehlversuchen, -- Patchday als tägliches Normalverhalten, -- einmalige Migration als Normalzustand, -- SYSTEM-Noise in Benutzerregeln. - ---- - -## 21.3 Symptome einer schlechten Baseline - -- echte Auffälligkeiten erzeugen keine Alerts, -- viele irrelevante Alerts, -- Baseline-Werte sind extrem hoch, -- Z-Scores bleiben niedrig trotz auffälligem Verhalten, -- bestimmte Hosts erscheinen nie auffällig, -- andere Hosts sind ständig auffällig. - ---- - -## 21.4 Was tun bei schlechter Baseline? - -Möglichkeiten: - -1. Baseline-Einträge gezielt löschen. -2. Baseline für bestimmte Kombinationen neu aufbauen. -3. Baseline-Exclusions setzen. -4. Regeln schärfen. -5. technische Accounts ausschließen. -6. Retention/Reset-Konzept einführen. - ---- - -# 22. Auswirkungen auf UEBA-Qualität - -## 22.1 Gute UEBA-Baseline - -Eine gute UEBA-Baseline enthält echte, normale Benutzerkontexte. - -Beispiele: - -- Benutzer A arbeitet auf Client A. -- Admin B arbeitet auf Jump Host B. -- Service Account C nutzt Server C. -- VPN-Benutzer haben bekannte Quellbereiche. - ---- - -## 22.2 Schlechte UEBA-Baseline - -Eine schlechte UEBA-Baseline enthält verdächtige oder falsche Kontexte. - -Beispiele: - -- kompromittierter Admin auf fremdem Client, -- SYSTEM als Benutzerkontext, -- Maschinenkonten als normale Benutzer, -- RDP-Lateral-Movement als normaler Kontext, -- einmalige Notfallaktion als dauerhaft normal. - ---- - -## 22.3 Wann UEBA-Kontexte bereinigt werden sollten - -Manuell bereinigen oder später Funktion ergänzen, wenn: - -- ein Account kompromittiert war, -- ein Admin versehentlich auf vielen Clients angemeldet war, -- ein Dienst falsch lief, -- Testdaten importiert wurden, -- technische Accounts eingelernt wurden, -- VPN-/NAT-Fehler falsche Quell-IPs erzeugt haben. - ---- - -# 23. Empfohlene Bedienregeln für unwissende Admins - -## 23.1 Goldene Regel 1 - -```text -Nicht wegklicken. Erst verstehen. -``` - -Jede Detection sollte mindestens kurz fachlich eingeordnet werden. - ---- - -## 23.2 Goldene Regel 2 - -```text -Plausibel ist nicht dasselbe wie legitim. -``` - -`plausible` ist für temporär erklärbare Ereignisse. - -`legitimate` ist für dauerhaft akzeptiertes Verhalten. - ---- - -## 23.3 Goldene Regel 3 - -```text -False Positive heißt: Die Regel war falsch, nicht das Ereignis war harmlos. -``` - -Ein echter Patchday-Reboot ist kein False Positive. - ---- - -## 23.4 Goldene Regel 4 - -```text -Confirmed Incident immer als confirmed_incident markieren. -``` - -Nicht als `resolved`, nicht als `plausible`, nicht als `legitimate`. - -Erst später kann der Vorfall zusätzlich abgeschlossen werden, aber die historische Detection sollte als Incident erkennbar bleiben. - ---- - -## 23.5 Goldene Regel 5 - -```text -Baseline nicht mit Störungen trainieren. -``` - -Bei Wartung, Angriffen, Fehlkonfigurationen und Testfluten: - -```text -nicht lernen -``` - -verwenden. - ---- - -## 23.6 Goldene Regel 6 - -```text -Suppression nur bewusst und möglichst zeitlich begrenzt. -``` - -Dauerhafte Suppression nur nach Prüfung. - ---- - -## 23.7 Goldene Regel 7 - -```text -Notizen müssen später verständlich sein. -``` - -Immer mit Ursache, Kontext und ggf. Ticketnummer dokumentieren. - ---- - -# 24. Empfohlene Rechte- und Rollenverteilung - -## 24.1 Viewer - -Darf: - -- Detections ansehen, -- Events ansehen, -- SOC-Dashboard ansehen. - -Darf nicht: - -- Detections bewerten, -- Suppressions setzen, -- Baselines ausschließen, -- Regeln ändern. - -## 24.2 Analyst - -Darf: - -- Detections bewerten, -- Notizen setzen, -- Status ändern, -- temporäre Baseline-Exclusions setzen. - -Darf vorsichtig: - -- temporäre Suppressions setzen. - -Darf nicht ohne Freigabe: - -- dauerhafte Suppression, -- Dynamic Rules ändern, -- Baseline komplett löschen. - -## 24.3 Admin - -Darf: - -- Regeln pflegen, -- Privileged Users pflegen, -- Suppressions verwalten, -- Baseline-Exclusions verwalten, -- Dynamic Rules anpassen, -- Indizes/DB warten. - -## 24.4 Security Lead - -Darf entscheiden: - -- was als confirmed_incident gilt, -- welche Suppressions dauerhaft sind, -- welche Baselines zurückgesetzt werden, -- welche Regeln kritisch sind, -- welche Betriebsprozesse gelten. - ---- - -# 25. Tagesroutine für Admins - -## 25.1 Morgens - -1. SOC Dashboard öffnen. -2. Top Host Risk Scores prüfen. -3. Critical/High Detections prüfen. -4. Neue `confirmed_incident` prüfen. -5. Offene `investigating` weiterverfolgen. -6. Patch-/Wartungsereignisse bewerten. - -## 25.2 Während des Tages - -- Neue High/Critical Detections zeitnah prüfen. -- Bei ungewöhnlichen Admin-Logins Rückfrage halten. -- Bei Password-Spray Quell-IP prüfen. -- Bei Success-after-Failure Benutzer und Quelle prüfen. -- Notizen setzen. - -## 25.3 Nachmittags / Tagesabschluss - -- Offene Detections nicht unnötig offen lassen. -- Geklärte Fälle bewerten. -- False Positives in Regelverbesserung überführen. -- Suppressions dokumentieren. -- Incidents sauber markieren. - ---- - -# 26. Wochenroutine - -Einmal pro Woche: - -- häufigste Detections prüfen, -- häufigste False Positives prüfen, -- Suppressions prüfen, -- Baseline-Exclusions prüfen, -- Privileged Users prüfen, -- Dynamic Rules nachschärfen, -- technische Noise-Accounts ergänzen, -- Host Risk Score Trends prüfen. - ---- - -# 27. Monatsroutine - -Einmal pro Monat: - -- Baseline-Qualität prüfen, -- UEBA-Kontexte stichprobenartig prüfen, -- alte Suppressions entfernen, -- dauerhafte Suppressions rechtfertigen, -- Admin-Konten prüfen, -- Service Accounts prüfen, -- Regelwerk dokumentieren, -- Lessons Learned aus Incidents übernehmen. - ---- - -# 28. Umgang mit Wartungsfenstern - -## 28.1 Vor Wartung - -Vor größeren Wartungen dokumentieren: - -- Zeitraum, -- betroffene Systeme, -- erwartete Events, -- verantwortlicher Admin, -- Ticketnummer, -- erwartete Reboots, -- erwartete Service-Installationen. - -## 28.2 Während Wartung - -Detections nicht blind löschen. - -Bewerten mit: - -```text -plausible -``` - -Notiz: - -```text -Wartungsfenster Ticket #12345, Update Exchange Server SE, Reboots erwartet. -``` - -Baseline: - -```text -nicht lernen: 24h oder 7 Tage -``` - -Suppression: - -```text -optional zeitlich begrenzt -``` - -## 28.3 Nach Wartung - -Prüfen: - -- treten Detections weiter auf? -- gibt es unerwartete Hosts? -- gab es erfolgreiche Logons nach Fehlversuchen? -- gab es neue Admin-Kontexte? -- sind Services installiert worden? -- sind Security Logs gelöscht worden? - ---- - -# 29. Umgang mit Patchdays - -Patchdays erzeugen oft: - -- `reboot_spike`, -- `agent_offline`, -- `baseline_event_rate_anomaly`, -- `new_event_id`, -- Service-Events, -- neue Fehler durch defekte Updates. - -Empfohlene Bewertung: - -| Fall | Bewertung | -|---|---| -| erwartete Reboots | `plausible` | -| Agent kurz offline | `plausible` | -| viele neue EventIDs nach Update | `plausible`, prüfen | -| Security Log gelöscht | nicht automatisch plausibel | -| Admin-Login auf ungewöhnlichem Host | prüfen | -| Password-Spray | nicht mit Patchday erklären | - ---- - -# 30. Umgang mit Service Accounts - -Service Accounts können viele Events erzeugen. - -Empfehlungen: - -- Service Accounts dokumentieren. -- Namensschema verwenden, z. B. `svc-*`. -- Nicht als normale Benutzer bewerten. -- Bei Fehlversuchen Passwort/Secret prüfen. -- Bei Off-Hours-Regeln gesondert behandeln. -- Nicht pauschal alles suppressen. - -Bewertung: - -| Situation | Status | -|---|---| -| bekannter Backup-Service | `legitimate` | -| Service mit altem Passwort | `plausible` oder `investigating` | -| unbekannter Service Account | `investigating` | -| Service Account meldet sich interaktiv an | verdächtig | - ---- - -# 31. Umgang mit Admin-Konten - -Admin-Konten sind besonders kritisch. - -Empfehlungen: - -- alle bekannten Admin-Konten in `privileged_users` pflegen, -- Admin-Konten nicht für normale Arbeit nutzen, -- Admin-Logins auf Clients prüfen, -- RDP-Logins mit Admin-Konto priorisieren, -- Off-Hours-Admin-Logins prüfen, -- neue Host-Kontexte für Admins immer bewerten. - -Bewertung: - -| Situation | Status | -|---|---| -| Admin auf bekanntem Jump Host | `legitimate` | -| Admin auf neuem Server mit Ticket | `plausible` | -| Admin auf normalem Client ohne Erklärung | `investigating` | -| Admin nachts auf fremdem Host | mindestens `investigating` | -| kompromittierter Admin | `confirmed_incident` | - ---- - -# 32. Umgang mit False Positives - -False Positives sind nützlich, wenn sie zur Regelverbesserung führen. - -## 32.1 Vorgehen - -1. Detection als `false_positive` markieren. -2. Notiz setzen. -3. Ursache beschreiben. -4. Regel anpassen. -5. Prüfen, ob neue Detections sauberer werden. - -## 32.2 Beispiele - -```text -SYSTEM wurde nicht ausgeschlossen. -``` - -```text -LogonType 3 erzeugt zu viel Rauschen. -``` - -```text -Service Account wird als Benutzer behandelt. -``` - -```text -EventID in falschem Channel ausgewertet. -``` - -```text -Parser liest TargetUserName falsch. -``` - ---- - -# 33. Umgang mit Confirmed Incidents - -## 33.1 Vorgehen - -Wenn eine Detection ein echter Vorfall ist: - -1. Status auf `confirmed_incident`. -2. Notiz mit Fakten schreiben. -3. Baseline nicht lernen lassen. -4. Kein Suppression setzen, solange Angriff aktiv sein könnte. -5. Betroffene Systeme prüfen. -6. Benutzer/Account prüfen. -7. Quell-IP prüfen. -8. Weitere Detections im gleichen Zeitraum prüfen. -9. Nach Abschluss Maßnahmen dokumentieren. - -## 33.2 Notizbeispiel - -```text -Confirmed Incident: adm.max kompromittiert. Unautorisierter RDP-Login auf CLIENT099 um 02:14 Uhr. Account deaktiviert, Passwort zurückgesetzt, Host isoliert. Ticket IR-2026-0042. -``` - -## 33.3 Warum nicht `resolved`? - -`resolved` bedeutet abgeschlossen, aber nicht zwingend bösartig. - -Ein echter Vorfall sollte historisch als `confirmed_incident` erkennbar bleiben. - -Optional kann später zusätzlich ein eigener Incident-Prozess außerhalb der Detection laufen. - ---- - -# 34. Typische Anfängerfehler - -## 34.1 Alles auf `resolved` setzen - -Problem: - -- Host Risk Score sinkt, -- Ursache bleibt unklar, -- spätere Auswertung wertlos. - -Besser: - -- erst fachlich bewerten, -- dann passenden Status setzen. - -## 34.2 `false_positive` für Wartung nutzen - -Problem: - -- Wartung war echt, nicht falsch erkannt. - -Besser: - -```text -plausible -``` - -## 34.3 Dauerhaft suppressen, weil es nervt - -Problem: - -- blinde Flecken. - -Besser: - -- Regel verbessern, -- technische Accounts ausschließen, -- Suppression zeitlich begrenzen. - -## 34.4 Keine Notizen schreiben - -Problem: - -- niemand weiß später, warum etwas bewertet wurde. - -Besser: - -- kurze, klare Ursache plus Ticket. - -## 34.5 Baseline während Incident lernen lassen - -Problem: - -- bösartiges Verhalten wird normalisiert. - -Besser: - -```text -confirmed_incident -Baseline: nicht lernen -``` - ---- - -# 35. Beispiel: vollständiger Bewertungsablauf - -Detection: - -```text -success_after_failures -Host: CLIENT099 -User: max.mustermann -Source IP: 10.2.50.77 -Severity: high -``` - -## Schritt 1: Eventkontext prüfen - -- Gab es vorher 4625? -- Wie viele Fehlversuche? -- Welche Quell-IP? -- War der erfolgreiche Login von derselben Quelle? -- Gab es weitere Benutzer? -- Ist die Quelle bekannt? - -## Schritt 2: Benutzer prüfen - -- Hat der Benutzer zu diesem Zeitpunkt gearbeitet? -- Ist der Benutzer im Urlaub? -- Gab es Passwortänderung? -- Ist MFA/VPN beteiligt? -- Ist es ein Admin? - -## Schritt 3: Host prüfen - -- Ist CLIENT099 der normale Client? -- Ist der Host ungewöhnlich? -- Gibt es weitere Detections auf dem Host? -- Host Risk Score erhöht? - -## Schritt 4: Bewertung - -### Benutzer bestätigt eigene Vertipper - -```text -Status: plausible -Notiz: Benutzer bestätigt mehrere Fehlversuche nach Passwortwechsel, danach erfolgreicher Login. -Baseline: keine Änderung -``` - -### Quelle unbekannt, Benutzer bestätigt es nicht - -```text -Status: investigating -Notiz: Benutzer bestätigt Login nicht, Quell-IP 10.2.50.77 wird geprüft. -Baseline: nicht lernen 24h -``` - -### Kompromittierung bestätigt - -```text -Status: confirmed_incident -Notiz: Benutzer bestätigt Login nicht, Quell-IP unbekannt, Account gesperrt, Host isoliert, Ticket IR-... -Baseline: nicht lernen 7d -``` - ---- - -# 36. Sicherheitsprinzip: Lernen ist Vertrauen - -Ein wichtiger Merksatz: - -```text -Was gelernt wird, dem vertraut die Software später eher. -``` - -Deshalb: - -- nur normales Verhalten lernen lassen, -- verdächtiges Verhalten untersuchen, -- bösartiges Verhalten ausschließen, -- temporäre Sonderfälle nicht dauerhaft normalisieren. - ---- - -# 37. Empfohlene UI-Texte für Admin-Schulung - -## 37.1 Plausible - -```text -Verwenden, wenn das Ereignis durch einen bekannten temporären Kontext erklärbar ist, z. B. Wartung, Patchday oder Rollout. Nicht für dauerhaft normales Verhalten. -``` - -## 37.2 Legitimate - -```text -Verwenden, wenn das Verhalten dauerhaft bekannt, erwartet und akzeptiert ist. Vorsichtig verwenden, da es aus der Risikobewertung fällt. -``` - -## 37.3 False Positive - -```text -Verwenden, wenn die Regel falsch oder technisch irreführend ausgelöst hat. Danach sollte die Regel verbessert werden. -``` - -## 37.4 Confirmed Incident - -```text -Verwenden, wenn ein echter Sicherheitsvorfall bestätigt wurde. Verhindert, dass entsprechendes Verhalten als normal eingelernt wird. -``` - -## 37.5 Suppressed - -```text -Verwenden, wenn diese Detection künftig bewusst nicht mehr gemeldet werden soll. Möglichst zeitlich begrenzen. -``` - ---- - -# 38. Empfohlene Startkonfiguration für sichere Anlernphase - -## 38.1 Baseline - -```text -BASELINE_ENABLED=true -BASELINE_WINDOW=5m -BASELINE_MIN_SAMPLES=24 -BASELINE_MIN_COUNT=10 -BASELINE_MEDIUM_Z=2.5 -BASELINE_HIGH_Z=4.0 -BASELINE_SUPPRESS_FOR=1h -``` - -## 38.2 UEBA - -```text -UEBA_ENABLED=true -UEBA_LOOKBACK=720h -UEBA_NEW_CONTEXT_WINDOW=10m -``` - -## 38.3 Detection - -```text -DETECTION_INTERVAL=1m -``` - -## 38.4 Empfehlung - -In den ersten 7 Tagen lieber: - -- mehr prüfen, -- weniger dauerhaft suppressen, -- gute Notizen schreiben, -- Baseline-Exclusions bei Sonderlagen setzen. - ---- - -# 39. Wann sollte man Baseline zurücksetzen? - -Ein kompletter oder teilweiser Reset kann sinnvoll sein nach: - -- großem Rollout, -- massiver Fehlkonfiguration, -- längerer Testphase mit falschen Daten, -- Umstellung der Audit Policies, -- Änderung der Agent-Filter, -- Korrektur von Parserfehlern, -- Kompromittierung während Anlernphase. - -## 39.1 Teilweiser Reset - -Besser als Komplettreset ist oft ein gezieltes Löschen: - -```text -für Host + Channel + EventID -``` - -Beispiel: - -```text -DC01 + Security + 4625 -``` - -## 39.2 Komplettreset - -Nur wenn die gesamte Baseline unbrauchbar ist. - -Vorher klären: - -- Warum ist sie unbrauchbar? -- Wurde die Ursache behoben? -- Wird danach sauber neu gelernt? -- Sind Admins informiert? - ---- - -# 40. Wann sollte man UEBA zurücksetzen? - -UEBA-Kontexte sollten bereinigt werden, wenn falsche oder gefährliche Kontexte gelernt wurden. - -Beispiele: - -- kompromittierter Account, -- Testdaten, -- SYSTEM/Service Accounts fälschlich gelernt, -- Admin hat versehentlich viele Clients besucht, -- Jump-Host-Konzept wurde geändert, -- VPN/NAT lieferte falsche Quell-IPs. - -Empfehlung: - -- gezielt pro Benutzer löschen, -- gezielt pro Host löschen, -- gezielt technische Accounts entfernen, -- nicht pauschal alles löschen, wenn nur einzelne Kontexte falsch sind. - ---- - -# 41. Wichtige technische Qualitätsprüfungen - -Admins sollten regelmäßig prüfen: - -## 41.1 Zeit - -- Sind Client-Zeiten korrekt? -- Ist Server-Zeit korrekt? -- Ist `TZ=Europe/Berlin` gesetzt? -- Sind UTC-Zeiten in DB konsistent? - -## 41.2 Eventfelder - -- Wird `target_user` korrekt gefüllt? -- Wird `subject_user` korrekt gefüllt? -- Wird `src_ip` korrekt gefüllt? -- Wird `logon_type` korrekt gefüllt? -- Werden Maschinenkonten erkannt? -- Werden Domain-Präfixe normalisiert? - -## 41.3 Noise - -- Taucht `SYSTEM` als Benutzer-Detection auf? -- Tauchen `DWM-*` oder `UMFD-*` auf? -- Tauchen viele Service-Logons auf? -- Werden LogonType 3 und 5 unnötig alarmiert? - -## 41.4 Datenmenge - -- Kommen alle Agents? -- Gibt es Hosts ohne Events? -- Gibt es extreme Eventfluten? -- Sind Indizes vorhanden? -- Laufen Queries schnell genug? - ---- - -# 42. Beispielhafte Schulungsübung - -Ein neuer Admin soll 10 Detections bewerten. - -## Aufgabe - -Für jede Detection beantworten: - -1. Welche Regel hat ausgelöst? -2. Welche Daten liegen zugrunde? -3. Ist das Ereignis technisch korrekt? -4. Ist es erwartbar? -5. Ist es einmalig oder dauerhaft? -6. Soll die Baseline lernen? -7. Ist Suppression sinnvoll? -8. Welche Notiz wird geschrieben? -9. Welcher Status ist korrekt? - -## Ziel - -Der Admin soll nicht nur klicken, sondern begründen. - ---- - -# 43. Kurze Checkliste pro Detection - -Vor Bewertung prüfen: - -```text -[ ] Regel verstanden -[ ] Host geprüft -[ ] Benutzer geprüft -[ ] Quell-IP geprüft -[ ] Zeit geprüft -[ ] Eventdetails geprüft -[ ] ähnliche Detections geprüft -[ ] Wartungsfenster geprüft -[ ] Ticket/Change geprüft -[ ] Status bewusst gewählt -[ ] Notiz geschrieben -[ ] Baseline-Aktion geprüft -[ ] Suppression bewusst entschieden -``` - ---- - -# 44. Empfohlene Status-Definition für Betriebsdokumentation - -Diese Definitionen sollten intern verbindlich sein: - -```text -open: -Neu und ungeprüft. - -acknowledged: -Gesehen, aber noch nicht abschließend geprüft. - -investigating: -Aktive Untersuchung läuft. - -plausible: -Temporär erklärbar, aber nicht dauerhaft als normal bestätigt. - -legitimate: -Dauerhaft erwartetes und akzeptiertes Verhalten. - -false_positive: -Regel oder Erkennung war fachlich/technisch falsch. - -resolved: -Bearbeitet und abgeschlossen, kein weiterer Handlungsbedarf. - -suppressed: -Bewusst unterdrückt, künftig nicht mehr oder vorübergehend nicht relevant. - -confirmed_incident: -Bestätigter Sicherheitsvorfall. -``` - ---- - -# 45. Zusammenfassung - -Die Software lernt aus beobachtetem Verhalten. Dadurch wird sie mit der Zeit besser, aber nur, wenn die Daten sauber sind und die Admins korrekt bewerten. - -Wichtigste Punkte: - -1. Die Anlernphase ist kritisch. -2. Nicht alles am Anfang ist automatisch normal. -3. Baseline lernt Eventmengen nach Host, Channel, EventID, Wochentag und Stunde. -4. UEBA lernt Benutzerkontexte. -5. Dynamic Rules lernen nicht selbst, sondern müssen gepflegt werden. -6. `plausible` ist für temporär erklärbare Ereignisse. -7. `legitimate` ist für dauerhaft akzeptiertes Verhalten. -8. `false_positive` ist für falsche Regeln/Erkennungen. -9. `confirmed_incident` ist für echte Vorfälle. -10. Baseline darf Incidents nicht lernen. -11. Suppression erzeugt blinde Flecken und muss bewusst eingesetzt werden. -12. Gute Notizen sind Pflicht. -13. Host Risk Score hängt stark von korrekter Bewertung ab. -14. Technische Accounts und LogonTypes müssen sauber gefiltert werden. -15. Regelmäßige Pflege verbessert die Erkennungsqualität. - -Der wichtigste Satz für Admins: - -```text -Bewertung ist Teil der Sicherheit. Falsch bewertete Detections können die Erkennung verschlechtern. -``` - - -# SIEM-lite Regel-Dokumentation - -Stand: 2026-04-27 -Projekt: `siem-backend` / `SIEM-lite Security Operations` - -Diese Dokumentation beschreibt die im Backend implementierten Detection-Regeln, ihre Datenbasis, Schwellenwerte, Statuslogik, Suppression-Mechanismen und empfohlene Bewertung im SOC. - ---- - -## 1. Überblick - -Das Backend nimmt Windows Eventlog-Daten von Agents über `/ingest` entgegen, normalisiert wichtige Felder aus dem Event-XML und speichert sie in MySQL. In regelmäßigen Intervallen läuft ein Detection-Loop, der verschiedene statische, dynamische, Baseline- und UEBA-Regeln auswertet. - -Der zentrale Detection-Loop befindet sich in: - -```go -func (s *server) runDetectionsOnce() -``` - -Aktuell vorgesehene Regeln: - -```go -agent_offline -failed_logon_spike -reboot_spike -new_event_id -password_spray -success_after_failures -new_source_ip_for_user -dynamic_rules -baseline_anomaly -baseline_update -ueba_admin_new_host -ueba_offhours_login -ueba_first_privileged_use -ueba_new_user_context -ueba_update -host_risk_score -``` - ---- - -## 2. Datenbasis - -### 2.1 Tabelle `event_logs` - -Die meisten Regeln basieren auf normalisierten Windows-Events in `event_logs`. - -Wichtige Felder: - -| Feld | Bedeutung | -|---|---| -| `hostname` | Hostname des meldenden Systems oder aus dem Event-XML | -| `channel_name` | Windows Eventlog-Channel, z. B. `Security`, `System`, `Application` | -| `event_id` | Windows Event ID | -| `target_user` | Zielbenutzer, meist aus `TargetUserName` | -| `subject_user` | ausführender Benutzer, meist aus `SubjectUserName` | -| `src_ip` | Quell-IP aus `IpAddress` | -| `workstation` | Workstation aus `WorkstationName` | -| `logon_type` | Windows LogonType | -| `process_name` | Prozessname, sofern vorhanden | -| `ts` | Event-Zeitpunkt | -| `received_at` | Empfangszeitpunkt | -| `msg` | Rohes Event-XML | -| `msg_sha256` | Hash des Event-XML | - -### 2.2 Tabelle `detections` - -Alle Regeln schreiben Findings in die Tabelle `detections`. - -Wichtige Felder: - -| Feld | Bedeutung | -|---|---| -| `rule_name` | Name der Regel | -| `severity` | `info`, `low`, `medium`, `high`, `critical` | -| `hostname` | betroffener Host | -| `channel_name` | Eventlog-Channel | -| `event_id` | relevante Event ID | -| `score` | numerischer Risiko-/Anomalie-Score | -| `window_start` | Beginn des betrachteten Zeitfensters | -| `window_end` | Ende des betrachteten Zeitfensters | -| `summary` | menschenlesbare Zusammenfassung | -| `details_json` | technische Details als JSON | -| `status` | SOC-Bewertung | -| `analyst_note` | Analystenkommentar | -| `reviewed_by` | Bearbeiter | -| `reviewed_at` | Bewertungszeitpunkt | -| `is_false_positive` | Flag für False Positive | -| `is_legitimate` | Flag für legitimes/plausibles Verhalten | - ---- - -## 3. Globale Konfiguration - -Die wichtigsten Regelparameter werden per Environment-Variablen gesetzt. - -| Variable | Default | Bedeutung | -|---|---:|---| -| `DETECTION_INTERVAL` | `1m` | Intervall des Detection-Loops | -| `OFFLINE_AFTER` | `10m` | Agent gilt nach dieser Zeit als offline | -| `OFFLINE_ALERT_MAX` | `120m` | Maximales Offline-Alter für Offline-Alerts | -| `FAILED_LOGON_WINDOW` | `5m` | Zeitfenster für fehlgeschlagene Logons | -| `FAILED_LOGON_THRESHOLD` | `25` | Mindestanzahl fehlgeschlagener Logons pro Host | -| `REBOOT_WINDOW` | `15m` | Zeitfenster für Reboot-Spikes | -| `REBOOT_THRESHOLD` | `3` | Mindestanzahl Reboot-/Shutdown-Events | -| `PASSWORD_SPRAY_WINDOW` | `5m` | Zeitfenster für Password-Spray | -| `PASSWORD_SPRAY_MIN_USERS` | `5` | Mindestanzahl unterschiedlicher Zielbenutzer | -| `PASSWORD_SPRAY_MIN_ATTEMPTS` | `15` | Mindestanzahl Fehlversuche | -| `SUCCESS_AFTER_FAILURE_WINDOW` | `10m` | Zeitfenster für Erfolg nach Fehlversuchen | -| `NEW_SOURCE_IP_LOOKBACK` | `720h` / 30 Tage | Rückblick für bekannte Quell-IPs | -| `NEW_SOURCE_IP_WINDOW` | `10m` | aktuelles Fenster für neue Quell-IP | -| `BASELINE_ENABLED` | `true` | aktiviert Baseline-Regeln | -| `BASELINE_WINDOW` | `5m` | Baseline-Bucket-Zeitfenster | -| `BASELINE_MIN_SAMPLES` | `24` | Mindestanzahl Baseline-Samples | -| `BASELINE_MIN_COUNT` | `10` | Mindestanzahl Events für Baseline-Alert | -| `BASELINE_MEDIUM_Z` | `2.5` | Z-Score ab Medium | -| `BASELINE_HIGH_Z` | `4.0` | Z-Score ab High | -| `BASELINE_SUPPRESS_FOR` | `1h` | automatische Unterdrückung gleicher Baseline-Findings | -| `UEBA_ENABLED` | `true` | aktiviert UEBA-Regeln | -| `UEBA_LOOKBACK` | `720h` / 30 Tage | UEBA-Rückblick | -| `UEBA_NEW_CONTEXT_WINDOW` | `10m` | Fenster für neue Kontexte | -| `RISK_SCORE_WINDOW` | `24h` | Zeitraum für Host Risk Score | -| `TZ` | `Europe/Berlin` | Anzeige-/Bewertungszeitzone | - ---- - -## 4. Severity-Logik - -### 4.1 `severityFromScore` - -Mehrere Regeln nutzen diese Score-zu-Severity-Logik: - -```go -func severityFromScore(score, low, medium, high, critical float64) string -``` - -Beispielaufruf: - -```go -severityFromScore(score, 1.0, 2.0, 4.0, 8.0) -``` - -Bedeutung: - -| Score | Severity | -|---:|---| -| `>= 8.0` | `critical` | -| `>= 4.0` | `high` | -| `>= 2.0` | `medium` | -| `>= 1.0` | `low` | -| `< 1.0` | `info` | - -### 4.2 Host Risk Severity - -Der Host Risk Score nutzt eine separate Einteilung: - -| Risk Score | Severity | -|---:|---| -| `>= 120` | `critical` | -| `>= 60` | `high` | -| `>= 20` | `medium` | -| `>= 5` | `low` | -| `< 5` | `info` | - ---- - -# 5. Regel: `agent_offline` - -## Zweck - -Erkennt Agents, die länger als `OFFLINE_AFTER` nicht mehr gesehen wurden, aber noch nicht älter als `OFFLINE_ALERT_MAX` offline sind. - -## Datenbasis - -Tabelle: `agents` - -Relevante Felder: - -- `hostname` -- `last_seen` -- `is_enabled` - -## Logik - -Ein Agent wird gemeldet, wenn: - -```sql -is_enabled = 1 -AND last_seen < now - OFFLINE_AFTER -AND last_seen >= now - OFFLINE_ALERT_MAX -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `OFFLINE_AFTER` | `10m` | -| `OFFLINE_ALERT_MAX` | `120m` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `agent_offline` | -| Severity | `low` | -| Score | abhängig von Offline-Dauer | -| Channel | leer | -| EventID | `0` | - -## Beispiel-Summary - -```text -Agent CLIENT01 ist seit 17 Minuten offline -``` - -## SOC-Bewertung - -Empfohlene Bewertung: - -- `legitimate`, wenn Gerät heruntergefahren, neu installiert oder absichtlich deaktiviert wurde. -- `investigating`, wenn Server oder kritischer Client unerwartet offline ist. -- `suppressed`, wenn bestimmte Systeme dauerhaft nicht überwacht werden sollen. - ---- - -# 6. Regel: `failed_logon_spike` - -## Zweck - -Erkennt Hosts mit ungewöhnlich vielen fehlgeschlagenen Logons innerhalb kurzer Zeit. - -## Datenbasis - -Tabelle: `event_logs` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4625` | - -## Logik - -Gruppierung nach Host: - -```sql -channel_name = 'Security' -AND event_id = 4625 -AND ts >= window_start -AND ts < window_end -GROUP BY hostname -HAVING COUNT(*) >= FAILED_LOGON_THRESHOLD -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `FAILED_LOGON_WINDOW` | `5m` | -| `FAILED_LOGON_THRESHOLD` | `25` | - -## Score - -```text -score = count / FAILED_LOGON_THRESHOLD -``` - -Severity wird über `severityFromScore(score, 1.0, 2.0, 4.0, 8.0)` berechnet. - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `failed_logon_spike` | -| EventID | `4625` | -| Channel | `Security` | -| Severity | score-basiert | - -## Beispiel-Summary - -```text -Host DC01 hatte 48 fehlgeschlagene Logons in 5 Minuten -``` - -## Typische Ursachen - -- Brute-Force-Versuch -- falsch gespeichertes Passwort in Diensten -- altes Passwort in RDP-/VPN-/Outlook-Sessions -- Scanner oder Monitoring mit falschen Credentials -- Lockout-Situation nach Passwortwechsel - -## SOC-Bewertung - -- `investigating`, wenn unbekannte Quelle, viele Accounts oder externe IP beteiligt. -- `legitimate`, wenn bekannter Dienst nach Passwortwechsel betroffen ist. -- `false_positive`, wenn Event-Parsing oder Testdaten Ursache sind. - ---- - -# 7. Regel: `reboot_spike` - -## Zweck - -Erkennt Hosts mit mehreren Reboot-/Shutdown-Events in kurzer Zeit. - -## Datenbasis - -Tabelle: `event_logs` - -Windows Events: - -| Channel | EventID | Bedeutung | -|---|---:|---| -| `System` | `1074` | geplanter Neustart/Shutdown | -| `System` | `6005` | Eventlog-Service gestartet | -| `System` | `6006` | Eventlog-Service beendet | - -## Logik - -```sql -channel_name = 'System' -AND event_id IN (1074, 6005, 6006) -AND ts >= window_start -AND ts < window_end -GROUP BY hostname -HAVING COUNT(*) >= REBOOT_THRESHOLD -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `REBOOT_WINDOW` | `15m` | -| `REBOOT_THRESHOLD` | `3` | - -## Score - -```text -score = count / REBOOT_THRESHOLD -``` - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `reboot_spike` | -| Channel | `System` | -| Severity | score-basiert | - -## Beispiel-Summary - -```text -Host CLIENT01 hatte 4 Reboot-/Shutdown-Events in 15 Minuten -``` - -## Typische Ursachen - -- Patchinstallation -- Treiberproblem -- instabiler Client -- Stromproblem -- absichtlicher Neustart durch Admin -- Malware/Manipulation eher selten, aber möglich - ---- - -# 8. Regel: `new_event_id` - -## Zweck - -Erkennt, wenn ein Host erstmals eine bestimmte EventID in einem bestimmten Channel sendet. - -## Datenbasis - -Tabelle: `event_logs` - -## Logik - -Die Regel betrachtet nur das aktuelle Detection-Intervall und prüft, ob es davor für denselben Host, Channel und EventID bereits Events gab. - -Vereinfacht: - -```sql -WHERE e.ts >= window_start -AND e.ts < window_end -AND NOT EXISTS ( - SELECT 1 - FROM event_logs old - WHERE old.hostname = e.hostname - AND old.channel_name = e.channel_name - AND old.event_id = e.event_id - AND old.ts < window_start -) -GROUP BY e.hostname, e.channel_name, e.event_id -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `DETECTION_INTERVAL` | `1m` | - -## Score - -```text -score = 1.0 + log10(count + 1) -``` - -## Severity - -| Bedingung | Severity | -|---|---| -| `count >= 10` | `high` | -| sonst | `medium` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `new_event_id` | -| Channel | aus Event | -| EventID | aus Event | - -## Beispiel-Summary - -```text -Host CLIENT01 sendet erstmals Event-ID 7045 im Channel System -``` - -## Typische Ursachen - -- neue Software -- neue Windows-Funktion -- neues Audit-Policy-Setting -- Treiberinstallation -- Service-Installation -- sicherheitsrelevante Aktivität, wenn EventID auffällig ist - -## SOC-Bewertung - -Besonders prüfen bei: - -- `System 7045` Service Installation -- `Security 1102` Auditlog gelöscht -- `Security 4720` User erstellt -- `Security 4697` Service installiert -- `Security 4698` Scheduled Task erstellt - ---- - -# 9. Regel: `password_spray` - -## Zweck - -Erkennt mögliche Password-Spray-Angriffe anhand vieler fehlgeschlagener Logons gegen mehrere verschiedene Benutzer von derselben Quell-IP. - -## Datenbasis - -Tabelle: `event_logs` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4625` | - -## Logik - -Gruppierung nach Host und Quell-IP: - -```sql -channel_name = 'Security' -AND event_id = 4625 -AND src_ip <> '' -AND src_ip <> '-' -AND src_ip <> '::1' -AND src_ip <> '127.0.0.1' -AND target_user <> '' -AND target_user <> '-' -GROUP BY hostname, src_ip -HAVING COUNT(*) >= PASSWORD_SPRAY_MIN_ATTEMPTS - AND COUNT(DISTINCT target_user) >= PASSWORD_SPRAY_MIN_USERS -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `PASSWORD_SPRAY_WINDOW` | `5m` | -| `PASSWORD_SPRAY_MIN_USERS` | `5` | -| `PASSWORD_SPRAY_MIN_ATTEMPTS` | `15` | - -## Score - -```text -score = max( - attempts / PASSWORD_SPRAY_MIN_ATTEMPTS, - users / PASSWORD_SPRAY_MIN_USERS -) -``` - -Severity wird über `severityFromScore(score, 1.0, 2.0, 4.0, 8.0)` berechnet. - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `password_spray` | -| Channel | `Security` | -| EventID | `4625` | - -## Beispiel-Summary - -```text -Möglicher Password-Spray auf DC01 von 10.10.10.50: 30 Fehlversuche gegen 12 Benutzer -``` - -## Typische Ursachen - -- echter Password-Spray -- Scanner mit falschem Passwort -- altes Passwort in zentralem Dienst -- falsch konfigurierte Applikation -- VPN-/RADIUS-/LDAP-Fehlversuche - -## Empfohlene Prüfung - -- Quell-IP identifizieren. -- Prüfen, ob Quell-IP intern, VPN, Server oder extern ist. -- Betroffene Benutzer auf Lockout oder spätere erfolgreiche Logons prüfen. -- In Kombination mit `success_after_failures` hoch priorisieren. - ---- - -# 10. Regel: `success_after_failures` - -## Zweck - -Erkennt erfolgreiche Logons, denen kurz zuvor fehlgeschlagene Logons für denselben Benutzer vorausgingen. - -## Datenbasis - -Tabelle: `event_logs` - -Windows Events: - -| EventID | Bedeutung | -|---:|---| -| `4625` | fehlgeschlagener Logon | -| `4624` | erfolgreicher Logon | - -## Logik - -Ein erfolgreicher Logon wird verdächtig, wenn vorher im konfigurierten Fenster ein passender fehlgeschlagener Logon existiert. - -Vereinfacht: - -```sql -s.event_id = 4624 -AND EXISTS ( - SELECT 1 - FROM event_logs f - WHERE f.hostname = s.hostname - AND f.event_id = 4625 - AND f.target_user = s.target_user - AND f.ts >= DATE_SUB(s.ts, INTERVAL ? SECOND) - AND f.ts < s.ts -) -``` - -Die Quell-IP wird berücksichtigt, wenn sie vorhanden ist. - -## Default-Wert - -| Parameter | Default | -|---|---:| -| `SUCCESS_AFTER_FAILURE_WINDOW` | `10m` | - -## Score - -```text -score = 2.0 + log10(success_count + 1) -``` - -## Severity - -Aktuell fest: - -```text -high -``` - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `success_after_failures` | -| Channel | `Security` | -| EventID | `4624` | -| Severity | `high` | - -## Beispiel-Summary - -```text -Erfolgreicher Logon nach Fehlversuchen auf CLIENT01 für Benutzer max.mustermann -``` - -## Typische Ursachen - -- Benutzer hat sich vertippt und danach korrekt angemeldet -- Angreifer hat Passwort erraten -- Passwort-Spray mit anschließendem Treffer -- Passwortänderung und gespeicherte Sessions - -## Empfohlene Bewertung - -Diese Regel sollte priorisiert werden, wenn zusätzlich eines davon zutrifft: - -- neue Quell-IP -- ungewöhnlicher Host -- privilegierter Benutzer -- außerhalb Arbeitszeit -- mehrere Zielbenutzer betroffen -- externer oder unbekannter Ursprung - ---- - -# 11. Regel: `new_source_ip_for_user` - -## Zweck - -Erkennt erfolgreiche Logons eines Benutzers von einer bisher unbekannten Quell-IP. - -## Datenbasis - -Tabelle: `event_logs` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4624` | - -## Logik - -Die Regel betrachtet erfolgreiche Logons im aktuellen Fenster und prüft, ob dieselbe Quell-IP für denselben Benutzer auf demselben Host im Lookback-Zeitraum bereits bekannt war. - -```sql -event_id = 4624 -AND src_ip NOT IN ('', '-', '::1', '127.0.0.1') -AND NOT EXISTS ( - SELECT 1 - FROM event_logs old - WHERE old.hostname = e.hostname - AND old.event_id = 4624 - AND old.target_user = e.target_user - AND old.src_ip = e.src_ip - AND old.ts >= lookback_start - AND old.ts < window_start -) -``` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `NEW_SOURCE_IP_WINDOW` | `10m` | -| `NEW_SOURCE_IP_LOOKBACK` | `30d` | - -## Score - -```text -score = 1.5 + log10(count + 1) -``` - -## Severity - -| Bedingung | Severity | -|---|---| -| `count >= 5` | `high` | -| sonst | `medium` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `new_source_ip_for_user` | -| Channel | `Security` | -| EventID | `4624` | - -## Beispiel-Summary - -```text -Benutzer max.mustermann meldet sich auf CLIENT01 von neuer Quell-IP 10.10.50.20 an -``` - -## Typische Ursachen - -- neuer Client -- VPN-Wechsel -- DHCP-Wechsel -- RDP von ungewohntem System -- kompromittierter Account - ---- - -# 12. Regel: `dynamic_rules` - -## Zweck - -Ermöglicht frei konfigurierbare Regeln über die Weboberfläche und Tabelle `detection_rules`. - -## Datenbasis - -Tabelle: - -```text -detection_rules -``` - -Ausgewertet gegen: - -```text -event_logs -``` - -## Regeltypen - -Es gibt zwei Betriebsarten: - -1. Einzelereignis-Regel -2. Threshold-Regel - -### 12.1 Einzelereignis-Regel - -Wird verwendet, wenn: - -```text -threshold_count <= 1 -ODER -threshold_window_seconds <= 0 -``` - -Sie prüft Events im aktuellen `DETECTION_INTERVAL`. - -### 12.2 Threshold-Regel - -Wird verwendet, wenn: - -```text -threshold_count > 1 -UND -threshold_window_seconds > 0 -``` - -Sie zählt passende Events im angegebenen Zeitfenster pro Host. - -## Konfigurierbare Felder - -| Feld | Bedeutung | -|---|---| -| `name` | Regelname | -| `description` | Beschreibung | -| `severity` | Severity | -| `channel` | ein oder mehrere Channels, kommasepariert | -| `event_ids` | ein oder mehrere EventIDs, kommasepariert | -| `match_field` | optionales Feld für Filter | -| `match_operator` | `eq`, `contains`, `in` | -| `match_value` | Vergleichswert | -| `threshold_count` | Mindestanzahl Events | -| `threshold_window_seconds` | Zeitfenster | -| `suppress_for_seconds` | Suppression-Zeit | -| `enabled` | aktiv/inaktiv | - -## Unterstützte Match-Felder - -| `match_field` | DB-Spalte | -|---|---| -| `hostname` | `hostname` | -| `channel` | `channel_name` | -| `event_id` | `event_id` | -| `target_user` | `target_user` | -| `subject_user` | `subject_user` | -| `src_ip` | `src_ip` | -| `workstation` | `workstation` | -| `process_name` | `process_name` | -| `msg` | `msg` | - -## Unterstützte Operatoren - -| Operator | Bedeutung | -|---|---| -| `eq` | exakter Vergleich | -| `contains` | Teilstring-Suche | -| `in` | Wert ist in kommaseparierter Liste enthalten | - -## Detection-Name - -Dynamische Regeln werden mit Prefix gespeichert: - -```text -dynamic_ -``` - -Beispiel: - -```text -dynamic_security_log_cleared -``` - -## Beispiel-Regeln - -### Auditlog gelöscht - -| Feld | Wert | -|---|---| -| Name | `security_log_cleared` | -| Severity | `critical` | -| Channel | `Security` | -| Event IDs | `1102` | -| Threshold Count | `1` | -| Suppress Seconds | `3600` | - -### Neuer lokaler Benutzer - -| Feld | Wert | -|---|---| -| Name | `local_user_created` | -| Severity | `high` | -| Channel | `Security` | -| Event IDs | `4720` | -| Threshold Count | `1` | - -### Service installiert - -| Feld | Wert | -|---|---| -| Name | `service_installed` | -| Severity | `high` | -| Channel | `System,Security` | -| Event IDs | `7045,4697` | - ---- - -# 13. Regel: `baseline_event_rate_anomaly` - -## Zweck - -Erkennt statistische Abweichungen bei Event-Häufigkeiten je Host, Channel, EventID, Wochentag und Stunde. - -## Komponenten - -Die Baseline besteht aus zwei Teilen: - -1. `baseline_update` -2. `baseline_anomaly` - -## 13.1 `baseline_update` - -Aktualisiert die Statistik in `baseline_event_stats`. - -Gruppierung: - -```text -hostname -channel_name -event_id -hour_of_day -day_of_week -``` - -Es wird ein Online-Mittelwert mit M2/Stddev gepflegt. - -## 13.2 `baseline_anomaly` - -Vergleicht aktuelle Counts gegen die gespeicherte Baseline. - -## Datenbasis - -Tabellen: - -- `event_logs` -- `baseline_event_stats` -- `baseline_exclusions` -- `detections` - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `BASELINE_WINDOW` | `5m` | -| `BASELINE_MIN_SAMPLES` | `24` | -| `BASELINE_MIN_COUNT` | `10` | -| `BASELINE_MEDIUM_Z` | `2.5` | -| `BASELINE_HIGH_Z` | `4.0` | -| `BASELINE_SUPPRESS_FOR` | `1h` | - -## Berechnung - -```text -z = (current_count - avg_count) / stddev_count -``` - -## Voraussetzungen für Alert - -Ein Alert entsteht nur, wenn: - -```text -sample_count >= BASELINE_MIN_SAMPLES -count >= BASELINE_MIN_COUNT -stddev_count > 0 -z >= BASELINE_MEDIUM_Z -``` - -## Severity - -| Bedingung | Severity | -|---|---| -| `z >= BASELINE_HIGH_Z` | `high` | -| `z >= BASELINE_MEDIUM_Z` | `medium` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `baseline_event_rate_anomaly` | -| Score | Z-Score | -| Severity | z-score-basiert | - -## Beispiel-Summary - -```text -Baseline-Anomalie auf DC01: Security EventID 4625 kam 80-mal in 5 Minuten, normal 12.40 ± 5.10, z=13.25 -``` - -## Ausschlüsse - -Baseline-Lernen wird übersprungen, wenn: - -- passender Eintrag in `baseline_exclusions` aktiv ist -- ein `confirmed_incident` im aktuellen Fenster existiert - -## UI-Bewertung - -In der Detection-UI kann über `Baseline` gesteuert werden: - -| Aktion | Effekt | -|---|---| -| `nicht lernen: 24h` | temporärer Baseline-Ausschluss | -| `nicht lernen: 7 Tage` | temporärer Baseline-Ausschluss | -| `nicht lernen: 30 Tage` | temporärer Baseline-Ausschluss | -| `nicht lernen: dauerhaft` | permanenter Baseline-Ausschluss | - -Bei `confirmed_incident` wird automatisch ein 7-Tage-Ausschluss erzeugt, sofern keine andere Baseline-Aktion gewählt wurde. - ---- - -# 14. Regel: `ueba_admin_new_host` - -## Zweck - -Erkennt, wenn sich ein privilegierter Benutzer erstmals auf einem Host anmeldet. - -## Datenbasis - -Tabellen: - -- `event_logs` -- `ueba_user_baseline` -- `privileged_users` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4624` | - -## Privilegierte Benutzer - -Ein Benutzer gilt als privilegiert, wenn: - -1. er in `privileged_users` aktiviert ist, oder -2. sein Name einem Admin-Muster entspricht: - -```text -adm-* -*-adm -*.adm -``` - -Maschinenkonten werden ignoriert. - -## Logik - -Es werden erfolgreiche Logons im aktuellen UEBA-Fenster betrachtet. Für Benutzer/Host-Kombinationen wird geprüft, ob es im Lookback bereits einen passenden Baseline-Kontext gab. - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `UEBA_NEW_CONTEXT_WINDOW` | `10m` | -| `UEBA_LOOKBACK` | `30d` | - -## Score und Severity - -| Bedingung | Score | Severity | -|---|---:|---| -| `count < 3` | `6.0` | `high` | -| `count >= 3` | `9.0` | `critical` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `ueba_admin_new_host` | -| Channel | `Security` | -| EventID | `4624` | - -## Beispiel-Summary - -```text -UEBA: Privilegierter Benutzer adm.max meldet sich erstmals auf Host SERVER01 an -``` - -## Prometheus-Metriken - -Bei neuer Detection wird erhöht: - -```text -siem_privileged_new_host_total{user,host} -``` - -## SOC-Bewertung - -Diese Regel ist besonders relevant, wenn: - -- der Host ein Client statt Admin-Jump-Host ist -- der Login außerhalb Arbeitszeit erfolgt -- vorher Fehlversuche auftraten -- die Quell-IP neu ist -- der Benutzer Domain Admin oder Server Admin ist - ---- - -# 15. Regel: `ueba_new_user_context` - -## Zweck - -Erkennt neue Login-Kontexte für normale Benutzer. - -Ein Kontext besteht aus: - -```text -username + hostname + src_ip + workstation -``` - -## Datenbasis - -Tabellen: - -- `event_logs` -- `ueba_user_baseline` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4624` | - -## Logik - -Ein Alert entsteht, wenn für einen erfolgreichen Logon im aktuellen Fenster kein bekannter Baseline-Kontext im Lookback existiert. - -## Default-Werte - -| Parameter | Default | -|---|---:| -| `UEBA_NEW_CONTEXT_WINDOW` | `10m` | -| `UEBA_LOOKBACK` | `30d` | - -## Score und Severity - -| Bedingung | Score | Severity | -|---|---:|---| -| `count < 5` | `2.0` | `medium` | -| `count >= 5` | `4.0` | `high` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `ueba_new_user_context` | -| Channel | `Security` | -| EventID | `4624` | - -## Beispiel-Summary - -```text -UEBA: Benutzer max.mustermann meldet sich in neuem Kontext an: Host=CLIENT01 IP=10.10.20.30 Workstation=CLIENT99 -``` - -## Typische Ursachen - -- neuer Arbeitsplatz -- VPN -- DHCP-Wechsel -- Remotezugriff -- Helpdesk-Anmeldung -- kompromittierter Account - ---- - -# 16. Regel: `ueba_update` - -## Zweck - -Aktualisiert die UEBA-Baseline für Benutzerkontexte. - -## Datenbasis - -Tabelle: - -```text -ueba_user_baseline -``` - -Quelle: - -```text -event_logs -``` - -## Logik - -Erfolgreiche Logons im aktuellen UEBA-Fenster werden in die Baseline übernommen. - -Kontext: - -```text -username -hostname -src_ip -workstation -``` - -Bei neuem Kontext: - -```sql -INSERT INTO ueba_user_baseline (...) -``` - -Bei bekanntem Kontext: - -```sql -last_seen = UTC_TIMESTAMP(6) -seen_count = seen_count + VALUES(seen_count) -``` - -## Wichtig - -Die Reihenfolge im Detection-Loop ist relevant. - -Empfohlen: - -```text -ueba_admin_new_host -ueba_offhours_login -ueba_first_privileged_use -ueba_new_user_context -ueba_update -``` - -So werden neue Kontexte zuerst erkannt und danach gelernt. - ---- - -# 17. Regel: `ueba_offhours_login` - -## Zweck - -Erkennt erfolgreiche Benutzeranmeldungen außerhalb der definierten Arbeitszeit. - -## Aktuelle Arbeitszeit - -```text -06:00 bis 20:00 Uhr -``` - -Außerhalb dieser Zeit wird geprüft. - -## Datenbasis - -Tabelle: - -```text -event_logs -``` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4624` | - -## Empfohlene Filter - -Um Rauschen durch technische Accounts zu vermeiden, sollten technische Accounts ausgeschlossen werden. - -### Empfohlene Go-Hilfsfunktion - -```go -func isNoiseAccount(username string) bool { - u := normalizeUsername(username) - - if u == "" || isMachineAccount(u) { - return true - } - - switch u { - case "system", - "localsystem", - "local service", - "network service", - "anonymous logon", - "dwm-1", - "dwm-2", - "dwm-3", - "umfd-0", - "umfd-1", - "umfd-2", - "umfd-3": - return true - } - - return false -} -``` - -`NT AUTHORITY\SYSTEM` wird durch `normalizeUsername()` zu `system` und damit sauber ignoriert. - -## Empfohlener SQL-Filter - -```sql -AND target_user <> '' -AND target_user <> '-' -AND target_user NOT LIKE '%$' -AND logon_type IN ('2', '7', '10', '11') -AND LOWER(target_user) NOT IN ( - 'system', - 'localsystem', - 'local service', - 'network service', - 'anonymous logon' -) -``` - -## Relevante LogonTypes - -| LogonType | Bedeutung | Bewertung | -|---:|---|---| -| `2` | Interactive | relevant | -| `7` | Unlock | meist relevant | -| `10` | RemoteInteractive/RDP | relevant | -| `11` | CachedInteractive | relevant | -| `3` | Network | oft Rauschen | -| `5` | Service | meist Rauschen | - -## Score und Severity - -Empfohlene Logik: - -| Bedingung | Score | Severity | -|---|---:|---| -| `count < 5` | `2.0` | `medium` | -| `count >= 5` | `4.0` | `high` | - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `ueba_offhours_login` | -| Channel | `Security` | -| EventID | `4624` | - -## Beispiel-Summary - -```text -UEBA: Login außerhalb der Arbeitszeit: Benutzer max.mustermann auf Host CLIENT01 -``` - -## Typische legitime Ursachen - -- Bereitschaftsdienst -- geplante Wartung -- Remote-Arbeit -- automatisierte Admin-Tätigkeit -- Zeitzonen-/Serverzeitproblem - -## Typische verdächtige Ursachen - -- kompromittierte Zugangsdaten -- RDP-Zugriff nach Feierabend -- laterale Bewegung -- Admin-Account außerhalb normaler Betriebszeit - ---- - -# 18. Regel: `ueba_first_privileged_use` - -## Zweck - -Erkennt, wenn ein Benutzer erstmals privilegierte Rechte verwendet. - -## Datenbasis - -Tabellen: - -- `event_logs` -- `user_privilege_baseline` - -Windows Event: - -| Channel | EventID | -|---|---:| -| `Security` | `4672` | - -Event `4672` steht für „Special privileges assigned to new logon“. - -## Logik - -Für jedes `4672`-Event im Fenster wird geprüft, ob der Benutzer bereits in `user_privilege_baseline` existiert. - -Wenn nicht: - -1. Detection erzeugen -2. Benutzer in Baseline übernehmen - -## Empfohlene Tabelle - -```sql -CREATE TABLE IF NOT EXISTS user_privilege_baseline ( - username VARCHAR(255) NOT NULL PRIMARY KEY, - first_seen DATETIME(6) NOT NULL DEFAULT UTC_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT UTC_TIMESTAMP(6), - seen_count BIGINT NOT NULL DEFAULT 1 -); -``` - -## Detection - -| Feld | Wert | -|---|---| -| Rule | `ueba_first_privileged_use` | -| Channel | `Security` | -| EventID | `4672` | -| Severity | `high` | -| Score | `6.0` | - -## Beispiel-Summary - -```text -UEBA: Benutzer adm.max nutzt erstmals privilegierte Rechte auf Host SERVER01 -``` - -## Wichtiger Hinweis - -Diese Regel ist nur sinnvoll, wenn die Baseline gepflegt wird. Sonst wird derselbe Benutzer immer wieder als „erstmals privilegiert“ erkannt. - ---- - -# 19. Regel: `host_risk_score` - -## Zweck - -Aggregiert offene Detections pro Host zu einem Host-Risiko-Score. - -## Datenbasis - -Tabelle: - -```text -detections -``` - -Ausgabe: - -```text -host_risk_scores -``` - -## Zeitfenster - -| Parameter | Default | -|---|---:| -| `RISK_SCORE_WINDOW` | `24h` | - -## Ignorierte Detection-Status - -Diese Status werden nicht in den Risk Score einbezogen: - -```text -false_positive -suppressed -legitimate -resolved -plausible -``` - -## Severity-Gewichte - -| Severity | Gewicht | -|---|---:| -| `critical` | `25` | -| `high` | `10` | -| `medium` | `2` | -| `low` | `0.3` | -| `info` | `0.05` | -| sonst | `0.2` | - -## Status-Modifikatoren - -| Status | Wirkung | -|---|---| -| `confirmed_incident` | `+75` zusätzlich pro Gruppe | -| `investigating` | Gewicht `* 2` | -| `acknowledged` | Gewicht `* 0.5` | -| `open` | Gewicht `* 0.35` | - -## Dämpfung - -Gleiche Events zählen nicht linear, sondern gedämpft: - -```text -score += weight * sqrt(count) -``` - -Dadurch erzeugen 100 gleiche offene Events nicht 100-faches Risiko. - -## Severity aus Risk Score - -| Risk Score | Severity | -|---:|---| -| `>= 120` | `critical` | -| `>= 60` | `high` | -| `>= 20` | `medium` | -| `>= 5` | `low` | -| `< 5` | `info` | - -## Ausgabe-Felder - -| Feld | Bedeutung | -|---|---| -| `hostname` | Host | -| `risk_score` | berechneter Score | -| `severity` | abgeleitete Severity | -| `open_detections` | offene relevante Detections | -| `high_detections` | Anzahl High | -| `critical_detections` | Anzahl Critical | -| `confirmed_incidents` | bestätigte Incidents | -| `last_detection_at` | letzter relevanter Detection-Zeitpunkt | - ---- - -# 20. Suppression-Mechanismen - -## 20.1 Detection-Suppression - -Tabelle: - -```text -detection_suppressions -``` - -Die Funktion: - -```go -isDetectionSuppressed(ctx, det) -``` - -prüft vor dem Insert, ob eine passende aktive Suppression existiert. - -Match-Kriterien: - -```text -rule_name -hostname oder leer -channel_name oder leer -event_id oder 0 -expires_at IS NULL oder expires_at > now -enabled = 1 -``` - -Wenn Suppression aktiv ist, wird keine Detection erzeugt. - -## 20.2 Dynamic Rule Suppression - -Dynamische Regeln prüfen zusätzlich `suppress_for_seconds`. - -Dadurch wird verhindert, dass dieselbe dynamische Regel pro Host zu oft auslöst. - -## 20.3 Baseline-Suppression - -Baseline-Anomalien werden über `BASELINE_SUPPRESS_FOR` zeitlich gedämpft. - -## 20.4 Baseline-Exclusion - -Tabelle: - -```text -baseline_exclusions +```env +EVENT_RETENTION_DAYS=90 +ROLLUP_RETENTION_DAYS=730 +RAW_RETENTION=720h +KAFKA_RETENTION_MS=86400000 ``` -Diese Tabelle verhindert, dass bestimmte Events in die Baseline eingelernt werden. - -Nützlich bei: - -- bestätigten Incidents -- Wartungsfenstern -- bekannten Ausnahmesituationen -- Testläufen -- Rollouts - ---- - -# 21. Detection-Status - -Unterstützte Status: - -| Status | Bedeutung | -|---|---| -| `open` | neu/offen | -| `acknowledged` | gesehen, noch nicht untersucht | -| `investigating` | in Untersuchung | -| `plausible` | plausibel, aber nicht unbedingt dauerhaft legitim | -| `legitimate` | legitim/erwartet | -| `false_positive` | Fehlalarm | -| `resolved` | erledigt | -| `suppressed` | bewusst unterdrückt | -| `confirmed_incident` | bestätigter Sicherheitsvorfall | - -## Auswirkungen - -| Status | Wirkung | -|---|---| -| `false_positive` | `is_false_positive = true`, aus Risk Score ausgeschlossen | -| `legitimate` | `is_legitimate = true`, aus Risk Score ausgeschlossen | -| `plausible` | aus Risk Score ausgeschlossen | -| `resolved` | aus Risk Score ausgeschlossen | -| `suppressed` | aus Risk Score ausgeschlossen | -| `confirmed_incident` | stark erhöhter Risk Score, Baseline-Lernen wird temporär verhindert | +## Detections in Version 1 ---- +Der Detector läuft unabhängig vom UI alle 30 Sekunden mit einem begrenzten Lookback und schreibt nur Findings in PostgreSQL. -# 22. Privileged Users +Enthalten: -## Zweck +- Audit Log gelöscht (1102) +- Dienst installiert (7045) +- Account Lockout (4740) +- Failed Logon Burst (4625) +- Password Spray (4625 gegen viele Benutzer) +- privilegierte Gruppenmitgliedschaft geändert (4728 / 4732 / 4756) -Die Tabelle `privileged_users` erweitert die automatische Erkennung privilegierter Benutzer. +**Es gibt bewusst keine pauschale `new_event_id`-Detection mehr.** Event 5857 und vergleichbare Betriebsereignisse sind normale Events, keine Incidents. ## UI -Pfad: +`/ui` enthält: -```text -/ui/privileged-users -``` +- Eventzahl 24h aus Rollup +- aktive Hosts 24h +- offene High-/Critical-Detections +- Eventsuche nach Zeitraum, Host, User, IP, Event-ID und Channel +- Detection-Liste mit Statusänderung +- Agent-Liste / Last Seen -Funktionen: +Die initiale Eventsuche ist auf 24 Stunden und 500 Ergebnisse begrenzt. Der ClickHouse-Sortierschlüssel beginnt mit Tenant und Event-Zeit; alle UI-Suchen besitzen zusätzlich ein hart begrenztes Zeitfenster und Ergebnislimit. Das verhindert wiederkehrende Vollscans über die gesamte Historie. -- Benutzer hinzufügen -- Grund dokumentieren -- aktivieren/deaktivieren +## Raw-Event wiederherstellen -## Automatische Erkennung +Die Eventliste enthält `raw_object_key`. Das Original ist ein gzip-komprimierter Ingress-Batch. -Auch ohne Tabelle gelten Benutzer als privilegiert, wenn sie folgendem Namensschema entsprechen: - -```text -adm-* -*-adm -*.adm -``` - -## Maschinenkonten - -Maschinenkonten werden ignoriert: - -```text -*$ -``` - ---- - -# 23. Empfohlene Dynamic Rules für Windows Security Monitoring - -Diese Regeln lassen sich über `/ui/rules` pflegen. - -## 23.1 Security Audit Log gelöscht - -| Feld | Wert | -|---|---| -| Name | `security_log_cleared` | -| Severity | `critical` | -| Channel | `Security` | -| Event IDs | `1102` | -| Threshold | `1` | -| Suppress | `3600` | - -## 23.2 Benutzer erstellt - -| Feld | Wert | -|---|---| -| Name | `user_created` | -| Severity | `high` | -| Channel | `Security` | -| Event IDs | `4720` | - -## 23.3 Benutzer aktiviert - -| Feld | Wert | -|---|---| -| Name | `user_enabled` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4722` | - -## 23.4 Benutzer deaktiviert - -| Feld | Wert | -|---|---| -| Name | `user_disabled` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4725` | - -## 23.5 Benutzer gelöscht - -| Feld | Wert | -|---|---| -| Name | `user_deleted` | -| Severity | `high` | -| Channel | `Security` | -| Event IDs | `4726` | - -## 23.6 Sicherheitsgruppe erstellt - -| Feld | Wert | -|---|---| -| Name | `security_group_created` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4727,4731` | - -## 23.7 Mitglied zu Sicherheitsgruppe hinzugefügt - -| Feld | Wert | -|---|---| -| Name | `security_group_member_added` | -| Severity | `high` | -| Channel | `Security` | -| Event IDs | `4728,4732,4756` | - -## 23.8 Mitglied aus Sicherheitsgruppe entfernt - -| Feld | Wert | -|---|---| -| Name | `security_group_member_removed` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4729,4733,4757` | - -## 23.9 Konto gesperrt - -| Feld | Wert | -|---|---| -| Name | `account_locked` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4740` | - -## 23.10 Kerberos Pre-Auth fehlgeschlagen - -| Feld | Wert | -|---|---| -| Name | `kerberos_preauth_failed` | -| Severity | `low` | -| Channel | `Security` | -| Event IDs | `4771` | -| Threshold Count | `10` | -| Threshold Window Seconds | `300` | - -## 23.11 Service installiert - -| Feld | Wert | -|---|---| -| Name | `service_installed` | -| Severity | `high` | -| Channel | `System,Security` | -| Event IDs | `7045,4697` | - -## 23.12 Scheduled Task erstellt - -| Feld | Wert | -|---|---| -| Name | `scheduled_task_created` | -| Severity | `medium` | -| Channel | `Security` | -| Event IDs | `4698` | - -## 23.13 Audit Policy geändert - -| Feld | Wert | -|---|---| -| Name | `audit_policy_changed` | -| Severity | `high` | -| Channel | `Security` | -| Event IDs | `4719` | - ---- - -# 24. Empfohlene Indizes - -Für Performance sollten mindestens diese Indizes vorhanden sein. - -```sql -CREATE INDEX idx_event_logs_ts ON event_logs (ts); -CREATE INDEX idx_event_logs_host_ts ON event_logs (hostname, ts); -CREATE INDEX idx_event_logs_channel_event_ts ON event_logs (channel_name, event_id, ts); -CREATE INDEX idx_event_logs_security_logon ON event_logs (channel_name, event_id, target_user, src_ip, ts); -CREATE INDEX idx_event_logs_host_channel_event_ts ON event_logs (hostname, channel_name, event_id, ts); - -CREATE INDEX idx_detections_created ON detections (created_at); -CREATE INDEX idx_detections_rule_host_created ON detections (rule_name, hostname, created_at); -CREATE INDEX idx_detections_status_created ON detections (status, created_at); -CREATE INDEX idx_detections_host_status_created ON detections (hostname, status, created_at); - -CREATE INDEX idx_agents_enabled_lastseen ON agents (is_enabled, last_seen); - -CREATE INDEX idx_ueba_user_baseline_lookup -ON ueba_user_baseline (username, hostname, src_ip, workstation, first_seen, last_seen); - -CREATE INDEX idx_baseline_event_stats_lookup -ON baseline_event_stats (hostname, channel_name, event_id, hour_of_day, day_of_week); - -CREATE INDEX idx_detection_suppressions_lookup -ON detection_suppressions (enabled, rule_name, hostname, channel_name, event_id, expires_at); - -CREATE INDEX idx_baseline_exclusions_lookup -ON baseline_exclusions (enabled, hostname, channel_name, event_id, expires_at); -``` - ---- - -# 25. Empfohlene SOC-Priorisierung - -## Sofort prüfen - -- `confirmed_incident` -- `critical` -- `success_after_failures` -- `password_spray` -- `security_log_cleared` -- `ueba_admin_new_host` -- `ueba_first_privileged_use` - -## Hoch priorisieren - -- `new_source_ip_for_user` bei privilegierten Benutzern -- `ueba_offhours_login` bei Admin-Konten -- `baseline_event_rate_anomaly` mit hohem Z-Score -- `service_installed` auf Clients oder Domain Controllern -- `scheduled_task_created` außerhalb Wartungsfenster - -## Eher Rauschen / Kontextabhängig - -- `agent_offline` bei normalen Clients -- `reboot_spike` während Patchfenster -- `new_event_id` direkt nach Audit-Policy-Änderungen -- `ueba_new_user_context` bei VPN-/DHCP-Änderungen -- `ueba_offhours_login` ohne LogonType-Filter - ---- - -# 26. Bekannte Rauschquellen - -## SYSTEM / technische Accounts - -Besonders bei `4624` können technische Accounts sehr viele Events erzeugen. - -Empfohlene Ausschlüsse: - -```text -SYSTEM -NT AUTHORITY\SYSTEM -LOCAL SERVICE -NETWORK SERVICE -ANONYMOUS LOGON -DWM-* -UMFD-* -Maschinenkonten mit $ -``` - -## LogonType - -Für echte Benutzeranmeldungen sind meist relevant: - -```text -2, 7, 10, 11 -``` - -Oft rauschanfällig: - -```text -3, 5 -``` - -## Patchfenster - -Während Patchfenstern können diese Regeln anschlagen: - -- `reboot_spike` -- `new_event_id` -- `baseline_event_rate_anomaly` -- `agent_offline` - -Empfehlung: - -- Batch-Bewertung auf `plausible` -- temporäre Baseline-Exclusion -- temporäre Suppression für bekannte Regeln - ---- - -# 27. Empfohlene Weiterentwicklung - -## 27.1 Regelkonfiguration für Off-Hours - -Aktuell ist die Arbeitszeit hart im Code: - -```text -06:00 bis 20:00 -``` - -Empfohlen wäre eine Konfiguration über ENV: - -```text -OFFHOURS_ENABLED=true -WORK_HOUR_START=6 -WORK_HOUR_END=20 -OFFHOURS_LOGON_TYPES=2,7,10,11 -``` - -## 27.2 Noise-Accounts in Tabelle auslagern - -Statt hartem Code könnte eine Tabelle genutzt werden: - -```sql -CREATE TABLE ignored_accounts ( - username VARCHAR(255) NOT NULL PRIMARY KEY, - reason VARCHAR(255) NULL, - enabled BOOLEAN NOT NULL DEFAULT TRUE, - created_at DATETIME(6) NOT NULL DEFAULT UTC_TIMESTAMP(6), - updated_at DATETIME(6) NOT NULL DEFAULT UTC_TIMESTAMP(6) ON UPDATE UTC_TIMESTAMP(6) -); -``` - -## 27.3 Dynamic Rules um Negativfilter erweitern - -Nützlich wären: - -- `exclude_field` -- `exclude_operator` -- `exclude_value` - -Beispiel: - -```text -target_user NOT IN system,local service,network service -``` - -## 27.4 MITRE ATT&CK Mapping - -Jede Detection könnte um MITRE-Technik erweitert werden: - -| Regel | Mögliche Technik | -|---|---| -| `password_spray` | T1110.003 Password Spraying | -| `success_after_failures` | T1110 Brute Force | -| `service_installed` | T1543.003 Windows Service | -| `scheduled_task_created` | T1053.005 Scheduled Task | -| `security_log_cleared` | T1070.001 Clear Windows Event Logs | -| `ueba_admin_new_host` | T1078 Valid Accounts | - ---- - -# 28. Kurze Betriebscheckliste - -## Täglich - -- SOC Dashboard prüfen -- Top Host Risk Scores prüfen -- offene `critical` und `high` Detections bewerten -- False Positives markieren -- legitime Wartungsereignisse als `plausible` oder `legitimate` markieren - -## Wöchentlich - -- häufige False Positives auswerten -- Suppressions prüfen -- Baseline-Exclusions prüfen -- Dynamic Rules nachschärfen -- neue EventIDs bewerten - -## Monatlich - -- Privileged Users prüfen -- UEBA-Baseline-Qualität prüfen -- Indizes und Query-Laufzeiten prüfen -- Prometheus/Grafana-Dashboards kontrollieren -- Retention-Konzept prüfen - ---- - -# 29. Glossar - -| Begriff | Bedeutung | -|---|---| -| Detection | Ein ausgelöster Regel-Treffer | -| Suppression | Unterdrückung bekannter oder akzeptierter Detections | -| Baseline | gelernter Normalzustand | -| UEBA | User and Entity Behavior Analytics | -| Z-Score | statistischer Abstand zum Mittelwert | -| Host Risk Score | aggregierter Risikowert pro Host | -| LogonType | Windows-Klassifikation der Anmeldung | -| `4624` | erfolgreicher Logon | -| `4625` | fehlgeschlagener Logon | -| `4672` | Special Privileges Assigned | -| `1102` | Security Audit Log gelöscht | -| `7045` | Service installiert | - ---- - -# 30. Kurzfazit - -Das Regelwerk kombiniert klassische EventID-Regeln, dynamische Regeln, Baseline-Anomalien und UEBA-Kontextregeln. - -Die wichtigsten Qualitätshebel sind: - -1. technische Accounts konsequent ausschließen, -2. LogonTypes sauber filtern, -3. Baselines nicht mit Incidents trainieren, -4. Status sauber pflegen, -5. legitime Wartungsfenster als `plausible` oder `legitimate` markieren, -6. echte Incidents als `confirmed_incident` markieren, -7. Dynamic Rules regelmäßig nachschärfen. - -Besonders wichtig für die aktuelle `runOffHoursLoginRule`: - -```text -SYSTEM, LOCAL SERVICE, NETWORK SERVICE, ANONYMOUS LOGON und Maschinenkonten sollten ignoriert werden. -Zusätzlich sollte die Regel auf LogonType 2, 7, 10 und 11 begrenzt werden. -``` - ---- - -## Metadata-first Ergänzung: neue Event-IDs realistisch bewerten - -Eine bislang unbekannte Event-ID ist zunächst eine Änderung des Event-Inventars und noch kein Sicherheitsvorfall. Die Regel `new_event_id` besitzt deshalb nun drei Betriebsarten: - -- `NEW_EVENT_ID_MODE=off`: Regel vollständig deaktiviert. `event_catalog` wird weiterhin gepflegt. -- `NEW_EVENT_ID_MODE=inventory` (Standard): bestätigte neue IDs werden höchstens als `info` mit Status `plausible` abgelegt. -- `NEW_EVENT_ID_MODE=alert`: neue IDs in den konfigurierten Alert-Channels werden als `low` beziehungsweise bei hoher Wiederholung als `medium` geöffnet. Erst im Modus `alert` werden explizit konfigurierte High-Risk-IDs zu `medium`/`high` hochgestuft. - -Zusätzliche Schutzmechanismen: - -```env -NEW_EVENT_ID_LEARNING_PERIOD=24h -NEW_EVENT_ID_CONFIRM_WINDOW=15m -NEW_EVENT_ID_MIN_COUNT=3 -NEW_EVENT_ID_IGNORE_CHANNELS=Microsoft-Windows-WMI-Activity/Operational -NEW_EVENT_ID_ALERT_CHANNELS=Security,System -NEW_EVENT_ID_HIGH_RISK_IDS=1102,4697,4719,7045 -``` - -Damit erzeugt Event-ID `5857` im Channel `Microsoft-Windows-WMI-Activity/Operational` standardmäßig keine Detection mehr. Neue Agenten dürfen zunächst eine Baseline lernen. Ein einmaliges Event wird ebenfalls nicht gemeldet. Die stabile Detection-Zeitspanne basiert auf `event_catalog.first_seen`, sodass die Regel bei wiederholten Läufen keine neuen Duplikate erzeugt. - -Für bestehende Installationen kann anschließend ausgeführt werden: +Einfacher über das mitgelieferte Hilfsskript: ```bash -mariadb -u root -p eventcollector < deploy/mariadb/migrations/003-new-event-id-classification.sql +./raw-event.sh '2026/07/23/12/.json.gz' ``` -Die Migration stuft bereits offene `new_event_id`-Meldungen aus dem WMI-Activity/Operational-Channel als `info` und `legitimate` ein. +Ohne `raw_index` wird der gesamte dekomprimierte Ingress-Batch ausgegeben. `raw_index` bezeichnet das Event innerhalb dieses JSON-Batches. -## Begrenzter Stress-Test-Agent +## Stress-Test -Der Agent erzeugt ausschließlich klar markierte Metadaten (`source` und `provider` sind `SIEM-Stress-Agent`). Er besitzt harte Grenzen für Rate, Laufzeit, Worker und Gesamtzahl und startet nur mit expliziter Bestätigung. - -Direkt mit Go: +Der Stress-Agent ist im selben Image enthalten und benötigt eine explizite Bestätigung: ```bash -go run ./cmd/siem-stress-agent \ - --url http://127.0.0.1:8090/ingest \ - --host SIEM-STRESS-01 \ - --api-key 'EIN-EIGENER-TEST-API-KEY' \ - --enrollment-key 'DER-ENROLLMENT-KEY' \ +docker compose run --rm ingress stress-agent \ + --url http://ingress:8080/ingest \ + --api-key stress-agent-key \ + --enrollment-key "$(grep '^ENROLLMENT_KEY=' .env | cut -d= -f2-)" \ --scenario mixed \ - --rate 500 \ - --batch 100 \ - --workers 4 \ - --duration 2m \ - --max-events 60000 \ + --rate 5000 \ + --batch 250 \ + --duration 5m \ --confirm-load-test ``` -Über das opt-in Compose-Profil: +Szenarien: `mixed`, `failed-logon`, `lockout`, `catalog`. + +**Nur in einer kontrollierten Umgebung beziehungsweise gegen das eigene SIEM verwenden.** + +## Betriebschecks + +Der schnellste Gesamtcheck: ```bash -cp .env.example .env -# Kennwörter, ENROLLMENT_KEY und SIEM_STRESS_API_KEY setzen -docker compose --profile stress run --rm stress-agent +./doctor.sh ``` -Szenarien: +Er prüft Containerstatus, HTTP-Readiness, Redpanda/Consumer-Lag, ClickHouse-Zeilen und Speicher, aktuelle Ingest-Verzögerung, Control-Plane-Zähler, Raw-Spool und aktuelle Fehler. -- `normal`: erfolgreiche Logons, um den normalen Ingest-Durchsatz zu messen. -- `failed-logon`: Event 4625 mit wechselnden Benutzern und Test-Netz-IP-Adressen. -- `lockout`: Event 4740 mit Testbenutzern und Test-Workstations. -- `mixed`: Mischung aus normalen Logons, Fehlern, Lockouts, Reboots und WMI 5857. -- `catalog`: bis zu 5.000 künstliche Event-IDs im eigenen Channel `SIEM-Stress-Agent/Operational`; geeignet zum Test des Event-Katalogs, nicht für einen normalen Lasttest. +Zusätzlich: -Der Agent gibt akzeptierte/fehlgeschlagene Events, tatsächlichen Durchsatz, HTTP-Status sowie p50/p95/p99-Latenzen aus. Er sollte nur gegen ein System ausgeführt werden, für das eine ausdrückliche Lasttest-Freigabe besteht. +```bash +docker compose ps +docker compose logs -f ingress processor detector api +``` + +Ingress: + +```bash +curl http://127.0.0.1:8090/healthz +curl http://127.0.0.1:8090/readyz +``` + +ClickHouse Zeilenzahl / Speicher: + +```bash +docker compose exec clickhouse clickhouse-client \ + --user siem --password "$(grep '^CLICKHOUSE_PASSWORD=' .env | cut -d= -f2-)" \ + --query "SELECT formatReadableQuantity(sum(rows)), formatReadableSize(sum(bytes_on_disk)) FROM system.parts WHERE database='siem' AND table='events' AND active" +``` + +Queue: + +```bash +docker compose exec redpanda rpk group describe siem-processor-v1 +``` + +## Hardware für das Single-Node-Deployment + +Für einen Test reichen typischerweise 4 CPU-Kerne, 8 GB RAM und SSD-Speicher. Für eure bisher beobachtete Produktivrate würde ich **mindestens 8 Kerne, 16 GB RAM und schnelle lokale NVMe** einplanen. Die benötigte SSD-Größe hängt stark von Eventgröße, Commandlines und Raw-XML ab; `./doctor.sh` zeigt die reale Kompression und Wachstumsrate, sodass die Retention nach einigen Tagen anhand echter Werte dimensioniert werden kann. + +## Kapazität + +Die bisher beobachteten ca. 19,2 Mio. Events in 48 Stunden entsprechen nur rund 111 Events/s. Das ist für ClickHouse/Redpanda eine kleine Last. Der neue Pfad ist bewusst für deutlich höhere Raten gebaut; die reale Grenze hängt vor allem von CPU, SSD-Durchsatz, Eventgröße, Anzahl der Hosts und Query-Last ab. + +Für hohe Last zuerst skalieren: + +1. `KAFKA_PARTITIONS` erhöhen und mehrere Processor-Instanzen starten. +2. ClickHouse auf schnelle lokale NVMe legen. +3. Redpanda und ClickHouse nicht auf dieselbe langsame Netzwerkplatte legen. +4. Raw-Retention reduzieren, wenn XML sehr groß ist. +5. Erst ab echtem Bedarf ClickHouse / Redpanda / Garage als Mehrknoten-Cluster betreiben. + +## Single-node vs. HA + +`compose.yml` ist ein **One-click Single-node Deployment**. Es eliminiert die bisherigen Architekturprobleme, ist aber kein hochverfügbarer Cluster: fällt der eine Docker-Host aus, sind alle Komponenten betroffen. + +Für echte HA sind mindestens sinnvoll: + +- 3 Redpanda Broker, Replication Factor 3 +- repliziertes ClickHouse (oder ClickHouse Cloud) +- 3 Garage Nodes / anderes redundantes S3 +- PostgreSQL mit Backup/Replica +- Reverse Proxy/TLS und zentraler Identity Provider vor UI/API + +Die Anwendungskomponenten selbst sind stateless und können danach horizontal vervielfacht werden. + +## Sicherheit + +- Datenbank-, Redpanda-, Garage- und Prometheus-Adminports binden standardmäßig nur an `127.0.0.1`. +- Nur Ingress und UI binden standardmäßig an `0.0.0.0`. +- UI/API verlangen ein beim Deployment zufällig erzeugtes Basic-Auth-Passwort. +- Für produktiven externen Zugriff gehört trotzdem TLS/Reverse-Proxy vor Ingress und UI, da Basic Auth allein keine Transportverschlüsselung bietet. +- `.env` enthält Secrets und darf nicht ins Repository. +- Der Ingress speichert ausschließlich SHA-256-Hashes der Agent-API-Keys in PostgreSQL. +- Raw-Logs können personenbezogene Daten enthalten. Retention und Zugriffsrechte entsprechend festlegen. + +## Projektstruktur + +```text +cmd/siem ein Binary, mehrere Rollen +internal/ingress kompatible HTTP-Aufnahme +internal/queue Redpanda/Kafka Producer + Consumer +internal/processor Normalisierung + ClickHouse + Raw-Spool +internal/normalize Windows XML / Meta → kanonisches Event +internal/clickhouse ClickHouse HTTP-Client +internal/postgres Agenten + Detections +internal/detector kontinuierliche Regeln +internal/api Investigation API + UI +internal/stress begrenzter Lastgenerator +doctor.sh Pipeline-/Kapazitätsdiagnose +raw-event.sh Raw-Archiv abrufen +deploy/clickhouse analytisches Schema +deploy/postgres Control-Plane-Schema +deploy/garage S3-Archivkonfiguration +deploy/prometheus Betriebsmetriken +``` + +## Reset + +Nur wenn wirklich alle Daten weg sollen: + +```bash +./reset-DANGEROUS.sh +``` + +Das Skript verlangt zusätzlich die Eingabe `DELETE` und entfernt anschließend die Docker-Volumes. diff --git a/RELEASE-CHECKS.md b/RELEASE-CHECKS.md new file mode 100644 index 0000000..245e283 --- /dev/null +++ b/RELEASE-CHECKS.md @@ -0,0 +1,24 @@ +# Release checks + +Stand: 2026-07-23 + +Durchgeführt: + +- `go test ./...` gegen lokale Interface-Stubs für pgx/kafka-go mit der vorhandenen Go-1.23.2-Toolchain: erfolgreich. +- `go vet ./...` unter denselben isolierten Bedingungen: erfolgreich. +- Ingress-Tests: Batch-UID/Retry-Idempotenz und Validierung: erfolgreich. +- Normalizer-Tests: Windows-Metadaten/XML und Lockout-Workstation: erfolgreich. +- `compose.yml` und Prometheus-YAML syntaktisch geparst: erfolgreich. +- ClickHouse-Prometheus-XML geparst: erfolgreich. +- POSIX-Shellskripte mit `sh -n` geprüft: erfolgreich. +- `deploy.sh` mit simuliertem Docker/Healthcheck end-to-end ausgeführt: Secret-Erzeugung, bestehender Enrollment-Key, Template-Rendering und Compose-Aufrufe erfolgreich. +- ZIP-Integrität mit `unzip -t`: erfolgreich. + +Nicht in dieser Ausführungsumgebung möglich: + +- echter Docker-Compose-Start der Datenbanken/Broker/Object-Storage-Komponenten; +- Integrationstest gegen reale ClickHouse-, PostgreSQL-, Redpanda- und Garage-Container; +- Ausführung von `deploy.ps1`, da PowerShell hier nicht installiert ist; +- Build mit Go 1.26.5, da die lokale Umgebung keinen Netzwerkzugriff für die Toolchain/Module besitzt. Das Dockerfile verwendet Go 1.26.5. + +Der erste reale Deployment-Lauf sollte daher anschließend mit `./doctor.sh` und einem begrenzten Stress-Test geprüft werden. diff --git a/cmd/siem-stress-agent/main.go b/cmd/siem-stress-agent/main.go deleted file mode 100644 index 5df12ba..0000000 --- a/cmd/siem-stress-agent/main.go +++ /dev/null @@ -1,476 +0,0 @@ -package main - -import ( - "bytes" - "context" - "encoding/json" - "errors" - "flag" - "fmt" - "io" - "math" - "net/http" - "net/url" - "os" - "os/signal" - "sort" - "strconv" - "strings" - "sync" - "sync/atomic" - "syscall" - "time" -) - -const ( - maxRate = 100_000 - maxWorkers = 64 - maxDuration = time.Hour - maxEvents = 5_000_000 -) - -type config struct { - Endpoint string - Hostname string - APIKey string - EnrollmentKey string - Scenario string - Rate int - BatchSize int - Workers int - Duration time.Duration - EventLimit int64 - Timeout time.Duration - Confirm bool -} - -type event struct { - Hostname string `json:"host"` - Channel string `json:"channel"` - EventID uint32 `json:"id"` - Source string `json:"source"` - Time time.Time `json:"ts"` - Metadata metadata `json:"meta"` -} - -type metadata struct { - Computer string `json:"computer,omitempty"` - ProviderName string `json:"provider,omitempty"` - TargetUser string `json:"target_user,omitempty"` - TargetDomain string `json:"target_domain,omitempty"` - SubjectUser string `json:"subject_user,omitempty"` - Workstation string `json:"workstation,omitempty"` - Device string `json:"device,omitempty"` - SrcIP string `json:"src_ip,omitempty"` - SrcPort string `json:"src_port,omitempty"` - LogonType string `json:"logon_type,omitempty"` - ProcessName string `json:"process_name,omitempty"` - StatusText string `json:"status,omitempty"` - SubStatus string `json:"sub_status,omitempty"` - Failure string `json:"failure_reason,omitempty"` -} - -type result struct { - Events int - Duration time.Duration - Status int - Err error -} - -type stats struct { - requests atomic.Int64 - accepted atomic.Int64 - failed atomic.Int64 - bytesSent atomic.Int64 - mu sync.Mutex - latencies []time.Duration - statuses map[int]int64 - errors map[string]int64 -} - -func main() { - cfg := parseFlags() - if err := validateConfig(cfg); err != nil { - fmt.Fprintln(os.Stderr, "Konfigurationsfehler:", err) - os.Exit(2) - } - - runCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) - defer stop() - produceCtx, cancelProduction := context.WithTimeout(runCtx, cfg.Duration) - defer cancelProduction() - - transport := &http.Transport{ - MaxIdleConns: cfg.Workers * 2, - MaxIdleConnsPerHost: cfg.Workers, - IdleConnTimeout: 30 * time.Second, - } - client := &http.Client{Transport: transport, Timeout: cfg.Timeout} - defer transport.CloseIdleConnections() - - jobs := make(chan []event, cfg.Workers*2) - results := make(chan result, cfg.Workers*2) - st := &stats{statuses: make(map[int]int64), errors: make(map[string]int64)} - - var workers sync.WaitGroup - for i := 0; i < cfg.Workers; i++ { - workers.Add(1) - go func() { - defer workers.Done() - for batch := range jobs { - results <- sendBatch(runCtx, client, cfg, batch) - } - }() - } - - collectorDone := make(chan struct{}) - go func() { - defer close(collectorDone) - for r := range results { - st.requests.Add(1) - st.mu.Lock() - st.latencies = append(st.latencies, r.Duration) - if r.Status != 0 { - st.statuses[r.Status]++ - } - if r.Err != nil { - st.failed.Add(int64(r.Events)) - st.errors[shortError(r.Err)]++ - } else if r.Status >= 200 && r.Status < 300 { - st.accepted.Add(int64(r.Events)) - } else { - st.failed.Add(int64(r.Events)) - } - st.mu.Unlock() - } - }() - - fmt.Printf("SIEM-Stresstest startet: endpoint=%s host=%s scenario=%s rate=%d/s batch=%d workers=%d duration=%s limit=%d\n", - cfg.Endpoint, cfg.Hostname, cfg.Scenario, cfg.Rate, cfg.BatchSize, cfg.Workers, cfg.Duration, cfg.EventLimit) - started := time.Now() - produced := produce(produceCtx, cfg, jobs, st) - close(jobs) - workers.Wait() - close(results) - <-collectorDone - - printReport(st, produced, time.Since(started)) - if st.failed.Load() > 0 { - os.Exit(1) - } -} - -func parseFlags() config { - var cfg config - flag.StringVar(&cfg.Endpoint, "url", getenv("SIEM_STRESS_URL", "http://127.0.0.1:8090/ingest"), "vollständige Ingest-URL") - flag.StringVar(&cfg.Hostname, "host", getenv("SIEM_STRESS_HOST", "SIEM-STRESS-01"), "Test-Hostname; alle Events eines Batches verwenden diesen Host") - flag.StringVar(&cfg.APIKey, "api-key", os.Getenv("SIEM_STRESS_API_KEY"), "API-Key des Test-Agenten") - flag.StringVar(&cfg.EnrollmentKey, "enrollment-key", os.Getenv("SIEM_STRESS_ENROLLMENT_KEY"), "Enrollment-Key für die erste Registrierung") - flag.StringVar(&cfg.Scenario, "scenario", getenv("SIEM_STRESS_SCENARIO", "mixed"), "mixed, failed-logon, lockout, normal oder catalog") - flag.IntVar(&cfg.Rate, "rate", getenvInt("SIEM_STRESS_RATE", 200), "Zielrate in Events pro Sekunde") - flag.IntVar(&cfg.BatchSize, "batch", getenvInt("SIEM_STRESS_BATCH", 100), "Events pro HTTP-Request, maximal 1000") - flag.IntVar(&cfg.Workers, "workers", getenvInt("SIEM_STRESS_WORKERS", 4), "parallele HTTP-Worker") - flag.DurationVar(&cfg.Duration, "duration", getenvDuration("SIEM_STRESS_DURATION", 30*time.Second), "maximale Laufzeit") - flag.Int64Var(&cfg.EventLimit, "max-events", getenvInt64("SIEM_STRESS_MAX_EVENTS", 100_000), "zusätzliche harte Obergrenze der erzeugten Events") - flag.DurationVar(&cfg.Timeout, "timeout", getenvDuration("SIEM_STRESS_TIMEOUT", 20*time.Second), "HTTP-Timeout pro Request") - flag.BoolVar(&cfg.Confirm, "confirm-load-test", getenvBool("SIEM_STRESS_CONFIRM", false), "bestätigt, dass das Zielsystem für diesen Lasttest autorisiert ist") - flag.Parse() - return cfg -} - -func validateConfig(cfg config) error { - if !cfg.Confirm { - return errors.New("--confirm-load-test fehlt") - } - u, err := url.Parse(cfg.Endpoint) - if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" { - return errors.New("--url muss eine gültige HTTP(S)-Ingest-URL sein") - } - if strings.TrimSpace(cfg.Hostname) == "" || len(cfg.Hostname) > 191 { - return errors.New("--host ist leer oder länger als 191 Zeichen") - } - if strings.TrimSpace(cfg.APIKey) == "" { - return errors.New("--api-key ist erforderlich") - } - if cfg.Rate < 1 || cfg.Rate > maxRate { - return fmt.Errorf("--rate muss zwischen 1 und %d liegen", maxRate) - } - if cfg.BatchSize < 1 || cfg.BatchSize > 1000 { - return errors.New("--batch muss zwischen 1 und 1000 liegen") - } - if cfg.Workers < 1 || cfg.Workers > maxWorkers { - return fmt.Errorf("--workers muss zwischen 1 und %d liegen", maxWorkers) - } - if cfg.Duration <= 0 || cfg.Duration > maxDuration { - return fmt.Errorf("--duration muss größer 0 und höchstens %s sein", maxDuration) - } - if cfg.EventLimit < 1 || cfg.EventLimit > maxEvents { - return fmt.Errorf("--max-events muss zwischen 1 und %d liegen", maxEvents) - } - if cfg.Timeout <= 0 || cfg.Timeout > 2*time.Minute { - return errors.New("--timeout muss größer 0 und höchstens 2m sein") - } - switch strings.ToLower(cfg.Scenario) { - case "mixed", "failed-logon", "lockout", "normal", "catalog": - default: - return errors.New("unbekanntes --scenario") - } - return nil -} - -func produce(ctx context.Context, cfg config, jobs chan<- []event, st *stats) int64 { - batchInterval := time.Duration(float64(time.Second) * float64(cfg.BatchSize) / float64(cfg.Rate)) - if batchInterval < time.Millisecond { - batchInterval = time.Millisecond - } - ticker := time.NewTicker(batchInterval) - defer ticker.Stop() - - var produced int64 - var sequence int64 - for produced < cfg.EventLimit { - remaining := cfg.EventLimit - produced - n := cfg.BatchSize - if int64(n) > remaining { - n = int(remaining) - } - batch := make([]event, n) - now := time.Now().UTC() - for i := range batch { - sequence++ - batch[i] = generateEvent(cfg, sequence, now.Add(time.Duration(i)*time.Microsecond)) - } - encoded, _ := json.Marshal(batch) - st.bytesSent.Add(int64(len(encoded))) - select { - case jobs <- batch: - produced += int64(n) - case <-ctx.Done(): - return produced - } - select { - case <-ticker.C: - case <-ctx.Done(): - return produced - } - } - return produced -} - -func generateEvent(cfg config, seq int64, ts time.Time) event { - base := event{ - Hostname: cfg.Hostname, - Source: "SIEM-Stress-Agent", - Time: ts, - Metadata: metadata{ - Computer: cfg.Hostname, - ProviderName: "SIEM-Stress-Agent", - TargetDomain: "STRESS", - SubjectUser: "stress-agent$", - Workstation: fmt.Sprintf("STRESS-CLIENT-%03d", seq%25), - Device: fmt.Sprintf("STRESS-CLIENT-%03d", seq%25), - SrcIP: fmt.Sprintf("198.18.%d.%d", (seq/250)%250, seq%250+1), - SrcPort: strconv.FormatInt(40000+seq%20000, 10), - ProcessName: `C:\\Program Files\\SIEM-Stress-Agent\\stress.exe`, - }, - } - - scenario := strings.ToLower(cfg.Scenario) - if scenario == "mixed" { - switch seq % 10 { - case 0, 1, 2, 3: - scenario = "failed-logon" - case 4: - scenario = "lockout" - case 5, 6, 7: - scenario = "normal" - case 8: - base.Channel, base.EventID = "System", 1074 - return base - default: - base.Channel, base.EventID = "Microsoft-Windows-WMI-Activity/Operational", 5857 - return base - } - } - - switch scenario { - case "failed-logon": - base.Channel, base.EventID = "Security", 4625 - base.Metadata.TargetUser = fmt.Sprintf("stress-user-%03d", seq%100) - base.Metadata.LogonType = "3" - base.Metadata.StatusText = "0xC000006D" - base.Metadata.SubStatus = "0xC000006A" - base.Metadata.Failure = "Unknown user name or bad password" - case "lockout": - base.Channel, base.EventID = "Security", 4740 - base.Metadata.TargetUser = fmt.Sprintf("stress-lockout-%03d", seq%20) - case "normal": - base.Channel, base.EventID = "Security", 4624 - base.Metadata.TargetUser = fmt.Sprintf("stress-user-%03d", seq%100) - base.Metadata.LogonType = "3" - case "catalog": - base.Channel = "SIEM-Stress-Agent/Operational" - base.EventID = uint32(50_000 + seq%5_000) - base.Metadata.TargetUser = fmt.Sprintf("catalog-user-%03d", seq%50) - } - return base -} - -func sendBatch(ctx context.Context, client *http.Client, cfg config, batch []event) result { - body, err := json.Marshal(batch) - if err != nil { - return result{Events: len(batch), Err: err} - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, cfg.Endpoint, bytes.NewReader(body)) - if err != nil { - return result{Events: len(batch), Err: err} - } - req.Header.Set("Content-Type", "application/json") - req.Header.Set("X-API-Key", cfg.APIKey) - if cfg.EnrollmentKey != "" { - req.Header.Set("X-Enrollment-Key", cfg.EnrollmentKey) - } - req.Header.Set("User-Agent", "siem-stress-agent/1.0") - - started := time.Now() - resp, err := client.Do(req) - duration := time.Since(started) - if err != nil { - return result{Events: len(batch), Duration: duration, Err: err} - } - defer resp.Body.Close() - limited, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096)) - if readErr != nil { - return result{Events: len(batch), Duration: duration, Status: resp.StatusCode, Err: readErr} - } - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return result{Events: len(batch), Duration: duration, Status: resp.StatusCode, Err: fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(limited)))} - } - return result{Events: len(batch), Duration: duration, Status: resp.StatusCode} -} - -func printReport(st *stats, produced int64, elapsed time.Duration) { - st.mu.Lock() - latencies := append([]time.Duration(nil), st.latencies...) - statuses := cloneMap(st.statuses) - errorsByText := cloneStringMap(st.errors) - st.mu.Unlock() - sort.Slice(latencies, func(i, j int) bool { return latencies[i] < latencies[j] }) - - accepted := st.accepted.Load() - failed := st.failed.Load() - throughput := float64(accepted) / math.Max(elapsed.Seconds(), 0.001) - fmt.Println("\n--- Ergebnis ---") - fmt.Printf("Laufzeit: %s\n", elapsed.Round(time.Millisecond)) - fmt.Printf("Erzeugte Events: %d\n", produced) - fmt.Printf("Akzeptierte Events:%d\n", accepted) - fmt.Printf("Fehlgeschlagen: %d\n", failed) - fmt.Printf("HTTP-Requests: %d\n", st.requests.Load()) - fmt.Printf("Durchsatz: %.1f Events/s\n", throughput) - fmt.Printf("JSON gesendet: %.2f MiB\n", float64(st.bytesSent.Load())/(1024*1024)) - if len(latencies) > 0 { - fmt.Printf("Latenz p50/p95/p99:%s / %s / %s\n", percentile(latencies, 0.50), percentile(latencies, 0.95), percentile(latencies, 0.99)) - } - fmt.Printf("HTTP-Status: %v\n", statuses) - if len(errorsByText) > 0 { - fmt.Printf("Fehler: %v\n", errorsByText) - } -} - -func percentile(values []time.Duration, p float64) time.Duration { - if len(values) == 0 { - return 0 - } - idx := int(math.Ceil(float64(len(values))*p)) - 1 - if idx < 0 { - idx = 0 - } - if idx >= len(values) { - idx = len(values) - 1 - } - return values[idx].Round(time.Millisecond) -} - -func shortError(err error) string { - text := strings.TrimSpace(err.Error()) - if len(text) > 180 { - return text[:180] + "..." - } - return text -} - -func getenv(key, fallback string) string { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback - } - return value -} - -func getenvInt(key string, fallback int) int { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback - } - n, err := strconv.Atoi(value) - if err != nil { - fmt.Fprintf(os.Stderr, "ungültiger Integer in %s: %v\n", key, err) - os.Exit(2) - } - return n -} - -func getenvInt64(key string, fallback int64) int64 { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback - } - n, err := strconv.ParseInt(value, 10, 64) - if err != nil { - fmt.Fprintf(os.Stderr, "ungültiger Integer in %s: %v\n", key, err) - os.Exit(2) - } - return n -} - -func getenvDuration(key string, fallback time.Duration) time.Duration { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback - } - d, err := time.ParseDuration(value) - if err != nil { - fmt.Fprintf(os.Stderr, "ungültige Dauer in %s: %v\n", key, err) - os.Exit(2) - } - return d -} - -func getenvBool(key string, fallback bool) bool { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback - } - parsed, err := strconv.ParseBool(value) - if err != nil { - fmt.Fprintf(os.Stderr, "ungültiger Boolean in %s: %v\n", key, err) - os.Exit(2) - } - return parsed -} - -func cloneMap(in map[int]int64) map[int]int64 { - out := make(map[int]int64, len(in)) - for k, v := range in { - out[k] = v - } - return out -} - -func cloneStringMap(in map[string]int64) map[string]int64 { - out := make(map[string]int64, len(in)) - for k, v := range in { - out[k] = v - } - return out -} diff --git a/cmd/siem-stress-agent/main_test.go b/cmd/siem-stress-agent/main_test.go deleted file mode 100644 index ff897c9..0000000 --- a/cmd/siem-stress-agent/main_test.go +++ /dev/null @@ -1,52 +0,0 @@ -package main - -import ( - "context" - "encoding/json" - "net/http" - "net/http/httptest" - "testing" - "time" -) - -func TestSendBatch(t *testing.T) { - var got []event - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.Header.Get("X-API-Key") != "test-key" { - t.Fatalf("unexpected api key") - } - if err := json.NewDecoder(r.Body).Decode(&got); err != nil { - t.Fatalf("decode: %v", err) - } - w.WriteHeader(http.StatusAccepted) - })) - defer server.Close() - - cfg := config{Endpoint: server.URL, APIKey: "test-key", Timeout: time.Second} - batch := []event{generateEvent(config{Hostname: "SIEM-STRESS-01", Scenario: "lockout"}, 1, time.Now().UTC())} - res := sendBatch(context.Background(), server.Client(), cfg, batch) - if res.Err != nil || res.Status != http.StatusAccepted { - t.Fatalf("unexpected result: %+v", res) - } - if len(got) != 1 || got[0].EventID != 4740 || got[0].Source != "SIEM-Stress-Agent" { - t.Fatalf("unexpected payload: %+v", got) - } -} - -func TestValidateConfigRequiresConfirmation(t *testing.T) { - cfg := config{ - Endpoint: "http://127.0.0.1:8080/ingest", - Hostname: "SIEM-STRESS-01", - APIKey: "x", - Scenario: "mixed", - Rate: 1, - BatchSize: 1, - Workers: 1, - Duration: time.Second, - EventLimit: 1, - Timeout: time.Second, - } - if err := validateConfig(cfg); err == nil { - t.Fatal("expected missing confirmation to fail") - } -} diff --git a/cmd/siem/main.go b/cmd/siem/main.go new file mode 100644 index 0000000..14f94bb --- /dev/null +++ b/cmd/siem/main.go @@ -0,0 +1,47 @@ +package main + +import ( + "context" + "fmt" + "log" + "os" + "os/signal" + "syscall" + + "example.com/siem-greenfield/internal/api" + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/detector" + "example.com/siem-greenfield/internal/ingress" + "example.com/siem-greenfield/internal/processor" + "example.com/siem-greenfield/internal/stress" +) + +func main() { + if len(os.Args) < 2 { + fmt.Fprintln(os.Stderr, "usage: siem ") + os.Exit(2) + } + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + cfg := config.Load() + var err error + switch os.Args[1] { + case "ingress": + err = ingress.Run(ctx, cfg) + case "processor": + err = processor.Run(ctx, cfg) + case "detector": + err = detector.Run(ctx, cfg) + case "api": + err = api.Run(ctx, cfg) + case "stress-agent": + err = stress.Run(ctx, cfg, os.Args[2:]) + default: + err = fmt.Errorf("unknown command %q", os.Args[1]) + } + if err != nil { + log.Printf("fatal: %v", err) + os.Exit(1) + } +} diff --git a/compose.yml b/compose.yml index 2ede329..4518f8c 100644 --- a/compose.yml +++ b/compose.yml @@ -1,179 +1,256 @@ +name: greenfield-siem + services: - mariadb: - image: mariadb:11.8 - container_name: siem-mariadb - restart: unless-stopped - ports: - - 3307:3306 - env_file: - - .env + postgres: + image: postgres:18.4-alpine environment: - MARIADB_DATABASE: ${MARIADB_DATABASE} - MARIADB_USER: ${MARIADB_USER} - MARIADB_PASSWORD: ${MARIADB_PASSWORD} - MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD} - TZ: ${TZ} - command: - - --character-set-server=utf8mb4 - - --collation-server=utf8mb4_unicode_ci - - --innodb-buffer-pool-size=512M - - --max-connections=300 + POSTGRES_DB: siem + POSTGRES_USER: siem + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: - - mariadb_data:/var/lib/mysql - - ./deploy/mariadb/init:/docker-entrypoint-initdb.d:ro + - postgres_data:/var/lib/postgresql + - ./deploy/postgres/init.sql:/docker-entrypoint-initdb.d/001-init.sql:ro healthcheck: - test: - - CMD-SHELL - - mariadb-admin ping -h 127.0.0.1 -u root -p$$MARIADB_ROOT_PASSWORD - --silent - interval: 20s - timeout: 5s - retries: 10 - start_period: 30s - networks: - - dockge_default - siem-backend: - image: git.send.nrw/sendnrw/siem-backend:latest - container_name: siem-backend + test: ["CMD-SHELL", "pg_isready -U siem -d siem"] + interval: 5s + timeout: 3s + retries: 30 restart: unless-stopped - env_file: - - .env - environment: - LISTEN_ADDR: ${LISTEN_ADDR} - DB_DSN: ${DB_DSN} - DB_MAX_OPEN_CONNS: ${DB_MAX_OPEN_CONNS} - DB_MAX_IDLE_CONNS: ${DB_MAX_IDLE_CONNS} - DB_CONN_MAX_LIFETIME: ${DB_CONN_MAX_LIFETIME} - DB_CONN_MAX_IDLE_TIME: ${DB_CONN_MAX_IDLE_TIME} - MAX_BODY_BYTES: ${MAX_BODY_BYTES} - HTTP_READ_TIMEOUT: ${HTTP_READ_TIMEOUT} - HTTP_WRITE_TIMEOUT: ${HTTP_WRITE_TIMEOUT} - HTTP_IDLE_TIMEOUT: ${HTTP_IDLE_TIMEOUT} - DETECTION_INTERVAL: ${DETECTION_INTERVAL} - OFFLINE_AFTER: ${OFFLINE_AFTER} - FAILED_LOGON_WINDOW: ${FAILED_LOGON_WINDOW} - FAILED_LOGON_THRESHOLD: ${FAILED_LOGON_THRESHOLD} - REBOOT_WINDOW: ${REBOOT_WINDOW} - REBOOT_THRESHOLD: ${REBOOT_THRESHOLD} - PASSWORD_SPRAY_WINDOW: ${PASSWORD_SPRAY_WINDOW} - PASSWORD_SPRAY_MIN_USERS: ${PASSWORD_SPRAY_MIN_USERS} - PASSWORD_SPRAY_MIN_ATTEMPTS: ${PASSWORD_SPRAY_MIN_ATTEMPTS} - SUCCESS_AFTER_FAILURE_WINDOW: ${SUCCESS_AFTER_FAILURE_WINDOW} - NEW_SOURCE_IP_LOOKBACK: ${NEW_SOURCE_IP_LOOKBACK} - NEW_SOURCE_IP_WINDOW: ${NEW_SOURCE_IP_WINDOW} - DETECTIONS_LIMIT: ${DETECTIONS_LIMIT} - NEW_EVENT_ID_MODE: ${NEW_EVENT_ID_MODE} - NEW_EVENT_ID_LEARNING_PERIOD: ${NEW_EVENT_ID_LEARNING_PERIOD} - NEW_EVENT_ID_CONFIRM_WINDOW: ${NEW_EVENT_ID_CONFIRM_WINDOW} - NEW_EVENT_ID_MIN_COUNT: ${NEW_EVENT_ID_MIN_COUNT} - NEW_EVENT_ID_IGNORE_CHANNELS: ${NEW_EVENT_ID_IGNORE_CHANNELS} - NEW_EVENT_ID_ALERT_CHANNELS: ${NEW_EVENT_ID_ALERT_CHANNELS} - NEW_EVENT_ID_HIGH_RISK_IDS: ${NEW_EVENT_ID_HIGH_RISK_IDS} - STORE_EVENT_ROWS: ${STORE_EVENT_ROWS} - STORE_RAW_XML: ${STORE_RAW_XML} - METADATA_CONTEXT_EVENT_IDS: ${METADATA_CONTEXT_EVENT_IDS} - METADATA_BUCKET: ${METADATA_BUCKET} - EVENT_RETENTION: ${EVENT_RETENTION} - RAW_RETENTION: ${RAW_RETENTION} - METADATA_RETENTION: ${METADATA_RETENTION} - PARTITION_MAINTENANCE_ENABLED: ${PARTITION_MAINTENANCE_ENABLED} - PARTITION_MAINTENANCE_INTERVAL: ${PARTITION_MAINTENANCE_INTERVAL} - PARTITION_INTERVAL: ${PARTITION_INTERVAL} - PARTITION_AHEAD: ${PARTITION_AHEAD} - PARTITION_BEHIND: ${PARTITION_BEHIND} - PARTITION_RETENTION: ${PARTITION_RETENTION} - TZ: ${TZ} - depends_on: - mariadb: - condition: service_healthy ports: - - 8090:8080 - healthcheck: - test: - - CMD-SHELL - - wget -qO- http://127.0.0.1:8080/healthz >/dev/null 2>&1 || exit 1 - interval: 30s - timeout: 5s - retries: 5 - start_period: 20s - networks: - - dockge_default - stress-agent: - profiles: ["stress"] - build: - context: . - dockerfile: Dockerfile.stress - container_name: siem-stress-agent - restart: "no" + - "127.0.0.1:${POSTGRES_PORT:-5432}:5432" + + clickhouse: + image: clickhouse/clickhouse-server:26.6.2.81 environment: - SIEM_STRESS_URL: ${SIEM_STRESS_URL} - SIEM_STRESS_HOST: ${SIEM_STRESS_HOST} - SIEM_STRESS_API_KEY: ${SIEM_STRESS_API_KEY} - SIEM_STRESS_ENROLLMENT_KEY: ${ENROLLMENT_KEY} - SIEM_STRESS_SCENARIO: ${SIEM_STRESS_SCENARIO} - SIEM_STRESS_RATE: ${SIEM_STRESS_RATE} - SIEM_STRESS_BATCH: ${SIEM_STRESS_BATCH} - SIEM_STRESS_WORKERS: ${SIEM_STRESS_WORKERS} - SIEM_STRESS_DURATION: ${SIEM_STRESS_DURATION} - SIEM_STRESS_MAX_EVENTS: ${SIEM_STRESS_MAX_EVENTS} - SIEM_STRESS_TIMEOUT: ${SIEM_STRESS_TIMEOUT} - SIEM_STRESS_CONFIRM: ${SIEM_STRESS_CONFIRM} + CLICKHOUSE_DB: siem + CLICKHOUSE_USER: siem + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD} + CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1" + volumes: + - clickhouse_data:/var/lib/clickhouse + - clickhouse_logs:/var/log/clickhouse-server + - ./deploy/clickhouse/init.sql:/docker-entrypoint-initdb.d/001-init.sql:ro + - ./deploy/clickhouse/prometheus.xml:/etc/clickhouse-server/config.d/prometheus.xml:ro + ulimits: + nofile: + soft: 262144 + hard: 262144 + healthcheck: + test: ["CMD-SHELL", "clickhouse-client --user siem --password $$CLICKHOUSE_PASSWORD --query 'SELECT 1' >/dev/null"] + interval: 5s + timeout: 5s + retries: 30 + restart: unless-stopped + ports: + - "127.0.0.1:${CLICKHOUSE_HTTP_PORT:-8123}:8123" + + redpanda: + image: docker.redpanda.com/redpandadata/redpanda:v26.1.13 + command: + - redpanda + - start + - --kafka-addr + - internal://0.0.0.0:9092,external://0.0.0.0:19092 + - --advertise-kafka-addr + - internal://redpanda:9092,external://localhost:19092 + - --pandaproxy-addr + - internal://0.0.0.0:8082,external://0.0.0.0:18082 + - --advertise-pandaproxy-addr + - internal://redpanda:8082,external://localhost:18082 + - --schema-registry-addr + - internal://0.0.0.0:8081,external://0.0.0.0:18081 + - --rpc-addr + - redpanda:33145 + - --advertise-rpc-addr + - redpanda:33145 + - --mode + - dev-container + - --smp + - "2" + - --default-log-level=info + volumes: + - redpanda_data:/var/lib/redpanda/data + healthcheck: + test: ["CMD-SHELL", "RPK_BROKERS=localhost:9092 rpk cluster health | grep -q 'Healthy:.*true'"] + interval: 5s + timeout: 5s + retries: 40 + restart: unless-stopped + ports: + - "127.0.0.1:${REDPANDA_KAFKA_PORT:-19092}:19092" + - "127.0.0.1:${REDPANDA_ADMIN_PORT:-19644}:9644" + + redpanda-init: + image: docker.redpanda.com/redpandadata/redpanda:v26.1.13 depends_on: - siem-backend: + redpanda: condition: service_healthy - networks: - - dockge_default + entrypoint: ["/bin/sh", "-ec"] + environment: + RPK_BROKERS: redpanda:9092 + command: >- + rpk topic create --if-not-exists -p ${KAFKA_PARTITIONS:-6} -r 1 + -c retention.ms=${KAFKA_RETENTION_MS:-86400000} + siem-events + restart: "no" + + redpanda-console: + image: docker.redpanda.com/redpandadata/console:v3.8.0 + depends_on: + redpanda: + condition: service_healthy + entrypoint: /bin/sh + command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console' + environment: + CONFIG_FILEPATH: /tmp/config.yml + CONSOLE_CONFIG_FILE: | + kafka: + brokers: ["redpanda:9092"] + schemaRegistry: + enabled: true + urls: ["http://redpanda:8081"] + redpanda: + adminApi: + enabled: true + urls: ["http://redpanda:9644"] + restart: unless-stopped + ports: + - "127.0.0.1:${REDPANDA_CONSOLE_PORT:-8081}:8080" + + garage: + image: dxflrs/garage:v2.3.0 + command: ["/garage", "server", "--single-node", "--default-bucket"] + environment: + GARAGE_DEFAULT_ACCESS_KEY: ${GARAGE_ACCESS_KEY} + GARAGE_DEFAULT_SECRET_KEY: ${GARAGE_SECRET_KEY} + GARAGE_DEFAULT_BUCKET: ${GARAGE_BUCKET:-siem-raw} + volumes: + - garage_meta:/var/lib/garage/meta + - garage_data:/var/lib/garage/data + - ./deploy/garage/garage.toml:/etc/garage.toml:ro + restart: unless-stopped + ports: + - "127.0.0.1:${GARAGE_S3_PORT:-3900}:3900" + - "127.0.0.1:${GARAGE_ADMIN_PORT:-3903}:3903" + + ingress: + image: greenfield-siem-app:local + build: . + command: ["ingress"] + environment: &app-env + SERVICE_ADDR: :8080 + POSTGRES_URL: postgres://siem:${POSTGRES_PASSWORD}@postgres:5432/siem?sslmode=disable + CLICKHOUSE_URL: http://clickhouse:8123 + CLICKHOUSE_USER: siem + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD} + KAFKA_BROKERS: redpanda:9092 + KAFKA_TOPIC: siem-events + TENANT_ID: ${TENANT_ID:-default} + UI_USERNAME: ${UI_USERNAME:-admin} + UI_PASSWORD: ${UI_PASSWORD} + ENROLLMENT_KEY: ${ENROLLMENT_KEY} + RAW_ARCHIVE_ENABLED: ${RAW_ARCHIVE_ENABLED:-true} + RAW_SPOOL_DIR: /var/spool/siem-raw + RAW_RETENTION: ${RAW_RETENTION:-720h} + depends_on: + postgres: + condition: service_healthy + redpanda-init: + condition: service_completed_successfully + restart: unless-stopped + ports: + - "${INGRESS_BIND:-0.0.0.0}:${INGRESS_PORT:-8090}:8080" + + processor: + image: greenfield-siem-app:local + command: ["processor"] + user: "0:0" + environment: + <<: *app-env + KAFKA_GROUP: siem-processor-v1 + volumes: + - raw_spool:/var/spool/siem-raw + depends_on: + clickhouse: + condition: service_healthy + redpanda-init: + condition: service_completed_successfully + restart: unless-stopped + + detector: + image: greenfield-siem-app:local + command: ["detector"] + environment: + <<: *app-env + DETECTOR_INTERVAL: ${DETECTOR_INTERVAL:-30s} + DETECTOR_LOOKBACK: ${DETECTOR_LOOKBACK:-20m} + depends_on: + clickhouse: + condition: service_healthy + postgres: + condition: service_healthy + processor: + condition: service_started + restart: unless-stopped + + api: + image: greenfield-siem-app:local + command: ["api"] + environment: + <<: *app-env + UI_QUERY_LIMIT: ${UI_QUERY_LIMIT:-500} + depends_on: + clickhouse: + condition: service_healthy + postgres: + condition: service_healthy + restart: unless-stopped + ports: + - "${UI_BIND:-0.0.0.0}:${UI_PORT:-8080}:8080" + + archive-uploader: + image: rclone/rclone:1.74.4 + entrypoint: ["/bin/sh", "-ec"] + command: | + while true; do + rclone move /spool garage:${GARAGE_BUCKET:-siem-raw} --exclude '*.tmp' --create-empty-src-dirs --transfers 8 --checkers 8 --retries 5 || true + rclone delete garage:${GARAGE_BUCKET:-siem-raw} --min-age ${RAW_RETENTION:-720h} --retries 5 || true + sleep 30 + done + environment: + RCLONE_CONFIG_GARAGE_TYPE: s3 + RCLONE_CONFIG_GARAGE_PROVIDER: Other + RCLONE_CONFIG_GARAGE_ACCESS_KEY_ID: ${GARAGE_ACCESS_KEY} + RCLONE_CONFIG_GARAGE_SECRET_ACCESS_KEY: ${GARAGE_SECRET_KEY} + RCLONE_CONFIG_GARAGE_ENDPOINT: http://garage:3900 + RCLONE_CONFIG_GARAGE_REGION: garage + RCLONE_CONFIG_GARAGE_FORCE_PATH_STYLE: "true" + volumes: + - raw_spool:/spool + depends_on: + - garage + - processor + restart: unless-stopped prometheus: - image: prom/prometheus:latest - container_name: siem-prometheus - restart: unless-stopped - env_file: - - .env - environment: - TZ: ${TZ} - command: - - --config.file=/etc/prometheus/prometheus.yml - - --storage.tsdb.path=/prometheus - - --storage.tsdb.retention.time=30d - - --web.enable-lifecycle - depends_on: - siem-backend: - condition: service_healthy + image: prom/prometheus:v3.13.1 + command: ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.retention.time=30d"] volumes: - ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - ./deploy/prometheus/rules:/etc/prometheus/rules:ro - prometheus_data:/prometheus - ports: - - 9090:9090 - networks: - - dockge_default - grafana: - image: grafana/grafana:latest - container_name: siem-grafana - restart: unless-stopped - env_file: - - .env - environment: - GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER} - GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD} - GF_USERS_ALLOW_SIGN_UP: "false" - GF_SERVER_ROOT_URL: http://localhost:3000 - TZ: ${TZ} depends_on: - - prometheus - volumes: - - grafana_data:/var/lib/grafana - - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro - - ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro + - ingress + - redpanda + restart: unless-stopped ports: - - 3090:3000 - networks: - - dockge_default + - "127.0.0.1:${PROMETHEUS_PORT:-9090}:9090" + volumes: - mariadb_data: null - prometheus_data: null - grafana_data: null -networks: - dockge_default: - external: true + postgres_data: + clickhouse_data: + clickhouse_logs: + redpanda_data: + garage_meta: + garage_data: + raw_spool: + prometheus_data: diff --git a/deploy.ps1 b/deploy.ps1 new file mode 100644 index 0000000..a1f8117 --- /dev/null +++ b/deploy.ps1 @@ -0,0 +1,70 @@ +$ErrorActionPreference = "Stop" +Set-Location $PSScriptRoot +if (-not (Get-Command docker -ErrorAction SilentlyContinue)) { throw "Docker fehlt." } +docker compose version | Out-Null +function Hex([int]$bytes) { $b = New-Object byte[] $bytes; [Security.Cryptography.RandomNumberGenerator]::Fill($b); return ([BitConverter]::ToString($b)).Replace("-","").ToLower() } +if (-not (Test-Path .env)) { + $presetEnrollment = $env:ENROLLMENT_KEY + $enrollment = if ([string]::IsNullOrWhiteSpace($presetEnrollment)) { Hex 32 } else { $presetEnrollment } + $vals = @{ + POSTGRES_PASSWORD=Hex 24; CLICKHOUSE_PASSWORD=Hex 24; ENROLLMENT_KEY=$enrollment; UI_PASSWORD=Hex 18; + GARAGE_ACCESS_KEY="GK$(Hex 16)"; GARAGE_SECRET_KEY=Hex 32; GARAGE_RPC_SECRET=Hex 32; + GARAGE_ADMIN_TOKEN=Hex 32; GARAGE_METRICS_TOKEN=Hex 32 + } + @" +INGRESS_BIND=0.0.0.0 +INGRESS_PORT=8090 +UI_BIND=0.0.0.0 +UI_PORT=8080 +TENANT_ID=default +UI_USERNAME=admin +UI_PASSWORD=$($vals.UI_PASSWORD) +POSTGRES_PASSWORD=$($vals.POSTGRES_PASSWORD) +CLICKHOUSE_PASSWORD=$($vals.CLICKHOUSE_PASSWORD) +ENROLLMENT_KEY=$($vals.ENROLLMENT_KEY) +GARAGE_ACCESS_KEY=$($vals.GARAGE_ACCESS_KEY) +GARAGE_SECRET_KEY=$($vals.GARAGE_SECRET_KEY) +GARAGE_RPC_SECRET=$($vals.GARAGE_RPC_SECRET) +GARAGE_ADMIN_TOKEN=$($vals.GARAGE_ADMIN_TOKEN) +GARAGE_METRICS_TOKEN=$($vals.GARAGE_METRICS_TOKEN) +GARAGE_BUCKET=siem-raw +EVENT_RETENTION_DAYS=90 +ROLLUP_RETENTION_DAYS=730 +RAW_RETENTION=720h +RAW_ARCHIVE_ENABLED=true +KAFKA_RETENTION_MS=86400000 +KAFKA_PARTITIONS=6 +DETECTOR_INTERVAL=30s +DETECTOR_LOOKBACK=20m +UI_QUERY_LIMIT=500 +"@ | Set-Content .env +} +$envMap=@{}; Get-Content .env | ForEach-Object { if ($_ -match '^([^#=]+)=(.*)$') {$envMap[$matches[1]]=$matches[2]} } +if (-not $envMap.ContainsKey('UI_USERNAME')) { $envMap['UI_USERNAME']='admin'; Add-Content .env "UI_USERNAME=admin" } +if (-not $envMap.ContainsKey('UI_PASSWORD') -or [string]::IsNullOrWhiteSpace($envMap['UI_PASSWORD'])) { $envMap['UI_PASSWORD']=Hex 18; Add-Content .env "UI_PASSWORD=$($envMap.UI_PASSWORD)" } +(Get-Content deploy/garage/garage.toml.template -Raw).Replace('__GARAGE_RPC_SECRET__',$envMap.GARAGE_RPC_SECRET).Replace('__GARAGE_ADMIN_TOKEN__',$envMap.GARAGE_ADMIN_TOKEN).Replace('__GARAGE_METRICS_TOKEN__',$envMap.GARAGE_METRICS_TOKEN) | Set-Content deploy/garage/garage.toml +(Get-Content deploy/clickhouse/init.sql.template -Raw).Replace('__EVENT_RETENTION_DAYS__',$envMap.EVENT_RETENTION_DAYS).Replace('__ROLLUP_RETENTION_DAYS__',$envMap.ROLLUP_RETENTION_DAYS) | Set-Content deploy/clickhouse/init.sql +Write-Host "Baue und starte Greenfield SIEM ..." +docker compose build --pull +docker compose up -d --remove-orphans + +Write-Host "Pruefe Dienste ..." +$healthy = $false +for ($i = 0; $i -lt 60; $i++) { + try { + Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 "http://127.0.0.1:$($envMap.UI_PORT)/readyz" | Out-Null + Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 "http://127.0.0.1:$($envMap.INGRESS_PORT)/readyz" | Out-Null + $healthy = $true + break + } catch { + Start-Sleep -Seconds 2 + } +} +if (-not $healthy) { throw "Deployment gestartet, Healthcheck noch nicht gruen. Pruefe: docker compose ps; docker compose logs" } +Write-Host "" +Write-Host "Greenfield SIEM laeuft." +Write-Host "UI: http://127.0.0.1:$($envMap.UI_PORT)/ui" +Write-Host "Ingress: http://127.0.0.1:$($envMap.INGRESS_PORT)/ingest" +Write-Host "UI-Login: $($envMap.UI_USERNAME) / $($envMap.UI_PASSWORD)" +Write-Host "Enrollment-Key: $($envMap.ENROLLMENT_KEY)" +Write-Host "Secrets stehen in .env (Datei schuetzen und sichern)." diff --git a/deploy.sh b/deploy.sh new file mode 100755 index 0000000..46f16b7 --- /dev/null +++ b/deploy.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env sh +set -eu +cd "$(dirname "$0")" +command -v docker >/dev/null 2>&1 || { echo "Docker fehlt." >&2; exit 1; } +docker compose version >/dev/null 2>&1 || { echo "Docker Compose Plugin fehlt." >&2; exit 1; } + +PRESET_ENROLLMENT_KEY="${ENROLLMENT_KEY:-}" +randhex(){ n="$1"; if command -v openssl >/dev/null 2>&1; then openssl rand -hex "$n"; else dd if=/dev/urandom bs="$n" count=1 2>/dev/null | od -An -tx1 | tr -d ' \n'; fi; } +http_ok(){ + url="$1" + if command -v curl >/dev/null 2>&1; then curl -fsS "$url" >/dev/null 2>&1; return $?; fi + if command -v wget >/dev/null 2>&1; then wget -q -O /dev/null "$url" >/dev/null 2>&1; return $?; fi + return 2 +} +if [ ! -f .env ]; then + POSTGRES_PASSWORD="$(randhex 24)" + CLICKHOUSE_PASSWORD="$(randhex 24)" + ENROLLMENT_KEY="${PRESET_ENROLLMENT_KEY:-$(randhex 32)}" + UI_PASSWORD="$(randhex 18)" + GARAGE_ACCESS_KEY="GK$(randhex 16)" + GARAGE_SECRET_KEY="$(randhex 32)" + GARAGE_RPC_SECRET="$(randhex 32)" + GARAGE_ADMIN_TOKEN="$(randhex 32)" + GARAGE_METRICS_TOKEN="$(randhex 32)" + cat > .env </dev/null || true +fi +set -a; . ./.env; set +a +UI_USERNAME="${UI_USERNAME:-admin}" +if [ -z "${UI_PASSWORD:-}" ]; then + UI_PASSWORD="$(randhex 18)" + printf '\nUI_USERNAME=%s\nUI_PASSWORD=%s\n' "$UI_USERNAME" "$UI_PASSWORD" >> .env + export UI_USERNAME UI_PASSWORD +fi +sed -e "s/__GARAGE_RPC_SECRET__/$GARAGE_RPC_SECRET/g" -e "s/__GARAGE_ADMIN_TOKEN__/$GARAGE_ADMIN_TOKEN/g" -e "s/__GARAGE_METRICS_TOKEN__/$GARAGE_METRICS_TOKEN/g" deploy/garage/garage.toml.template > deploy/garage/garage.toml +sed -e "s/__EVENT_RETENTION_DAYS__/${EVENT_RETENTION_DAYS:-90}/g" -e "s/__ROLLUP_RETENTION_DAYS__/${ROLLUP_RETENTION_DAYS:-730}/g" deploy/clickhouse/init.sql.template > deploy/clickhouse/init.sql +chmod 600 deploy/garage/garage.toml 2>/dev/null || true + +echo "Baue und starte Greenfield SIEM ..." +docker compose build --pull +docker compose up -d --remove-orphans + +echo "Prüfe Dienste ..." +if command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1; then + i=0 + while [ "$i" -lt 60 ]; do + if http_ok "http://127.0.0.1:${UI_PORT:-8080}/readyz" && http_ok "http://127.0.0.1:${INGRESS_PORT:-8090}/readyz"; then break; fi + i=$((i+1)); sleep 2 + done + if [ "$i" -ge 60 ]; then echo "Deployment gestartet, Healthcheck noch nicht grün. Prüfe: docker compose ps && docker compose logs" >&2; exit 1; fi +else + echo "Hinweis: curl/wget fehlt; HTTP-Healthcheck wird übersprungen." + docker compose ps +fi + +echo +echo "Greenfield SIEM läuft." +echo "UI: http://127.0.0.1:${UI_PORT:-8080}/ui" +echo "Ingress: http://127.0.0.1:${INGRESS_PORT:-8090}/ingest" +echo "UI-Login: $UI_USERNAME / $UI_PASSWORD" +echo "Enrollment-Key: $ENROLLMENT_KEY" +echo "Secrets stehen in .env (Datei schützen und sichern)." diff --git a/deploy/clickhouse/init.sql b/deploy/clickhouse/init.sql new file mode 100644 index 0000000..ed67cc7 --- /dev/null +++ b/deploy/clickhouse/init.sql @@ -0,0 +1,82 @@ +CREATE DATABASE IF NOT EXISTS siem; + +CREATE TABLE IF NOT EXISTS siem.events +( + event_uid String, + queue_partition Int32, + queue_offset Int64, + tenant_id LowCardinality(String), + event_time DateTime64(3, 'UTC'), + ingest_time DateTime64(3, 'UTC'), + event_date Date MATERIALIZED toDate(event_time), + agent_id String, + host_name LowCardinality(String), + source_type LowCardinality(String), + channel LowCardinality(String), + provider LowCardinality(String), + event_code UInt32, + category LowCardinality(String), + action LowCardinality(String), + outcome LowCardinality(String), + severity UInt8, + user_name String, + user_domain LowCardinality(String), + subject_user String, + subject_domain LowCardinality(String), + target_user String, + target_domain LowCardinality(String), + source_ip String, + source_port UInt16, + destination_ip String, + destination_port UInt16, + workstation String, + logon_type LowCardinality(String), + authentication_package LowCardinality(String), + logon_process LowCardinality(String), + status_code LowCardinality(String), + sub_status_code LowCardinality(String), + failure_reason LowCardinality(String), + process_path String, + parent_process_path String, + command_line String CODEC(ZSTD(3)), + message String CODEC(ZSTD(3)), + attributes Map(String, String) CODEC(ZSTD(3)), + raw_object_key String, + raw_index UInt32, + payload_hash FixedString(64), + schema_version UInt16, + parser_version UInt16, + ingest_delay_ms Int64, + INDEX idx_host host_name TYPE set(10000) GRANULARITY 4, + INDEX idx_event_code event_code TYPE set(256) GRANULARITY 4, + INDEX idx_user user_name TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_target target_user TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_source_ip source_ip TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_process process_path TYPE bloom_filter(0.01) GRANULARITY 8 +) +ENGINE = ReplacingMergeTree(ingest_time) +PARTITION BY toYYYYMM(event_time) +ORDER BY (tenant_id, event_time, host_name, event_code, event_uid) +TTL event_time + INTERVAL 90 DAY DELETE +SETTINGS index_granularity = 8192; + +CREATE TABLE IF NOT EXISTS siem.events_5m +( + tenant_id LowCardinality(String), + bucket DateTime('UTC'), + host_name LowCardinality(String), + event_code UInt32, + category LowCardinality(String), + action LowCardinality(String), + outcome LowCardinality(String), + cnt_state AggregateFunction(uniqExact, String) +) +ENGINE = AggregatingMergeTree +PARTITION BY toYYYYMM(bucket) +ORDER BY (tenant_id, bucket, host_name, event_code, category, action, outcome) +TTL bucket + INTERVAL 730 DAY DELETE; + +CREATE MATERIALIZED VIEW IF NOT EXISTS siem.events_5m_mv TO siem.events_5m AS +SELECT tenant_id, toStartOfInterval(event_time, INTERVAL 5 MINUTE) AS bucket, host_name, event_code, category, action, outcome, uniqExactState(event_uid) AS cnt_state +FROM siem.events +GROUP BY tenant_id, bucket, host_name, event_code, category, action, outcome; diff --git a/deploy/clickhouse/init.sql.template b/deploy/clickhouse/init.sql.template new file mode 100644 index 0000000..433779b --- /dev/null +++ b/deploy/clickhouse/init.sql.template @@ -0,0 +1,82 @@ +CREATE DATABASE IF NOT EXISTS siem; + +CREATE TABLE IF NOT EXISTS siem.events +( + event_uid String, + queue_partition Int32, + queue_offset Int64, + tenant_id LowCardinality(String), + event_time DateTime64(3, 'UTC'), + ingest_time DateTime64(3, 'UTC'), + event_date Date MATERIALIZED toDate(event_time), + agent_id String, + host_name LowCardinality(String), + source_type LowCardinality(String), + channel LowCardinality(String), + provider LowCardinality(String), + event_code UInt32, + category LowCardinality(String), + action LowCardinality(String), + outcome LowCardinality(String), + severity UInt8, + user_name String, + user_domain LowCardinality(String), + subject_user String, + subject_domain LowCardinality(String), + target_user String, + target_domain LowCardinality(String), + source_ip String, + source_port UInt16, + destination_ip String, + destination_port UInt16, + workstation String, + logon_type LowCardinality(String), + authentication_package LowCardinality(String), + logon_process LowCardinality(String), + status_code LowCardinality(String), + sub_status_code LowCardinality(String), + failure_reason LowCardinality(String), + process_path String, + parent_process_path String, + command_line String CODEC(ZSTD(3)), + message String CODEC(ZSTD(3)), + attributes Map(String, String) CODEC(ZSTD(3)), + raw_object_key String, + raw_index UInt32, + payload_hash FixedString(64), + schema_version UInt16, + parser_version UInt16, + ingest_delay_ms Int64, + INDEX idx_host host_name TYPE set(10000) GRANULARITY 4, + INDEX idx_event_code event_code TYPE set(256) GRANULARITY 4, + INDEX idx_user user_name TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_target target_user TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_source_ip source_ip TYPE bloom_filter(0.01) GRANULARITY 4, + INDEX idx_process process_path TYPE bloom_filter(0.01) GRANULARITY 8 +) +ENGINE = ReplacingMergeTree(ingest_time) +PARTITION BY toYYYYMM(event_time) +ORDER BY (tenant_id, event_time, host_name, event_code, event_uid) +TTL event_time + INTERVAL __EVENT_RETENTION_DAYS__ DAY DELETE +SETTINGS index_granularity = 8192; + +CREATE TABLE IF NOT EXISTS siem.events_5m +( + tenant_id LowCardinality(String), + bucket DateTime('UTC'), + host_name LowCardinality(String), + event_code UInt32, + category LowCardinality(String), + action LowCardinality(String), + outcome LowCardinality(String), + cnt_state AggregateFunction(uniqExact, String) +) +ENGINE = AggregatingMergeTree +PARTITION BY toYYYYMM(bucket) +ORDER BY (tenant_id, bucket, host_name, event_code, category, action, outcome) +TTL bucket + INTERVAL __ROLLUP_RETENTION_DAYS__ DAY DELETE; + +CREATE MATERIALIZED VIEW IF NOT EXISTS siem.events_5m_mv TO siem.events_5m AS +SELECT tenant_id, toStartOfInterval(event_time, INTERVAL 5 MINUTE) AS bucket, host_name, event_code, category, action, outcome, uniqExactState(event_uid) AS cnt_state +FROM siem.events +GROUP BY tenant_id, bucket, host_name, event_code, category, action, outcome; diff --git a/deploy/clickhouse/prometheus.xml b/deploy/clickhouse/prometheus.xml new file mode 100644 index 0000000..73d51be --- /dev/null +++ b/deploy/clickhouse/prometheus.xml @@ -0,0 +1,10 @@ + + + /metrics + 9363 + true + true + true + true + + diff --git a/deploy/garage/garage.toml.template b/deploy/garage/garage.toml.template new file mode 100644 index 0000000..e1d1bbb --- /dev/null +++ b/deploy/garage/garage.toml.template @@ -0,0 +1,17 @@ +metadata_dir = "/var/lib/garage/meta" +data_dir = "/var/lib/garage/data" +db_engine = "sqlite" +replication_factor = 1 +rpc_bind_addr = "[::]:3901" +rpc_public_addr = "garage:3901" +rpc_secret = "__GARAGE_RPC_SECRET__" + +[s3_api] +s3_region = "garage" +api_bind_addr = "[::]:3900" +root_domain = ".s3.garage.localhost" + +[admin] +api_bind_addr = "[::]:3903" +admin_token = "__GARAGE_ADMIN_TOKEN__" +metrics_token = "__GARAGE_METRICS_TOKEN__" diff --git a/deploy/grafana/provisioning/dashboards/dashboards.yml b/deploy/grafana/provisioning/dashboards/dashboards.yml deleted file mode 100644 index e95f332..0000000 --- a/deploy/grafana/provisioning/dashboards/dashboards.yml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: 1 - -providers: - - name: SIEM Dashboards - orgId: 1 - folder: SIEM - type: file - disableDeletion: false - editable: true - updateIntervalSeconds: 30 - options: - path: /var/lib/grafana/dashboards \ No newline at end of file diff --git a/deploy/grafana/provisioning/dashboards/siem-overview.json b/deploy/grafana/provisioning/dashboards/siem-overview.json deleted file mode 100644 index a0b0bfe..0000000 --- a/deploy/grafana/provisioning/dashboards/siem-overview.json +++ /dev/null @@ -1,745 +0,0 @@ -{ - "annotations": { - "list": [] - }, - "editable": true, - "fiscalYearStartMonth": 0, - "graphTooltip": 1, - "links": [], - "liveNow": false, - "panels": [ - { - "type": "stat", - "title": "Active Agents", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 }, - "targets": [ - { - "expr": "eventcollector_active_agents", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "red", "value": null }, - { "color": "green", "value": 1 } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "Events/s", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 }, - "targets": [ - { - "expr": "sum(rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "eps", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "textMode": "auto" - } - }, - { - "type": "stat", - "title": "High Detections 5m", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 8, "y": 0 }, - "targets": [ - { - "expr": "sum(increase(eventcollector_detection_hits_total{severity=\"high\",rule=~\"$rule\"}[5m]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "red", "value": 1 } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "textMode": "auto" - } - }, - { - "type": "stat", - "title": "Baseline Max Z-Score", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 12, "y": 0 }, - "targets": [ - { - "expr": "max(eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "decimals": 2, - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "orange", "value": 3 }, - { "color": "red", "value": 5 } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "textMode": "auto" - } - }, - { - "type": "stat", - "title": "Rule Errors 5m", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 16, "y": 0 }, - "targets": [ - { - "expr": "sum(increase(eventcollector_rule_errors_total{rule=~\"$rule\"}[5m]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "red", "value": 1 } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "textMode": "auto" - } - }, - { - "type": "stat", - "title": "DB Insert Failures 5m", - "datasource": "$datasource", - "gridPos": { "h": 4, "w": 4, "x": 20, "y": 0 }, - "targets": [ - { - "expr": "increase(eventcollector_db_insert_failures_total[5m])", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "red", "value": 1 } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto", - "orientation": "auto", - "textMode": "auto" - } - }, - - { - "type": "timeseries", - "title": "Ingested Events / Second by Channel", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 }, - "targets": [ - { - "expr": "sum by (channel) (rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m]))", - "legendFormat": "{{channel}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "eps", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "Detection Hits by Rule / Severity", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 }, - "targets": [ - { - "expr": "sum by (rule,severity) (increase(eventcollector_detection_hits_total{rule=~\"$rule\",severity=~\"$severity\"}[5m]))", - "legendFormat": "{{rule}} / {{severity}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short" - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - - { - "type": "timeseries", - "title": "Baseline: Current Count vs Average", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 }, - "targets": [ - { - "expr": "eventcollector_baseline_current_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}", - "legendFormat": "current {{host}} {{channel}} {{event_id}}", - "refId": "A" - }, - { - "expr": "eventcollector_baseline_avg_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}", - "legendFormat": "avg {{host}} {{channel}} {{event_id}}", - "refId": "B" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "Baseline Z-Score", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 }, - "targets": [ - { - "expr": "eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"}", - "legendFormat": "{{host}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "decimals": 2, - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "orange", "value": 3 }, - { "color": "red", "value": 5 } - ] - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - - { - "type": "bargauge", - "title": "Top Baseline Z-Scores", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 8, "x": 0, "y": 20 }, - "targets": [ - { - "expr": "topk(10, eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})", - "legendFormat": "{{host}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "decimals": 2, - "thresholds": { - "mode": "absolute", - "steps": [ - { "color": "green", "value": null }, - { "color": "orange", "value": 3 }, - { "color": "red", "value": 5 } - ] - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top EventIDs by Ingest Rate", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 8, "x": 8, "y": 20 }, - "targets": [ - { - "expr": "topk(15, sum by (channel,event_id) (rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m])))", - "legendFormat": "{{channel}} / {{event_id}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "eps", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top Detection Rules 1h", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 8, "x": 16, "y": 20 }, - "targets": [ - { - "expr": "topk(15, sum by (rule,severity) (increase(eventcollector_detection_hits_total{rule=~\"$rule\",severity=~\"$severity\"}[1h])))", - "legendFormat": "{{rule}} / {{severity}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short" - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - - { - "type": "timeseries", - "title": "HTTP Requests by Path / Status", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 }, - "targets": [ - { - "expr": "sum by (path,status) (rate(eventcollector_http_requests_total[5m]))", - "legendFormat": "{{path}} {{status}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "reqps", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "HTTP Latency p95", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 }, - "targets": [ - { - "expr": "histogram_quantile(0.95, sum by (le,path) (rate(eventcollector_http_request_duration_seconds_bucket[5m])))", - "legendFormat": "{{path}} p95", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s", - "decimals": 3 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - - { - "type": "timeseries", - "title": "DB Insert Transaction Latency p95", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 }, - "targets": [ - { - "expr": "histogram_quantile(0.95, sum by (le) (rate(eventcollector_db_tx_duration_seconds_bucket[5m])))", - "legendFormat": "db tx p95", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s", - "decimals": 3 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "DB Batch Size p95", - "datasource": "$datasource", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 }, - "targets": [ - { - "expr": "histogram_quantile(0.95, sum by (le) (rate(eventcollector_db_batch_size_bucket[5m])))", - "legendFormat": "batch size p95", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - - { - "type": "table", - "title": "Agent Last Seen", - "datasource": "$datasource", - "gridPos": { "h": 10, "w": 12, "x": 0, "y": 44 }, - "targets": [ - { - "expr": "time() - eventcollector_agent_last_seen_unixtime{host=~\"$host\"}", - "legendFormat": "{{host}}", - "refId": "A", - "instant": true, - "format": "table" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "showHeader": true - } - }, - { - "type": "table", - "title": "Baseline Samples", - "datasource": "$datasource", - "gridPos": { "h": 10, "w": 12, "x": 12, "y": 44 }, - "targets": [ - { - "expr": "eventcollector_baseline_sample_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}", - "legendFormat": "{{host}} {{channel}} {{event_id}}", - "refId": "A", - "instant": true, - "format": "table" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "showHeader": true - } - } - ], - "refresh": "30s", - "schemaVersion": 39, - "style": "dark", - "tags": ["siem", "baseline", "ad"], - "templating": { - "list": [ - { - "name": "datasource", - "type": "datasource", - "query": "prometheus", - "current": {}, - "hide": 0, - "label": "Datasource" - }, - { - "name": "host", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_agent_last_seen_unixtime, host)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Host" - }, - { - "name": "channel", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_ingest_events_total, channel)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Channel" - }, - { - "name": "event_id", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_ingest_events_total, event_id)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Event ID" - }, - { - "name": "rule", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_detection_hits_total, rule)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Rule" - }, - { - "name": "severity", - "type": "custom", - "query": "low,medium,high", - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Severity" - } - ] - }, - "time": { - "from": "now-6h", - "to": "now" - }, - "timezone": "browser", - "title": "SIEM Overview Extended", - "uid": "siem-overview-extended", - "version": 1 -} \ No newline at end of file diff --git a/deploy/grafana/provisioning/dashboards/siem-soc-privileged.json b/deploy/grafana/provisioning/dashboards/siem-soc-privileged.json deleted file mode 100644 index 62878dd..0000000 --- a/deploy/grafana/provisioning/dashboards/siem-soc-privileged.json +++ /dev/null @@ -1,1006 +0,0 @@ -{ - "annotations": { - "list": [] - }, - "editable": true, - "fiscalYearStartMonth": 0, - "graphTooltip": 1, - "links": [], - "liveNow": false, - "panels": [ - { - "type": "row", - "title": "Privileged Account Overview", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "collapsed": false, - "panels": [] - }, - { - "type": "stat", - "title": "Privileged Logons 15m", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 0, - "y": 1 - }, - "targets": [ - { - "expr": "sum(increase(siem_privileged_logons_total{user=~\"$user\",host=~\"$host\"}[15m]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 10 - }, - { - "color": "red", - "value": 50 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "Failed Admin Logons 15m", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 4, - "y": 1 - }, - "targets": [ - { - "expr": "sum(increase(siem_privileged_logon_failures_total{user=~\"$user\",host=~\"$host\"}[15m]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 1 - }, - { - "color": "red", - "value": 5 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "Admin New Hosts 1h", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 8, - "y": 1 - }, - "targets": [ - { - "expr": "sum(increase(siem_privileged_new_host_total{user=~\"$user\",host=~\"$host\"}[1h]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 1 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "High/Critical Detections 1h", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 12, - "y": 1 - }, - "targets": [ - { - "expr": "sum(increase(eventcollector_detection_hits_total{severity=~\"high|critical\"}[1h]))", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 1 - }, - { - "color": "red", - "value": 5 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "Max Host Risk", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 16, - "y": 1 - }, - "targets": [ - { - "expr": "max(eventcollector_host_risk_score{host=~\"$host\"})", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 2, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 20 - }, - { - "color": "red", - "value": 60 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "stat", - "title": "Baseline Max Z-Score", - "datasource": "$datasource", - "gridPos": { - "h": 4, - "w": 4, - "x": 20, - "y": 1 - }, - "targets": [ - { - "expr": "max(eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 2, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 3 - }, - { - "color": "red", - "value": 5 - } - ] - } - }, - "overrides": [] - }, - "options": { - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "orientation": "auto", - "textMode": "auto", - "colorMode": "value", - "graphMode": "area", - "justifyMode": "auto" - } - }, - { - "type": "timeseries", - "title": "Privileged Logons by User", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 5 - }, - "targets": [ - { - "expr": "sum by (user) (rate(siem_privileged_logons_total{user=~\"$user\",host=~\"$host\"}[5m]))", - "legendFormat": "{{user}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "eps", - "decimals": 3 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "Failed Privileged Logons by User", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 5 - }, - "targets": [ - { - "expr": "sum by (user) (increase(siem_privileged_logon_failures_total{user=~\"$user\",host=~\"$host\"}[5m]))", - "legendFormat": "{{user}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "bargauge", - "title": "Top Admins by Logons 1h", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 13 - }, - "targets": [ - { - "expr": "topk(15, sum by (user) (increase(siem_privileged_logons_total{user=~\"$user\",host=~\"$host\"}[1h])))", - "legendFormat": "{{user}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top Hosts with Admin Activity 1h", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 13 - }, - "targets": [ - { - "expr": "topk(15, sum by (host) (increase(siem_privileged_logons_total{user=~\"$user\",host=~\"$host\"}[1h])))", - "legendFormat": "{{host}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top Admin New Hosts 24h", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 13 - }, - "targets": [ - { - "expr": "topk(15, sum by (user,host) (increase(siem_privileged_new_host_total{user=~\"$user\",host=~\"$host\"}[24h])))", - "legendFormat": "{{user}} → {{host}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "row", - "title": "SOC Risk & Detections", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 21 - }, - "collapsed": false, - "panels": [] - }, - { - "type": "bargauge", - "title": "Top Host Risk Scores", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 22 - }, - "targets": [ - { - "expr": "topk(20, eventcollector_host_risk_score{host=~\"$host\"})", - "legendFormat": "{{host}} / {{severity}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 1 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top Detection Rules 24h", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 22 - }, - "targets": [ - { - "expr": "topk(20, sum by (rule,severity) (increase(eventcollector_detection_hits_total{rule=~\"$rule\",severity=~\"$severity\"}[24h])))", - "legendFormat": "{{rule}} / {{severity}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "bargauge", - "title": "Top Baseline Z-Scores", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 22 - }, - "targets": [ - { - "expr": "topk(20, eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})", - "legendFormat": "{{host}}", - "refId": "A", - "instant": true - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ], - "fields": "", - "values": false - }, - "showUnfilled": true - } - }, - { - "type": "timeseries", - "title": "Detection Hits by Severity", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 30 - }, - "targets": [ - { - "expr": "sum by (severity) (increase(eventcollector_detection_hits_total{severity=~\"$severity\",rule=~\"$rule\"}[5m]))", - "legendFormat": "{{severity}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "Baseline Current vs Average", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 30 - }, - "targets": [ - { - "expr": "eventcollector_baseline_current_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}", - "legendFormat": "current {{host}} {{channel}} {{event_id}}", - "refId": "A" - }, - { - "expr": "eventcollector_baseline_avg_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}", - "legendFormat": "avg {{host}} {{channel}} {{event_id}}", - "refId": "B" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "list", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "row", - "title": "Operations", - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 38 - }, - "collapsed": false, - "panels": [] - }, - { - "type": "timeseries", - "title": "Ingested Events / Second by Channel", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 39 - }, - "targets": [ - { - "expr": "sum by (channel) (rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m]))", - "legendFormat": "{{channel}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "eps", - "decimals": 2 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "timeseries", - "title": "HTTP Latency p95", - "datasource": "$datasource", - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 39 - }, - "targets": [ - { - "expr": "histogram_quantile(0.95, sum by (le,path) (rate(eventcollector_http_request_duration_seconds_bucket[5m])))", - "legendFormat": "{{path}} p95", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s", - "decimals": 3 - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "mode": "multi", - "sort": "desc" - } - } - }, - { - "type": "table", - "title": "Privileged Users / Hosts - Current Activity", - "datasource": "$datasource", - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 47 - }, - "targets": [ - { - "expr": "sum by (user,host) (increase(siem_privileged_logons_total{user=~\"$user\",host=~\"$host\"}[1h]))", - "legendFormat": "{{user}} {{host}}", - "refId": "A", - "instant": true, - "format": "table" - } - ], - "fieldConfig": { - "defaults": { - "unit": "short", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "showHeader": true - } - }, - { - "type": "table", - "title": "Agent Last Seen Age", - "datasource": "$datasource", - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 47 - }, - "targets": [ - { - "expr": "time() - eventcollector_agent_last_seen_unixtime{host=~\"$host\"}", - "legendFormat": "{{host}}", - "refId": "A", - "instant": true, - "format": "table" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s", - "decimals": 0 - }, - "overrides": [] - }, - "options": { - "showHeader": true - } - } - ], - "refresh": "30s", - "schemaVersion": 39, - "style": "dark", - "tags": [ - "siem", - "soc", - "ueba", - "privileged-accounts" - ], - "templating": { - "list": [ - { - "name": "datasource", - "type": "datasource", - "query": "prometheus", - "current": {}, - "hide": 0, - "label": "Datasource" - }, - { - "name": "user", - "type": "query", - "datasource": "$datasource", - "query": "label_values(siem_privileged_logons_total, user)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Privileged User" - }, - { - "name": "host", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_agent_last_seen_unixtime, host)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Host" - }, - { - "name": "channel", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_ingest_events_total, channel)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Channel" - }, - { - "name": "event_id", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_ingest_events_total, event_id)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Event ID" - }, - { - "name": "rule", - "type": "query", - "datasource": "$datasource", - "query": "label_values(eventcollector_detection_hits_total, rule)", - "refresh": 1, - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Rule" - }, - { - "name": "severity", - "type": "custom", - "query": "info,low,medium,high,critical", - "includeAll": true, - "multi": true, - "allValue": ".*", - "current": { - "selected": true, - "text": "All", - "value": "$__all" - }, - "label": "Severity" - } - ] - }, - "time": { - "from": "now-6h", - "to": "now" - }, - "timezone": "browser", - "title": "SIEM SOC - Privileged Accounts & UEBA", - "uid": "siem-soc-privileged-ueba", - "version": 1 -} \ No newline at end of file diff --git a/deploy/grafana/provisioning/datasources/datasource.yml b/deploy/grafana/provisioning/datasources/datasource.yml deleted file mode 100644 index fe7fa20..0000000 --- a/deploy/grafana/provisioning/datasources/datasource.yml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: 1 - -datasources: - - name: Prometheus - uid: prometheus - type: prometheus - access: proxy - url: http://prometheus:9090 - isDefault: true - editable: true \ No newline at end of file diff --git a/deploy/mariadb/cardinality-diagnostics.sql b/deploy/mariadb/cardinality-diagnostics.sql deleted file mode 100644 index 29a7adc..0000000 --- a/deploy/mariadb/cardinality-diagnostics.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Shows where rows are growing and how effective aggregation currently is. -SET time_zone = '+00:00'; - -SELECT table_name, - table_rows, - ROUND(data_length/1024/1024,1) AS data_mb, - ROUND(index_length/1024/1024,1) AS index_mb, - ROUND((data_length+index_length)/1024/1024,1) AS total_mb -FROM information_schema.tables -WHERE table_schema = DATABASE() -ORDER BY data_length + index_length DESC; - -SELECT 'event_count_buckets' AS source, - COUNT(*) AS rows_24h, - COALESCE(SUM(cnt),0) AS represented_events_24h, - ROUND(COALESCE(SUM(cnt),0) / NULLIF(COUNT(*),0), 1) AS events_per_row -FROM event_count_buckets -WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR -UNION ALL -SELECT 'event_occurrences', - COUNT(*), - COALESCE(SUM(cnt),0), - ROUND(COALESCE(SUM(cnt),0) / NULLIF(COUNT(*),0), 1) -FROM event_occurrences -WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR; - -SELECT channel_name, event_id, - COUNT(*) AS context_rows_24h, - SUM(cnt) AS represented_events_24h, - ROUND(SUM(cnt)/NULLIF(COUNT(*),0),1) AS events_per_row -FROM event_occurrences -WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR -GROUP BY channel_name, event_id -ORDER BY context_rows_24h DESC -LIMIT 50; diff --git a/deploy/mariadb/diagnostics.sql b/deploy/mariadb/diagnostics.sql deleted file mode 100644 index 6bed80e..0000000 --- a/deploy/mariadb/diagnostics.sql +++ /dev/null @@ -1,52 +0,0 @@ --- Read-only diagnostics for sizing and partition verification. -SET time_zone = '+00:00'; - -SELECT - TABLE_NAME, - ENGINE, - TABLE_ROWS, - ROUND(DATA_LENGTH / 1024 / 1024, 1) AS data_mb, - ROUND(INDEX_LENGTH / 1024 / 1024, 1) AS index_mb, - ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024, 1) AS total_mb -FROM information_schema.TABLES -WHERE TABLE_SCHEMA = DATABASE() -ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC; - -SELECT - TABLE_NAME, - COUNT(*) AS partitions, - MIN(PARTITION_NAME) AS oldest_partition, - MAX(PARTITION_NAME) AS newest_partition, - ROUND(SUM(DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024, 1) AS total_mb -FROM information_schema.PARTITIONS -WHERE TABLE_SCHEMA = DATABASE() - AND PARTITION_NAME IS NOT NULL -GROUP BY TABLE_NAME -ORDER BY total_mb DESC; - -SELECT - channel_name, - event_id, - SUM(cnt) AS events_24h, - COUNT(*) AS metadata_rows_24h, - ROUND(SUM(cnt) / NULLIF(COUNT(*), 0), 2) AS compression_factor -FROM event_occurrences -WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR -GROUP BY channel_name, event_id -ORDER BY events_24h DESC -LIMIT 50; - -SELECT - first_event_ts, - last_event_ts, - cnt, - hostname, - target_user, - workstation, - src_ip -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4740 - AND bucket_start >= UTC_TIMESTAMP() - INTERVAL 7 DAY -ORDER BY bucket_start DESC -LIMIT 200; diff --git a/deploy/mariadb/emergency-compact-cleanup.sql b/deploy/mariadb/emergency-compact-cleanup.sql deleted file mode 100644 index 7d1fbe4..0000000 --- a/deploy/mariadb/emergency-compact-cleanup.sql +++ /dev/null @@ -1,43 +0,0 @@ --- EMERGENCY CLEANUP FOR THE PREVIOUS METADATA-FIRST BUILD --- --- Run ONLY AFTER the new backend is deployed with: --- STORE_EVENT_ROWS=false --- STORE_RAW_XML=false --- --- This intentionally removes redundant per-event/full-context history generated --- by older builds. Aggregate event history remains in event_count_buckets and --- event_catalog. Existing detections and UEBA baselines are not deleted. --- --- Take a database backup/snapshot first if old raw/event detail may still matter. - -SET NAMES utf8mb4; -SET time_zone = '+00:00'; - -SELECT 'before' AS phase, - table_name, - table_rows, - ROUND((data_length + index_length) / 1024 / 1024, 1) AS size_mb -FROM information_schema.tables -WHERE table_schema = DATABASE() - AND table_name IN ('event_logs','event_log_raw','event_occurrences','event_count_buckets','event_catalog','detections') -ORDER BY size_mb DESC; - --- Raw XML must be cleared before its logical parent event rows. -TRUNCATE TABLE event_log_raw; -TRUNCATE TABLE event_logs; - --- Old versions used too many dimensions and could make this table nearly as --- large as event_logs. The new backend will repopulate only selected Security --- contexts in 5-minute buckets. -TRUNCATE TABLE event_occurrences; - -ANALYZE TABLE event_count_buckets, event_catalog, detections; - -SELECT 'after' AS phase, - table_name, - table_rows, - ROUND((data_length + index_length) / 1024 / 1024, 1) AS size_mb -FROM information_schema.tables -WHERE table_schema = DATABASE() - AND table_name IN ('event_logs','event_log_raw','event_occurrences','event_count_buckets','event_catalog','detections') -ORDER BY size_mb DESC; diff --git a/deploy/mariadb/init/001-schema.sql b/deploy/mariadb/init/001-schema.sql deleted file mode 100644 index febe2ab..0000000 --- a/deploy/mariadb/init/001-schema.sql +++ /dev/null @@ -1,2239 +0,0 @@ --- SIEM-lite vollständiges MariaDB-kompatibles Datenbankschema --- Stand: Partitionierung event_logs/event_log_raw, 3h-Partitionen, Raw-XML-Auslagerung, --- Baseline-Buckets, UEBA, SOC/Risk, UI-Bewertungen. --- --- Getestet/ausgelegt für MariaDB/MySQL InnoDB. --- --- WICHTIG: --- 1. Dieses Script löscht bestehende Tabellen. --- 2. Partitionierung erfolgt nach DATETIME(6)-Spalte ts. --- 3. MariaDB erlaubt ON UPDATE UTC_TIMESTAMP(6) nicht zuverlässig. --- Deshalb nutzt dieses Schema DEFAULT CURRENT_TIMESTAMP(6) / ON UPDATE CURRENT_TIMESTAMP(6). --- 4. Stelle für echte UTC-Speicherung im Go-DSN zusätzlich die DB-Session auf UTC: --- ?parseTime=true&loc=UTC&time_zone=%27%2B00%3A00%27 --- oder setze beim Verbindungsaufbau SET time_zone = '+00:00'. --- 5. Spalten in zusammengesetzten Keys wurden auf VARCHAR(191) begrenzt, --- damit utf8mb4 und 3072-Byte-Key-Limit keine Fehler erzeugen. - -SET NAMES utf8mb4; -SET time_zone = '+00:00'; - -SET FOREIGN_KEY_CHECKS = 0; - -DROP PROCEDURE IF EXISTS ensure_siem_partitions; - -DROP TABLE IF EXISTS event_catalog; -DROP TABLE IF EXISTS event_occurrences; -DROP TABLE IF EXISTS event_count_buckets; -DROP TABLE IF EXISTS ueba_context_buckets; -DROP TABLE IF EXISTS host_risk_scores; -DROP TABLE IF EXISTS baseline_exclusions; -DROP TABLE IF EXISTS baseline_event_stats; -DROP TABLE IF EXISTS detection_suppressions; -DROP TABLE IF EXISTS detections; -DROP TABLE IF EXISTS detection_rules; -DROP TABLE IF EXISTS user_privilege_baseline; -DROP TABLE IF EXISTS user_source_ip_seen; -DROP TABLE IF EXISTS ueba_user_baseline; -DROP TABLE IF EXISTS privileged_users; -DROP TABLE IF EXISTS event_log_raw; -DROP TABLE IF EXISTS event_logs; -DROP TABLE IF EXISTS agents; - -SET FOREIGN_KEY_CHECKS = 1; - --- --------------------------------------------------------------------- --- Agents --- --------------------------------------------------------------------- - -CREATE TABLE agents ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - hostname VARCHAR(191) NOT NULL, - api_key_hash CHAR(64) NOT NULL, - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_ip VARCHAR(64) NOT NULL DEFAULT '', - is_enabled TINYINT(1) NOT NULL DEFAULT 1, - - PRIMARY KEY (id), - UNIQUE KEY uq_agents_hostname (hostname), - KEY idx_agents_enabled_last_seen (is_enabled, last_seen), - KEY idx_agents_last_seen (last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Event Logs, optionaler kurzlebiger Full-Event-Hotstore (STORE_EVENT_ROWS=true) --- --------------------------------------------------------------------- - -CREATE TABLE event_logs ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - ts DATETIME(6) NOT NULL, - agent_id BIGINT UNSIGNED NOT NULL, - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - source VARCHAR(191) NOT NULL DEFAULT '', - - computer VARCHAR(191) NOT NULL DEFAULT '', - provider_name VARCHAR(191) NOT NULL DEFAULT '', - - level_value INT UNSIGNED NOT NULL DEFAULT 0, - task_value INT UNSIGNED NOT NULL DEFAULT 0, - opcode_value INT UNSIGNED NOT NULL DEFAULT 0, - keywords VARCHAR(191) NOT NULL DEFAULT '', - - target_user VARCHAR(191) NOT NULL DEFAULT '', - target_user_norm VARCHAR(191) NOT NULL DEFAULT '', - target_domain VARCHAR(191) NOT NULL DEFAULT '', - - subject_user VARCHAR(191) NOT NULL DEFAULT '', - subject_user_norm VARCHAR(191) NOT NULL DEFAULT '', - subject_domain VARCHAR(191) NOT NULL DEFAULT '', - - workstation VARCHAR(191) NOT NULL DEFAULT '', - src_ip VARCHAR(64) NOT NULL DEFAULT '', - src_port VARCHAR(32) NOT NULL DEFAULT '', - logon_type VARCHAR(32) NOT NULL DEFAULT '', - process_name VARCHAR(768) NOT NULL DEFAULT '', - authentication_package VARCHAR(128) NOT NULL DEFAULT '', - logon_process VARCHAR(128) NOT NULL DEFAULT '', - status_text VARCHAR(128) NOT NULL DEFAULT '', - sub_status_text VARCHAR(128) NOT NULL DEFAULT '', - failure_reason VARCHAR(768) NOT NULL DEFAULT '', - - received_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - msg_sha256 CHAR(64) NOT NULL, - - -- Optionaler Kompatibilitätsrest. Raw XML gehört in event_log_raw. - msg MEDIUMTEXT NULL, - - PRIMARY KEY (id, ts), - - KEY idx_event_logs_id (id), - KEY idx_event_logs_agent_ts (agent_id, ts), - KEY idx_event_logs_ts_host_channel_event (ts, hostname, channel_name, event_id), - KEY idx_event_logs_hostname_ts (hostname, ts), - KEY idx_event_logs_channel_event_ts (channel_name, event_id, ts), - KEY idx_event_logs_target_user_norm_ts (target_user_norm, ts), - KEY idx_event_logs_subject_user_norm_ts (subject_user_norm, ts), - KEY idx_event_logs_src_ip_ts (src_ip, ts), - - KEY idx_event_logs_security_logon_user ( - channel_name, - event_id, - ts, - target_user_norm, - hostname, - src_ip - ), - - KEY idx_event_logs_ueba_context ( - channel_name, - event_id, - ts, - hostname, - target_user_norm, - src_ip, - workstation - ), - - KEY idx_event_logs_password_spray ( - channel_name, - event_id, - ts, - src_ip, - target_user_norm, - hostname - ), - - KEY idx_event_logs_dedupe_lookup ( - hostname, - channel_name, - event_id, - ts, - msg_sha256 - ) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(ts) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- --------------------------------------------------------------------- --- Raw XML Tabelle --- --------------------------------------------------------------------- - -CREATE TABLE event_log_raw ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - event_log_id BIGINT UNSIGNED NOT NULL, - ts DATETIME(6) NOT NULL, - msg_sha256 CHAR(64) NOT NULL, - msg LONGBLOB NOT NULL, - is_gzip TINYINT(1) NOT NULL DEFAULT 0, - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - - PRIMARY KEY (id, ts), - - KEY idx_event_log_raw_id (id), - KEY idx_event_log_raw_event_log_id (event_log_id), - KEY idx_event_log_raw_ts (ts), - KEY idx_event_log_raw_sha (msg_sha256), - UNIQUE KEY uq_event_log_raw_event_ts (event_log_id, ts) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(ts) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- --------------------------------------------------------------------- --- Kleine Event-Landkarte: erstes/letztes Auftreten pro Host/Channel/EventID. --- Verhindert teure NOT EXISTS-Scans über die Hot-Event-Tabelle. --- --------------------------------------------------------------------- - -CREATE TABLE event_catalog ( - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - first_seen DATETIME(6) NOT NULL, - last_seen DATETIME(6) NOT NULL, - total_count BIGINT UNSIGNED NOT NULL DEFAULT 0, - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - PRIMARY KEY (hostname, channel_name, event_id), - KEY idx_event_catalog_first_seen (first_seen), - KEY idx_event_catalog_last_seen (last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Kompakte Security-Kontext-Metadaten für eine kleine Event-ID-Allowlist. --- Alle Events werden universell in event_count_buckets gezählt; diese Tabelle --- speichert nur Benutzer/IP/Workstation-Kontext für ausgewählte Security-Events. --- --------------------------------------------------------------------- - -CREATE TABLE event_occurrences ( - bucket_start DATETIME(6) NOT NULL, - bucket_end DATETIME(6) NOT NULL, - dimension_key BINARY(16) NOT NULL, - - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - provider_name VARCHAR(191) NOT NULL DEFAULT '', - - target_user VARCHAR(191) NOT NULL DEFAULT '', - subject_user VARCHAR(191) NOT NULL DEFAULT '', - src_ip VARCHAR(64) NOT NULL DEFAULT '', - workstation VARCHAR(191) NOT NULL DEFAULT '', - logon_type VARCHAR(32) NOT NULL DEFAULT '', - status_text VARCHAR(128) NOT NULL DEFAULT '', - failure_reason VARCHAR(255) NOT NULL DEFAULT '', - - cnt BIGINT UNSIGNED NOT NULL DEFAULT 0, - first_event_ts DATETIME(6) NOT NULL, - last_event_ts DATETIME(6) NOT NULL, - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY (bucket_start, dimension_key), - KEY idx_occurrences_time_host_event (bucket_start, hostname, channel_name, event_id), - KEY idx_occurrences_host_event_time (hostname, channel_name, event_id, bucket_start), - KEY idx_occurrences_target_user_time (target_user, bucket_start), - KEY idx_occurrences_subject_user_time (subject_user, bucket_start), - KEY idx_occurrences_src_ip_time (src_ip, bucket_start), - KEY idx_occurrences_security_event_time ( - channel_name, - event_id, - bucket_start, - hostname, - target_user, - src_ip - ) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(bucket_start) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- --------------------------------------------------------------------- --- Detection-Regeln --- --------------------------------------------------------------------- - -CREATE TABLE detection_rules ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - name VARCHAR(191) NOT NULL, - description TEXT NULL, - severity ENUM('info','low','medium','high','critical') NOT NULL DEFAULT 'medium', - channel VARCHAR(512) NOT NULL DEFAULT 'Security', - event_ids VARCHAR(1024) NOT NULL, - - match_field VARCHAR(128) NOT NULL DEFAULT '', - match_operator ENUM('', 'eq', 'contains', 'in') NOT NULL DEFAULT '', - match_value TEXT NULL, - - threshold_count INT NOT NULL DEFAULT 1, - threshold_window_seconds INT NOT NULL DEFAULT 0, - suppress_for_seconds INT NOT NULL DEFAULT 3600, - - enabled TINYINT(1) NOT NULL DEFAULT 1, - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY (id), - UNIQUE KEY uq_detection_rules_name (name), - KEY idx_detection_rules_enabled (enabled) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Detections --- --------------------------------------------------------------------- - -CREATE TABLE detections ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - rule_name VARCHAR(191) NOT NULL, - severity ENUM('info','low','medium','high','critical') NOT NULL DEFAULT 'medium', - hostname VARCHAR(191) NOT NULL DEFAULT '', - channel_name VARCHAR(128) NOT NULL DEFAULT '', - event_id INT UNSIGNED NOT NULL DEFAULT 0, - score DOUBLE NOT NULL DEFAULT 0, - - window_start DATETIME(6) NOT NULL, - window_end DATETIME(6) NOT NULL, - - summary TEXT NOT NULL, - details_json LONGTEXT NULL CHECK (JSON_VALID(details_json)), - - status ENUM( - 'open', - 'acknowledged', - 'investigating', - 'plausible', - 'legitimate', - 'false_positive', - 'resolved', - 'suppressed', - 'confirmed_incident' - ) NOT NULL DEFAULT 'open', - - analyst_note TEXT NULL, - reviewed_by VARCHAR(191) NOT NULL DEFAULT '', - reviewed_at DATETIME(6) NULL, - is_false_positive TINYINT(1) NOT NULL DEFAULT 0, - is_legitimate TINYINT(1) NOT NULL DEFAULT 0, - - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - - PRIMARY KEY (id), - - UNIQUE KEY uq_detection_dedupe ( - rule_name, - hostname, - channel_name, - event_id, - window_start, - window_end - ), - - KEY idx_detections_created (created_at), - KEY idx_detections_status_created (status, created_at), - KEY idx_detections_severity_created (severity, created_at), - KEY idx_detections_host_created (hostname, created_at), - KEY idx_detections_rule_host_created (rule_name, hostname, created_at), - KEY idx_detections_created_status_severity_host ( - created_at, - status, - severity, - hostname - ), - KEY idx_detections_window ( - hostname, - window_start, - window_end - ), - KEY idx_detections_soc ( - status, - severity, - hostname, - created_at - ) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Detection Suppressions --- --------------------------------------------------------------------- - -CREATE TABLE detection_suppressions ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - rule_name VARCHAR(191) NOT NULL, - hostname VARCHAR(191) NOT NULL DEFAULT '', - channel_name VARCHAR(128) NOT NULL DEFAULT '', - event_id INT UNSIGNED NOT NULL DEFAULT 0, - - reason TEXT NULL, - created_by VARCHAR(191) NOT NULL DEFAULT '', - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - expires_at DATETIME(6) NULL, - enabled TINYINT(1) NOT NULL DEFAULT 1, - - PRIMARY KEY (id), - KEY idx_detection_suppressions_lookup ( - enabled, - rule_name, - hostname, - channel_name, - event_id, - expires_at - ), - KEY idx_detection_suppressions_expires (expires_at) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Event Count Buckets für Baseline --- --------------------------------------------------------------------- - -CREATE TABLE event_count_buckets ( - bucket_start DATETIME(6) NOT NULL, - bucket_end DATETIME(6) NOT NULL, - - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - - cnt BIGINT UNSIGNED NOT NULL DEFAULT 0, - - first_event_ts DATETIME(6) NULL, - last_event_ts DATETIME(6) NULL, - - finalized TINYINT(1) NOT NULL DEFAULT 0, - anomaly_checked_at DATETIME(6) NULL, - learned_at DATETIME(6) NULL, - - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY (bucket_start, hostname, channel_name, event_id), - - KEY idx_event_count_buckets_lookup ( - hostname, - channel_name, - event_id, - bucket_start - ), - - KEY idx_event_count_buckets_anomaly ( - finalized, - anomaly_checked_at, - bucket_start - ), - - KEY idx_event_count_buckets_learn ( - finalized, - learned_at, - bucket_start - ), - - KEY idx_event_count_buckets_time ( - bucket_start, - bucket_end - ), - - KEY idx_event_count_buckets_event_time ( - channel_name, - event_id, - bucket_start, - hostname - ) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(bucket_start) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- --------------------------------------------------------------------- --- Baseline Stats --- --------------------------------------------------------------------- - -CREATE TABLE baseline_event_stats ( - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - hour_of_day TINYINT UNSIGNED NOT NULL, - day_of_week TINYINT UNSIGNED NOT NULL, - - avg_count DOUBLE NOT NULL DEFAULT 0, - m2_count DOUBLE NOT NULL DEFAULT 0, - stddev_count DOUBLE NOT NULL DEFAULT 0, - sample_count INT NOT NULL DEFAULT 0, - - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_updated DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY ( - hostname, - channel_name, - event_id, - hour_of_day, - day_of_week - ), - - KEY idx_baseline_event_stats_event ( - channel_name, - event_id, - hour_of_day, - day_of_week - ), - - KEY idx_baseline_event_stats_updated (last_updated) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE baseline_exclusions ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - hostname VARCHAR(191) NOT NULL DEFAULT '', - channel_name VARCHAR(128) NOT NULL DEFAULT '', - event_id INT UNSIGNED NOT NULL DEFAULT 0, - - reason TEXT NULL, - created_by VARCHAR(191) NOT NULL DEFAULT '', - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - expires_at DATETIME(6) NULL, - enabled TINYINT(1) NOT NULL DEFAULT 1, - - PRIMARY KEY (id), - - KEY idx_baseline_exclusions_lookup ( - enabled, - hostname, - channel_name, - event_id, - expires_at - ), - KEY idx_baseline_exclusions_expires (expires_at) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- UEBA Tabellen --- --------------------------------------------------------------------- - -CREATE TABLE ueba_user_baseline ( - username VARCHAR(191) NOT NULL, - hostname VARCHAR(191) NOT NULL, - src_ip VARCHAR(64) NOT NULL, - workstation VARCHAR(191) NOT NULL, - - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - seen_count BIGINT UNSIGNED NOT NULL DEFAULT 0, - - PRIMARY KEY (username, hostname, src_ip, workstation), - - KEY idx_ueba_user_baseline_last_seen (last_seen), - KEY idx_ueba_user_baseline_user_last_seen (username, last_seen), - KEY idx_ueba_user_baseline_host_last_seen (hostname, last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE user_source_ip_seen ( - username VARCHAR(191) NOT NULL, - src_ip VARCHAR(64) NOT NULL, - hostname VARCHAR(191) NOT NULL, - - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - seen_count BIGINT UNSIGNED NOT NULL DEFAULT 0, - - PRIMARY KEY (username, src_ip, hostname), - - KEY idx_user_source_ip_seen_last_seen (last_seen), - KEY idx_user_source_ip_seen_user_last_seen (username, last_seen), - KEY idx_user_source_ip_seen_src_ip_last_seen (src_ip, last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE user_privilege_baseline ( - username VARCHAR(191) NOT NULL, - - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - seen_count BIGINT UNSIGNED NOT NULL DEFAULT 0, - - PRIMARY KEY (username), - KEY idx_user_privilege_baseline_last_seen (last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE ueba_context_buckets ( - bucket_start DATETIME(6) NOT NULL, - bucket_end DATETIME(6) NOT NULL, - - username VARCHAR(191) NOT NULL, - hostname VARCHAR(191) NOT NULL, - src_ip VARCHAR(64) NOT NULL, - workstation VARCHAR(191) NOT NULL, - - cnt BIGINT UNSIGNED NOT NULL DEFAULT 0, - first_event_ts DATETIME(6) NULL, - last_event_ts DATETIME(6) NULL, - - finalized TINYINT(1) NOT NULL DEFAULT 0, - checked_at DATETIME(6) NULL, - learned_at DATETIME(6) NULL, - - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY ( - bucket_start, - username, - hostname, - src_ip, - workstation - ), - - KEY idx_ueba_context_check ( - finalized, - checked_at, - bucket_start - ), - - KEY idx_ueba_context_learn ( - finalized, - learned_at, - bucket_start - ), - - KEY idx_ueba_context_lookup ( - username, - hostname, - src_ip, - workstation, - bucket_start - ) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(bucket_start) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- --------------------------------------------------------------------- --- Privileged Users --- --------------------------------------------------------------------- - -CREATE TABLE privileged_users ( - username VARCHAR(191) NOT NULL, - reason TEXT NULL, - enabled TINYINT(1) NOT NULL DEFAULT 1, - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY (username), - KEY idx_privileged_users_enabled (enabled) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Host Risk Scores / SOC --- --------------------------------------------------------------------- - -CREATE TABLE host_risk_scores ( - hostname VARCHAR(191) NOT NULL, - risk_score DOUBLE NOT NULL DEFAULT 0, - severity ENUM('info','low','medium','high','critical') NOT NULL DEFAULT 'info', - - open_detections INT NOT NULL DEFAULT 0, - high_detections INT NOT NULL DEFAULT 0, - critical_detections INT NOT NULL DEFAULT 0, - confirmed_incidents INT NOT NULL DEFAULT 0, - - last_detection_at DATETIME(6) NULL, - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - - PRIMARY KEY (hostname), - KEY idx_host_risk_scores_risk (risk_score), - KEY idx_host_risk_scores_severity_risk (severity, risk_score), - KEY idx_host_risk_scores_updated (updated_at) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- --------------------------------------------------------------------- --- Partition Management Stored Procedure --- Erzeugt 3h-Partitionen von UTC now - 6h bis UTC now + 24h. --- --------------------------------------------------------------------- - -DELIMITER $$ - -CREATE PROCEDURE ensure_siem_partitions() -BEGIN - DECLARE v_start DATETIME; - DECLARE v_end DATETIME; - DECLARE v_current DATETIME; - DECLARE v_part_end DATETIME; - DECLARE v_part_name VARCHAR(32); - DECLARE v_exists INT DEFAULT 0; - - SET v_start = FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(UTC_TIMESTAMP()) / 10800) * 10800); - SET v_start = DATE_SUB(v_start, INTERVAL 6 HOUR); - SET v_end = DATE_ADD(FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(UTC_TIMESTAMP()) / 10800) * 10800), INTERVAL 27 HOUR); - - SET v_current = v_start; - - WHILE v_current < v_end DO - SET v_part_end = DATE_ADD(v_current, INTERVAL 3 HOUR); - SET v_part_name = CONCAT('p', DATE_FORMAT(v_current, '%Y%m%d%H')); - - SELECT COUNT(*) - INTO v_exists - FROM information_schema.PARTITIONS - WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = 'event_logs' - AND PARTITION_NAME = v_part_name; - - IF v_exists = 0 THEN - SET @sql_event_logs = CONCAT( - 'ALTER TABLE event_logs REORGANIZE PARTITION pmax INTO (', - 'PARTITION ', v_part_name, ' VALUES LESS THAN (''', DATE_FORMAT(v_part_end, '%Y-%m-%d %H:%i:%s'), '''),', - 'PARTITION pmax VALUES LESS THAN (MAXVALUE))' - ); - - PREPARE stmt_event_logs FROM @sql_event_logs; - EXECUTE stmt_event_logs; - DEALLOCATE PREPARE stmt_event_logs; - END IF; - - SELECT COUNT(*) - INTO v_exists - FROM information_schema.PARTITIONS - WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = 'event_log_raw' - AND PARTITION_NAME = v_part_name; - - IF v_exists = 0 THEN - SET @sql_event_log_raw = CONCAT( - 'ALTER TABLE event_log_raw REORGANIZE PARTITION pmax INTO (', - 'PARTITION ', v_part_name, ' VALUES LESS THAN (''', DATE_FORMAT(v_part_end, '%Y-%m-%d %H:%i:%s'), '''),', - 'PARTITION pmax VALUES LESS THAN (MAXVALUE))' - ); - - PREPARE stmt_event_log_raw FROM @sql_event_log_raw; - EXECUTE stmt_event_log_raw; - DEALLOCATE PREPARE stmt_event_log_raw; - END IF; - - SET v_current = v_part_end; - END WHILE; -END$$ - -DELIMITER ; - -CALL ensure_siem_partitions(); --- --------------------------------------------------------------------- --- Initiale Beispielregeln --- Optional. Kann gelöscht werden, wenn Regeln nur über UI gepflegt werden sollen. --- --------------------------------------------------------------------- - -INSERT INTO detection_rules -( - name, - description, - severity, - channel, - event_ids, - match_field, - match_operator, - match_value, - threshold_count, - threshold_window_seconds, - suppress_for_seconds, - enabled -) -VALUES - --- ============================================================ --- Kritische Manipulationen / Spurenverwischung --- ============================================================ - -( - 'v1_security_log_cleared', - 'Security Log wurde gelöscht. Das ist fast immer sicherheitsrelevant.', - 'high', - 'Security', - '1102', - '', - '', - '', - 1, - 0, - 86400, - 1 -), - -( - 'v1_audit_policy_changed', - 'Audit Policy wurde geändert.', - 'high', - 'Security', - '4719', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_system_audit_policy_changed', - 'System Audit Policy wurde geändert.', - 'high', - 'Security', - '4902,4904,4905,4906,4907,4908,4912', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_special_logon', - 'Anmeldung mit speziellen Rechten erkannt.', - 'medium', - 'Security', - '4672', - '', - '', - '', - 20, - 300, - 1800, - 1 -), - -( - 'v1_privileged_service_called', - 'Privilegierter Dienst wurde aufgerufen.', - 'medium', - 'Security', - '4673,4674', - '', - '', - '', - 10, - 300, - 1800, - 1 -), - --- ============================================================ --- Benutzerverwaltung --- ============================================================ - -( - 'v1_user_created', - 'Neuer AD-Benutzer wurde erstellt.', - 'medium', - 'Security', - '4720', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_user_enabled', - 'AD-Benutzer wurde aktiviert.', - 'medium', - 'Security', - '4722', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_user_disabled', - 'AD-Benutzer wurde deaktiviert.', - 'low', - 'Security', - '4725', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_user_deleted', - 'AD-Benutzer wurde gelöscht.', - 'medium', - 'Security', - '4726', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_user_changed', - 'AD-Benutzerobjekt wurde geändert.', - 'low', - 'Security', - '4738', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_password_changed_by_user', - 'Benutzer hat eigenes Passwort geändert.', - 'low', - 'Security', - '4723', - '', - '', - '', - 1, - 0, - 900, - 1 -), - -( - 'v1_password_reset', - 'Passwort wurde durch Admin oder berechtigten Benutzer zurückgesetzt.', - 'medium', - 'Security', - '4724', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_account_lockout', - 'Benutzerkonto wurde gesperrt.', - 'low', - 'Security', - '4740', - '', - '', - '', - 1, - 0, - 900, - 1 -), - -( - 'v1_account_lockout_spike', - 'Viele Account Lockouts innerhalb kurzer Zeit.', - 'medium', - 'Security', - '4740', - '', - '', - '', - 5, - 300, - 1800, - 1 -), - --- ============================================================ --- Gruppenverwaltung allgemein --- ============================================================ - -( - 'v1_security_group_created', - 'Security-Gruppe wurde erstellt.', - 'medium', - 'Security', - '4727,4731,4754', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_security_group_deleted', - 'Security-Gruppe wurde gelöscht.', - 'medium', - 'Security', - '4730,4734,4758', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_security_group_changed', - 'Security-Gruppe wurde geändert.', - 'medium', - 'Security', - '4735,4737,4755', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_group_member_added', - 'Mitglied wurde zu einer Security-Gruppe hinzugefügt.', - 'medium', - 'Security', - '4728,4732,4756', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_group_member_removed', - 'Mitglied wurde aus einer Security-Gruppe entfernt.', - 'medium', - 'Security', - '4729,4733,4757', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - --- ============================================================ --- Privilegierte Gruppen --- Achtung: target_user ist hier normalerweise der Gruppenname. --- ============================================================ - -( - 'v1_privileged_group_member_added', - 'Mitglied wurde zu einer privilegierten AD-Gruppe hinzugefügt.', - 'high', - 'Security', - '4728,4732,4756', - 'target_user', - 'in', - 'Domain Admins,Enterprise Admins,Schema Admins,Administrators,Account Operators,Server Operators,Backup Operators,Print Operators,DNSAdmins,Group Policy Creator Owners,Cert Publishers,Key Admins,Enterprise Key Admins', - 1, - 0, - 3600, - 1 -), - -( - 'v1_privileged_group_member_removed', - 'Mitglied wurde aus einer privilegierten AD-Gruppe entfernt.', - 'medium', - 'Security', - '4729,4733,4757', - 'target_user', - 'in', - 'Domain Admins,Enterprise Admins,Schema Admins,Administrators,Account Operators,Server Operators,Backup Operators,Print Operators,DNSAdmins,Group Policy Creator Owners,Cert Publishers,Key Admins,Enterprise Key Admins', - 1, - 0, - 3600, - 1 -), - -( - 'v1_domain_admins_changed', - 'Änderung an Domain Admins erkannt.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Domain Admins', - 1, - 0, - 3600, - 1 -), - -( - 'v1_enterprise_admins_changed', - 'Änderung an Enterprise Admins erkannt.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Enterprise Admins', - 1, - 0, - 3600, - 1 -), - -( - 'v1_schema_admins_changed', - 'Änderung an Schema Admins erkannt.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Schema Admins', - 1, - 0, - 3600, - 1 -), - -( - 'v1_dnsadmins_changed', - 'Änderung an DNSAdmins erkannt. DNSAdmins können in vielen Umgebungen kritisch sein.', - 'high', - 'Security', - '4728,4729,4732,4733,4756,4757', - 'target_user', - 'eq', - 'DNSAdmins', - 1, - 0, - 3600, - 1 -), - --- ============================================================ --- Computer-Konten --- ============================================================ - -( - 'v1_computer_account_created', - 'Computer-Konto wurde erstellt.', - 'low', - 'Security', - '4741', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_computer_account_changed', - 'Computer-Konto wurde geändert.', - 'low', - 'Security', - '4742', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_computer_account_deleted', - 'Computer-Konto wurde gelöscht.', - 'medium', - 'Security', - '4743', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - --- ============================================================ --- Kerberos / Authentifizierung --- ============================================================ - -( - 'v1_kerberos_preauth_failure', - 'Kerberos PreAuth Fehler erkannt.', - 'low', - 'Security', - '4771', - '', - '', - '', - 1, - 0, - 300, - 1 -), - -( - 'v1_kerberos_preauth_spike', - 'Viele Kerberos PreAuth Fehler. Möglicher Password-Spray oder Bruteforce.', - 'high', - 'Security', - '4771', - '', - '', - '', - 20, - 300, - 1800, - 1 -), - -( - 'v1_kerberos_tgt_failed_spike', - 'Viele fehlgeschlagene Kerberos-TGT-Anfragen.', - 'medium', - 'Security', - '4768', - '', - '', - '', - 30, - 300, - 1800, - 1 -), - -( - 'v1_kerberos_service_ticket_spike', - 'Viele Kerberos-Service-Ticket-Anfragen. Kann auf Kerberoasting oder ungewöhnliche Service-Nutzung hindeuten.', - 'medium', - 'Security', - '4769', - '', - '', - '', - 100, - 300, - 1800, - 1 -), - -( - 'v1_ntlm_authentication_spike', - 'Viele NTLM-Authentifizierungen erkannt.', - 'medium', - 'Security', - '4776', - '', - '', - '', - 50, - 300, - 1800, - 1 -), - -( - 'v1_logon_failed_spike', - 'Viele fehlgeschlagene Logons.', - 'medium', - 'Security', - '4625', - '', - '', - '', - 25, - 300, - 1800, - 1 -), - -( - 'v1_explicit_credentials_used', - 'Anmeldung mit expliziten Anmeldeinformationen erkannt.', - 'medium', - 'Security', - '4648', - '', - '', - '', - 1, - 0, - 900, - 1 -), - -( - 'v1_many_explicit_credentials_used', - 'Viele Anmeldungen mit expliziten Anmeldeinformationen.', - 'high', - 'Security', - '4648', - '', - '', - '', - 20, - 300, - 1800, - 1 -), - --- ============================================================ --- Dienste, Tasks, Persistenz --- ============================================================ - -( - 'v1_service_installed_security', - 'Neuer Dienst laut Security Log installiert.', - 'high', - 'Security', - '4697', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_service_installed_system', - 'Neuer Dienst laut System Log installiert.', - 'high', - 'System', - '7045', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_scheduled_task_created', - 'Geplanter Task wurde erstellt.', - 'high', - 'Security', - '4698', - '', - '', - '', - 1, - 0, - 3600, - 1 -), - -( - 'v1_scheduled_task_updated', - 'Geplanter Task wurde geändert.', - 'medium', - 'Security', - '4702', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_scheduled_task_deleted', - 'Geplanter Task wurde gelöscht.', - 'medium', - 'Security', - '4699', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_scheduled_task_enabled_disabled', - 'Geplanter Task wurde aktiviert oder deaktiviert.', - 'medium', - 'Security', - '4700,4701', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - --- ============================================================ --- Prozess / PowerShell / Script Logging --- Nur aktivieren, wenn diese Events bei dir tatsächlich gesammelt werden. --- ============================================================ - -( - 'v1_process_created', - 'Neuer Prozess wurde erstellt. Nur sinnvoll bei aktiviertem Process Creation Auditing.', - 'low', - 'Security', - '4688', - '', - '', - '', - 1, - 0, - 300, - 0 -), - -( - 'v1_suspicious_powershell_process', - 'Verdächtiger PowerShell-Aufruf im Prozess-Event.', - 'high', - 'Security', - '4688', - 'msg', - 'contains', - 'powershell', - 1, - 0, - 1800, - 0 -), - -( - 'v1_powershell_script_block', - 'PowerShell Script Block Logging Event erkannt.', - 'medium', - 'Windows PowerShell,Microsoft-Windows-PowerShell/Operational', - '4104', - '', - '', - '', - 1, - 0, - 900, - 0 -), - -( - 'v1_suspicious_powershell_encodedcommand', - 'PowerShell EncodedCommand erkannt.', - 'high', - 'Windows PowerShell,Microsoft-Windows-PowerShell/Operational,Security', - '4104,4688', - 'msg', - 'contains', - 'EncodedCommand', - 1, - 0, - 1800, - 0 -), - -( - 'v1_suspicious_powershell_downloadstring', - 'PowerShell DownloadString erkannt.', - 'high', - 'Windows PowerShell,Microsoft-Windows-PowerShell/Operational,Security', - '4104,4688', - 'msg', - 'contains', - 'DownloadString', - 1, - 0, - 1800, - 0 -), - -( - 'v1_suspicious_powershell_iex', - 'PowerShell Invoke-Expression / IEX erkannt.', - 'high', - 'Windows PowerShell,Microsoft-Windows-PowerShell/Operational,Security', - '4104,4688', - 'msg', - 'contains', - 'Invoke-Expression', - 1, - 0, - 1800, - 0 -), - --- ============================================================ --- GPO / AD Objektänderungen --- Hinweis: 5136/5137/5141 kommen aus Directory Service Changes. --- Dafür muss das passende Auditing auf DCs aktiv sein. --- ============================================================ - -( - 'v1_directory_object_modified', - 'AD-Objekt wurde geändert.', - 'medium', - 'Security', - '5136', - '', - '', - '', - 1, - 0, - 900, - 0 -), - -( - 'v1_directory_object_created', - 'AD-Objekt wurde erstellt.', - 'medium', - 'Security', - '5137', - '', - '', - '', - 1, - 0, - 900, - 0 -), - -( - 'v1_directory_object_deleted', - 'AD-Objekt wurde gelöscht.', - 'medium', - 'Security', - '5141', - '', - '', - '', - 1, - 0, - 1800, - 0 -), - -( - 'v1_gpo_changed', - 'Mögliche GPO-Änderung erkannt.', - 'high', - 'Security', - '5136', - 'msg', - 'contains', - 'CN=Policies,CN=System', - 1, - 0, - 3600, - 0 -), - -( - 'v1_adminsdholder_changed', - 'AdminSDHolder wurde geändert. Sehr sicherheitsrelevant.', - 'high', - 'Security', - '5136', - 'msg', - 'contains', - 'CN=AdminSDHolder', - 1, - 0, - 86400, - 0 -), - -( - 'v1_domain_root_changed', - 'Domain-Root-Objekt wurde geändert.', - 'high', - 'Security', - '5136', - 'msg', - 'contains', - 'DC=', - 1, - 0, - 3600, - 0 -), - --- ============================================================ --- Objektzugriff / ACL --- Hinweis: 4662/4670 brauchen passende SACLs. --- ============================================================ - -( - 'v1_object_permissions_changed', - 'Berechtigungen eines Objekts wurden geändert.', - 'high', - 'Security', - '4670', - '', - '', - '', - 1, - 0, - 3600, - 0 -), - -( - 'v1_directory_service_object_access', - 'Directory-Service-Objektzugriff erkannt.', - 'medium', - 'Security', - '4662', - '', - '', - '', - 10, - 300, - 1800, - 0 -), - --- ============================================================ --- Share / SMB / SYSVOL / administrative Zugriffe --- Hinweis: Kann sehr laut werden. --- ============================================================ - -( - 'v1_network_share_accessed', - 'Netzwerkfreigabe wurde genutzt.', - 'low', - 'Security', - '5140', - '', - '', - '', - 50, - 300, - 900, - 0 -), - -( - 'v1_network_share_object_checked', - 'Detaillierter Netzwerkfreigabezugriff erkannt.', - 'low', - 'Security', - '5145', - '', - '', - '', - 100, - 300, - 900, - 0 -), - -( - 'v1_sysvol_access_spike', - 'Viele Zugriffe auf SYSVOL erkannt.', - 'low', - 'Security', - '5140,5145', - 'msg', - 'contains', - 'SYSVOL', - 100, - 300, - 900, - 0 -), - -( - 'v1_admin_share_access', - 'Zugriff auf administrative Freigabe erkannt.', - 'medium', - 'Security', - '5140,5145', - 'msg', - 'contains', - 'ADMIN$', - 1, - 0, - 1800, - 0 -), - -( - 'v1_c_share_access', - 'Zugriff auf C$ erkannt.', - 'medium', - 'Security', - '5140,5145', - 'msg', - 'contains', - 'C$', - 1, - 0, - 1800, - 0 -), - --- ============================================================ --- System / Neustart / Agent-Kontext --- ============================================================ - -( - 'v1_system_startup', - 'System wurde gestartet.', - 'low', - 'System', - '6005', - '', - '', - '', - 1, - 0, - 300, - 0 -), - -( - 'v1_system_shutdown', - 'System wurde heruntergefahren.', - 'low', - 'System', - '6006', - '', - '', - '', - 1, - 0, - 300, - 0 -), - -( - 'v1_planned_shutdown_or_restart', - 'Geplanter Shutdown oder Neustart erkannt.', - 'low', - 'System', - '1074', - '', - '', - '', - 1, - 0, - 900, - 1 -), - -( - 'v1_unexpected_shutdown', - 'Unerwarteter Shutdown erkannt.', - 'medium', - 'System', - '6008', - '', - '', - '', - 1, - 0, - 1800, - 1 -), - -( - 'v1_reboot_spike_dynamic', - 'Viele Neustart-/Shutdown-Events in kurzer Zeit.', - 'medium', - 'System', - '1074,6005,6006,6008', - '', - '', - '', - 3, - 900, - 1800, - 0 -), - --- ============================================================ --- Windows Defender / Security Center --- Event IDs können je nach Version/Quelle variieren. --- ============================================================ - -( - 'v1_defender_malware_detected', - 'Microsoft Defender hat Malware erkannt.', - 'high', - 'Microsoft-Windows-Windows Defender/Operational', - '1116', - '', - '', - '', - 1, - 0, - 3600, - 0 -), - -( - 'v1_defender_malware_remediated', - 'Microsoft Defender hat Malware bereinigt.', - 'medium', - 'Microsoft-Windows-Windows Defender/Operational', - '1117', - '', - '', - '', - 1, - 0, - 1800, - 0 -), - -( - 'v1_defender_action_failed', - 'Microsoft Defender Aktion fehlgeschlagen.', - 'high', - 'Microsoft-Windows-Windows Defender/Operational', - '1118,1119', - '', - '', - '', - 1, - 0, - 3600, - 0 -), - -( - 'v1_defender_disabled_or_config_changed', - 'Defender-Konfiguration wurde geändert oder Schutz deaktiviert.', - 'high', - 'Microsoft-Windows-Windows Defender/Operational', - '5007,5013', - '', - '', - '', - 1, - 0, - 3600, - 0 -), - --- ============================================================ --- Zertifikatsdienste / AD CS --- Nur aktivieren, wenn AD CS vorhanden und Events gesammelt werden. --- ============================================================ - -( - 'v1_certificate_request_issued', - 'Zertifikat wurde ausgestellt.', - 'medium', - 'Security', - '4886,4887', - '', - '', - '', - 1, - 0, - 1800, - 0 -), - -( - 'v1_certificate_template_changed', - 'Zertifikatstemplate wurde geändert.', - 'high', - 'Security', - '4898,4899,4900', - '', - '', - '', - 1, - 0, - 86400, - 0 -), - -( - 'v1_cert_services_config_changed', - 'AD-CS-Konfiguration wurde geändert.', - 'high', - 'Security', - '4882,4885,4890,4891,4892', - '', - '', - '', - 1, - 0, - 86400, - 0 -), - -( - 'v2_security_log_cleared', - 'Security Log wurde gelöscht. Sehr wahrscheinlich Spurenverwischung.', - 'high', - 'Security', - '1102', - '', '', '', - 1, 0, 86400, - 1 -), - -( - 'v2_audit_policy_changed', - 'Audit Policy wurde geändert.', - 'high', - 'Security', - '4719', - '', '', '', - 1, 0, 3600, - 1 -), - -( - 'v2_system_audit_policy_changed', - 'System Audit Policy wurde geändert.', - 'high', - 'Security', - '4902,4904,4905,4906,4907,4908,4912', - '', '', '', - 1, 0, 3600, - 1 -), - --- ============================================================ --- PRIVILEGIERTE GRUPPEN (WICHTIGSTER BLOCK!) --- ============================================================ - -( - 'v2_privileged_group_member_added', - 'Mitglied wurde zu privilegierter AD-Gruppe hinzugefügt.', - 'high', - 'Security', - '4728,4732,4756', - 'target_user', - 'in', - 'Domain Admins,Enterprise Admins,Schema Admins,Administrators,DNSAdmins', - 1, 0, 3600, - 1 -), - -( - 'v2_privileged_group_member_removed', - 'Mitglied wurde aus privilegierter AD-Gruppe entfernt.', - 'medium', - 'Security', - '4729,4733,4757', - 'target_user', - 'in', - 'Domain Admins,Enterprise Admins,Schema Admins,Administrators,DNSAdmins', - 1, 0, 3600, - 1 -), - -( - 'v2_domain_admins_changed', - 'Änderung an Domain Admins.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Domain Admins', - 1, 0, 3600, - 1 -), - -( - 'v2_enterprise_admins_changed', - 'Änderung an Enterprise Admins.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Enterprise Admins', - 1, 0, 3600, - 1 -), - -( - 'v2_schema_admins_changed', - 'Änderung an Schema Admins.', - 'high', - 'Security', - '4728,4729,4735,4737', - 'target_user', - 'eq', - 'Schema Admins', - 1, 0, 3600, - 1 -), - -( - 'v2_dnsadmins_changed', - 'Änderung an DNSAdmins erkannt.', - 'high', - 'Security', - '4728,4729,4732,4733,4756,4757', - 'target_user', - 'eq', - 'DNSAdmins', - 1, 0, 3600, - 1 -), - --- ============================================================ --- USER MANAGEMENT --- ============================================================ - -( - 'v2_user_created', - 'Neuer AD-Benutzer wurde erstellt.', - 'medium', - 'Security', - '4720', - '', '', '', - 1, 0, 1800, - 1 -), - -( - 'v2_user_enabled', - 'AD-Benutzer wurde aktiviert.', - 'medium', - 'Security', - '4722', - '', '', '', - 1, 0, 1800, - 1 -), - -( - 'v2_user_deleted', - 'AD-Benutzer wurde gelöscht.', - 'medium', - 'Security', - '4726', - '', '', '', - 1, 0, 3600, - 1 -), - -( - 'v2_password_reset', - 'Passwort wurde zurückgesetzt.', - 'medium', - 'Security', - '4724', - '', '', '', - 1, 0, 1800, - 1 -), - --- ============================================================ --- AUTHENTIFIZIERUNG (ANOMALIEN) --- ============================================================ - -( - 'v2_account_lockout_spike', - 'Viele Account Lockouts.', - 'medium', - 'Security', - '4740', - '', '', '', - 5, 300, 1800, - 1 -), - -( - 'v2_kerberos_preauth_spike', - 'Viele Kerberos PreAuth Fehler.', - 'high', - 'Security', - '4771', - '', '', '', - 20, 300, 1800, - 1 -), - -( - 'v2_kerberos_service_ticket_spike', - 'Viele Kerberos Service Tickets (Kerberoasting möglich).', - 'medium', - 'Security', - '4769', - '', '', '', - 100, 300, 1800, - 1 -), - -( - 'v2_ntlm_authentication_spike', - 'Viele NTLM Authentifizierungen.', - 'medium', - 'Security', - '4776', - '', '', '', - 50, 300, 1800, - 1 -), - -( - 'v2_logon_failed_spike', - 'Viele fehlgeschlagene Logons.', - 'medium', - 'Security', - '4625', - '', '', '', - 25, 300, 1800, - 1 -), - -( - 'v2_many_explicit_credentials_used', - 'Viele Logons mit expliziten Credentials.', - 'high', - 'Security', - '4648', - '', '', '', - 20, 300, 1800, - 1 -), - --- ============================================================ --- PERSISTENZ / LATERAL MOVEMENT --- ============================================================ - -( - 'v2_service_installed_security', - 'Neuer Dienst (Security Log).', - 'high', - 'Security', - '4697', - '', '', '', - 1, 0, 3600, - 1 -), - -( - 'v2_service_installed_system', - 'Neuer Dienst (System Log).', - 'high', - 'System', - '7045', - '', '', '', - 1, 0, 3600, - 1 -), - -( - 'v2_scheduled_task_created', - 'Scheduled Task erstellt.', - 'high', - 'Security', - '4698', - '', '', '', - 1, 0, 3600, - 1 -), - -( - 'v2_scheduled_task_updated', - 'Scheduled Task geändert.', - 'medium', - 'Security', - '4702', - '', '', '', - 1, 0, 1800, - 1 -), - -( - 'v2_scheduled_task_deleted', - 'Scheduled Task gelöscht.', - 'medium', - 'Security', - '4699', - '', '', '', - 1, 0, 1800, - 1 -), - -( - 'v2_scheduled_task_enabled_disabled', - 'Scheduled Task aktiviert/deaktiviert.', - 'medium', - 'Security', - '4700,4701', - '', '', '', - 1, 0, 1800, - 1 -), - --- ============================================================ --- SYSTEM EVENTS --- ============================================================ - -( - 'v2_unexpected_shutdown', - 'Unerwarteter Shutdown erkannt.', - 'medium', - 'System', - '6008', - '', '', '', - 1, 0, 1800, - 1 -) -ON DUPLICATE KEY UPDATE - description = VALUES(description), - severity = VALUES(severity), - channel = VALUES(channel), - event_ids = VALUES(event_ids), - match_field = VALUES(match_field), - match_operator = VALUES(match_operator), - match_value = VALUES(match_value), - threshold_count = VALUES(threshold_count), - threshold_window_seconds = VALUES(threshold_window_seconds), - suppress_for_seconds = VALUES(suppress_for_seconds), - enabled = VALUES(enabled); - -UPDATE detection_rules -SET enabled = 0 -WHERE name LIKE 'v1_%'; - --- --------------------------------------------------------------------- --- Diagnose-Queries nach Restore --- --------------------------------------------------------------------- --- SELECT TABLE_NAME, PARTITION_NAME, PARTITION_DESCRIPTION, TABLE_ROWS --- FROM information_schema.PARTITIONS --- WHERE TABLE_SCHEMA = DATABASE() --- AND TABLE_NAME IN ('event_logs', 'event_log_raw') --- ORDER BY TABLE_NAME, PARTITION_ORDINAL_POSITION; --- --- SHOW INDEX FROM event_logs; --- SHOW INDEX FROM event_log_raw; --- SHOW INDEX FROM detections; diff --git a/deploy/mariadb/migrations/002-metadata-first.sql b/deploy/mariadb/migrations/002-metadata-first.sql deleted file mode 100644 index 85c727f..0000000 --- a/deploy/mariadb/migrations/002-metadata-first.sql +++ /dev/null @@ -1,98 +0,0 @@ --- Metadata-first migration for an existing SIEM-lite database. --- IMPORTANT: ALTER TABLE ... PARTITION BY can rebuild and lock large tables. --- Run during a maintenance window and take a backup first. - -SET NAMES utf8mb4; -SET time_zone = '+00:00'; - -CREATE TABLE IF NOT EXISTS event_catalog ( - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - first_seen DATETIME(6) NOT NULL, - last_seen DATETIME(6) NOT NULL, - total_count BIGINT UNSIGNED NOT NULL DEFAULT 0, - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - PRIMARY KEY (hostname, channel_name, event_id), - KEY idx_event_catalog_first_seen (first_seen), - KEY idx_event_catalog_last_seen (last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - --- Warm the catalog from the much smaller baseline buckets so the --- new_event_id rule does not treat every known event type as new after migration. -INSERT INTO event_catalog -(hostname, channel_name, event_id, first_seen, last_seen, total_count, updated_at) -SELECT hostname, - channel_name, - event_id, - MIN(COALESCE(first_event_ts, bucket_start)), - MAX(COALESCE(last_event_ts, bucket_end)), - SUM(cnt), - UTC_TIMESTAMP(6) -FROM event_count_buckets -GROUP BY hostname, channel_name, event_id -ON DUPLICATE KEY UPDATE - first_seen = LEAST(first_seen, VALUES(first_seen)), - last_seen = GREATEST(last_seen, VALUES(last_seen)), - total_count = GREATEST(total_count, VALUES(total_count)), - updated_at = UTC_TIMESTAMP(6); - -CREATE TABLE IF NOT EXISTS event_occurrences ( - bucket_start DATETIME(6) NOT NULL, - bucket_end DATETIME(6) NOT NULL, - dimension_key BINARY(16) NOT NULL, - hostname VARCHAR(191) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - provider_name VARCHAR(191) NOT NULL DEFAULT '', - target_user VARCHAR(191) NOT NULL DEFAULT '', - subject_user VARCHAR(191) NOT NULL DEFAULT '', - src_ip VARCHAR(64) NOT NULL DEFAULT '', - workstation VARCHAR(191) NOT NULL DEFAULT '', - logon_type VARCHAR(32) NOT NULL DEFAULT '', - status_text VARCHAR(128) NOT NULL DEFAULT '', - failure_reason VARCHAR(255) NOT NULL DEFAULT '', - cnt BIGINT UNSIGNED NOT NULL DEFAULT 0, - first_event_ts DATETIME(6) NOT NULL, - last_event_ts DATETIME(6) NOT NULL, - updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), - PRIMARY KEY (bucket_start, dimension_key), - KEY idx_occurrences_time_host_event (bucket_start, hostname, channel_name, event_id), - KEY idx_occurrences_host_event_time (hostname, channel_name, event_id, bucket_start), - KEY idx_occurrences_target_user_time (target_user, bucket_start), - KEY idx_occurrences_subject_user_time (subject_user, bucket_start), - KEY idx_occurrences_src_ip_time (src_ip, bucket_start) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci -PARTITION BY RANGE COLUMNS(bucket_start) ( - PARTITION pmax VALUES LESS THAN (MAXVALUE) -); - --- Partition bucket tables only when needed, so this migration can be rerun. -DELIMITER $$ -CREATE PROCEDURE partition_bucket_table_if_needed(IN p_table VARCHAR(64)) -BEGIN - DECLARE v_partition_count INT DEFAULT 0; - - SELECT COUNT(*) INTO v_partition_count - FROM information_schema.PARTITIONS - WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = p_table - AND PARTITION_NAME IS NOT NULL; - - IF v_partition_count = 0 THEN - SET @partition_sql = CONCAT( - 'ALTER TABLE `', p_table, '` ', - 'PARTITION BY RANGE COLUMNS(bucket_start) ', - '(PARTITION pmax VALUES LESS THAN (MAXVALUE))' - ); - PREPARE partition_stmt FROM @partition_sql; - EXECUTE partition_stmt; - DEALLOCATE PREPARE partition_stmt; - END IF; -END$$ -DELIMITER ; - -CALL partition_bucket_table_if_needed('event_occurrences'); -CALL partition_bucket_table_if_needed('event_count_buckets'); -CALL partition_bucket_table_if_needed('ueba_context_buckets'); -DROP PROCEDURE partition_bucket_table_if_needed; diff --git a/deploy/mariadb/migrations/003-new-event-id-classification.sql b/deploy/mariadb/migrations/003-new-event-id-classification.sql deleted file mode 100644 index ed664ab..0000000 --- a/deploy/mariadb/migrations/003-new-event-id-classification.sql +++ /dev/null @@ -1,27 +0,0 @@ --- Metadata-first SIEM: realistischere Einordnung der Regel new_event_id --- Diese Migration verändert kein Schema. Sie bereinigt nur bereits offene --- WMI-Operational-Inventarmeldungen, die von der alten Regel als Medium-Alarm --- angelegt wurden. Vor produktiver Ausführung wie üblich ein Backup erstellen. - -START TRANSACTION; - -UPDATE detections -SET severity = 'info', - status = 'legitimate', - is_legitimate = 1, - reviewed_by = 'system:migration-003', - reviewed_at = UTC_TIMESTAMP(6), - analyst_note = CASE - WHEN analyst_note IS NULL OR TRIM(analyst_note) = '' THEN - 'Automatisch neu klassifiziert: neue Event-IDs im WMI-Activity/Operational-Channel sind Inventar-/Diagnoseinformationen und allein kein Incident.' - ELSE CONCAT( - analyst_note, - '\nAutomatisch neu klassifiziert: neue Event-IDs im WMI-Activity/Operational-Channel sind Inventar-/Diagnoseinformationen und allein kein Incident.' - ) - END -WHERE rule_name = 'new_event_id' - AND channel_name = 'Microsoft-Windows-WMI-Activity/Operational' - AND status IN ('open', 'acknowledged', 'investigating', 'plausible') - AND is_legitimate = 0; - -COMMIT; diff --git a/deploy/mariadb/migrations/004-compact-storage.sql b/deploy/mariadb/migrations/004-compact-storage.sql deleted file mode 100644 index 0500e03..0000000 --- a/deploy/mariadb/migrations/004-compact-storage.sql +++ /dev/null @@ -1,21 +0,0 @@ --- Compact-storage migration. --- Safe to apply without deleting event history. The destructive cleanup of old --- per-event rows is intentionally kept in ../emergency-compact-cleanup.sql. - -SET NAMES utf8mb4; -SET time_zone = '+00:00'; - -CREATE INDEX IF NOT EXISTS idx_event_count_buckets_event_time - ON event_count_buckets (channel_name, event_id, bucket_start, hostname); - -CREATE INDEX IF NOT EXISTS idx_occurrences_security_event_time - ON event_occurrences ( - channel_name, - event_id, - bucket_start, - hostname, - target_user, - src_ip - ); - -ANALYZE TABLE event_count_buckets, event_occurrences, event_catalog, detections; diff --git a/deploy/postgres/init.sql b/deploy/postgres/init.sql new file mode 100644 index 0000000..ac21565 --- /dev/null +++ b/deploy/postgres/init.sql @@ -0,0 +1,38 @@ +CREATE EXTENSION IF NOT EXISTS pgcrypto; +CREATE TABLE IF NOT EXISTS agents ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + tenant_id text NOT NULL, + hostname text NOT NULL, + api_key_hash char(64) NOT NULL, + enabled boolean NOT NULL DEFAULT true, + last_ip text NOT NULL DEFAULT '', + first_seen timestamptz NOT NULL DEFAULT now(), + last_seen timestamptz NOT NULL DEFAULT now(), + UNIQUE (tenant_id, hostname) +); +CREATE INDEX IF NOT EXISTS agents_last_seen_idx ON agents(tenant_id,last_seen DESC); + +CREATE TABLE IF NOT EXISTS detections ( + id bigserial PRIMARY KEY, + tenant_id text NOT NULL, + fingerprint char(64) NOT NULL, + rule_name text NOT NULL, + severity text NOT NULL, + status text NOT NULL DEFAULT 'open', + hostname text NOT NULL DEFAULT '', + user_name text NOT NULL DEFAULT '', + source_ip text NOT NULL DEFAULT '', + workstation text NOT NULL DEFAULT '', + event_code integer NOT NULL DEFAULT 0, + score double precision NOT NULL DEFAULT 0, + window_start timestamptz NOT NULL, + window_end timestamptz NOT NULL, + summary text NOT NULL, + hit_count bigint NOT NULL DEFAULT 1, + first_seen timestamptz NOT NULL DEFAULT now(), + last_seen timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (tenant_id, fingerprint) +); +CREATE INDEX IF NOT EXISTS detections_open_idx ON detections(tenant_id,status,last_seen DESC); +CREATE INDEX IF NOT EXISTS detections_rule_idx ON detections(tenant_id,rule_name,last_seen DESC); diff --git a/deploy/prometheus/prometheus.yml b/deploy/prometheus/prometheus.yml index 8f80b65..e0327f9 100644 --- a/deploy/prometheus/prometheus.yml +++ b/deploy/prometheus/prometheus.yml @@ -1,18 +1,11 @@ global: scrape_interval: 15s - evaluation_interval: 15s - -rule_files: - - /etc/prometheus/rules/*.yml - scrape_configs: - - job_name: siem-backend + - job_name: siem-ingress + static_configs: [{targets: ['ingress:8080']}] + - job_name: redpanda + metrics_path: /public_metrics + static_configs: [{targets: ['redpanda:9644']}] + - job_name: clickhouse metrics_path: /metrics - static_configs: - - targets: - - siem-backend:8080 - - - job_name: prometheus - static_configs: - - targets: - - localhost:9090 \ No newline at end of file + static_configs: [{targets: ['clickhouse:9363']}] diff --git a/deploy/prometheus/rules/siem-alerts.yml b/deploy/prometheus/rules/siem-alerts.yml deleted file mode 100644 index edaeff1..0000000 --- a/deploy/prometheus/rules/siem-alerts.yml +++ /dev/null @@ -1,132 +0,0 @@ -groups: - - name: siem-backend-availability - rules: - - alert: SiemBackendDown - expr: up{job="siem-backend"} == 0 - for: 2m - labels: - severity: critical - annotations: - summary: "SIEM backend nicht erreichbar" - description: "Prometheus kann das SIEM-Backend seit mindestens 2 Minuten nicht scrapen." - - - alert: SiemNoIngestEvents - expr: sum(rate(eventcollector_ingest_events_total[15m])) == 0 - for: 15m - labels: - severity: warning - annotations: - summary: "Keine eingehenden SIEM Events" - description: "Seit mindestens 15 Minuten wurden keine Events mehr ingestiert." - - - alert: SiemTooFewActiveAgents - expr: eventcollector_active_agents < 1 - for: 5m - labels: - severity: warning - annotations: - summary: "Zu wenige aktive Agents" - description: "Es wurden weniger aktive Agents erkannt als erwartet." - - - alert: SiemCriticalDetections - expr: increase(eventcollector_detection_hits_total{severity="critical"}[5m]) > 0 - for: 1m - labels: - severity: critical - annotations: - summary: "Neue Critical Detection" - description: "Es wurde mindestens eine Critical-Detection erzeugt." - - - alert: SiemHighDetections - expr: increase(eventcollector_detection_hits_total{severity="high"}[5m]) > 0 - for: 1m - labels: - severity: high - annotations: - summary: "Neue High-Severity Detection" - description: "Es wurde mindestens eine High-Severity-Detection erzeugt." - - - alert: SiemManyMediumDetections - expr: sum(increase(eventcollector_detection_hits_total{severity="medium"}[15m])) > 10 - for: 2m - labels: - severity: warning - annotations: - summary: "Viele Medium-Detections" - description: "Es wurden mehr als 10 Medium-Detections in 15 Minuten erzeugt." - - - alert: SiemBaselineHighAnomaly - expr: eventcollector_anomaly_score{rule="baseline_event_rate_anomaly"} >= 5 - for: 2m - labels: - severity: high - annotations: - summary: "Hohe Baseline-Anomalie" - description: "Host {{ $labels.host }} hat einen hohen Baseline-Z-Score: {{ $value }}." - - - alert: SiemBaselineMediumAnomaly - expr: eventcollector_anomaly_score{rule="baseline_event_rate_anomaly"} >= 3 - for: 5m - labels: - severity: warning - annotations: - summary: "Baseline-Anomalie" - description: "Host {{ $labels.host }} hat einen erhöhten Baseline-Z-Score: {{ $value }}." - - - alert: SiemRuleErrors - expr: increase(eventcollector_rule_errors_total[5m]) > 0 - for: 1m - labels: - severity: warning - annotations: - summary: "Fehler in Detection-Regeln" - description: "Mindestens eine Detection-Regel hat in den letzten 5 Minuten einen Fehler erzeugt." - - - name: siem-backend-ingest - rules: - - alert: SiemIngestRejected - expr: sum(increase(eventcollector_ingest_rejected_total[5m])) > 0 - for: 1m - labels: - severity: warning - annotations: - summary: "Ingest Requests abgelehnt" - description: "In den letzten 5 Minuten wurden Ingest Requests abgelehnt." - - - alert: SiemDBInsertFailures - expr: increase(eventcollector_db_insert_failures_total[5m]) > 0 - for: 1m - labels: - severity: high - annotations: - summary: "DB Insert Fehler" - description: "Das SIEM-Backend konnte Events nicht in die Datenbank schreiben." - - - alert: SiemHighIngestRate - expr: sum(rate(eventcollector_ingest_events_total[5m])) > 500 - for: 5m - labels: - severity: warning - annotations: - summary: "Sehr hohe Eventrate" - description: "Die Eventrate liegt seit 5 Minuten über 500 Events/s." - - - name: siem-backend-baseline - rules: - - alert: SiemBaselineNotEnoughSamples - expr: eventcollector_baseline_sample_count > 0 and eventcollector_baseline_sample_count < 24 - for: 30m - labels: - severity: info - annotations: - summary: "Baseline lernt noch" - description: "Für {{ $labels.host }} / {{ $labels.channel }} / {{ $labels.event_id }} gibt es erst {{ $value }} Samples." - - - alert: SiemBaselineCurrentFarAboveAverage - expr: eventcollector_baseline_avg_count > 0 and (eventcollector_baseline_current_count / eventcollector_baseline_avg_count) > 10 - for: 2m - labels: - severity: warning - annotations: - summary: "Eventrate deutlich über Baseline" - description: "{{ $labels.host }} / {{ $labels.channel }} / {{ $labels.event_id }} liegt mehr als 10x über Durchschnitt." \ No newline at end of file diff --git a/doctor.sh b/doctor.sh new file mode 100755 index 0000000..a78b2de --- /dev/null +++ b/doctor.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env sh +set -u +cd "$(dirname "$0")" +[ -f .env ] && { set -a; . ./.env; set +a; } + +echo "== Docker ==" +docker compose ps || true + +echo "\n== Host filesystem ==" +df -h . 2>/dev/null || true + +echo "\n== HTTP readiness ==" +if command -v curl >/dev/null 2>&1; then + printf "Ingress: "; curl -fsS "http://127.0.0.1:${INGRESS_PORT:-8090}/readyz" || true; echo + printf "API: "; curl -fsS "http://127.0.0.1:${UI_PORT:-8080}/readyz" || true; echo +else + echo "curl fehlt; HTTP-Prüfung übersprungen." +fi + +echo "\n== Redpanda ==" +docker compose exec -T redpanda rpk cluster health 2>/dev/null || true +docker compose exec -T redpanda rpk group describe siem-processor-v1 2>/dev/null || true + +echo "\n== ClickHouse events ==" +docker compose exec -T clickhouse clickhouse-client --user siem --password "${CLICKHOUSE_PASSWORD:-}" --query "SELECT formatReadableQuantity(sum(rows)) AS rows, formatReadableSize(sum(bytes_on_disk)) AS disk, count() AS active_parts FROM system.parts WHERE database='siem' AND table='events' AND active" 2>/dev/null || true +docker compose exec -T clickhouse clickhouse-client --user siem --password "${CLICKHOUSE_PASSWORD:-}" --query "SELECT max(event_time) AS latest_event, max(ingest_time) AS latest_ingest, quantile(0.95)(abs(ingest_delay_ms))/1000 AS p95_ingest_delay_seconds FROM siem.events WHERE event_time >= now() - INTERVAL 1 DAY" 2>/dev/null || true + +echo "\n== PostgreSQL control plane ==" +docker compose exec -T postgres psql -U siem -d siem -Atc "SELECT 'agents='||count(*) FROM agents; SELECT 'open_detections='||count(*) FROM detections WHERE status IN ('open','investigating');" 2>/dev/null || true + +echo "\n== Raw spool ==" +docker compose exec -T archive-uploader sh -c 'du -sh /spool 2>/dev/null || true; find /spool -type f 2>/dev/null | wc -l' 2>/dev/null || true + +echo "\n== Recent errors ==" +docker compose logs --since=15m ingress processor detector api archive-uploader 2>&1 | grep -Ei 'error|fatal|failed|panic|not_ready' | tail -80 || true diff --git a/dot_env b/dot_env deleted file mode 100644 index 5f733c9..0000000 --- a/dot_env +++ /dev/null @@ -1,93 +0,0 @@ -TZ=Europe/Berlin - -LISTEN_ADDR=:8080 -DB_DSN=eventuser:DEINPASSWORT@tcp(mariadb:3306)/eventcollector?parseTime=true&charset=utf8mb4,utf8&collation=utf8mb4_unicode_ci&loc=UTC - -DB_MAX_OPEN_CONNS=50 -DB_MAX_IDLE_CONNS=25 -DB_CONN_MAX_LIFETIME=3m -DB_CONN_MAX_IDLE_TIME=1m - -MAX_BODY_BYTES=10485760 -HTTP_READ_TIMEOUT=15s -HTTP_WRITE_TIMEOUT=30s -HTTP_IDLE_TIMEOUT=60s - -DETECTION_INTERVAL=1m -OFFLINE_AFTER=10m -OFFLINE_ALERT_MAX=120m -FAILED_LOGON_WINDOW=5m -FAILED_LOGON_THRESHOLD=25 -REBOOT_WINDOW=15m -REBOOT_THRESHOLD=3 -PASSWORD_SPRAY_WINDOW=5m -PASSWORD_SPRAY_MIN_USERS=5 -PASSWORD_SPRAY_MIN_ATTEMPTS=15 -SUCCESS_AFTER_FAILURE_WINDOW=10m -NEW_SOURCE_IP_LOOKBACK=720h -NEW_SOURCE_IP_WINDOW=10m -DETECTIONS_LIMIT=100 - -MARIADB_DATABASE=eventcollector -MARIADB_USER=eventuser -MARIADB_PASSWORD=DEINPASSWORT -MARIADB_ROOT_PASSWORD=ROOTPASSWORT - -GRAFANA_ADMIN_USER=admin -GRAFANA_ADMIN_PASSWORD=admin - -ENROLLMENT_KEY=BITTE_SEHR_LANG_UND_ZUFAELLIG - -BASELINE_ENABLED=true -BASELINE_WINDOW=5m -BASELINE_MIN_SAMPLES=24 -BASELINE_MIN_COUNT=10 -BASELINE_MEDIUM_Z=2.5 -BASELINE_HIGH_Z=4.0 -BASELINE_SUPPRESS_FOR=1h - - -#BASELINE_MIN_SAMPLES=84 -#BASELINE_MEDIUM_Z=3.0 -#BASELINE_HIGH_Z=5.0 -#BASELINE_MIN_COUNT=20 -# Metadata-first storage -# Raw XML is disabled by default. Enable only for short forensic retention. -STORE_EVENT_ROWS=false -STORE_RAW_XML=false -METADATA_CONTEXT_EVENT_IDS=4624,4625,4648,4672,4720,4726,4728,4732,4740,4756,4768,4769,4771,4776 -METADATA_BUCKET=5m -EVENT_RETENTION=6h -RAW_RETENTION=6h -METADATA_RETENTION=720h - -# Partition maintenance -PARTITION_MAINTENANCE_ENABLED=true -PARTITION_MAINTENANCE_INTERVAL=15m -PARTITION_INTERVAL=3h -PARTITION_AHEAD=24h -PARTITION_BEHIND=6h -# Compatibility fallback; table-specific retention values above take precedence. -PARTITION_RETENTION=720h - -# New Event-ID classification -NEW_EVENT_ID_MODE=inventory -NEW_EVENT_ID_LEARNING_PERIOD=24h -NEW_EVENT_ID_CONFIRM_WINDOW=15m -NEW_EVENT_ID_MIN_COUNT=3 -NEW_EVENT_ID_IGNORE_CHANNELS=Microsoft-Windows-WMI-Activity/Operational -NEW_EVENT_ID_ALERT_CHANNELS=Security,System -NEW_EVENT_ID_HIGH_RISK_IDS=1102,4697,4719,7045 - -# Optional stress-agent profile -SIEM_STRESS_URL=http://siem-backend:8080/ingest -SIEM_STRESS_HOST=SIEM-STRESS-01 -SIEM_STRESS_API_KEY=CHANGE-ME-STRESS-API-KEY -SIEM_STRESS_SCENARIO=mixed -SIEM_STRESS_RATE=200 -SIEM_STRESS_BATCH=100 -SIEM_STRESS_WORKERS=4 -SIEM_STRESS_DURATION=30s -SIEM_STRESS_MAX_EVENTS=100000 -SIEM_STRESS_TIMEOUT=20s -SIEM_STRESS_CONFIRM=true diff --git a/go.mod b/go.mod index 1a2f53c..a326dd4 100644 --- a/go.mod +++ b/go.mod @@ -1,22 +1,8 @@ -module git.send.nrw/sendnrw/siem-backend +module example.com/siem-greenfield -go 1.26.1 - -require github.com/go-sql-driver/mysql v1.9.3 +go 1.26 require ( - github.com/beorn7/perks v1.0.1 // indirect - github.com/cespare/xxhash/v2 v2.3.0 // indirect - github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/prometheus/client_model v0.6.2 // indirect - github.com/prometheus/common v0.66.1 // indirect - github.com/prometheus/procfs v0.16.1 // indirect - go.yaml.in/yaml/v2 v2.4.2 // indirect - golang.org/x/sys v0.35.0 // indirect - google.golang.org/protobuf v1.36.8 // indirect -) - -require ( - filippo.io/edwards25519 v1.1.0 // indirect - github.com/prometheus/client_golang v1.23.2 + github.com/jackc/pgx/v5 v5.9.2 + github.com/segmentio/kafka-go v0.4.51 ) diff --git a/go.sum b/go.sum deleted file mode 100644 index 2163bbe..0000000 --- a/go.sum +++ /dev/null @@ -1,25 +0,0 @@ -filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= -filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= -github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= -github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo= -github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= -github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= -github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= -github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= -github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= -github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs= -github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= -github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= -github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= -go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= -go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= -golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= -golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= -google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc= -google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/internal/api/server.go b/internal/api/server.go new file mode 100644 index 0000000..b3cc9c1 --- /dev/null +++ b/internal/api/server.go @@ -0,0 +1,187 @@ +package api + +import ( + "context" + "crypto/subtle" + "encoding/json" + "fmt" + "html/template" + "log" + "net/http" + "strconv" + "strings" + "time" + + "example.com/siem-greenfield/internal/clickhouse" + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/postgres" +) + +func Run(ctx context.Context, cfg config.Config) error { + pg, e := postgres.Open(ctx, cfg.PostgresURL) + if e != nil { + return e + } + defer pg.Close() + ch := clickhouse.New(cfg) + tpl, e := template.ParseFiles("/app/web/index.html") + if e != nil { + return e + } + mux := http.NewServeMux() + mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { j(w, 200, map[string]string{"status": "ok"}) }) + mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) { + cctx, cancel := context.WithTimeout(r.Context(), 3*time.Second) + defer cancel() + if e := pg.Pool.Ping(cctx); e != nil { + j(w, 503, map[string]string{"status": "not_ready", "component": "postgres"}) + return + } + if e := ch.Exec(cctx, "SELECT 1"); e != nil { + j(w, 503, map[string]string{"status": "not_ready", "component": "clickhouse"}) + return + } + j(w, 200, map[string]string{"status": "ready"}) + }) + mux.HandleFunc("/ui", func(w http.ResponseWriter, r *http.Request) { + _ = tpl.Execute(w, map[string]string{"Tenant": cfg.TenantID}) + }) + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/" { + http.Redirect(w, r, "/ui", http.StatusFound) + return + } + http.NotFound(w, r) + }) + mux.HandleFunc("/api/summary", func(w http.ResponseWriter, r *http.Request) { summary(w, r, cfg, pg, ch) }) + mux.HandleFunc("/api/events", func(w http.ResponseWriter, r *http.Request) { events(w, r, cfg, ch) }) + mux.HandleFunc("/api/detections", func(w http.ResponseWriter, r *http.Request) { detections(w, r, cfg, pg) }) + mux.HandleFunc("/api/detections/status", func(w http.ResponseWriter, r *http.Request) { detectionStatus(w, r, cfg, pg) }) + mux.HandleFunc("/api/agents", func(w http.ResponseWriter, r *http.Request) { + x, e := pg.ListAgents(r.Context(), cfg.TenantID) + if e != nil { + j(w, 500, map[string]string{"error": e.Error()}) + return + } + j(w, 200, x) + }) + srv := &http.Server{Addr: cfg.ServiceAddr, Handler: security(basicAuth(mux, cfg.UIUsername, cfg.UIPassword)), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 30 * time.Second, IdleTimeout: 60 * time.Second} + go func() { + <-ctx.Done() + c, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = srv.Shutdown(c) + }() + log.Printf("api/ui listening on %s", cfg.ServiceAddr) + e = srv.ListenAndServe() + if e == http.ErrServerClosed { + return nil + } + return e +} +func summary(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store, ch *clickhouse.Client) { + ctx, cancel := context.WithTimeout(r.Context(), 8*time.Second) + defer cancel() + q := fmt.Sprintf(`SELECT uniqExactMerge(cnt_state) events_24h, uniqExact(host_name) active_hosts FROM siem.events_5m WHERE tenant_id=%s AND bucket>=now()-INTERVAL 24 HOUR`, clickhouse.Q(cfg.TenantID)) + rows, e := ch.QueryJSON(ctx, q) + if e != nil { + j(w, 500, map[string]string{"error": e.Error()}) + return + } + out := map[string]any{"events_24h": 0, "active_hosts": 0} + if len(rows) > 0 { + for k, v := range rows[0] { + out[k] = v + } + } + dc, _ := pg.DetectionCounts(ctx, cfg.TenantID) + out["detections"] = dc + j(w, 200, out) +} +func events(w http.ResponseWriter, r *http.Request, cfg config.Config, ch *clickhouse.Client) { + limit := cfg.UIQueryLimit + if n, e := strconv.Atoi(r.URL.Query().Get("limit")); e == nil && n > 0 && n <= 2000 { + limit = n + } + hours := 24 + if n, e := strconv.Atoi(r.URL.Query().Get("hours")); e == nil && n > 0 && n <= 2160 { + hours = n + } + where := []string{"tenant_id=" + clickhouse.Q(cfg.TenantID), fmt.Sprintf("event_time>=now()-INTERVAL %d HOUR", hours)} + for key, col := range map[string]string{"host": "host_name", "user": "user_name", "ip": "source_ip", "channel": "channel", "action": "action"} { + if v := strings.TrimSpace(r.URL.Query().Get(key)); v != "" { + where = append(where, col+"="+clickhouse.Q(v)) + } + } + if v := strings.TrimSpace(r.URL.Query().Get("event_code")); v != "" { + if _, e := strconv.ParseUint(v, 10, 32); e == nil { + where = append(where, "event_code="+v) + } + } + q := fmt.Sprintf(`SELECT event_uid,event_time,host_name,channel,event_code,category,action,outcome,severity,user_name,subject_user,target_user,source_ip,workstation,process_path,message,raw_object_key,raw_index FROM siem.events WHERE %s ORDER BY event_time DESC, ingest_time DESC LIMIT 1 BY event_uid LIMIT %d`, strings.Join(where, " AND "), limit) + ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second) + defer cancel() + rows, e := ch.QueryJSON(ctx, q) + if e != nil { + j(w, 500, map[string]string{"error": e.Error()}) + return + } + j(w, 200, rows) +} +func detections(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store) { + x, e := pg.ListDetections(r.Context(), cfg.TenantID, 500) + if e != nil { + j(w, 500, map[string]string{"error": e.Error()}) + return + } + j(w, 200, x) +} +func detectionStatus(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store) { + if r.Method != http.MethodPost { + j(w, 405, map[string]string{"error": "method not allowed"}) + return + } + var v struct { + ID int64 `json:"id"` + Status string `json:"status"` + } + if e := json.NewDecoder(r.Body).Decode(&v); e != nil { + j(w, 400, map[string]string{"error": "invalid json"}) + return + } + if e := pg.UpdateDetectionStatus(r.Context(), cfg.TenantID, v.ID, v.Status); e != nil { + j(w, 400, map[string]string{"error": e.Error()}) + return + } + j(w, 200, map[string]string{"status": "ok"}) +} +func j(w http.ResponseWriter, s int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(s) + _ = json.NewEncoder(w).Encode(v) +} +func basicAuth(next http.Handler, user, pass string) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/healthz" || r.URL.Path == "/readyz" { + next.ServeHTTP(w, r) + return + } + u, p, ok := r.BasicAuth() + if !ok || subtle.ConstantTimeCompare([]byte(u), []byte(user)) != 1 || subtle.ConstantTimeCompare([]byte(p), []byte(pass)) != 1 { + w.Header().Set("WWW-Authenticate", `Basic realm="Greenfield SIEM", charset="UTF-8"`) + http.Error(w, "Unauthorized", http.StatusUnauthorized) + return + } + next.ServeHTTP(w, r) + }) +} + +func security(n http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("X-Frame-Options", "DENY") + w.Header().Set("Referrer-Policy", "no-referrer") + w.Header().Set("Content-Security-Policy", "default-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'") + n.ServeHTTP(w, r) + }) +} diff --git a/internal/clickhouse/client.go b/internal/clickhouse/client.go new file mode 100644 index 0000000..0eed875 --- /dev/null +++ b/internal/clickhouse/client.go @@ -0,0 +1,98 @@ +package clickhouse + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "time" + + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/contracts" +) + +type Client struct { + base, user, pass string + hc *http.Client +} + +func New(cfg config.Config) *Client { + return &Client{base: cfg.ClickHouseURL, user: cfg.ClickHouseUser, pass: cfg.ClickHousePassword, hc: &http.Client{Timeout: 30 * time.Second}} +} +func (c *Client) request(ctx context.Context, query string, body io.Reader) (*http.Response, error) { + u := c.base + "/?query=" + url.QueryEscape(query) + req, e := http.NewRequestWithContext(ctx, http.MethodPost, u, body) + if e != nil { + return nil, e + } + req.SetBasicAuth(c.user, c.pass) + return c.hc.Do(req) +} +func (c *Client) Exec(ctx context.Context, q string) error { + r, e := c.request(ctx, q, nil) + if e != nil { + return e + } + defer r.Body.Close() + if r.StatusCode/100 != 2 { + b, _ := io.ReadAll(io.LimitReader(r.Body, 8192)) + return fmt.Errorf("clickhouse %s: %s", r.Status, string(b)) + } + return nil +} +func (c *Client) InsertEvents(ctx context.Context, events []contracts.CanonicalEvent) error { + if len(events) == 0 { + return nil + } + var b bytes.Buffer + enc := json.NewEncoder(&b) + enc.SetEscapeHTML(false) + for i := range events { + if e := enc.Encode(events[i]); e != nil { + return e + } + } + q := `INSERT INTO siem.events FORMAT JSONEachRow` + r, e := c.request(ctx, q, &b) + if e != nil { + return e + } + defer r.Body.Close() + if r.StatusCode/100 != 2 { + x, _ := io.ReadAll(io.LimitReader(r.Body, 16384)) + return fmt.Errorf("clickhouse insert %s: %s", r.Status, string(x)) + } + return nil +} +func (c *Client) QueryJSON(ctx context.Context, q string) ([]map[string]any, error) { + if !strings.Contains(strings.ToUpper(q), "FORMAT") { + q += " FORMAT JSONEachRow" + } + r, e := c.request(ctx, q, nil) + if e != nil { + return nil, e + } + defer r.Body.Close() + if r.StatusCode/100 != 2 { + x, _ := io.ReadAll(io.LimitReader(r.Body, 16384)) + return nil, fmt.Errorf("clickhouse query %s: %s", r.Status, string(x)) + } + var out []map[string]any + s := bufio.NewScanner(r.Body) + buf := make([]byte, 0, 64*1024) + s.Buffer(buf, 4*1024*1024) + for s.Scan() { + var m map[string]any + if e := json.Unmarshal(s.Bytes(), &m); e != nil { + return nil, e + } + out = append(out, m) + } + return out, s.Err() +} +func Q(s string) string { return "'" + strings.ReplaceAll(s, "'", "''") + "'" } diff --git a/internal/config/config.go b/internal/config/config.go new file mode 100644 index 0000000..8066560 --- /dev/null +++ b/internal/config/config.go @@ -0,0 +1,103 @@ +package config + +import ( + "os" + "strconv" + "strings" + "time" +) + +type Config struct { + ServiceAddr string + PostgresURL string + ClickHouseURL string + ClickHouseUser string + ClickHousePassword string + KafkaBrokers []string + KafkaTopic string + KafkaGroup string + EnrollmentKey string + TenantID string + MaxBodyBytes int64 + MaxBatchEvents int + RawArchiveEnabled bool + RawSpoolDir string + RawRetention time.Duration + EventRetention time.Duration + DetectorInterval time.Duration + DetectorLookback time.Duration + UIQueryLimit int + UIUsername string + UIPassword string +} + +func Load() Config { + return Config{ + ServiceAddr: env("SERVICE_ADDR", ":8080"), + PostgresURL: env("POSTGRES_URL", "postgres://siem:siem@postgres:5432/siem?sslmode=disable"), + ClickHouseURL: strings.TrimRight(env("CLICKHOUSE_URL", "http://clickhouse:8123"), "/"), + ClickHouseUser: env("CLICKHOUSE_USER", "siem"), + ClickHousePassword: env("CLICKHOUSE_PASSWORD", "siem"), + KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "redpanda:9092")), + KafkaTopic: env("KAFKA_TOPIC", "siem-events"), + KafkaGroup: env("KAFKA_GROUP", "siem-processor-v1"), + EnrollmentKey: env("ENROLLMENT_KEY", "change-me"), + TenantID: env("TENANT_ID", "default"), + MaxBodyBytes: int64(envInt("MAX_BODY_BYTES", 8*1024*1024)), + MaxBatchEvents: envInt("MAX_BATCH_EVENTS", 1000), + RawArchiveEnabled: envBool("RAW_ARCHIVE_ENABLED", true), + RawSpoolDir: env("RAW_SPOOL_DIR", "/var/spool/siem-raw"), + RawRetention: envDuration("RAW_RETENTION", 30*24*time.Hour), + EventRetention: envDuration("EVENT_RETENTION", 90*24*time.Hour), + DetectorInterval: envDuration("DETECTOR_INTERVAL", 30*time.Second), + DetectorLookback: envDuration("DETECTOR_LOOKBACK", 20*time.Minute), + UIQueryLimit: envInt("UI_QUERY_LIMIT", 500), + UIUsername: env("UI_USERNAME", "admin"), + UIPassword: env("UI_PASSWORD", "change-me"), + } +} + +func env(k, d string) string { + if v := strings.TrimSpace(os.Getenv(k)); v != "" { + return v + } + return d +} +func envInt(k string, d int) int { + v := strings.TrimSpace(os.Getenv(k)) + if v == "" { + return d + } + n, e := strconv.Atoi(v) + if e != nil { + return d + } + return n +} +func envBool(k string, d bool) bool { + v := strings.ToLower(strings.TrimSpace(os.Getenv(k))) + if v == "" { + return d + } + return v == "1" || v == "true" || v == "yes" || v == "on" +} +func envDuration(k string, d time.Duration) time.Duration { + v := strings.TrimSpace(os.Getenv(k)) + if v == "" { + return d + } + x, e := time.ParseDuration(v) + if e != nil { + return d + } + return x +} +func splitCSV(v string) []string { + var out []string + for _, p := range strings.Split(v, ",") { + if p = strings.TrimSpace(p); p != "" { + out = append(out, p) + } + } + return out +} diff --git a/internal/contracts/event.go b/internal/contracts/event.go new file mode 100644 index 0000000..24bcc4e --- /dev/null +++ b/internal/contracts/event.go @@ -0,0 +1,94 @@ +package contracts + +import "time" + +type EventMetadataPayload struct { + Computer string `json:"computer,omitempty"` + ProviderName string `json:"provider,omitempty"` + TargetUser string `json:"target_user,omitempty"` + TargetDomain string `json:"target_domain,omitempty"` + SubjectUser string `json:"subject_user,omitempty"` + SubjectDomain string `json:"subject_domain,omitempty"` + Workstation string `json:"workstation,omitempty"` + Device string `json:"device,omitempty"` + SrcIP string `json:"src_ip,omitempty"` + SrcPort string `json:"src_port,omitempty"` + DstIP string `json:"dst_ip,omitempty"` + DstPort string `json:"dst_port,omitempty"` + LogonType string `json:"logon_type,omitempty"` + ProcessName string `json:"process_name,omitempty"` + ParentProcessName string `json:"parent_process_name,omitempty"` + CommandLine string `json:"command_line,omitempty"` + AuthenticationPackage string `json:"authentication_package,omitempty"` + LogonProcess string `json:"logon_process,omitempty"` + StatusText string `json:"status,omitempty"` + SubStatusText string `json:"sub_status,omitempty"` + FailureReason string `json:"failure_reason,omitempty"` +} + +type LogPayload struct { + Hostname string `json:"host"` + Channel string `json:"channel"` + EventID uint32 `json:"id"` + Source string `json:"source"` + Time time.Time `json:"ts"` + Message string `json:"msg,omitempty"` + Metadata *EventMetadataPayload `json:"meta,omitempty"` +} + +type IngestEnvelope struct { + Version int `json:"version"` + TenantID string `json:"tenant_id"` + AgentID string `json:"agent_id"` + BatchUID string `json:"batch_uid"` + RemoteIP string `json:"remote_ip"` + ReceivedAt time.Time `json:"received_at"` + Events []LogPayload `json:"events"` +} + +type CanonicalEvent struct { + EventUID string `json:"event_uid"` + QueuePartition int32 `json:"queue_partition"` + QueueOffset int64 `json:"queue_offset"` + TenantID string `json:"tenant_id"` + EventTime string `json:"event_time"` + IngestTime string `json:"ingest_time"` + AgentID string `json:"agent_id"` + HostName string `json:"host_name"` + SourceType string `json:"source_type"` + Channel string `json:"channel"` + Provider string `json:"provider"` + EventCode uint32 `json:"event_code"` + Category string `json:"category"` + Action string `json:"action"` + Outcome string `json:"outcome"` + Severity uint8 `json:"severity"` + UserName string `json:"user_name"` + UserDomain string `json:"user_domain"` + SubjectUser string `json:"subject_user"` + SubjectDomain string `json:"subject_domain"` + TargetUser string `json:"target_user"` + TargetDomain string `json:"target_domain"` + SourceIP string `json:"source_ip"` + SourcePort uint16 `json:"source_port"` + DestinationIP string `json:"destination_ip"` + DestinationPort uint16 `json:"destination_port"` + Workstation string `json:"workstation"` + LogonType string `json:"logon_type"` + AuthenticationPackage string `json:"authentication_package"` + LogonProcess string `json:"logon_process"` + StatusCode string `json:"status_code"` + SubStatusCode string `json:"sub_status_code"` + FailureReason string `json:"failure_reason"` + ProcessPath string `json:"process_path"` + ParentProcessPath string `json:"parent_process_path"` + CommandLine string `json:"command_line"` + Message string `json:"message"` + Attributes map[string]string `json:"attributes"` + RawObjectKey string `json:"raw_object_key"` + RawIndex uint32 `json:"raw_index"` + PayloadHash string `json:"payload_hash"` + SchemaVersion uint16 `json:"schema_version"` + ParserVersion uint16 `json:"parser_version"` + IngestDelayMS int64 `json:"ingest_delay_ms"` +} diff --git a/internal/detector/detector.go b/internal/detector/detector.go new file mode 100644 index 0000000..b2ef709 --- /dev/null +++ b/internal/detector/detector.go @@ -0,0 +1,152 @@ +package detector + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "log" + "strconv" + "strings" + "time" + + "example.com/siem-greenfield/internal/clickhouse" + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/postgres" +) + +type rule struct { + name, severity string + eventCode uint32 + score float64 + query func(time.Time, time.Time, string) string + summary func(map[string]any) string +} + +func Run(ctx context.Context, cfg config.Config) error { + pg, e := postgres.Open(ctx, cfg.PostgresURL) + if e != nil { + return e + } + defer pg.Close() + ch := clickhouse.New(cfg) + ticker := time.NewTicker(cfg.DetectorInterval) + defer ticker.Stop() + run := func() { + if e := runAll(ctx, cfg, pg, ch); e != nil { + log.Printf("detector cycle: %v", e) + } + } + run() + for { + select { + case <-ctx.Done(): + return nil + case <-ticker.C: + run() + } + } +} +func runAll(ctx context.Context, cfg config.Config, pg *postgres.Store, ch *clickhouse.Client) error { + end := time.Now().UTC() + start := end.Add(-cfg.DetectorLookback) + for _, r := range rules() { + q := r.query(start, end, cfg.TenantID) + rows, e := ch.QueryJSON(ctx, q) + if e != nil { + return fmt.Errorf("%s: %w", r.name, e) + } + for _, row := range rows { + host := str(row["host_name"]) + user := str(row["user_name"]) + ip := str(row["source_ip"]) + workstation := str(row["workstation"]) + count := int64(num(row["cnt"])) + ws := timeVal(row["window_start"], start) + we := timeVal(row["window_end"], end) + fp := fingerprint(r.name, host, user, ip, workstation, strconv.FormatInt(ws.Unix()/300, 10)) + d := postgres.Detection{Fingerprint: fp, RuleName: r.name, Severity: r.severity, Hostname: host, UserName: user, SourceIP: ip, Workstation: workstation, EventCode: r.eventCode, Score: r.score, WindowStart: ws, WindowEnd: we, Summary: r.summary(row), Count: max64(1, count)} + if e := pg.UpsertDetection(ctx, d, cfg.TenantID); e != nil { + return e + } + } + } + return nil +} +func rules() []rule { + return []rule{ + {name: "audit_log_cleared", severity: "critical", eventCode: 1102, score: 9.8, query: simpleEvent(1102, 1), summary: func(m map[string]any) string { + return fmt.Sprintf("Audit-Log auf %s wurde gelöscht", str(m["host_name"])) + }}, + {name: "service_installed", severity: "high", eventCode: 7045, score: 8.0, query: simpleEvent(7045, 1), summary: func(m map[string]any) string { + return fmt.Sprintf("Neuer Dienst auf %s installiert", str(m["host_name"])) + }}, + {name: "account_lockout", severity: "medium", eventCode: 4740, score: 5.5, query: func(s, e time.Time, t string) string { + return fmt.Sprintf(`SELECT host_name, target_user AS user_name, '' AS source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4740 GROUP BY host_name,user_name,workstation HAVING cnt>=1`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e))) + }, summary: func(m map[string]any) string { + return fmt.Sprintf("Account-Lockout: %s; Caller %s; DC/Host %s (%d×)", str(m["user_name"]), fallback(str(m["workstation"]), "unbekannt"), str(m["host_name"]), int64(num(m["cnt"]))) + }}, + {name: "failed_logon_burst", severity: "high", eventCode: 4625, score: 7.5, query: func(s, e time.Time, t string) string { + return fmt.Sprintf(`SELECT host_name, target_user AS user_name, source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4625 AND target_user!='' GROUP BY host_name,user_name,source_ip,workstation HAVING cnt>=20`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e))) + }, summary: func(m map[string]any) string { + return fmt.Sprintf("%d fehlgeschlagene Logons für %s auf %s", int64(num(m["cnt"])), str(m["user_name"]), str(m["host_name"])) + }}, + {name: "password_spray", severity: "high", eventCode: 4625, score: 8.5, query: func(s, e time.Time, t string) string { + return fmt.Sprintf(`SELECT '' AS host_name, '' AS user_name, source_ip, '' AS workstation, uniqExact(target_user) users, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4625 AND source_ip!='' AND target_user!='' GROUP BY source_ip HAVING users>=10 AND cnt>=20`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e))) + }, summary: func(m map[string]any) string { + return fmt.Sprintf("Password-Spray von %s gegen %.0f Benutzer (%d Versuche)", str(m["source_ip"]), num(m["users"]), int64(num(m["cnt"]))) + }}, + {name: "privileged_group_change", severity: "critical", eventCode: 4728, score: 9.2, query: func(s, e time.Time, t string) string { + return fmt.Sprintf(`SELECT host_name, target_user AS user_name, '' AS source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code IN (4728,4732,4756) GROUP BY host_name,user_name,workstation HAVING cnt>=1`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e))) + }, summary: func(m map[string]any) string { + return fmt.Sprintf("Privilegierte Gruppenmitgliedschaft geändert: %s auf %s", str(m["user_name"]), str(m["host_name"])) + }}, + } +} +func simpleEvent(id uint32, min int) func(time.Time, time.Time, string) string { + return func(s, e time.Time, t string) string { + return fmt.Sprintf(`SELECT host_name, '' AS user_name, '' AS source_ip, '' AS workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=%d GROUP BY host_name HAVING cnt>=%d`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)), id, min) + } +} +func ts(t time.Time) string { return t.UTC().Format("2006-01-02 15:04:05.000") } +func str(v any) string { + if v == nil { + return "" + } + return fmt.Sprint(v) +} +func num(v any) float64 { + switch x := v.(type) { + case float64: + return x + case jsonNumber: + return x.Float() + default: + f, _ := strconv.ParseFloat(fmt.Sprint(v), 64) + return f + } +} + +type jsonNumber string + +func (n jsonNumber) Float() float64 { f, _ := strconv.ParseFloat(string(n), 64); return f } +func timeVal(v any, d time.Time) time.Time { + s := str(v) + for _, layout := range []string{"2006-01-02 15:04:05.999999", "2006-01-02 15:04:05", "2006-01-02T15:04:05Z07:00"} { + if t, e := time.Parse(layout, s); e == nil { + return t.UTC() + } + } + return d +} +func fingerprint(v ...string) string { + h := sha256.Sum256([]byte(strings.Join(v, "|"))) + return hex.EncodeToString(h[:]) +} +func fallback(v, d string) string { if strings.TrimSpace(v) == "" { return d }; return v } +func max64(a, b int64) int64 { + if a > b { + return a + } + return b +} diff --git a/internal/ingress/server.go b/internal/ingress/server.go new file mode 100644 index 0000000..64b3ea5 --- /dev/null +++ b/internal/ingress/server.go @@ -0,0 +1,178 @@ +package ingress + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "net/http" + "strings" + "time" + + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/contracts" + "example.com/siem-greenfield/internal/metrics" + "example.com/siem-greenfield/internal/postgres" + "example.com/siem-greenfield/internal/queue" +) + +func Run(ctx context.Context, cfg config.Config) error { + pg, err := postgres.Open(ctx, cfg.PostgresURL) + if err != nil { + return fmt.Errorf("postgres: %w", err) + } + defer pg.Close() + prod := queue.NewProducer(cfg.KafkaBrokers, cfg.KafkaTopic) + defer prod.Close() + m := metrics.New() + accepted := m.Counter("siem_ingress_events_accepted_total") + rejected := m.Counter("siem_ingress_requests_rejected_total") + batches := m.Counter("siem_ingress_batches_total") + mux := http.NewServeMux() + mux.Handle("/metrics", m.Handler()) + mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, 200, map[string]string{"status": "ok"}) }) + mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) { + cctx, cancel := context.WithTimeout(r.Context(), 2*time.Second) + defer cancel() + if e := pg.Pool.Ping(cctx); e != nil { + writeJSON(w, 503, map[string]string{"status": "not_ready"}) + return + } + writeJSON(w, 200, map[string]string{"status": "ready"}) + }) + mux.HandleFunc("/ingest", func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + rejected.Add(1) + writeJSON(w, 405, map[string]string{"error": "method not allowed"}) + return + } + apiKey := strings.TrimSpace(r.Header.Get("X-API-Key")) + if apiKey == "" { + rejected.Add(1) + writeJSON(w, 401, map[string]string{"error": "missing api key"}) + return + } + r.Body = http.MaxBytesReader(w, r.Body, cfg.MaxBodyBytes) + defer r.Body.Close() + dec := json.NewDecoder(r.Body) + dec.DisallowUnknownFields() + var batch []contracts.LogPayload + if e := dec.Decode(&batch); e != nil || (func() bool { var x any; return dec.Decode(&x) != io.EOF })() { + rejected.Add(1) + writeJSON(w, 400, map[string]string{"error": "invalid json"}) + return + } + if len(batch) == 0 || len(batch) > cfg.MaxBatchEvents { + rejected.Add(1) + writeJSON(w, 400, map[string]string{"error": "invalid batch size"}) + return + } + host := strings.TrimSpace(batch[0].Hostname) + for i := range batch { + if e := validate(&batch[i]); e != nil { + rejected.Add(1) + writeJSON(w, 400, map[string]string{"error": fmt.Sprintf("invalid payload at index %d: %v", i, e)}) + return + } + if batch[i].Hostname != host { + rejected.Add(1) + writeJSON(w, 400, map[string]string{"error": "all events in a batch must use the same hostname"}) + return + } + } + cctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) + defer cancel() + agentID, e := pg.AuthenticateOrEnroll(cctx, cfg.TenantID, host, apiKey, strings.TrimSpace(r.Header.Get("X-Enrollment-Key")), cfg.EnrollmentKey, postgres.ClientIP(r.RemoteAddr)) + if e != nil { + rejected.Add(1) + if errors.Is(e, postgres.ErrUnauthorized) { + writeJSON(w, 401, map[string]string{"error": "invalid api key or hostname"}) + } else { + log.Printf("auth: %v", e) + writeJSON(w, 503, map[string]string{"error": "control plane unavailable"}) + } + return + } + env := contracts.IngestEnvelope{Version: 1, TenantID: cfg.TenantID, AgentID: agentID, BatchUID: batchUID(agentID, batch), RemoteIP: postgres.ClientIP(r.RemoteAddr), ReceivedAt: time.Now().UTC(), Events: batch} + payload, e := json.Marshal(env) + if e != nil { + writeJSON(w, 500, map[string]string{"error": "internal error"}) + return + } + qctx, qcancel := context.WithTimeout(r.Context(), 8*time.Second) + defer qcancel() + if e = prod.Write(qctx, agentID, payload); e != nil { + log.Printf("queue write: %v", e) + writeJSON(w, 503, map[string]string{"error": "ingest queue unavailable"}) + return + } + accepted.Add(uint64(len(batch))) + batches.Add(1) + writeJSON(w, 202, map[string]int{"accepted": len(batch)}) + }) + srv := &http.Server{Addr: cfg.ServiceAddr, Handler: withLimits(mux), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second} + go func() { + <-ctx.Done() + cctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = srv.Shutdown(cctx) + }() + log.Printf("ingress listening on %s", cfg.ServiceAddr) + e := srv.ListenAndServe() + if e == http.ErrServerClosed { + return nil + } + return e +} +func batchUID(agentID string, batch []contracts.LogPayload) string { + b, _ := json.Marshal(batch) + h := sha256.New() + _, _ = h.Write([]byte(agentID)) + _, _ = h.Write([]byte{0}) + _, _ = h.Write(b) + return hex.EncodeToString(h.Sum(nil)) +} + +func validate(p *contracts.LogPayload) error { + p.Hostname = strings.TrimSpace(p.Hostname) + p.Channel = strings.TrimSpace(p.Channel) + p.Source = strings.TrimSpace(p.Source) + if p.Hostname == "" || len(p.Hostname) > 255 { + return fmt.Errorf("invalid host") + } + if p.Channel == "" || len(p.Channel) > 128 { + return fmt.Errorf("invalid channel") + } + if p.Source == "" || len(p.Source) > 255 { + return fmt.Errorf("invalid source") + } + if p.EventID == 0 { + return fmt.Errorf("event id required") + } + if p.Time.IsZero() { + return fmt.Errorf("ts required") + } + if strings.TrimSpace(p.Message) == "" && p.Metadata == nil { + return fmt.Errorf("either msg or meta required") + } + if len(p.Message) > 2*1024*1024 { + return fmt.Errorf("msg too large") + } + return nil +} +func writeJSON(w http.ResponseWriter, status int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(v) +} +func withLimits(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("Cache-Control", "no-store") + next.ServeHTTP(w, r) + }) +} diff --git a/internal/ingress/server_test.go b/internal/ingress/server_test.go new file mode 100644 index 0000000..9c8938b --- /dev/null +++ b/internal/ingress/server_test.go @@ -0,0 +1,29 @@ +package ingress + +import ( + "testing" + "time" + + "example.com/siem-greenfield/internal/contracts" +) + +func TestValidateCompatiblePayload(t *testing.T) { + p := contracts.LogPayload{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Now().UTC(), Metadata: &contracts.EventMetadataPayload{TargetUser: "alice"}} + if err := validate(&p); err != nil { + t.Fatalf("valid payload rejected: %v", err) + } +} + +func TestValidateRequiresMessageOrMetadata(t *testing.T) { + p := contracts.LogPayload{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Now().UTC()} + if err := validate(&p); err == nil { + t.Fatal("expected validation error") + } +} + +func TestBatchUIDIsDeterministic(t *testing.T) { + b := []contracts.LogPayload{{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Date(2026,7,23,12,0,0,0,time.UTC), Metadata: &contracts.EventMetadataPayload{TargetUser: "alice"}}} + a := batchUID("agent-1", b) + if a == "" || a != batchUID("agent-1", b) { t.Fatalf("batch uid is not deterministic") } + if a == batchUID("agent-2", b) { t.Fatalf("batch uid must be scoped to agent") } +} diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go new file mode 100644 index 0000000..cba5b46 --- /dev/null +++ b/internal/metrics/metrics.go @@ -0,0 +1,67 @@ +package metrics + +import ( + "fmt" + "net/http" + "sort" + "strings" + "sync" + "sync/atomic" +) + +type Registry struct { + mu sync.RWMutex + counters map[string]*atomic.Uint64 + gauges map[string]*atomic.Int64 +} + +func New() *Registry { + return &Registry{counters: map[string]*atomic.Uint64{}, gauges: map[string]*atomic.Int64{}} +} +func (r *Registry) Counter(name string) *atomic.Uint64 { + r.mu.Lock() + defer r.mu.Unlock() + if c := r.counters[name]; c != nil { + return c + } + c := &atomic.Uint64{} + r.counters[name] = c + return c +} +func (r *Registry) Gauge(name string) *atomic.Int64 { + r.mu.Lock() + defer r.mu.Unlock() + if g := r.gauges[name]; g != nil { + return g + } + g := &atomic.Int64{} + r.gauges[name] = g + return g +} +func (r *Registry) Handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain; version=0.0.4") + r.mu.RLock() + defer r.mu.RUnlock() + var lines []string + for n, c := range r.counters { + lines = append(lines, fmt.Sprintf("%s %d", sanitize(n), c.Load())) + } + for n, g := range r.gauges { + lines = append(lines, fmt.Sprintf("%s %d", sanitize(n), g.Load())) + } + sort.Strings(lines) + fmt.Fprintln(w, strings.Join(lines, "\n")) + }) +} +func sanitize(s string) string { + var b strings.Builder + for i, r := range s { + if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9' && i > 0) || r == '_' || r == ':' { + b.WriteRune(r) + } else { + b.WriteByte('_') + } + } + return b.String() +} diff --git a/internal/normalize/normalize.go b/internal/normalize/normalize.go new file mode 100644 index 0000000..50f3a48 --- /dev/null +++ b/internal/normalize/normalize.go @@ -0,0 +1,297 @@ +package normalize + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "encoding/xml" + "io" + "net" + "strconv" + "strings" + "time" + + "example.com/siem-greenfield/internal/contracts" +) + +type Parsed struct { + Computer, Provider, TargetUser, TargetDomain, SubjectUser, SubjectDomain, Workstation, SrcIP, SrcPort, DstIP, DstPort, LogonType, ProcessName, ParentProcessName, CommandLine, AuthPackage, LogonProcess, Status, SubStatus, FailureReason string + Attributes map[string]string +} + +func Event(env contracts.IngestEnvelope, p contracts.LogPayload, rawKey string, partition int, offset int64, idx int) contracts.CanonicalEvent { + parsed := parseXML(p.Message) + if p.Metadata != nil { + overlay(&parsed, *p.Metadata) + } + host := first(parsed.Computer, p.Hostname) + provider := first(parsed.Provider, p.Source) + cat, action, outcome, sev := classify(p.Channel, p.EventID) + user := first(parsed.TargetUser, parsed.SubjectUser) + msg := strings.TrimSpace(p.Message) + if strings.HasPrefix(msg, "<") { + msg = "" + } + if len(msg) > 4096 { + msg = msg[:4096] + } + attrs := parsed.Attributes + if attrs == nil { + attrs = map[string]string{} + } + eventUID := env.BatchUID + if eventUID == "" { + eventUID = strconv.Itoa(partition) + ":" + strconv.FormatInt(offset, 10) + } + eventUID += ":" + strconv.Itoa(idx) + return contracts.CanonicalEvent{ + EventUID: eventUID, QueuePartition: int32(partition), QueueOffset: offset, + TenantID: env.TenantID, EventTime: p.Time.UTC().Format("2006-01-02 15:04:05.000"), IngestTime: env.ReceivedAt.UTC().Format("2006-01-02 15:04:05.000"), AgentID: env.AgentID, HostName: host, SourceType: p.Source, Channel: p.Channel, Provider: provider, EventCode: p.EventID, Category: cat, Action: action, Outcome: outcome, Severity: sev, + UserName: cleanUser(user), UserDomain: first(parsed.TargetDomain, parsed.SubjectDomain), SubjectUser: cleanUser(parsed.SubjectUser), SubjectDomain: parsed.SubjectDomain, TargetUser: cleanUser(parsed.TargetUser), TargetDomain: parsed.TargetDomain, SourceIP: normalizeIP(parsed.SrcIP), SourcePort: parsePort(parsed.SrcPort), DestinationIP: normalizeIP(parsed.DstIP), DestinationPort: parsePort(parsed.DstPort), Workstation: parsed.Workstation, LogonType: parsed.LogonType, AuthenticationPackage: parsed.AuthPackage, LogonProcess: parsed.LogonProcess, StatusCode: parsed.Status, SubStatusCode: parsed.SubStatus, FailureReason: parsed.FailureReason, ProcessPath: parsed.ProcessName, ParentProcessPath: parsed.ParentProcessName, CommandLine: truncate(parsed.CommandLine, 8192), Message: msg, Attributes: attrs, RawObjectKey: rawKey, RawIndex: uint32(idx), PayloadHash: fingerprint(p), SchemaVersion: 1, ParserVersion: 1, IngestDelayMS: env.ReceivedAt.Sub(p.Time).Milliseconds(), + } +} + +func overlay(p *Parsed, m contracts.EventMetadataPayload) { + p.Computer = first(m.Computer, p.Computer) + p.Provider = first(m.ProviderName, p.Provider) + p.TargetUser = first(m.TargetUser, p.TargetUser) + p.TargetDomain = first(m.TargetDomain, p.TargetDomain) + p.SubjectUser = first(m.SubjectUser, p.SubjectUser) + p.SubjectDomain = first(m.SubjectDomain, p.SubjectDomain) + p.Workstation = first(m.Workstation, m.Device, p.Workstation) + p.SrcIP = first(m.SrcIP, p.SrcIP) + p.SrcPort = first(m.SrcPort, p.SrcPort) + p.DstIP = first(m.DstIP, p.DstIP) + p.DstPort = first(m.DstPort, p.DstPort) + p.LogonType = first(m.LogonType, p.LogonType) + p.ProcessName = first(m.ProcessName, p.ProcessName) + p.ParentProcessName = first(m.ParentProcessName, p.ParentProcessName) + p.CommandLine = first(m.CommandLine, p.CommandLine) + p.AuthPackage = first(m.AuthenticationPackage, p.AuthPackage) + p.LogonProcess = first(m.LogonProcess, p.LogonProcess) + p.Status = first(m.StatusText, p.Status) + p.SubStatus = first(m.SubStatusText, p.SubStatus) + p.FailureReason = first(m.FailureReason, p.FailureReason) +} + +func parseXML(s string) Parsed { + out := Parsed{Attributes: map[string]string{}} + if !strings.HasPrefix(strings.TrimSpace(s), "<") { + return out + } + dec := xml.NewDecoder(strings.NewReader(s)) + var path []string + dataName := "" + for { + tok, e := dec.Token() + if e == io.EOF { + break + } + if e != nil { + return out + } + switch t := tok.(type) { + case xml.StartElement: + path = append(path, t.Name.Local) + if t.Name.Local == "Provider" { + for _, a := range t.Attr { + if a.Name.Local == "Name" { + out.Provider = strings.TrimSpace(a.Value) + } + } + } + if t.Name.Local == "Data" { + dataName = "" + for _, a := range t.Attr { + if a.Name.Local == "Name" { + dataName = strings.TrimSpace(a.Value) + } + } + } + case xml.EndElement: + if len(path) > 0 { + path = path[:len(path)-1] + } + if t.Name.Local == "Data" { + dataName = "" + } + case xml.CharData: + v := strings.TrimSpace(string(t)) + if v == "" { + continue + } + if ends(path, "System", "Computer") { + out.Computer = v + continue + } + if dataName != "" { + if !promotedField(dataName) && len(out.Attributes) < 32 { + out.Attributes[dataName] = truncate(v, 1024) + } + switch dataName { + case "TargetUserName": + out.TargetUser = v + case "TargetDomainName": + out.TargetDomain = v + case "SubjectUserName": + out.SubjectUser = v + case "SubjectDomainName": + out.SubjectDomain = v + case "WorkstationName", "CallerComputerName": + out.Workstation = v + case "IpAddress", "SourceAddress": + out.SrcIP = v + case "IpPort", "SourcePort": + out.SrcPort = v + case "DestinationAddress", "DestAddress": + out.DstIP = v + case "DestinationPort", "DestPort": + out.DstPort = v + case "LogonType": + out.LogonType = v + case "ProcessName", "NewProcessName": + out.ProcessName = v + case "ParentProcessName": + out.ParentProcessName = v + case "CommandLine", "ProcessCommandLine": + out.CommandLine = v + case "AuthenticationPackageName": + out.AuthPackage = v + case "LogonProcessName": + out.LogonProcess = v + case "Status": + out.Status = v + case "SubStatus": + out.SubStatus = v + case "FailureReason": + out.FailureReason = v + } + } + } + } + return out +} +func promotedField(name string) bool { + switch name { + case "TargetUserName", "TargetDomainName", "SubjectUserName", "SubjectDomainName", + "WorkstationName", "CallerComputerName", "IpAddress", "SourceAddress", "IpPort", "SourcePort", + "DestinationAddress", "DestAddress", "DestinationPort", "DestPort", "LogonType", + "ProcessName", "NewProcessName", "ParentProcessName", "CommandLine", "ProcessCommandLine", + "AuthenticationPackageName", "LogonProcessName", "Status", "SubStatus", "FailureReason": + return true + default: + return false + } +} + +func ends(path []string, p ...string) bool { + if len(path) < len(p) { + return false + } + o := len(path) - len(p) + for i := range p { + if path[o+i] != p[i] { + return false + } + } + return true +} +func first(v ...string) string { + for _, s := range v { + if s = strings.TrimSpace(s); s != "" && s != "-" { + return s + } + } + return "" +} +func cleanUser(v string) string { + v = strings.TrimSpace(v) + if v == "-" { + return "" + } + return v +} +func normalizeIP(v string) string { + v = strings.TrimSpace(v) + if v == "" || v == "-" || v == "::1" || v == "127.0.0.1" { + return "" + } + ip := net.ParseIP(v) + if ip == nil { + return "" + } + if x := ip.To4(); x != nil { + return x.String() + } + return ip.String() +} +func parsePort(v string) uint16 { + n, e := strconv.ParseUint(strings.TrimSpace(v), 10, 16) + if e != nil { + return 0 + } + return uint16(n) +} +func truncate(v string, n int) string { + if len(v) > n { + return v[:n] + } + return v +} +func fingerprint(p contracts.LogPayload) string { + b, _ := json.Marshal(p) + h := sha256.Sum256(b) + return hex.EncodeToString(h[:]) +} + +func classify(channel string, id uint32) (string, string, string, uint8) { + if strings.EqualFold(channel, "Security") { + switch id { + case 4624: + return "authentication", "logon", "success", 2 + case 4625: + return "authentication", "logon", "failure", 3 + case 4648: + return "authentication", "explicit_credentials", "unknown", 3 + case 4672: + return "iam", "special_privileges", "success", 3 + case 4720: + return "iam", "user_created", "success", 4 + case 4726: + return "iam", "user_deleted", "success", 4 + case 4728, 4732, 4756: + return "iam", "privileged_group_membership_changed", "success", 5 + case 4740: + return "authentication", "account_locked", "failure", 4 + case 4768: + return "authentication", "kerberos_tgt", "unknown", 2 + case 4769: + return "authentication", "kerberos_service_ticket", "unknown", 2 + case 4771: + return "authentication", "kerberos_preauth", "failure", 3 + case 4776: + return "authentication", "credential_validation", "unknown", 2 + case 1102: + return "audit", "audit_log_cleared", "success", 5 + } + } + switch id { + case 7045: + return "persistence", "service_installed", "success", 5 + case 1074: + return "host", "shutdown_requested", "success", 2 + case 6005: + return "host", "eventlog_started", "success", 1 + case 6006: + return "host", "eventlog_stopped", "success", 2 + } + return "event", "observed", "unknown", 1 +} + +func ArchiveKey(t time.Time, agent, batchUID string, partition int, offset int64) string { + id := strings.TrimSpace(batchUID) + if id == "" { + id = "p" + strconv.Itoa(partition) + "-o" + strconv.FormatInt(offset, 10) + } + return t.UTC().Format("2006/01/02/15") + "/" + strings.ReplaceAll(agent, "/", "_") + "-" + id + ".json.gz" +} diff --git a/internal/normalize/normalize_test.go b/internal/normalize/normalize_test.go new file mode 100644 index 0000000..46197a3 --- /dev/null +++ b/internal/normalize/normalize_test.go @@ -0,0 +1,37 @@ +package normalize + +import ( + "testing" + "time" + + "example.com/siem-greenfield/internal/contracts" +) + +func TestEvent4740ExtractsCallerComputer(t *testing.T) { + xml := `DC01.example.localaliceCLIENT-42` + env := contracts.IngestEnvelope{TenantID: "default", AgentID: "agent-1", BatchUID: "batch-abc", ReceivedAt: time.Date(2026, 7, 23, 12, 0, 1, 0, time.UTC)} + p := contracts.LogPayload{Hostname: "DC01", Channel: "Security", EventID: 4740, Source: "windows-agent", Time: time.Date(2026, 7, 23, 12, 0, 0, 0, time.UTC), Message: xml} + e := Event(env, p, "2026/07/raw.json.gz", 3, 99, 4) + if e.EventUID != "batch-abc:4" { + t.Fatalf("event uid = %q", e.EventUID) + } + if e.TargetUser != "alice" || e.Workstation != "CLIENT-42" { + t.Fatalf("unexpected normalized event: %#v", e) + } + if e.Action != "account_locked" || e.Severity != 4 { + t.Fatalf("unexpected classification: action=%s severity=%d", e.Action, e.Severity) + } + if e.Message != "" { + t.Fatalf("raw XML must not be stored as message") + } + if len(e.Attributes) != 0 { + t.Fatalf("promoted fields must not be duplicated in attributes: %#v", e.Attributes) + } +} + +func TestWMI5857IsGenericNotDetectionClass(t *testing.T) { + cat, action, _, sev := classify("Microsoft-Windows-WMI-Activity/Operational", 5857) + if cat != "event" || action != "observed" || sev != 1 { + t.Fatalf("unexpected WMI classification: %s %s %d", cat, action, sev) + } +} diff --git a/internal/postgres/store.go b/internal/postgres/store.go new file mode 100644 index 0000000..4a18f78 --- /dev/null +++ b/internal/postgres/store.go @@ -0,0 +1,184 @@ +package postgres + +import ( + "context" + "crypto/sha256" + "crypto/subtle" + "encoding/hex" + "errors" + "fmt" + "net" + "strings" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +var ErrUnauthorized = errors.New("unauthorized") + +type Store struct{ Pool *pgxpool.Pool } + +type Agent struct { + ID string `json:"id"` + Hostname string `json:"hostname"` + LastIP string `json:"last_ip"` + Enabled bool `json:"enabled"` + FirstSeen time.Time `json:"first_seen"` + LastSeen time.Time `json:"last_seen"` +} +type Detection struct { + ID int64 `json:"id"` + Fingerprint string `json:"fingerprint"` + RuleName string `json:"rule_name"` + Severity string `json:"severity"` + Status string `json:"status"` + Hostname string `json:"hostname"` + UserName string `json:"user_name"` + SourceIP string `json:"source_ip"` + Workstation string `json:"workstation"` + Summary string `json:"summary"` + EventCode uint32 `json:"event_code"` + Score float64 `json:"score"` + WindowStart time.Time `json:"window_start"` + WindowEnd time.Time `json:"window_end"` + FirstSeen time.Time `json:"first_seen"` + LastSeen time.Time `json:"last_seen"` + Count int64 `json:"count"` +} + +func Open(ctx context.Context, url string) (*Store, error) { + p, e := pgxpool.New(ctx, url) + if e != nil { + return nil, e + } + cctx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + if e = p.Ping(cctx); e != nil { + p.Close() + return nil, e + } + return &Store{Pool: p}, nil +} +func (s *Store) Close() { s.Pool.Close() } + +func (s *Store) AuthenticateOrEnroll(ctx context.Context, tenant, hostname, apiKey, enrollmentKey, expectedEnrollment, remoteIP string) (string, error) { + hostname = strings.TrimSpace(hostname) + apiKey = strings.TrimSpace(apiKey) + var id, hash string + var enabled bool + err := s.Pool.QueryRow(ctx, `SELECT id::text, api_key_hash, enabled FROM agents WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname).Scan(&id, &hash, &enabled) + if err != nil { + if !errors.Is(err, pgx.ErrNoRows) { + return "", err + } + if enrollmentKey == "" || !secureEqual(hashHex(enrollmentKey), hashHex(expectedEnrollment)) { + return "", ErrUnauthorized + } + err = s.Pool.QueryRow(ctx, `INSERT INTO agents(tenant_id,hostname,api_key_hash,last_ip) VALUES($1,$2,$3,$4) ON CONFLICT(tenant_id,hostname) DO NOTHING RETURNING id::text`, tenant, hostname, hashHex(apiKey), remoteIP).Scan(&id) + if err == nil { + return id, nil + } + if !errors.Is(err, pgx.ErrNoRows) { + return "", err + } + // A concurrent first request may have enrolled the host between SELECT and INSERT. + err = s.Pool.QueryRow(ctx, `SELECT id::text, api_key_hash, enabled FROM agents WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname).Scan(&id, &hash, &enabled) + if err != nil || !enabled || !secureEqual(strings.ToLower(hash), hashHex(apiKey)) { + return "", ErrUnauthorized + } + return id, nil + } + if !enabled || !secureEqual(strings.ToLower(hash), hashHex(apiKey)) { + return "", ErrUnauthorized + } + _, err = s.Pool.Exec(ctx, `UPDATE agents SET last_seen=now(), last_ip=$3 WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname, remoteIP) + return id, err +} + +func (s *Store) ListAgents(ctx context.Context, tenant string) ([]Agent, error) { + rows, e := s.Pool.Query(ctx, `SELECT id::text,hostname,enabled,COALESCE(last_ip,''),first_seen,last_seen FROM agents WHERE tenant_id=$1 ORDER BY last_seen DESC`, tenant) + if e != nil { + return nil, e + } + defer rows.Close() + var out []Agent + for rows.Next() { + var a Agent + if e := rows.Scan(&a.ID, &a.Hostname, &a.Enabled, &a.LastIP, &a.FirstSeen, &a.LastSeen); e != nil { + return nil, e + } + out = append(out, a) + } + return out, rows.Err() +} + +func (s *Store) UpsertDetection(ctx context.Context, d Detection, tenant string) error { + _, e := s.Pool.Exec(ctx, `INSERT INTO detections(tenant_id,fingerprint,rule_name,severity,status,hostname,user_name,source_ip,workstation,event_code,score,window_start,window_end,summary,hit_count,first_seen,last_seen) +VALUES($1,$2,$3,$4,'open',$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$11,$12) +ON CONFLICT(tenant_id,fingerprint) DO UPDATE SET + first_seen=LEAST(detections.first_seen,EXCLUDED.first_seen), + last_seen=GREATEST(detections.last_seen,EXCLUDED.last_seen), + window_start=LEAST(detections.window_start,EXCLUDED.window_start), + window_end=GREATEST(detections.window_end,EXCLUDED.window_end), + hit_count=GREATEST(detections.hit_count,EXCLUDED.hit_count), + score=GREATEST(detections.score,EXCLUDED.score), + summary=EXCLUDED.summary, workstation=EXCLUDED.workstation, updated_at=now()`, tenant, d.Fingerprint, d.RuleName, d.Severity, d.Hostname, d.UserName, d.SourceIP, d.Workstation, d.EventCode, d.Score, d.WindowStart, d.WindowEnd, d.Summary, d.Count) + return e +} + +func (s *Store) ListDetections(ctx context.Context, tenant string, limit int) ([]Detection, error) { + rows, e := s.Pool.Query(ctx, `SELECT id,fingerprint,rule_name,severity,status,hostname,user_name,source_ip,workstation,event_code,score,window_start,window_end,summary,hit_count,first_seen,last_seen FROM detections WHERE tenant_id=$1 ORDER BY last_seen DESC LIMIT $2`, tenant, limit) + if e != nil { + return nil, e + } + defer rows.Close() + var out []Detection + for rows.Next() { + var d Detection + if e := rows.Scan(&d.ID, &d.Fingerprint, &d.RuleName, &d.Severity, &d.Status, &d.Hostname, &d.UserName, &d.SourceIP, &d.Workstation, &d.EventCode, &d.Score, &d.WindowStart, &d.WindowEnd, &d.Summary, &d.Count, &d.FirstSeen, &d.LastSeen); e != nil { + return nil, e + } + out = append(out, d) + } + return out, rows.Err() +} +func (s *Store) UpdateDetectionStatus(ctx context.Context, tenant string, id int64, status string) error { + if status != "open" && status != "investigating" && status != "closed" && status != "false_positive" { + return fmt.Errorf("invalid status") + } + _, e := s.Pool.Exec(ctx, `UPDATE detections SET status=$3, updated_at=now() WHERE tenant_id=$1 AND id=$2`, tenant, id, status) + return e +} +func (s *Store) DetectionCounts(ctx context.Context, tenant string) (map[string]int64, error) { + rows, e := s.Pool.Query(ctx, `SELECT severity,count(*) FROM detections WHERE tenant_id=$1 AND status IN ('open','investigating') GROUP BY severity`, tenant) + if e != nil { + return nil, e + } + defer rows.Close() + out := map[string]int64{} + for rows.Next() { + var k string + var n int64 + if e := rows.Scan(&k, &n); e != nil { + return nil, e + } + out[k] = n + } + return out, rows.Err() +} + +func hashHex(v string) string { h := sha256.Sum256([]byte(v)); return hex.EncodeToString(h[:]) } +func secureEqual(a, b string) bool { + if len(a) != len(b) { + return false + } + return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1 +} +func ClientIP(remote string) string { + h, _, e := net.SplitHostPort(remote) + if e == nil { + return h + } + return remote +} diff --git a/internal/processor/processor.go b/internal/processor/processor.go new file mode 100644 index 0000000..a4be3fa --- /dev/null +++ b/internal/processor/processor.go @@ -0,0 +1,88 @@ +package processor + +import ( + "compress/gzip" + "context" + "encoding/json" + "fmt" + "log" + "os" + "path/filepath" + "time" + + "example.com/siem-greenfield/internal/clickhouse" + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/contracts" + "example.com/siem-greenfield/internal/normalize" + "example.com/siem-greenfield/internal/queue" +) + +func Run(ctx context.Context, cfg config.Config) error { + ch := clickhouse.New(cfg) + con := queue.NewConsumer(cfg.KafkaBrokers, cfg.KafkaTopic, cfg.KafkaGroup) + defer con.Close() + log.Printf("processor started topic=%s group=%s", cfg.KafkaTopic, cfg.KafkaGroup) + for { + m, e := con.Fetch(ctx) + if e != nil { + if ctx.Err() != nil { + return nil + } + return e + } + var env contracts.IngestEnvelope + if e = json.Unmarshal(m.Value, &env); e != nil { + log.Printf("dropping malformed queue message offset=%d: %v", m.Offset, e) + _ = con.Commit(ctx, m) + continue + } + rawKey := "" + if cfg.RawArchiveEnabled { + rawKey = normalize.ArchiveKey(env.ReceivedAt, env.AgentID, env.BatchUID, m.Partition, m.Offset) + if e = spool(cfg, rawKey, m.Value); e != nil { + log.Printf("raw spool failed offset=%d (message will be retried): %v", m.Offset, e) + time.Sleep(time.Second) + continue + } + } + events := make([]contracts.CanonicalEvent, 0, len(env.Events)) + for i, p := range env.Events { + events = append(events, normalize.Event(env, p, rawKey, m.Partition, m.Offset, i)) + } + cctx, cancel := context.WithTimeout(ctx, 30*time.Second) + e = ch.InsertEvents(cctx, events) + cancel() + if e != nil { + log.Printf("clickhouse insert failed offset=%d: %v", m.Offset, e) + time.Sleep(time.Second) + continue + } + if e = con.Commit(ctx, m); e != nil { + return fmt.Errorf("commit queue offset: %w", e) + } + } +} +func spool(cfg config.Config, key string, payload []byte) error { + path := filepath.Join(cfg.RawSpoolDir, filepath.FromSlash(key)) + if e := os.MkdirAll(filepath.Dir(path), 0750); e != nil { + return e + } + tmp := path + ".tmp" + f, e := os.Create(tmp) + if e != nil { + return e + } + gz := gzip.NewWriter(f) + _, e = gz.Write(payload) + if ce := gz.Close(); e == nil { + e = ce + } + if ce := f.Close(); e == nil { + e = ce + } + if e != nil { + _ = os.Remove(tmp) + return e + } + return os.Rename(tmp, path) +} diff --git a/internal/queue/queue.go b/internal/queue/queue.go new file mode 100644 index 0000000..d8748d6 --- /dev/null +++ b/internal/queue/queue.go @@ -0,0 +1,29 @@ +package queue + +import ( + "context" + "time" + + "github.com/segmentio/kafka-go" +) + +type Producer struct{ w *kafka.Writer } + +func NewProducer(brokers []string, topic string) *Producer { + return &Producer{w: &kafka.Writer{Addr: kafka.TCP(brokers...), Topic: topic, Balancer: &kafka.Hash{}, BatchSize: 100, BatchTimeout: 10 * time.Millisecond, RequiredAcks: kafka.RequireAll, Async: false}} +} +func (p *Producer) Write(ctx context.Context, key string, value []byte) error { + return p.w.WriteMessages(ctx, kafka.Message{Key: []byte(key), Value: value, Time: time.Now().UTC()}) +} +func (p *Producer) Close() error { return p.w.Close() } + +type Consumer struct{ r *kafka.Reader } + +func NewConsumer(brokers []string, topic, group string) *Consumer { + return &Consumer{r: kafka.NewReader(kafka.ReaderConfig{Brokers: brokers, Topic: topic, GroupID: group, MinBytes: 1e3, MaxBytes: 16e6, MaxWait: 500 * time.Millisecond, CommitInterval: 0})} +} +func (c *Consumer) Fetch(ctx context.Context) (kafka.Message, error) { return c.r.FetchMessage(ctx) } +func (c *Consumer) Commit(ctx context.Context, m kafka.Message) error { + return c.r.CommitMessages(ctx, m) +} +func (c *Consumer) Close() error { return c.r.Close() } diff --git a/internal/stress/stress.go b/internal/stress/stress.go new file mode 100644 index 0000000..929d057 --- /dev/null +++ b/internal/stress/stress.go @@ -0,0 +1,132 @@ +package stress + +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "math/rand/v2" + "net/http" + "strconv" + "sync" + "sync/atomic" + "time" + + "example.com/siem-greenfield/internal/config" + "example.com/siem-greenfield/internal/contracts" +) + +func Run(ctx context.Context, _ config.Config, args []string) error { + fs := flag.NewFlagSet("stress-agent", flag.ContinueOnError) + url := fs.String("url", "http://127.0.0.1:8090/ingest", "") + api := fs.String("api-key", "stress-agent-key", "") + enroll := fs.String("enrollment-key", "", "") + host := fs.String("host", "SIEM-STRESS-01", "") + rate := fs.Int("rate", 1000, "events per second") + batch := fs.Int("batch", 100, "events per request") + workers := fs.Int("workers", 8, "max concurrent requests") + duration := fs.Duration("duration", time.Minute, "") + maxEvents := fs.Int64("max-events", 5_000_000, "hard event limit") + scenario := fs.String("scenario", "mixed", "") + confirm := fs.Bool("confirm-load-test", false, "") + if e := fs.Parse(args); e != nil { + return e + } + if !*confirm { + return fmt.Errorf("refusing to start without --confirm-load-test") + } + if *rate < 1 || *rate > 100000 || *batch < 1 || *batch > 1000 || *workers < 1 || *workers > 64 || *duration <= 0 || *duration > time.Hour || *maxEvents < 1 || *maxEvents > 50_000_000 { + return fmt.Errorf("unsafe load-test parameters") + } + + client := &http.Client{Timeout: 10 * time.Second} + end := time.Now().Add(*duration) + var sent, failed, scheduled atomic.Uint64 + sem := make(chan struct{}, *workers) + var wg sync.WaitGroup + + for time.Now().Before(end) && int64(scheduled.Load()) < *maxEvents { + select { + case <-ctx.Done(): + wg.Wait() + return nil + default: + } + remaining := *maxEvents - int64(scheduled.Load()) + n := min(*batch, *rate) + if int64(n) > remaining { + n = int(remaining) + } + if n <= 0 { + break + } + events := make([]contracts.LogPayload, 0, n) + for i := 0; i < n; i++ { + events = append(events, makeEvent(*host, *scenario)) + } + scheduled.Add(uint64(n)) + sem <- struct{}{} + wg.Add(1) + go func(b []contracts.LogPayload) { + defer wg.Done() + defer func() { <-sem }() + data, _ := json.Marshal(b) + req, _ := http.NewRequestWithContext(ctx, http.MethodPost, *url, bytes.NewReader(data)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-API-Key", *api) + if *enroll != "" { + req.Header.Set("X-Enrollment-Key", *enroll) + } + resp, e := client.Do(req) + if e != nil { + failed.Add(uint64(len(b))) + return + } + resp.Body.Close() + if resp.StatusCode/100 != 2 { + failed.Add(uint64(len(b))) + return + } + sent.Add(uint64(len(b))) + }(events) + // Pace by event count, not requests. This remains correct when rate < batch. + time.Sleep(time.Duration(float64(time.Second) * float64(n) / float64(*rate))) + } + wg.Wait() + fmt.Printf("stress test finished: accepted=%d failed=%d scheduled=%d duration=%s\n", sent.Load(), failed.Load(), scheduled.Load(), *duration) + return nil +} + +func makeEvent(host, scenario string) contracts.LogPayload { + id := uint32(4624) + switch scenario { + case "failed-logon": + id = 4625 + case "lockout": + id = 4740 + case "catalog": + id = uint32(10000 + rand.IntN(30000)) + case "mixed": + ids := []uint32{4624, 4625, 4740, 4768, 4769, 7045, 5857} + id = ids[rand.IntN(len(ids))] + } + u := "user" + strconv.Itoa(rand.IntN(500)) + ip := fmt.Sprintf("10.%d.%d.%d", rand.IntN(250)+1, rand.IntN(250)+1, rand.IntN(250)+1) + return contracts.LogPayload{Hostname: host, Channel: func() string { + if id == 7045 { + return "System" + } + if id == 5857 { + return "Microsoft-Windows-WMI-Activity/Operational" + } + return "Security" + }(), EventID: id, Source: "SIEM-Stress-Agent", Time: time.Now().UTC(), Metadata: &contracts.EventMetadataPayload{ProviderName: "SIEM-Stress-Agent", TargetUser: u, Workstation: "STRESS-CLIENT-" + strconv.Itoa(rand.IntN(50)), SrcIP: ip, LogonType: "3"}} +} + +func min(a, b int) int { + if a < b { + return a + } + return b +} diff --git a/main.go b/main.go deleted file mode 100644 index 4e05be9..0000000 --- a/main.go +++ /dev/null @@ -1,7797 +0,0 @@ -package main - -import ( - "context" - "crypto/sha256" - "database/sql" - "encoding/hex" - "encoding/json" - "encoding/xml" - "errors" - "fmt" - "html/template" - "io" - "log" - "math" - "net" - "net/http" - "net/url" - "os" - "os/signal" - "strconv" - "strings" - "syscall" - "time" - - _ "github.com/go-sql-driver/mysql" - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/promhttp" -) - -const uiTemplates = ` -{{define "header"}} - - - - - - {{.Title}} - - - -
-
SIEM-lite Security Operations
- -
-
-{{end}} - -{{define "footer"}} -
- - -{{end}} - -{{define "privileged_users"}} -{{template "header" .}} -

{{.Title}}

-

Privilegierte Benutzer für UEBA-Regeln wie „Admin auf neuem Host“.

- -
-

Benutzer hinzufügen

-
-
-
- - -
-
- - -
-
- -
-
- -

Privileged Users

-
- - - - - - - - - - {{range .Users}} - - - - - - - - - {{end}} -
UsernameGrundStatusCreatedUpdatedAktion
{{.Username}}{{.Reason}} - {{if .Enabled}} - aktiv - {{else}} - inaktiv - {{end}} - {{fmtTime .CreatedAt}}{{fmtTime .UpdatedAt}} -
- - {{if .Enabled}} - - - {{else}} - - - {{end}} -
-
-
- -{{template "footer" .}} -{{end}} - -{{define "dashboard"}} -{{template "header" .}} -

{{.Title}}

-

Stand: {{fmtTime .Now}}

-
-
Agents gesamt
{{.Stats.AgentsTotal}}
-
Agents aktiv
{{.Stats.AgentsActive}}
-
Events 24h
{{.Stats.Events24h}}
-
Detections 24h
{{.Stats.Detections24h}}
-
High Detections 24h
{{.Stats.HighDetections24h}}
-
Open Detections
{{.Stats.OpenDetections}}
-
Investigating
{{.Stats.InvestigatingDetections}}
-
Critical 24h
{{.Stats.CriticalDetections24h}}
-
False Positive 24h
{{.Stats.FalsePositive24h}}
-
Legitim 24h
{{.Stats.Legitimate24h}}
-
- -

Neueste Detections

-
- - - {{range .RecentDetections}} - - - - - - - - {{end}} -
ZeitRuleSeverityHostZusammenfassung
{{fmtTime .CreatedAt}}{{.RuleName}}{{.Severity}}{{.Hostname}}{{.Summary}}
-
- -

Neueste Event-Metadaten

-
- - - {{range .RecentEvents}} - - - - - - - - - - {{end}} -
ZuletztHostChannelEventIDUserIPAnzahl
{{fmtTime .LastSeen}}{{.Hostname}}{{.Channel}}{{.EventID}}{{if .TargetUser}}{{.TargetUser}}{{else}}{{.SubjectUser}}{{end}}{{.SrcIP}}{{.Count}}
-
-{{template "footer" .}} -{{end}} - -{{define "detections"}} -{{template "header" .}} -

{{.Title}}

-
-
-
-
-
-
- - -
-
-
- -
-
- Open - Critical - High - Investigating - False Positives - Legitim - Plausibel - Resolved - Confirmed Incidents -
- -
-

Batch-Bewertung

-

Bearbeitet mehrere Detections anhand von Rule, Host, Channel, EventID oder Zeitfenster.

- -
-
-
-
-
-
-
-
-
-
- - -
-
- - - - -

- -

-
-
- -
- - - - - - - - - - - - - {{range .Detections}} - - - - - - - - - - - - {{end}} -
ZeitRuleSeverityStatusHostScoreSummaryBewertungEvents
{{fmtTime .CreatedAt}}{{.RuleName}}{{.Severity}}{{.Status}}{{.Hostname}}{{printf "%.2f" .Score}} - {{.Summary}} - {{if .AnalystNote}} -
Notiz: {{.AnalystNote}}
- {{end}} - 		-
- - - - - - - - - - - - - - -
-
- - - - - - - - -
- 		- anzeigen -
-
-{{template "footer" .}} -{{end}} - -{{define "rules"}} -{{template "header" .}} -

{{.Title}}

-

Dynamische Regeln für einfache EventID-, Feld- und Threshold-Erkennung.

- -

Neue Regel

-
-
-
-
- -
-
-
-
-
- -
-
-
-
-
-
-
-

-
- -

Bestehende Regeln

-
- - - - - {{range .Rules}} - - - - - - - - - - - {{end}} -
NameSeverityChannelEventsFilterThresholdStatusAktion
{{.Name}}
{{.Description}}		{{.Severity}}{{.Channel}}{{.EventIDs}}{{.MatchField}} {{.MatchOperator}} {{.MatchValue}}{{.ThresholdCount}} / {{.ThresholdWindowSeconds}}s - {{if .Enabled}} - aktiv - {{else}} - inaktiv - {{end}} - -
- - {{if .Enabled}} - - - {{else}} - - - {{end}} -
-
-
-{{template "footer" .}} -{{end}} - -{{define "soc"}} -{{template "header" .}} -

{{.Title}}

-

Stand: {{fmtTime .Now}}

- -

Top Host Risk Scores

-
- - - - - - - - - - - - {{range .TopHosts}} - - - - - - - - - - - {{end}} -
HostRiskSeverityOpenHighCriticalConfirmedLast Detection
{{.Hostname}}{{printf "%.1f" .RiskScore}}{{.Severity}}{{.OpenDetections}}{{.HighDetections}}{{.CriticalDetections}}{{.ConfirmedIncidents}}{{if .LastDetectionAt.Valid}}{{fmtTime .LastDetectionAt.Time}}{{end}}
-
- -

Recent SOC Relevant Detections

-
- - - - - - - - - - {{range .RecentIncidents}} - - - - - - - - - {{end}} -
ZeitRuleSeverityStatusHostSummary
{{fmtTime .CreatedAt}}{{.RuleName}}{{.Severity}}{{.Status}}{{.Hostname}}{{.Summary}}
-
- -{{template "footer" .}} -{{end}} - -{{define "baseline"}} -{{template "header" .}} -

{{.Title}}

-

Baseline-Anomalien aus der Regel baseline_event_rate_anomaly.

- -
-
-
-
-
-
-
-
- -
- -
- - - - - - - - - - - - - - {{range .Anomalies}} - - - - - - - - - - - - - {{end}} -
ZeitHostChannelEventIDSeverityAktuellBaselineZ-ScoreSamplesBucket
{{fmtTime .CreatedAt}}{{.Hostname}}{{.Channel}}{{.EventID}}{{.Severity}}{{.Count}}{{printf "%.2f" .AvgCount}} ± {{printf "%.2f" .StddevCount}}{{printf "%.2f" .ZScore}}{{.SampleCount}}Tag {{.DayOfWeek}}, Stunde {{.HourOfDay}}
-
-{{template "footer" .}} -{{end}} - -{{define "events"}} -{{template "header" .}} -

{{.Title}}

-
-
-
-
-
-
-
-
-
-
-
-
-
- -
-
- - - - - {{range .Events}} - - - - - - - - - - - - - - {{end}} -
Erstes EventLetztes EventAnzahlHostChannelEventIDTarget UserSubject UserIPWorkstationStatus
{{fmtTime .FirstSeen}}{{fmtTime .LastSeen}}{{.Count}}{{.Hostname}}{{.Channel}}{{.EventID}}{{.TargetUser}}{{.SubjectUser}}{{.SrcIP}}{{.Workstation}}{{.StatusText}}
-
-{{template "footer" .}} -{{end}} - -{{define "agents"}} -{{template "header" .}} -

{{.Title}}

-

Stand: {{fmtTime .Now}}

-
- - - - - - - - - - - - {{range .Agents}} - - - - - - - - - - - {{end}} -
HostnameStatusAktiviertFirst SeenLast SeenOffline MinutenLast IPAktion
{{.Hostname}} - {{if .IsOnline}} - online - {{else}} - offline - {{end}} - - {{if .IsEnabled}} - aktiv - {{else}} - inaktiv - {{end}} - {{fmtTime .FirstSeen}}{{fmtTime .LastSeen}}{{.OfflineMinutes}}{{.LastIP}} -
- - {{if .IsEnabled}} - - - {{else}} - - - {{end}} -
-
-
-{{template "footer" .}} -{{end}} - -{{define "event_detail"}} -{{template "header" .}} -

{{.Title}}

-
-
Host
{{.Event.Hostname}}
-
Channel
{{.Event.Channel}}
-
EventID
{{.Event.EventID}}
-
Zeit
{{fmtTime .Event.Time}}
-
Target User
{{.Event.TargetUser}}
-
Subject User
{{.Event.SubjectUser}}
-
Source IP
{{.Event.SrcIP}}
-
Workstation
{{.Event.Workstation}}
-
Logon Type
{{.Event.LogonType}}
-
Process
{{.Event.ProcessName}}
-
Status
{{.Event.StatusText}}
-
SubStatus
{{.Event.SubStatusText}}
-
- -

Rohes Event XML (nur bei aktivierter Kurzzeit-Speicherung)

-
{{.Event.Message}}
-{{template "footer" .}} -{{end}} -` - -type Config struct { - ListenAddr string - DBDSN string - MaxBodyBytes int64 - HTTPReadTimeout time.Duration - HTTPWriteTimeout time.Duration - HTTPIdleTimeout time.Duration - DBMaxOpenConns int - DBMaxIdleConns int - DBConnMaxLifetime time.Duration - DBConnMaxIdleTime time.Duration - DetectionInterval time.Duration - OfflineAfter time.Duration - OfflineAlertMax time.Duration - FailedLogonWindow time.Duration - FailedLogonThreshold int - RebootWindow time.Duration - RebootThreshold int - PasswordSprayWindow time.Duration - PasswordSprayMinUsers int - PasswordSprayMinAttempts int - SuccessAfterFailureWindow time.Duration - NewSourceIPLookback time.Duration - NewSourceIPWindow time.Duration - DetectionsLimit int - - // A previously unseen Event ID is primarily inventory information, not an - // incident. The rule can be disabled, run in inventory mode, or explicitly - // promoted to alert mode for selected channels/Event IDs. - NewEventIDMode string - NewEventIDLearningPeriod time.Duration - NewEventIDConfirmWindow time.Duration - NewEventIDMinCount int - NewEventIDIgnoreChannels []string - NewEventIDAlertChannels []string - NewEventIDHighRiskIDs map[uint32]struct{} - - EnrollmentKey string - - BaselineEnabled bool - BaselineWindow time.Duration - BaselineMinSamples int - BaselineMinCount int - BaselineMediumZScore float64 - BaselineHighZScore float64 - BaselineSuppressFor time.Duration - - Timezone string - - UEBAEnabled bool - UEBALookback time.Duration - UEBANewContextWindow time.Duration - RiskScoreWindow time.Duration - - PartitionMaintenanceEnabled bool - PartitionMaintenanceInterval time.Duration - PartitionInterval time.Duration - PartitionAhead time.Duration - PartitionBehind time.Duration - PartitionRetention time.Duration // compatibility fallback - - // Metadata-first storage: raw XML is optional, compact event rows are short-lived, - // and aggregated occurrences remain queryable for a much longer period. - StoreEventRows bool - StoreRawXML bool - MetadataBucket time.Duration - EventRetention time.Duration - RawRetention time.Duration - MetadataRetention time.Duration - MetadataContextEventIDs map[uint32]struct{} -} - -type LogPayload struct { - Hostname string `json:"host"` - Channel string `json:"channel"` - EventID uint32 `json:"id"` - Source string `json:"source"` - Time time.Time `json:"ts"` - Message string `json:"msg,omitempty"` - Metadata *EventMetadataPayload `json:"meta,omitempty"` -} - -// EventMetadataPayload lets collectors send already-extracted fields without the -// original XML. This is the preferred ingestion format for metadata-first mode. -type EventMetadataPayload struct { - Computer string `json:"computer,omitempty"` - ProviderName string `json:"provider,omitempty"` - TargetUser string `json:"target_user,omitempty"` - TargetDomain string `json:"target_domain,omitempty"` - SubjectUser string `json:"subject_user,omitempty"` - SubjectDomain string `json:"subject_domain,omitempty"` - Workstation string `json:"workstation,omitempty"` - Device string `json:"device,omitempty"` - SrcIP string `json:"src_ip,omitempty"` - SrcPort string `json:"src_port,omitempty"` - LogonType string `json:"logon_type,omitempty"` - ProcessName string `json:"process_name,omitempty"` - AuthenticationPackage string `json:"authentication_package,omitempty"` - LogonProcess string `json:"logon_process,omitempty"` - StatusText string `json:"status,omitempty"` - SubStatusText string `json:"sub_status,omitempty"` - FailureReason string `json:"failure_reason,omitempty"` -} - -type NormalizedEvent struct { - Computer string - ProviderName string - LevelValue uint32 - TaskValue uint32 - OpcodeValue uint32 - Keywords string - TargetUser string - TargetDomain string - SubjectUser string - SubjectDomain string - Workstation string - SrcIP string - SrcPort string - LogonType string - ProcessName string - AuthenticationPackage string - LogonProcess string - StatusText string - SubStatusText string - FailureReason string - MemberName string - GroupName string -} - -type Detection struct { - ID uint64 `json:"id"` - RuleName string `json:"rule_name"` - Severity string `json:"severity"` - Hostname string `json:"hostname"` - Channel string `json:"channel"` - EventID uint32 `json:"event_id"` - Score float64 `json:"score"` - WindowStart time.Time `json:"window_start"` - WindowEnd time.Time `json:"window_end"` - Summary string `json:"summary"` - Details json.RawMessage `json:"details_json"` - CreatedAt time.Time `json:"created_at"` - - Status string `json:"status"` - AnalystNote string - ReviewedBy string - ReviewedAt sql.NullTime - IsFalsePositive bool - IsLegitimate bool -} - -type ingestResponse struct { - Accepted int `json:"accepted"` -} - -type server struct { - db *sql.DB - logger *log.Logger - cfg Config - registry *prometheus.Registry - detector *detector - startTime time.Time - templates *template.Template - - location *time.Location -} - -type detector struct { - db *sql.DB - cfg Config - logger *log.Logger - - lastSeenGauge *prometheus.GaugeVec - activeAgentsGauge prometheus.Gauge - anomalyScoreGauge *prometheus.GaugeVec - detectionHitsTotal *prometheus.CounterVec - ruleLastRunGauge *prometheus.GaugeVec - ruleRuntimeHist *prometheus.HistogramVec - ruleErrorsTotal *prometheus.CounterVec - - baselineCurrentCountGauge *prometheus.GaugeVec - baselineAverageGauge *prometheus.GaugeVec - baselineStddevGauge *prometheus.GaugeVec - baselineSamplesGauge *prometheus.GaugeVec - - hostRiskScoreGauge *prometheus.GaugeVec - - privilegedLogonsTotal *prometheus.CounterVec - privilegedLogonFailuresTotal *prometheus.CounterVec - privilegedNewHostTotal *prometheus.CounterVec -} - -type EventRow struct { - ID uint64 - Hostname string - Channel string - EventID uint32 - Source string - Computer string - ProviderName string - TargetUser string - TargetDomain string - SubjectUser string - SubjectDomain string - Workstation string - SrcIP string - SrcPort string - LogonType string - ProcessName string - AuthenticationPackage string - LogonProcess string - StatusText string - SubStatusText string - FailureReason string - Time time.Time - ReceivedAt time.Time - Message string - Count uint64 - FirstSeen time.Time - LastSeen time.Time - Aggregated bool -} - -type DashboardStats struct { - AgentsTotal int - AgentsActive int - Events24h int64 - Detections24h int64 - HighDetections24h int64 - - OpenDetections int64 - InvestigatingDetections int64 - CriticalDetections24h int64 - FalsePositive24h int64 - Legitimate24h int64 -} - -type DashboardPageData struct { - Title string - Now time.Time - Stats DashboardStats - RecentDetections []Detection - RecentEvents []EventRow -} - -type DetectionListPageData struct { - Title string - Now time.Time - Filters map[string]string - Detections []Detection -} - -type EventListPageData struct { - Title string - Now time.Time - Filters map[string]string - Events []EventRow -} - -type EventDetailPageData struct { - Title string - Now time.Time - Event EventRow -} - -type AgentRow struct { - ID uint64 - Hostname string - FirstSeen time.Time - LastSeen time.Time - LastIP string - IsEnabled bool - IsOnline bool - OfflineMinutes int -} - -type AgentListPageData struct { - Title string - Now time.Time - Agents []AgentRow -} - -type DynamicRule struct { - ID uint64 - Name string - Description string - Severity string - Channel string - EventIDs string - MatchField string - MatchOperator string - MatchValue string - ThresholdCount int - ThresholdWindowSeconds int - SuppressForSeconds int - Enabled bool - CreatedAt time.Time - UpdatedAt time.Time -} - -type DynamicRulePageData struct { - Title string - Now time.Time - Rules []DynamicRule -} - -type BaselineBucket struct { - Hostname string - Channel string - EventID uint32 - Hour int - DayOfWeek int - Count int -} - -type BaselineStat struct { - AvgCount float64 - M2Count float64 - StddevCount float64 - SampleCount int -} - -type BaselineAnomalyRow struct { - ID uint64 - CreatedAt time.Time - Hostname string - Channel string - EventID uint32 - Severity string - Score float64 - WindowStart time.Time - WindowEnd time.Time - Summary string - - Count int - AvgCount float64 - StddevCount float64 - ZScore float64 - SampleCount int - HourOfDay int - DayOfWeek int - WindowMin int -} - -type BaselinePageData struct { - Title string - Now time.Time - Filters map[string]string - Anomalies []BaselineAnomalyRow -} - -type baselineDetailsJSON struct { - Hostname string `json:"hostname"` - Channel string `json:"channel"` - EventID uint32 `json:"event_id"` - Count int `json:"count"` - AvgCount float64 `json:"avg_count"` - StddevCount float64 `json:"stddev_count"` - ZScore float64 `json:"z_score"` - SampleCount int `json:"sample_count"` - HourOfDay int `json:"hour_of_day"` - DayOfWeek int `json:"day_of_week"` - WindowMinutes int `json:"window_minutes"` -} - -type SOCHostRiskRow struct { - Hostname string - RiskScore float64 - Severity string - OpenDetections int - HighDetections int - CriticalDetections int - ConfirmedIncidents int - LastDetectionAt sql.NullTime - UpdatedAt time.Time -} - -type SOCRecentIncidentRow struct { - ID uint64 - CreatedAt time.Time - RuleName string - Severity string - Status string - Hostname string - Summary string -} - -type SOCPageData struct { - Title string - Now time.Time - TopHosts []SOCHostRiskRow - RecentIncidents []SOCRecentIncidentRow -} - -type PrivilegedUserRow struct { - Username string - Reason string - Enabled bool - CreatedAt time.Time - UpdatedAt time.Time -} - -type PrivilegedUsersPageData struct { - Title string - Now time.Time - Users []PrivilegedUserRow -} - -type RawEventInsert struct { - EventOffset int - Message string - SHA256 string - Time time.Time -} - -type EventCountBucketAgg struct { - BucketStart time.Time - BucketEnd time.Time - Hostname string - Channel string - EventID uint32 - Count uint64 - FirstTS time.Time - LastTS time.Time -} - -type MetadataOccurrenceAgg struct { - BucketStart time.Time - BucketEnd time.Time - DimensionKey []byte - Hostname string - Channel string - EventID uint32 - ProviderName string - TargetUser string - SubjectUser string - SrcIP string - Workstation string - LogonType string - StatusText string - FailureReason string - Count uint64 - FirstTS time.Time - LastTS time.Time -} - -type partitionedTable struct { - Name string - TimeColumn string - Retention time.Duration -} - -var ( - httpRequestsTotal = prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "eventcollector_http_requests_total", Help: "Total HTTP requests."}, - []string{"path", "method", "status"}, - ) - httpRequestDuration = prometheus.NewHistogramVec( - prometheus.HistogramOpts{Name: "eventcollector_http_request_duration_seconds", Help: "HTTP request latency.", Buckets: prometheus.DefBuckets}, - []string{"path", "method", "status"}, - ) - ingestBatchesTotal = prometheus.NewCounter( - prometheus.CounterOpts{Name: "eventcollector_ingest_batches_total", Help: "Total ingested batches."}, - ) - ingestEventsTotal = prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "eventcollector_ingest_events_total", Help: "Total ingested events."}, - []string{"channel", "event_id"}, - ) - ingestRejectedTotal = prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "eventcollector_ingest_rejected_total", Help: "Rejected ingest requests."}, - []string{"reason"}, - ) - dbInsertEventsTotal = prometheus.NewCounter( - prometheus.CounterOpts{Name: "eventcollector_db_insert_events_total", Help: "Total inserted events into database."}, - ) - dbInsertFailuresTotal = prometheus.NewCounter( - prometheus.CounterOpts{Name: "eventcollector_db_insert_failures_total", Help: "Failed database insert operations."}, - ) - dbBatchSizeHist = prometheus.NewHistogram( - prometheus.HistogramOpts{Name: "eventcollector_db_batch_size", Help: "Batch sizes written to database.", Buckets: []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000}}, - ) - dbTxDurationHist = prometheus.NewHistogram( - prometheus.HistogramOpts{Name: "eventcollector_db_tx_duration_seconds", Help: "Database transaction duration.", Buckets: prometheus.DefBuckets}, - ) -) - -func main() { - cfg := loadConfig() - logger := log.New(os.Stdout, "", log.LstdFlags|log.Lmicroseconds|log.LUTC) - logger.Printf("storage mode: store_event_rows=%t store_raw_xml=%t metadata_bucket=%s context_event_ids=%d", cfg.StoreEventRows, cfg.StoreRawXML, cfg.MetadataBucket, len(cfg.MetadataContextEventIDs)) - if cfg.StoreRawXML && !cfg.StoreEventRows { - logger.Printf("storage warning: STORE_RAW_XML=true implicitly requires per-event rows; high-cardinality hot storage is active") - } - loc, err := time.LoadLocation(cfg.Timezone) - if err != nil { - logger.Printf("invalid TZ %q, falling back to Local: %v", cfg.Timezone, err) - loc = time.Local - } - - db, err := sql.Open("mysql", cfg.DBDSN) - if err != nil { - logger.Fatalf("sql.Open: %v", err) - } - db.SetMaxOpenConns(cfg.DBMaxOpenConns) - db.SetMaxIdleConns(cfg.DBMaxIdleConns) - db.SetConnMaxLifetime(cfg.DBConnMaxLifetime) - db.SetConnMaxIdleTime(cfg.DBConnMaxIdleTime) - - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - if err := db.PingContext(ctx); err != nil { - logger.Fatalf("db.PingContext: %v", err) - } - - reg := prometheus.NewRegistry() - reg.MustRegister( - httpRequestsTotal, - httpRequestDuration, - ingestBatchesTotal, - ingestEventsTotal, - ingestRejectedTotal, - dbInsertEventsTotal, - dbInsertFailuresTotal, - dbBatchSizeHist, - dbTxDurationHist, - ) - - d := &detector{ - db: db, - cfg: cfg, - logger: logger, - lastSeenGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{Name: "eventcollector_agent_last_seen_unixtime", Help: "Unix time when agent was last seen."}, - []string{"host"}, - ), - activeAgentsGauge: prometheus.NewGauge( - prometheus.GaugeOpts{Name: "eventcollector_active_agents", Help: "Number of active agents seen within offline threshold."}, - ), - anomalyScoreGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{Name: "eventcollector_anomaly_score", Help: "Current anomaly score per host and rule."}, - []string{"host", "rule"}, - ), - detectionHitsTotal: prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "eventcollector_detection_hits_total", Help: "Total number of detections per rule and severity."}, - []string{"rule", "severity"}, - ), - ruleLastRunGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{Name: "eventcollector_rule_last_run_unixtime", Help: "Unix time of the last successful rule run."}, - []string{"rule"}, - ), - ruleRuntimeHist: prometheus.NewHistogramVec( - prometheus.HistogramOpts{Name: "eventcollector_rule_runtime_seconds", Help: "Rule runtime duration.", Buckets: prometheus.DefBuckets}, - []string{"rule"}, - ), - ruleErrorsTotal: prometheus.NewCounterVec( - prometheus.CounterOpts{Name: "eventcollector_rule_errors_total", Help: "Rule execution errors."}, - []string{"rule"}, - ), - baselineCurrentCountGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "eventcollector_baseline_current_count", - Help: "Current event count in baseline window.", - }, - []string{"host", "channel", "event_id"}, - ), - baselineAverageGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "eventcollector_baseline_avg_count", - Help: "Baseline average event count.", - }, - []string{"host", "channel", "event_id"}, - ), - baselineStddevGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "eventcollector_baseline_stddev_count", - Help: "Baseline standard deviation event count.", - }, - []string{"host", "channel", "event_id"}, - ), - baselineSamplesGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "eventcollector_baseline_sample_count", - Help: "Baseline sample count.", - }, - []string{"host", "channel", "event_id"}, - ), - hostRiskScoreGauge: prometheus.NewGaugeVec( - prometheus.GaugeOpts{ - Name: "eventcollector_host_risk_score", - Help: "Calculated risk score per host.", - }, - []string{"host", "severity"}, - ), - privilegedLogonsTotal: prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "siem_privileged_logons_total", - Help: "Successful logons by privileged users.", - }, - []string{"user", "host"}, - ), - privilegedLogonFailuresTotal: prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "siem_privileged_logon_failures_total", - Help: "Failed logons by privileged users.", - }, - []string{"user", "host"}, - ), - privilegedNewHostTotal: prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "siem_privileged_new_host_total", - Help: "Privileged user logged on to a new host.", - }, - []string{"user", "host"}, - ), - } - reg.MustRegister( - d.lastSeenGauge, - d.activeAgentsGauge, - d.anomalyScoreGauge, - d.detectionHitsTotal, - d.ruleLastRunGauge, - d.ruleRuntimeHist, - d.ruleErrorsTotal, - d.baselineCurrentCountGauge, - d.baselineAverageGauge, - d.baselineStddevGauge, - d.baselineSamplesGauge, - d.hostRiskScoreGauge, - d.privilegedLogonsTotal, - d.privilegedLogonFailuresTotal, - d.privilegedNewHostTotal, - ) - - s := &server{ - db: db, - logger: logger, - cfg: cfg, - registry: reg, - detector: d, - startTime: time.Now().UTC(), - location: loc, - } - - tmpl := template.Must(template.New("ui").Funcs(template.FuncMap{ - "q": url.QueryEscape, - "fmtTime": func(t time.Time) string { - if t.IsZero() { - return "" - } - return normalizeTime(t).In(loc).Format("2006-01-02 15:04:05 MST") - }, - "short": func(s string, n int) string { - if len(s) <= n { - return s - } - return s[:n] + "..." - }, - "eq": func(a, b string) bool { - return a == b - }, - }).Parse(uiTemplates)) - - s.templates = tmpl - - go s.runSOCLoop() - go s.runDetectionLoop() - go s.runBaselineLoop() - - go s.runPartitionMaintenanceLoop() - - mux := http.NewServeMux() - mux.HandleFunc("/healthz", s.handleHealthz) - mux.HandleFunc("/readyz", s.handleReadyz) - mux.HandleFunc("/ingest", s.handleIngest) - mux.HandleFunc("/detections", s.handleDetections) - mux.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) - mux.HandleFunc("/ui", s.handleUIIndex) - mux.HandleFunc("/ui/detections", s.handleUIDetections) - mux.HandleFunc("/ui/detection/update", s.handleUIDetectionUpdate) - mux.HandleFunc("/ui/events", s.handleUIEvents) - mux.HandleFunc("/ui/event", s.handleUIEventDetail) - mux.HandleFunc("/ui/agents", s.handleUIAgents) - mux.HandleFunc("/ui/agents/toggle", s.handleUIAgentToggle) - mux.HandleFunc("/ui/rules", s.handleUIRules) - mux.HandleFunc("/ui/rules/save", s.handleUIRuleSave) - mux.HandleFunc("/ui/rules/toggle", s.handleUIRuleToggle) - mux.HandleFunc("/ui/baseline", s.handleUIBaseline) - mux.HandleFunc("/ui/soc", s.handleUISOC) - mux.HandleFunc("/ui/detections/batch-update", s.handleUIDetectionsBatchUpdate) - mux.HandleFunc("/ui/privileged-users", s.handleUIPrivilegedUsers) - mux.HandleFunc("/ui/privileged-users/save", s.handleUIPrivilegedUserSave) - mux.HandleFunc("/ui/privileged-users/toggle", s.handleUIPrivilegedUserToggle) - - httpSrv := &http.Server{ - Addr: cfg.ListenAddr, - Handler: metricsMiddleware(logger, recoveryMiddleware(mux)), - ReadTimeout: cfg.HTTPReadTimeout, - WriteTimeout: cfg.HTTPWriteTimeout, - IdleTimeout: cfg.HTTPIdleTimeout, - } - - go func() { - logger.Printf("listening on %s", cfg.ListenAddr) - if err := httpSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - logger.Fatalf("ListenAndServe: %v", err) - } - }() - - stop := make(chan os.Signal, 1) - signal.Notify(stop, os.Interrupt, syscall.SIGTERM) - <-stop - - logger.Println("shutdown requested") - - shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 15*time.Second) - defer shutdownCancel() - - if err := httpSrv.Shutdown(shutdownCtx); err != nil { - logger.Printf("http shutdown error: %v", err) - } - if err := db.Close(); err != nil { - logger.Printf("db close error: %v", err) - } -} - -func bucketStart(t time.Time, size time.Duration) time.Time { - t = t.UTC() - - if size <= 0 { - size = 5 * time.Minute - } - - unix := t.Unix() - bucketSeconds := int64(size.Seconds()) - start := unix - (unix % bucketSeconds) - - return time.Unix(start, 0).UTC() -} - -func upsertEventCountBucketsTx(ctx context.Context, tx *sql.Tx, buckets map[string]*EventCountBucketAgg) error { - if len(buckets) == 0 { - return nil - } - - var sb strings.Builder - args := make([]any, 0, len(buckets)*8) - - sb.WriteString(` -INSERT INTO event_count_buckets -(bucket_start, bucket_end, hostname, channel_name, event_id, cnt, first_event_ts, last_event_ts) -VALUES -`) - - i := 0 - for _, b := range buckets { - if i > 0 { - sb.WriteString(",") - } - i++ - - sb.WriteString("(?,?,?,?,?,?,?,?)") - - args = append(args, - b.BucketStart.UTC(), - b.BucketEnd.UTC(), - b.Hostname, - b.Channel, - b.EventID, - b.Count, - b.FirstTS.UTC(), - b.LastTS.UTC(), - ) - } - - sb.WriteString(` -ON DUPLICATE KEY UPDATE - cnt = cnt + VALUES(cnt), - first_event_ts = LEAST(first_event_ts, VALUES(first_event_ts)), - last_event_ts = GREATEST(last_event_ts, VALUES(last_event_ts)), - finalized = 0, - updated_at = UTC_TIMESTAMP(6) -`) - - _, err := tx.ExecContext(ctx, sb.String(), args...) - if err != nil { - return fmt.Errorf("upsert event_count_buckets: %w", err) - } - - return nil -} - -func upsertEventCatalogTx(ctx context.Context, tx *sql.Tx, buckets map[string]*EventCountBucketAgg) error { - if len(buckets) == 0 { - return nil - } - var sb strings.Builder - args := make([]any, 0, len(buckets)*7) - sb.WriteString(` -INSERT INTO event_catalog -(hostname, channel_name, event_id, first_seen, last_seen, total_count, updated_at) -VALUES -`) - i := 0 - for _, b := range buckets { - if i > 0 { - sb.WriteString(",") - } - i++ - sb.WriteString("(?,?,?,?,?,?,UTC_TIMESTAMP(6))") - args = append(args, b.Hostname, b.Channel, b.EventID, b.FirstTS.UTC(), b.LastTS.UTC(), b.Count) - } - sb.WriteString(` -ON DUPLICATE KEY UPDATE - first_seen = LEAST(first_seen, VALUES(first_seen)), - last_seen = GREATEST(last_seen, VALUES(last_seen)), - total_count = total_count + VALUES(total_count), - updated_at = UTC_TIMESTAMP(6) -`) - if _, err := tx.ExecContext(ctx, sb.String(), args...); err != nil { - return fmt.Errorf("upsert event_catalog: %w", err) - } - return nil -} - -func metadataDimensionKey(parts ...string) []byte { - h := sha256.New() - for _, part := range parts { - _, _ = io.WriteString(h, part) - _, _ = h.Write([]byte{0}) - } - sum := h.Sum(nil) - key := make([]byte, 16) - copy(key, sum[:16]) - return key -} - -func upsertMetadataOccurrencesTx(ctx context.Context, tx *sql.Tx, occurrences map[string]*MetadataOccurrenceAgg) error { - if len(occurrences) == 0 { - return nil - } - - var sb strings.Builder - args := make([]any, 0, len(occurrences)*16) - sb.WriteString(` -INSERT INTO event_occurrences ( - bucket_start, bucket_end, dimension_key, - hostname, channel_name, event_id, provider_name, - target_user, subject_user, src_ip, workstation, - logon_type, status_text, failure_reason, - cnt, first_event_ts, last_event_ts -) VALUES -`) - - i := 0 - for _, o := range occurrences { - if i > 0 { - sb.WriteString(",") - } - i++ - sb.WriteString("(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)") - args = append(args, - o.BucketStart.UTC(), o.BucketEnd.UTC(), o.DimensionKey, - o.Hostname, o.Channel, o.EventID, o.ProviderName, - o.TargetUser, o.SubjectUser, o.SrcIP, o.Workstation, - o.LogonType, o.StatusText, o.FailureReason, - o.Count, o.FirstTS.UTC(), o.LastTS.UTC(), - ) - } - - sb.WriteString(` -ON DUPLICATE KEY UPDATE - cnt = cnt + VALUES(cnt), - first_event_ts = LEAST(first_event_ts, VALUES(first_event_ts)), - last_event_ts = GREATEST(last_event_ts, VALUES(last_event_ts)), - bucket_end = GREATEST(bucket_end, VALUES(bucket_end)), - provider_name = VALUES(provider_name), - failure_reason = VALUES(failure_reason), - updated_at = UTC_TIMESTAMP(6) -`) - - if _, err := tx.ExecContext(ctx, sb.String(), args...); err != nil { - return fmt.Errorf("upsert event_occurrences: %w", err) - } - return nil -} - -func (s *server) runSOCLoop() { - ticker := time.NewTicker(1 * time.Minute) - defer ticker.Stop() - - s.runSOCOnce() - - for range ticker.C { - s.runSOCOnce() - } -} - -func (s *server) runBaselineLoop() { - ticker := time.NewTicker(s.cfg.DetectionInterval) - defer ticker.Stop() - - s.runBaselineOnce() - - for range ticker.C { - s.runBaselineOnce() - } -} - -func (s *server) runBaselineOnce() { - if !s.cfg.BaselineEnabled { - return - } - - rules := []struct { - name string - timeout time.Duration - fn func(context.Context) error - }{ - {"baseline_finalize_buckets", 30 * time.Second, s.detector.finalizeEventCountBuckets}, - {"baseline_anomaly", 120 * time.Second, s.detector.runBaselineAnomalyRule}, - {"baseline_update", 120 * time.Second, s.detector.runBaselineUpdate}, - } - - for _, rule := range rules { - start := time.Now() - - ctx, cancel := context.WithTimeout(context.Background(), rule.timeout) - err := rule.fn(ctx) - cancel() - - dur := time.Since(start) - - if err != nil { - s.logger.Printf("baseline rule %s error after %s: %v", rule.name, dur, err) - s.detector.ruleErrorsTotal.WithLabelValues(rule.name).Inc() - continue - } - - s.logger.Printf("baseline rule %s completed in %s", rule.name, dur) - - s.detector.ruleLastRunGauge.WithLabelValues(rule.name).Set(float64(time.Now().Unix())) - s.detector.ruleRuntimeHist.WithLabelValues(rule.name).Observe(dur.Seconds()) - } -} - -func (s *server) runSOCOnce() { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - - start := time.Now() - - if err := s.detector.runHostRiskScoreUpdate(ctx); err != nil { - s.logger.Printf("soc host risk update error after %s: %v", time.Since(start), err) - s.detector.ruleErrorsTotal.WithLabelValues("host_risk_score").Inc() - return - } - - s.detector.ruleLastRunGauge.WithLabelValues("host_risk_score").Set(float64(time.Now().Unix())) - s.detector.ruleRuntimeHist.WithLabelValues("host_risk_score").Observe(time.Since(start).Seconds()) - - s.logger.Printf("soc host risk update completed in %s", time.Since(start)) -} - -func normalizeTime(t time.Time) time.Time { - if t.IsZero() { - return t - } - - // Fall 1: - // MySQL DATETIME wird mit loc=UTC gelesen. - // Dann ist t.Location() bereits UTC und alles ist gut. - if t.Location() == time.UTC { - return t - } - - // Fall 2: - // Der MySQL-Treiber hat die Zeit bereits als echten Zeitpunkt mit Location gelesen. - // Dann NICHT die Uhrzeit neu als UTC interpretieren, sondern Instant nach UTC konvertieren. - name := t.Location().String() - if name != "" && name != "Local" { - return t.UTC() - } - - // Fall 3: - // MySQL DATETIME ohne echte Zeitzone wurde als time.Local interpretiert. - // In deiner Anwendung speicherst du DB-Zeiten logisch als UTC. - // Deshalb interpretieren wir die Wandzeit als UTC. - return time.Date( - t.Year(), - t.Month(), - t.Day(), - t.Hour(), - t.Minute(), - t.Second(), - t.Nanosecond(), - time.UTC, - ) -} - -func (s *server) listPrivilegedUsers(ctx context.Context) ([]PrivilegedUserRow, error) { - rows, err := s.db.QueryContext(ctx, ` -SELECT username, COALESCE(reason, ''), enabled, created_at, updated_at -FROM privileged_users -ORDER BY username ASC -`) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []PrivilegedUserRow - for rows.Next() { - var u PrivilegedUserRow - if err := rows.Scan(&u.Username, &u.Reason, &u.Enabled, &u.CreatedAt, &u.UpdatedAt); err != nil { - return nil, err - } - out = append(out, u) - } - - return out, rows.Err() -} - -func (s *server) handleUIPrivilegedUsers(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - users, err := s.listPrivilegedUsers(ctx) - if err != nil { - s.logger.Printf("privileged users: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - s.renderTemplate(w, "privileged_users", PrivilegedUsersPageData{ - Title: "Privileged Users", - Now: time.Now().UTC(), - Users: users, - }) -} - -func (s *server) runPartitionMaintenanceLoop() { - if !s.cfg.PartitionMaintenanceEnabled { - s.logger.Printf("partition maintenance disabled") - return - } - - s.runPartitionMaintenanceOnce() - - ticker := time.NewTicker(s.cfg.PartitionMaintenanceInterval) - defer ticker.Stop() - - for range ticker.C { - s.runPartitionMaintenanceOnce() - } -} - -func (s *server) runPartitionMaintenanceOnce() { - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) - defer cancel() - - start := time.Now() - - if err := s.ensureConfiguredPartitions(ctx); err != nil { - s.logger.Printf("partition maintenance error after %s: %v", time.Since(start), err) - return - } - - s.logger.Printf("partition maintenance completed in %s", time.Since(start)) -} - -func (s *server) ensureTableIsPartitioned(ctx context.Context, tableName string) error { - var partitionCount int - - err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM information_schema.PARTITIONS -WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = ? - AND PARTITION_NAME IS NOT NULL -`, tableName).Scan(&partitionCount) - - if err != nil { - return fmt.Errorf("check partitioned table %s: %w", tableName, err) - } - - if partitionCount == 0 { - return fmt.Errorf( - "table %s is not partitioned; run the SQL migration first", - tableName, - ) - } - - return nil -} - -func (s *server) ensure3HourPartitions(ctx context.Context, tbl partitionedTable) error { - interval := s.cfg.PartitionInterval - if interval <= 0 { - interval = 3 * time.Hour - } - - ahead := s.cfg.PartitionAhead - if ahead <= 0 { - ahead = 24 * time.Hour - } - - behind := s.cfg.PartitionBehind - if behind < 0 { - behind = 0 - } - - now := time.Now().UTC() - - start := partitionFloor(now.Add(-behind), interval) - end := partitionFloor(now.Add(ahead), interval).Add(interval) - - for pStart := start; pStart.Before(end); pStart = pStart.Add(interval) { - pEnd := pStart.Add(interval) - - exists, err := s.partitionExists(ctx, tbl.Name, partitionName(pStart)) - if err != nil { - return err - } - if exists { - continue - } - - if err := s.addPartitionBeforePMax(ctx, tbl.Name, pStart, pEnd); err != nil { - return err - } - } - - return nil -} - -func partitionFloor(t time.Time, interval time.Duration) time.Time { - t = t.UTC() - - if interval <= 0 { - interval = 3 * time.Hour - } - - seconds := int64(interval.Seconds()) - unix := t.Unix() - floored := unix - (unix % seconds) - - return time.Unix(floored, 0).UTC() -} - -func partitionName(start time.Time) string { - return "p" + start.UTC().Format("2006010215") -} - -func mysqlDateTimeLiteral(t time.Time) string { - return t.UTC().Format("2006-01-02 15:04:05") -} - -func (s *server) partitionExists(ctx context.Context, tableName, partitionName string) (bool, error) { - var count int - - err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM information_schema.PARTITIONS -WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = ? - AND PARTITION_NAME = ? -`, tableName, partitionName).Scan(&count) - - if err != nil { - return false, fmt.Errorf("check partition exists %s.%s: %w", tableName, partitionName, err) - } - - return count > 0, nil -} - -func (s *server) addPartitionBeforePMax(ctx context.Context, tableName string, start, end time.Time) error { - pName := partitionName(start) - endLit := mysqlDateTimeLiteral(end) - - if !safeIdentifier(tableName) || !safeIdentifier(pName) { - return fmt.Errorf("unsafe partition/table identifier: table=%q partition=%q", tableName, pName) - } - - query := fmt.Sprintf(` -ALTER TABLE %s -REORGANIZE PARTITION pmax INTO ( - PARTITION %s VALUES LESS THAN ('%s'), - PARTITION pmax VALUES LESS THAN (MAXVALUE) -) -`, tableName, pName, endLit) - - s.logger.Printf("creating partition table=%s partition=%s less_than=%s", tableName, pName, endLit) - - if _, err := s.db.ExecContext(ctx, query); err != nil { - return fmt.Errorf("create partition %s on %s: %w", pName, tableName, err) - } - - return nil -} - -func safeIdentifier(v string) bool { - if v == "" { - return false - } - - for _, r := range v { - if r >= 'a' && r <= 'z' { - continue - } - if r >= 'A' && r <= 'Z' { - continue - } - if r >= '0' && r <= '9' { - continue - } - if r == '_' { - continue - } - return false - } - - return true -} - -func (s *server) dropOldPartitions(ctx context.Context, tableName string, retention time.Duration) error { - if retention <= 0 { - return nil - } - - cutoff := partitionFloor(time.Now().UTC().Add(-retention), s.cfg.PartitionInterval) - - rows, err := s.db.QueryContext(ctx, ` -SELECT PARTITION_NAME -FROM information_schema.PARTITIONS -WHERE TABLE_SCHEMA = DATABASE() - AND TABLE_NAME = ? - AND PARTITION_NAME IS NOT NULL - AND PARTITION_NAME <> 'pmax' -`, tableName) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var pName string - if err := rows.Scan(&pName); err != nil { - return err - } - - pStart, ok := parsePartitionName(pName) - if !ok { - continue - } - - if !pStart.Before(cutoff) { - continue - } - - if !safeIdentifier(tableName) || !safeIdentifier(pName) { - return fmt.Errorf("unsafe identifier while dropping partition: %s.%s", tableName, pName) - } - - query := fmt.Sprintf(`ALTER TABLE %s DROP PARTITION %s`, tableName, pName) - - s.logger.Printf("dropping old partition table=%s partition=%s", tableName, pName) - - if _, err := s.db.ExecContext(ctx, query); err != nil { - return fmt.Errorf("drop partition %s on %s: %w", pName, tableName, err) - } - } - - return rows.Err() -} - -func parsePartitionName(name string) (time.Time, bool) { - if len(name) != len("p2006010215") || !strings.HasPrefix(name, "p") { - return time.Time{}, false - } - - t, err := time.ParseInLocation("2006010215", strings.TrimPrefix(name, "p"), time.UTC) - if err != nil { - return time.Time{}, false - } - - return t.UTC(), true -} - -func (s *server) ensureConfiguredPartitions(ctx context.Context) error { - tables := []partitionedTable{ - {Name: "event_logs", TimeColumn: "ts", Retention: s.cfg.EventRetention}, - // Fixed: the actual table is event_log_raw (singular "log"). The old typo - // aborted every maintenance run and therefore prevented all partition drops. - {Name: "event_log_raw", TimeColumn: "ts", Retention: s.cfg.RawRetention}, - {Name: "event_occurrences", TimeColumn: "bucket_start", Retention: s.cfg.MetadataRetention}, - {Name: "event_count_buckets", TimeColumn: "bucket_start", Retention: s.cfg.MetadataRetention}, - {Name: "ueba_context_buckets", TimeColumn: "bucket_start", Retention: s.cfg.MetadataRetention}, - } - - for _, tbl := range tables { - if err := s.ensureTableIsPartitioned(ctx, tbl.Name); err != nil { - return err - } - - if err := s.ensure3HourPartitions(ctx, tbl); err != nil { - return err - } - - retention := tbl.Retention - if retention <= 0 { - retention = s.cfg.PartitionRetention - } - if retention > 0 { - if err := s.dropOldPartitions(ctx, tbl.Name, retention); err != nil { - return err - } - } - } - - return nil -} - -func (s *server) handleUIPrivilegedUserSave(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - username := normalizeUsername(r.FormValue("username")) - reason := strings.TrimSpace(r.FormValue("reason")) - - if username == "" || strings.HasSuffix(username, "$") { - writeError(w, http.StatusBadRequest, "invalid username") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - _, err := s.db.ExecContext(ctx, ` -INSERT INTO privileged_users (username, reason, enabled) -VALUES (?, ?, 1) -ON DUPLICATE KEY UPDATE -reason = VALUES(reason), -enabled = 1, -updated_at = UTC_TIMESTAMP(6) -`, username, reason) - if err != nil { - s.logger.Printf("save privileged user: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - http.Redirect(w, r, "/ui/privileged-users", http.StatusSeeOther) -} - -func (s *server) handleUIPrivilegedUserToggle(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - username := normalizeUsername(r.FormValue("username")) - enabled := strings.TrimSpace(r.FormValue("enabled")) == "1" - - if username == "" { - writeError(w, http.StatusBadRequest, "invalid username") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - _, err := s.db.ExecContext(ctx, ` -UPDATE privileged_users -SET enabled = ?, - updated_at = UTC_TIMESTAMP(6) -WHERE username = ? -`, enabled, username) - if err != nil { - s.logger.Printf("toggle privileged user: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - http.Redirect(w, r, "/ui/privileged-users", http.StatusSeeOther) -} - -func (s *server) handleUIDetectionsBatchUpdate(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - status := strings.TrimSpace(r.FormValue("status")) - note := strings.TrimSpace(r.FormValue("note")) - reviewedBy := strings.TrimSpace(r.FormValue("reviewed_by")) - - if reviewedBy == "" { - reviewedBy = "ui-batch" - } - - switch status { - case "open", "acknowledged", "investigating", "legitimate", "plausible", "false_positive", "resolved", "suppressed", "confirmed_incident": - default: - writeError(w, http.StatusBadRequest, "invalid status") - return - } - - rule := strings.TrimSpace(r.FormValue("rule")) - host := strings.TrimSpace(r.FormValue("host")) - channel := strings.TrimSpace(r.FormValue("channel")) - - var eventID uint32 - if v := strings.TrimSpace(r.FormValue("event_id")); v != "" { - n, err := strconv.ParseUint(v, 10, 32) - if err != nil { - writeError(w, http.StatusBadRequest, "invalid event_id") - return - } - eventID = uint32(n) - } - - from := strings.TrimSpace(r.FormValue("from")) - to := strings.TrimSpace(r.FormValue("to")) - - limit := atoiDefault(r.FormValue("limit"), 5000) - if limit <= 0 || limit > 50000 { - limit = 5000 - } - - if rule == "" && host == "" && channel == "" && eventID == 0 && from == "" && to == "" { - writeError(w, http.StatusBadRequest, "at least one filter required") - return - } - - isFalsePositive := status == "false_positive" - isLegitimate := status == "legitimate" || status == "plausible" - - query := ` -UPDATE detections -SET status = ?, - analyst_note = ?, - reviewed_by = ?, - reviewed_at = UTC_TIMESTAMP(6), - is_false_positive = ?, - is_legitimate = ? -WHERE id IN ( - SELECT id FROM ( - SELECT id - FROM detections - WHERE 1=1 -` - args := []any{ - status, - note, - reviewedBy, - isFalsePositive, - isLegitimate, - } - - if rule != "" { - query += " AND rule_name = ?" - args = append(args, rule) - } - if host != "" { - query += " AND hostname = ?" - args = append(args, host) - } - if channel != "" { - query += " AND channel_name = ?" - args = append(args, channel) - } - if eventID != 0 { - query += " AND event_id = ?" - args = append(args, eventID) - } - if from != "" { - if t, err := parseUIRFC3339(from); err == nil { - query += " AND created_at >= ?" - args = append(args, t.UTC()) - } - } - if to != "" { - if t, err := parseUIRFC3339(to); err == nil { - query += " AND created_at <= ?" - args = append(args, t.UTC()) - } - } - - query += ` - ORDER BY created_at DESC - LIMIT ? - ) x -) -` - args = append(args, limit) - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - res, err := s.db.ExecContext(ctx, query, args...) - if err != nil { - s.logger.Printf("batch update detections: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - affected, _ := res.RowsAffected() - - http.Redirect(w, r, fmt.Sprintf("/ui/detections?status=%s&updated=%d", url.QueryEscape(status), affected), http.StatusSeeOther) -} - -func (s *server) listSOCTopHosts(ctx context.Context, limit int) ([]SOCHostRiskRow, error) { - rows, err := s.db.QueryContext(ctx, ` -SELECT hostname, risk_score, severity, open_detections, - high_detections, critical_detections, confirmed_incidents, - last_detection_at, updated_at -FROM host_risk_scores -ORDER BY risk_score DESC -LIMIT ? -`, limit) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []SOCHostRiskRow - for rows.Next() { - var r SOCHostRiskRow - if err := rows.Scan( - &r.Hostname, - &r.RiskScore, - &r.Severity, - &r.OpenDetections, - &r.HighDetections, - &r.CriticalDetections, - &r.ConfirmedIncidents, - &r.LastDetectionAt, - &r.UpdatedAt, - ); err != nil { - return nil, err - } - out = append(out, r) - } - return out, rows.Err() -} - -func (s *server) getRawEventXMLByID(ctx context.Context, id uint64) (string, error) { - var msg string - - err := s.db.QueryRowContext(ctx, ` -SELECT msg -FROM event_log_raw -WHERE event_log_id = ? -LIMIT 1 -`, id).Scan(&msg) - - if err == nil { - return msg, nil - } - - if !errors.Is(err, sql.ErrNoRows) { - return "", err - } - - // Fallback für Alt-Daten, falls event_log_raw noch nicht vollständig befüllt ist. - err = s.db.QueryRowContext(ctx, ` -SELECT COALESCE(msg, '') -FROM event_logs -WHERE id = ? -LIMIT 1 -`, id).Scan(&msg) - - if err != nil { - return "", err - } - if strings.TrimSpace(msg) == "" { - return "Roh-XML wurde nicht gespeichert oder ist gemäß RAW_RETENTION bereits abgelaufen.", nil - } - - return msg, nil -} - -func eventListSummary(ev EventRow) string { - user := firstNonEmpty(ev.TargetUser, ev.SubjectUser) - parts := []string{ - fmt.Sprintf("%s EventID %d", ev.Channel, ev.EventID), - } - - if user != "" { - parts = append(parts, "User="+user) - } - if ev.SrcIP != "" { - parts = append(parts, "IP="+ev.SrcIP) - } - if ev.Workstation != "" { - parts = append(parts, "Workstation="+ev.Workstation) - } - if ev.ProcessName != "" { - parts = append(parts, "Process="+ev.ProcessName) - } - if ev.FailureReason != "" { - parts = append(parts, "Reason="+ev.FailureReason) - } - - return strings.Join(parts, " | ") -} - -func (s *server) listSOCRecentIncidents(ctx context.Context, limit int) ([]SOCRecentIncidentRow, error) { - if limit <= 0 || limit > 500 { - limit = 50 - } - - rows, err := s.db.QueryContext(ctx, ` -SELECT id, created_at, rule_name, severity, status, hostname, summary -FROM ( - SELECT id, created_at, rule_name, severity, status, hostname, summary - FROM ( - SELECT id, created_at, rule_name, severity, status, hostname, summary - FROM detections - WHERE status IN ('open', 'investigating', 'confirmed_incident') - ORDER BY created_at DESC - LIMIT ? - ) s - - UNION - - SELECT id, created_at, rule_name, severity, status, hostname, summary - FROM ( - SELECT id, created_at, rule_name, severity, status, hostname, summary - FROM detections - WHERE severity IN ('high', 'critical') - ORDER BY created_at DESC - LIMIT ? - ) sev -) x -ORDER BY created_at DESC -LIMIT ? -`, limit, limit, limit) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []SOCRecentIncidentRow - for rows.Next() { - var r SOCRecentIncidentRow - if err := rows.Scan( - &r.ID, - &r.CreatedAt, - &r.RuleName, - &r.Severity, - &r.Status, - &r.Hostname, - &r.Summary, - ); err != nil { - return nil, err - } - - r.CreatedAt = normalizeTime(r.CreatedAt) - out = append(out, r) - } - - return out, rows.Err() -} - -func (s *server) handleUISOC(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - topHosts, err := s.listSOCTopHosts(ctx, 20) - if err != nil { - s.logger.Printf("soc top hosts: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - incidents, err := s.listSOCRecentIncidents(ctx, 50) - if err != nil { - s.logger.Printf("soc recent incidents: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - s.renderTemplate(w, "soc", SOCPageData{ - Title: "SOC Dashboard", - Now: time.Now().UTC(), - TopHosts: topHosts, - RecentIncidents: incidents, - }) -} - -func (s *server) createBaselineExclusionFromDetection(ctx context.Context, det Detection, reason, createdBy string, hours int) error { - var expiresAt any = nil - if hours > 0 { - expiresAt = time.Now().UTC().Add(time.Duration(hours) * time.Hour) - } - - _, err := s.db.ExecContext(ctx, ` -INSERT INTO baseline_exclusions -(hostname, channel_name, event_id, reason, created_by, expires_at, enabled) -VALUES (?, ?, ?, ?, ?, ?, 1) -`, - det.Hostname, - det.Channel, - det.EventID, - reason, - createdBy, - expiresAt, - ) - - return err -} - -func (s *server) handleUIDetectionUpdate(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - id, err := strconv.ParseUint(strings.TrimSpace(r.FormValue("id")), 10, 64) - if err != nil || id == 0 { - writeError(w, http.StatusBadRequest, "invalid id") - return - } - - status := strings.TrimSpace(r.FormValue("status")) - note := strings.TrimSpace(r.FormValue("note")) - reviewedBy := strings.TrimSpace(r.FormValue("reviewed_by")) - - if reviewedBy == "" { - reviewedBy = "ui" - } - - switch status { - case "open", "acknowledged", "investigating", "legitimate", "plausible", "false_positive", "resolved", "suppressed", "confirmed_incident": - default: - writeError(w, http.StatusBadRequest, "invalid status") - return - } - - isFalsePositive := status == "false_positive" - isLegitimate := status == "legitimate" - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - _, err = s.db.ExecContext(ctx, ` -UPDATE detections -SET status = ?, - analyst_note = ?, - reviewed_by = ?, - reviewed_at = UTC_TIMESTAMP(6), - is_false_positive = ?, - is_legitimate = ? -WHERE id = ? -`, - status, - note, - reviewedBy, - isFalsePositive, - isLegitimate, - id, - ) - if err != nil { - s.logger.Printf("update detection: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - createSuppression := strings.TrimSpace(r.FormValue("create_suppression")) == "1" - - if createSuppression && (status == "false_positive" || status == "legitimate" || status == "suppressed") { - det, err := s.getDetectionByID(ctx, id) - if err != nil { - s.logger.Printf("get detection for suppression: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - hours := atoiDefault(r.FormValue("suppress_hours"), 24) - - reason := note - if reason == "" { - reason = fmt.Sprintf("Suppression via UI wegen Status %s", status) - } - - if err := s.createSuppressionFromDetection(ctx, det, reason, reviewedBy, hours); err != nil { - s.logger.Printf("create suppression: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - } - - baselineAction := strings.TrimSpace(r.FormValue("baseline_action")) - - if baselineAction != "" { - det, err := s.getDetectionByID(ctx, id) - if err != nil { - s.logger.Printf("get detection for baseline exclusion: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - hours := 0 - - switch baselineAction { - case "exclude_24h": - hours = 24 - case "exclude_7d": - hours = 168 - case "exclude_30d": - hours = 720 - case "exclude_forever": - hours = 0 - default: - writeError(w, http.StatusBadRequest, "invalid baseline action") - return - } - - reason := note - if reason == "" { - reason = fmt.Sprintf("Baseline exclusion via UI wegen Status %s", status) - } - - if err := s.createBaselineExclusionFromDetection(ctx, det, reason, reviewedBy, hours); err != nil { - s.logger.Printf("create baseline exclusion: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - } - - if status == "confirmed_incident" && baselineAction == "" { - det, err := s.getDetectionByID(ctx, id) - if err != nil { - s.logger.Printf("get detection for confirmed incident exclusion: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - reason := note - if reason == "" { - reason = "Confirmed incident: nicht in Baseline einlernen" - } - - if err := s.createBaselineExclusionFromDetection(ctx, det, reason, reviewedBy, 168); err != nil { - s.logger.Printf("create confirmed incident baseline exclusion: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - } - - redirect := strings.TrimSpace(r.FormValue("redirect")) - if redirect == "" { - redirect = "/ui/detections" - } - - http.Redirect(w, r, redirect, http.StatusSeeOther) -} - -func (d *detector) isBaselineExcluded(ctx context.Context, hostname, channel string, eventID uint32) (bool, error) { - var count int - - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM baseline_exclusions -WHERE enabled = 1 - AND (hostname = '' OR hostname = ?) - AND (channel_name = '' OR channel_name = ?) - AND (event_id = 0 OR event_id = ?) - AND (expires_at IS NULL OR expires_at > UTC_TIMESTAMP(6)) -`, - hostname, - channel, - eventID, - ).Scan(&count) - - if err != nil { - return false, err - } - - return count > 0, nil -} - -func (s *server) getDetectionByID(ctx context.Context, id uint64) (Detection, error) { - const q = ` -SELECT id, rule_name, severity, hostname, channel_name, event_id, score, - window_start, window_end, summary, details_json, created_at, - status, - COALESCE(analyst_note, ''), - COALESCE(reviewed_by, ''), - reviewed_at, - is_false_positive, - is_legitimate -FROM detections -WHERE id = ? -LIMIT 1 -` - - var d Detection - err := s.db.QueryRowContext(ctx, q, id).Scan( - &d.ID, - &d.RuleName, - &d.Severity, - &d.Hostname, - &d.Channel, - &d.EventID, - &d.Score, - &d.WindowStart, - &d.WindowEnd, - &d.Summary, - &d.Details, - &d.CreatedAt, - &d.Status, - &d.AnalystNote, - &d.ReviewedBy, - &d.ReviewedAt, - &d.IsFalsePositive, - &d.IsLegitimate, - ) - return d, err -} - -func (s *server) createSuppressionFromDetection(ctx context.Context, det Detection, reason, createdBy string, hours int) error { - var expiresAt any = nil - if hours > 0 { - expiresAt = time.Now().UTC().Add(time.Duration(hours) * time.Hour) - } - - _, err := s.db.ExecContext(ctx, ` -INSERT INTO detection_suppressions -(rule_name, hostname, channel_name, event_id, reason, created_by, expires_at, enabled) -VALUES (?, ?, ?, ?, ?, ?, ?, 1) -`, - det.RuleName, - det.Hostname, - det.Channel, - det.EventID, - reason, - createdBy, - expiresAt, - ) - - return err -} - -func (s *server) listBaselineAnomalies(ctx context.Context, host, channel, severity string, eventID uint32, limit int) ([]BaselineAnomalyRow, error) { - if limit <= 0 || limit > 1000 { - limit = 100 - } - - query := ` -SELECT id, severity, hostname, channel_name, event_id, score, - window_start, window_end, summary, details_json, created_at -FROM detections -WHERE rule_name = 'baseline_event_rate_anomaly' -` - args := make([]any, 0, 8) - - if host != "" { - query += ` AND hostname = ?` - args = append(args, host) - } - if channel != "" { - query += ` AND channel_name = ?` - args = append(args, channel) - } - if eventID != 0 { - query += ` AND event_id = ?` - args = append(args, eventID) - } - if severity != "" { - query += ` AND severity = ?` - args = append(args, severity) - } - - query += ` ORDER BY created_at DESC LIMIT ?` - args = append(args, limit) - - rows, err := s.db.QueryContext(ctx, query, args...) - if err != nil { - return nil, err - } - defer rows.Close() - - out := make([]BaselineAnomalyRow, 0) - - for rows.Next() { - var row BaselineAnomalyRow - var detailsRaw []byte - - if err := rows.Scan( - &row.ID, - &row.Severity, - &row.Hostname, - &row.Channel, - &row.EventID, - &row.Score, - &row.WindowStart, - &row.WindowEnd, - &row.Summary, - &detailsRaw, - &row.CreatedAt, - ); err != nil { - return nil, err - } - - var details baselineDetailsJSON - if err := json.Unmarshal(detailsRaw, &details); err == nil { - row.Count = details.Count - row.AvgCount = details.AvgCount - row.StddevCount = details.StddevCount - row.ZScore = details.ZScore - row.SampleCount = details.SampleCount - row.HourOfDay = details.HourOfDay - row.DayOfWeek = details.DayOfWeek - row.WindowMin = details.WindowMinutes - } else { - row.ZScore = row.Score - } - - out = append(out, row) - } - - return out, rows.Err() -} - -func (s *server) handleUIBaseline(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - filters := map[string]string{ - "host": strings.TrimSpace(r.URL.Query().Get("host")), - "channel": strings.TrimSpace(r.URL.Query().Get("channel")), - "event_id": strings.TrimSpace(r.URL.Query().Get("event_id")), - "severity": strings.TrimSpace(r.URL.Query().Get("severity")), - "limit": strings.TrimSpace(r.URL.Query().Get("limit")), - } - - limit := 100 - if filters["limit"] != "" { - if n, err := strconv.Atoi(filters["limit"]); err == nil && n > 0 && n <= 1000 { - limit = n - } - } - - var eventID uint32 - if filters["event_id"] != "" { - if n, err := strconv.ParseUint(filters["event_id"], 10, 32); err == nil { - eventID = uint32(n) - } - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - items, err := s.listBaselineAnomalies( - ctx, - filters["host"], - filters["channel"], - filters["severity"], - eventID, - limit, - ) - if err != nil { - s.logger.Printf("ui baseline: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - data := BaselinePageData{ - Title: "Baseline-Anomalien", - Now: time.Now().UTC(), - Filters: filters, - Anomalies: items, - } - - s.renderTemplate(w, "baseline", data) -} - -func (s *server) listDynamicRules(ctx context.Context) ([]DynamicRule, error) { - const q = ` -SELECT id, - name, - COALESCE(description, ''), - severity, - channel, - event_ids, - COALESCE(match_field, ''), - COALESCE(match_operator, ''), - COALESCE(match_value, ''), - threshold_count, - threshold_window_seconds, - suppress_for_seconds, - enabled, - created_at, - updated_at -FROM detection_rules -ORDER BY name ASC -` - rows, err := s.db.QueryContext(ctx, q) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []DynamicRule - for rows.Next() { - var r DynamicRule - if err := rows.Scan( - &r.ID, - &r.Name, - &r.Description, - &r.Severity, - &r.Channel, - &r.EventIDs, - &r.MatchField, - &r.MatchOperator, - &r.MatchValue, - &r.ThresholdCount, - &r.ThresholdWindowSeconds, - &r.SuppressForSeconds, - &r.Enabled, - &r.CreatedAt, - &r.UpdatedAt, - ); err != nil { - return nil, err - } - out = append(out, r) - } - return out, rows.Err() -} - -func (s *server) handleUIRules(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - rules, err := s.listDynamicRules(ctx) - if err != nil { - s.logger.Printf("ui rules: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - s.renderTemplate(w, "rules", DynamicRulePageData{ - Title: "Dynamic Rules", - Now: time.Now().UTC(), - Rules: rules, - }) -} - -func (s *server) handleUIRuleToggle(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - id, err := strconv.ParseUint(strings.TrimSpace(r.FormValue("id")), 10, 64) - if err != nil || id == 0 { - writeError(w, http.StatusBadRequest, "invalid id") - return - } - - enabled := strings.TrimSpace(r.FormValue("enabled")) == "1" - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - _, err = s.db.ExecContext(ctx, ` -UPDATE detection_rules -SET enabled = ? -WHERE id = ? -`, enabled, id) - if err != nil { - s.logger.Printf("toggle rule: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - http.Redirect(w, r, "/ui/rules", http.StatusSeeOther) -} - -func (s *server) handleUIRuleSave(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - rule := DynamicRule{ - Name: strings.TrimSpace(r.FormValue("name")), - Description: strings.TrimSpace(r.FormValue("description")), - Severity: strings.TrimSpace(r.FormValue("severity")), - Channel: strings.TrimSpace(r.FormValue("channel")), - EventIDs: strings.TrimSpace(r.FormValue("event_ids")), - MatchField: strings.TrimSpace(r.FormValue("match_field")), - MatchOperator: strings.TrimSpace(r.FormValue("match_operator")), - MatchValue: strings.TrimSpace(r.FormValue("match_value")), - ThresholdCount: atoiDefault(r.FormValue("threshold_count"), 1), - ThresholdWindowSeconds: atoiDefault(r.FormValue("threshold_window_seconds"), 0), - SuppressForSeconds: atoiDefault(r.FormValue("suppress_for_seconds"), 3600), - } - - if rule.Name == "" || rule.EventIDs == "" { - writeError(w, http.StatusBadRequest, "name and event_ids required") - return - } - if rule.Severity == "" { - rule.Severity = "medium" - } - if rule.Channel == "" { - rule.Channel = "Security" - } - if rule.ThresholdCount <= 0 { - rule.ThresholdCount = 1 - } - if rule.SuppressForSeconds < 0 { - rule.SuppressForSeconds = 0 - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - _, err := s.db.ExecContext(ctx, ` -INSERT INTO detection_rules -(name, description, severity, channel, event_ids, - match_field, match_operator, match_value, - threshold_count, threshold_window_seconds, suppress_for_seconds, enabled) -VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1) -ON DUPLICATE KEY UPDATE - description = VALUES(description), - severity = VALUES(severity), - channel = VALUES(channel), - event_ids = VALUES(event_ids), - match_field = VALUES(match_field), - match_operator = VALUES(match_operator), - match_value = VALUES(match_value), - threshold_count = VALUES(threshold_count), - threshold_window_seconds = VALUES(threshold_window_seconds), - suppress_for_seconds = VALUES(suppress_for_seconds), - enabled = VALUES(enabled) -`, - rule.Name, - rule.Description, - rule.Severity, - rule.Channel, - rule.EventIDs, - rule.MatchField, - rule.MatchOperator, - rule.MatchValue, - rule.ThresholdCount, - rule.ThresholdWindowSeconds, - rule.SuppressForSeconds, - ) - if err != nil { - s.logger.Printf("save rule: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - http.Redirect(w, r, "/ui/rules", http.StatusSeeOther) -} - -func atoiDefault(v string, def int) int { - n, err := strconv.Atoi(strings.TrimSpace(v)) - if err != nil { - return def - } - return n -} - -func (s *server) listAgents(ctx context.Context) ([]AgentRow, error) { - const q = ` -SELECT id, hostname, first_seen, last_seen, last_ip, is_enabled -FROM agents -ORDER BY hostname ASC -` - - rows, err := s.db.QueryContext(ctx, q) - if err != nil { - return nil, err - } - defer rows.Close() - - now := time.Now().UTC() - out := make([]AgentRow, 0) - - for rows.Next() { - var a AgentRow - if err := rows.Scan( - &a.ID, - &a.Hostname, - &a.FirstSeen, - &a.LastSeen, - &a.LastIP, - &a.IsEnabled, - ); err != nil { - return nil, err - } - - if !a.LastSeen.IsZero() { - a.OfflineMinutes = int(now.Sub(a.LastSeen.UTC()).Minutes()) - a.IsOnline = a.IsEnabled && now.Sub(a.LastSeen.UTC()) <= s.cfg.OfflineAfter - } - - out = append(out, a) - } - - return out, rows.Err() -} - -func (s *server) handleUIAgents(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - agents, err := s.listAgents(ctx) - if err != nil { - s.logger.Printf("ui agents: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - data := AgentListPageData{ - Title: "Agents", - Now: time.Now(), - Agents: agents, - } - - s.renderTemplate(w, "agents", data) -} - -func (s *server) handleUIAgentToggle(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - if err := r.ParseForm(); err != nil { - writeError(w, http.StatusBadRequest, "invalid form") - return - } - - id, err := strconv.ParseUint(strings.TrimSpace(r.FormValue("id")), 10, 64) - if err != nil || id == 0 { - writeError(w, http.StatusBadRequest, "invalid id") - return - } - - enabled := strings.TrimSpace(r.FormValue("enabled")) == "1" - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - res, err := s.db.ExecContext(ctx, ` -UPDATE agents -SET is_enabled = ? -WHERE id = ? -`, enabled, id) - if err != nil { - s.logger.Printf("toggle agent: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - affected, _ := res.RowsAffected() - if affected == 0 { - http.NotFound(w, r) - return - } - - http.Redirect(w, r, "/ui/agents", http.StatusSeeOther) -} - -func (s *server) handleUIIndex(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - stats, err := s.getDashboardStats(ctx) - if err != nil { - s.logger.Printf("dashboard stats: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - dets, err := s.listDetections(ctx, "", "", "", "", 20) - if err != nil { - s.logger.Printf("dashboard detections: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - events, err := s.listEvents(ctx, EventFilter{Limit: 20}) - if err != nil { - s.logger.Printf("dashboard events: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - data := DashboardPageData{ - Title: "SIEM-lite Dashboard", - Now: time.Now(), - Stats: stats, - RecentDetections: dets, - RecentEvents: events, - } - - s.renderTemplate(w, "dashboard", data) -} - -func (s *server) handleUIDetections(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - filters := map[string]string{ - "host": strings.TrimSpace(r.URL.Query().Get("host")), - "rule": strings.TrimSpace(r.URL.Query().Get("rule")), - "severity": strings.TrimSpace(r.URL.Query().Get("severity")), - "status": strings.TrimSpace(r.URL.Query().Get("status")), - "limit": strings.TrimSpace(r.URL.Query().Get("limit")), - } - - limit := s.cfg.DetectionsLimit - if filters["limit"] != "" { - if n, err := strconv.Atoi(filters["limit"]); err == nil && n > 0 && n <= 500 { - limit = n - } - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - items, err := s.listDetections(ctx, filters["host"], filters["rule"], filters["severity"], filters["status"], limit) - if err != nil { - s.logger.Printf("ui detections: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - data := DetectionListPageData{ - Title: "Detections", - Now: time.Now(), - Filters: filters, - Detections: items, - } - - s.renderTemplate(w, "detections", data) -} - -func (s *server) handleUIEvents(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - timeFrom := strings.TrimSpace(r.URL.Query().Get("from")) - timeTo := strings.TrimSpace(r.URL.Query().Get("to")) - - if timeFrom == "" && timeTo == "" { - timeFrom = time.Now().UTC().Add(-24 * time.Hour).Format(time.RFC3339) - } - - filter := EventFilter{ - Host: strings.TrimSpace(r.URL.Query().Get("host")), - Channel: strings.TrimSpace(r.URL.Query().Get("channel")), - Rule: strings.TrimSpace(r.URL.Query().Get("rule")), - User: strings.TrimSpace(r.URL.Query().Get("user")), - SrcIP: strings.TrimSpace(r.URL.Query().Get("src_ip")), - Severity: strings.TrimSpace(r.URL.Query().Get("severity")), - TimeFrom: timeFrom, - TimeTo: timeTo, - Limit: 100, - } - - if v := strings.TrimSpace(r.URL.Query().Get("event_id")); v != "" { - if n, err := strconv.Atoi(v); err == nil && n > 0 { - filter.EventID = uint32(n) - } - } - - if v := strings.TrimSpace(r.URL.Query().Get("limit")); v != "" { - if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 1000 { - filter.Limit = n - } - } - - ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) - defer cancel() - - events, err := s.listEvents(ctx, filter) - if err != nil { - s.logger.Printf("ui events: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - filters := map[string]string{ - "host": filter.Host, - "channel": filter.Channel, - "rule": filter.Rule, - "user": filter.User, - "src_ip": filter.SrcIP, - "severity": filter.Severity, - "from": filter.TimeFrom, - "to": filter.TimeTo, - "limit": strconv.Itoa(filter.Limit), - "event_id": func() string { - if filter.EventID == 0 { - return "" - } - return strconv.Itoa(int(filter.EventID)) - }(), - } - - data := EventListPageData{ - Title: "Events", - Now: time.Now(), - Filters: filters, - Events: events, - } - - s.renderTemplate(w, "events", data) -} - -func (s *server) handleUIEventDetail(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - idStr := strings.TrimSpace(r.URL.Query().Get("id")) - if idStr == "" { - writeError(w, http.StatusBadRequest, "missing id") - return - } - - id, err := strconv.ParseUint(idStr, 10, 64) - if err != nil { - writeError(w, http.StatusBadRequest, "invalid id") - return - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - ev, err := s.getEventByID(ctx, id) - if err != nil { - if errors.Is(err, sql.ErrNoRows) { - http.NotFound(w, r) - return - } - s.logger.Printf("ui event detail: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - data := EventDetailPageData{ - Title: "Event Detail", - Now: time.Now(), - Event: ev, - } - s.renderTemplate(w, "event_detail", data) -} - -func (s *server) renderTemplate(w http.ResponseWriter, name string, data any) { - w.Header().Set("Content-Type", "text/html; charset=utf-8") - if err := s.templates.ExecuteTemplate(w, name, data); err != nil { - s.logger.Printf("render template %s: %v", name, err) - http.Error(w, "template error", http.StatusInternalServerError) - } -} - -type EventFilter struct { - Host string - Channel string - Rule string - User string - SrcIP string - Severity string - TimeFrom string - TimeTo string - EventID uint32 - Limit int -} - -func (s *server) getDashboardStats(ctx context.Context) (DashboardStats, error) { - var stats DashboardStats - - if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM agents WHERE is_enabled = 1`).Scan(&stats.AgentsTotal); err != nil { - return stats, err - } - if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM agents WHERE is_enabled = 1 AND last_seen >= ?`, time.Now().UTC().Add(-s.cfg.OfflineAfter)).Scan(&stats.AgentsActive); err != nil { - return stats, err - } - if err := s.db.QueryRowContext(ctx, `SELECT COALESCE(SUM(cnt), 0) FROM event_count_buckets WHERE bucket_start >= ?`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.Events24h); err != nil { - return stats, err - } - if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM detections WHERE created_at >= ?`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.Detections24h); err != nil { - return stats, err - } - if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM detections WHERE created_at >= ? AND severity = 'high'`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.HighDetections24h); err != nil { - return stats, err - } - - if err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) FROM detections WHERE status = 'open' -`).Scan(&stats.OpenDetections); err != nil { - return stats, err - } - - if err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) FROM detections WHERE status = 'investigating' -`).Scan(&stats.InvestigatingDetections); err != nil { - return stats, err - } - - if err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) FROM detections -WHERE created_at >= ? AND severity = 'critical' -`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.CriticalDetections24h); err != nil { - return stats, err - } - - if err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) FROM detections -WHERE created_at >= ? AND status = 'false_positive' -`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.FalsePositive24h); err != nil { - return stats, err - } - - if err := s.db.QueryRowContext(ctx, ` -SELECT COUNT(*) FROM detections -WHERE created_at >= ? AND status = 'legitimate' -`, time.Now().UTC().Add(-24*time.Hour)).Scan(&stats.Legitimate24h); err != nil { - return stats, err - } - - return stats, nil -} - -func (s *server) listEvents(ctx context.Context, f EventFilter) ([]EventRow, error) { - if f.Limit <= 0 || f.Limit > 1000 { - f.Limit = 100 - } - - // User/IP filters need the deliberately small security-context table. Every - // other view reads the universal low-cardinality counter table. In particular - // the /ui start page never touches event_occurrences anymore. - useContext := f.User != "" || f.SrcIP != "" - args := make([]any, 0, 16) - var query string - - if useContext { - query = ` -SELECT hostname, channel_name, event_id, provider_name, - target_user, subject_user, src_ip, workstation, - logon_type, status_text, failure_reason, - cnt, first_event_ts, last_event_ts -FROM event_occurrences -WHERE 1=1 -` - } else { - query = ` -SELECT hostname, channel_name, event_id, '', - '', '', '', '', - '', '', '', - cnt, first_event_ts, last_event_ts -FROM event_count_buckets -WHERE 1=1 -` - } - - if f.Host != "" { - query += ` AND hostname = ?` - args = append(args, f.Host) - } - if f.Channel != "" { - query += ` AND channel_name = ?` - args = append(args, f.Channel) - } - if f.EventID != 0 { - query += ` AND event_id = ?` - args = append(args, f.EventID) - } - if useContext && f.User != "" { - query += ` AND (target_user = ? OR subject_user = ?)` - u := normalizeUsername(f.User) - args = append(args, u, u) - } - if useContext && f.SrcIP != "" { - query += ` AND src_ip = ?` - args = append(args, f.SrcIP) - } - - // Never let an unfiltered UI request range over the whole retention period. - // A user can still explicitly request an older range with from/to. - if f.TimeFrom != "" { - if t, err := parseUIRFC3339(f.TimeFrom); err == nil { - query += ` AND bucket_start >= ?` - bucketSize := s.cfg.BaselineWindow - if useContext { - bucketSize = s.cfg.MetadataBucket - } - args = append(args, bucketStart(t, bucketSize)) - } - } else { - query += ` AND bucket_start >= ?` - bucketSize := s.cfg.BaselineWindow - if useContext { - bucketSize = s.cfg.MetadataBucket - } - args = append(args, bucketStart(time.Now().UTC().Add(-24*time.Hour), bucketSize)) - } - if f.TimeTo != "" { - if t, err := parseUIRFC3339(f.TimeTo); err == nil { - query += ` AND bucket_start <= ?` - args = append(args, t) - } - } - if f.Rule != "" || f.Severity != "" { - tableName := "event_count_buckets" - if useContext { - tableName = "event_occurrences" - } - query += ` AND EXISTS ( - SELECT 1 FROM detections d - WHERE d.hostname = ` + tableName + `.hostname - AND ` + tableName + `.last_event_ts >= d.window_start - AND ` + tableName + `.first_event_ts <= d.window_end` - if f.Rule != "" { - query += ` AND d.rule_name = ?` - args = append(args, f.Rule) - } - if f.Severity != "" { - query += ` AND d.severity = ?` - args = append(args, f.Severity) - } - query += ` )` - } - query += ` ORDER BY bucket_start DESC LIMIT ?` - args = append(args, f.Limit) - - rows, err := s.db.QueryContext(ctx, query, args...) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []EventRow - for rows.Next() { - var ev EventRow - if err := rows.Scan( - &ev.Hostname, &ev.Channel, &ev.EventID, &ev.ProviderName, - &ev.TargetUser, &ev.SubjectUser, &ev.SrcIP, &ev.Workstation, - &ev.LogonType, &ev.StatusText, &ev.FailureReason, - &ev.Count, &ev.FirstSeen, &ev.LastSeen, - ); err != nil { - return nil, err - } - ev.FirstSeen = normalizeTime(ev.FirstSeen) - ev.LastSeen = normalizeTime(ev.LastSeen) - ev.Time = ev.LastSeen - ev.Aggregated = true - ev.Message = fmt.Sprintf("%s EventID %d: %d Vorkommen", ev.Channel, ev.EventID, ev.Count) - out = append(out, ev) - } - return out, rows.Err() -} - -func (s *server) getEventByID(ctx context.Context, id uint64) (EventRow, error) { - const q = ` -SELECT id, hostname, channel_name, event_id, source, computer, provider_name, - target_user, target_domain, subject_user, subject_domain, - workstation, src_ip, src_port, logon_type, process_name, - authentication_package, logon_process, status_text, sub_status_text, - failure_reason, ts, received_at -FROM event_logs -WHERE id = ? -LIMIT 1 -` - - var ev EventRow - err := s.db.QueryRowContext(ctx, q, id).Scan( - &ev.ID, &ev.Hostname, &ev.Channel, &ev.EventID, &ev.Source, - &ev.Computer, &ev.ProviderName, &ev.TargetUser, &ev.TargetDomain, - &ev.SubjectUser, &ev.SubjectDomain, &ev.Workstation, &ev.SrcIP, - &ev.SrcPort, &ev.LogonType, &ev.ProcessName, &ev.AuthenticationPackage, - &ev.LogonProcess, &ev.StatusText, &ev.SubStatusText, - &ev.FailureReason, &ev.Time, &ev.ReceivedAt, - ) - if err != nil { - return ev, err - } - - ev.Time = normalizeTime(ev.Time) - ev.ReceivedAt = normalizeTime(ev.ReceivedAt) - - raw, err := s.getRawEventXMLByID(ctx, id) - if err != nil { - return ev, err - } - - ev.Message = raw - return ev, nil -} - -func parseUIRFC3339(v string) (time.Time, error) { - return time.Parse(time.RFC3339, strings.TrimSpace(v)) -} - -func loadConfig() Config { - return Config{ - ListenAddr: getenv("LISTEN_ADDR", ":8080"), - DBDSN: mustGetenv("DB_DSN"), - MaxBodyBytes: getenvInt64("MAX_BODY_BYTES", 10*1024*1024), - HTTPReadTimeout: getenvDuration("HTTP_READ_TIMEOUT", 15*time.Second), - HTTPWriteTimeout: getenvDuration("HTTP_WRITE_TIMEOUT", 30*time.Second), - HTTPIdleTimeout: getenvDuration("HTTP_IDLE_TIMEOUT", 60*time.Second), - DBMaxOpenConns: getenvInt("DB_MAX_OPEN_CONNS", 50), - DBMaxIdleConns: getenvInt("DB_MAX_IDLE_CONNS", 25), - DBConnMaxLifetime: getenvDuration("DB_CONN_MAX_LIFETIME", 3*time.Minute), - DBConnMaxIdleTime: getenvDuration("DB_CONN_MAX_IDLE_TIME", 1*time.Minute), - DetectionInterval: getenvDuration("DETECTION_INTERVAL", 1*time.Minute), - OfflineAfter: getenvDuration("OFFLINE_AFTER", 10*time.Minute), - OfflineAlertMax: getenvDuration("OFFLINE_ALERT_MAX", 120*time.Minute), - FailedLogonWindow: getenvDuration("FAILED_LOGON_WINDOW", 5*time.Minute), - FailedLogonThreshold: getenvInt("FAILED_LOGON_THRESHOLD", 25), - RebootWindow: getenvDuration("REBOOT_WINDOW", 15*time.Minute), - RebootThreshold: getenvInt("REBOOT_THRESHOLD", 3), - PasswordSprayWindow: getenvDuration("PASSWORD_SPRAY_WINDOW", 5*time.Minute), - PasswordSprayMinUsers: getenvInt("PASSWORD_SPRAY_MIN_USERS", 5), - PasswordSprayMinAttempts: getenvInt("PASSWORD_SPRAY_MIN_ATTEMPTS", 15), - SuccessAfterFailureWindow: getenvDuration("SUCCESS_AFTER_FAILURE_WINDOW", 10*time.Minute), - NewSourceIPLookback: getenvDuration("NEW_SOURCE_IP_LOOKBACK", 30*24*time.Hour), - NewSourceIPWindow: getenvDuration("NEW_SOURCE_IP_WINDOW", 10*time.Minute), - DetectionsLimit: getenvInt("DETECTIONS_LIMIT", 100), - - NewEventIDMode: strings.ToLower(getenv("NEW_EVENT_ID_MODE", "inventory")), - NewEventIDLearningPeriod: getenvDuration("NEW_EVENT_ID_LEARNING_PERIOD", 24*time.Hour), - NewEventIDConfirmWindow: getenvDuration("NEW_EVENT_ID_CONFIRM_WINDOW", 15*time.Minute), - NewEventIDMinCount: getenvInt("NEW_EVENT_ID_MIN_COUNT", 3), - NewEventIDIgnoreChannels: getenvCSV("NEW_EVENT_ID_IGNORE_CHANNELS", "Microsoft-Windows-WMI-Activity/Operational"), - NewEventIDAlertChannels: getenvCSV("NEW_EVENT_ID_ALERT_CHANNELS", "Security,System"), - NewEventIDHighRiskIDs: getenvUint32Set("NEW_EVENT_ID_HIGH_RISK_IDS", "1102,4697,4719,7045"), - - EnrollmentKey: mustGetenv("ENROLLMENT_KEY"), - - BaselineEnabled: getenvBool("BASELINE_ENABLED", true), - BaselineWindow: getenvDuration("BASELINE_WINDOW", 5*time.Minute), - BaselineMinSamples: getenvInt("BASELINE_MIN_SAMPLES", 24), - BaselineMinCount: getenvInt("BASELINE_MIN_COUNT", 10), - BaselineMediumZScore: getenvFloat("BASELINE_MEDIUM_Z", 2.5), - BaselineHighZScore: getenvFloat("BASELINE_HIGH_Z", 4.0), - BaselineSuppressFor: getenvDuration("BASELINE_SUPPRESS_FOR", 1*time.Hour), - - Timezone: getenv("TZ", "Europe/Berlin"), - - UEBAEnabled: getenvBool("UEBA_ENABLED", true), - UEBALookback: getenvDuration("UEBA_LOOKBACK", 30*24*time.Hour), - UEBANewContextWindow: getenvDuration("UEBA_NEW_CONTEXT_WINDOW", 10*time.Minute), - RiskScoreWindow: getenvDuration("RISK_SCORE_WINDOW", 24*time.Hour), - - PartitionMaintenanceEnabled: getenvBool("PARTITION_MAINTENANCE_ENABLED", true), - PartitionMaintenanceInterval: getenvDuration("PARTITION_MAINTENANCE_INTERVAL", 15*time.Minute), - PartitionInterval: getenvDuration("PARTITION_INTERVAL", 3*time.Hour), - PartitionAhead: getenvDuration("PARTITION_AHEAD", 24*time.Hour), - PartitionBehind: getenvDuration("PARTITION_BEHIND", 6*time.Hour), - PartitionRetention: getenvDuration("PARTITION_RETENTION", 30*24*time.Hour), - - StoreEventRows: getenvBool("STORE_EVENT_ROWS", false), - StoreRawXML: getenvBool("STORE_RAW_XML", false), - MetadataBucket: getenvDuration("METADATA_BUCKET", 5*time.Minute), - EventRetention: getenvDuration("EVENT_RETENTION", 6*time.Hour), - RawRetention: getenvDuration("RAW_RETENTION", 6*time.Hour), - MetadataRetention: getenvDuration("METADATA_RETENTION", 30*24*time.Hour), - MetadataContextEventIDs: getenvUint32Set("METADATA_CONTEXT_EVENT_IDS", "4624,4625,4648,4672,4688,4697,4720,4726,4728,4732,4740,4756,4768,4769,4771,4776,7045"), - } -} - -func getenvCSV(key, def string) []string { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - v = def - } - if strings.TrimSpace(v) == "" { - return nil - } - parts := strings.Split(v, ",") - out := make([]string, 0, len(parts)) - for _, part := range parts { - part = strings.TrimSpace(part) - if part != "" { - out = append(out, part) - } - } - return out -} - -func getenvUint32Set(key, def string) map[uint32]struct{} { - values := getenvCSV(key, def) - out := make(map[uint32]struct{}, len(values)) - for _, value := range values { - n, err := strconv.ParseUint(value, 10, 32) - if err != nil || n == 0 { - log.Fatalf("invalid event id %q in %s", value, key) - } - out[uint32(n)] = struct{}{} - } - return out -} - -func containsFold(values []string, candidate string) bool { - candidate = strings.TrimSpace(candidate) - for _, value := range values { - if strings.EqualFold(strings.TrimSpace(value), candidate) { - return true - } - } - return false -} - -func maxInt(a, b int) int { - if a > b { - return a - } - return b -} - -func getenvBool(key string, def bool) bool { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - - switch strings.ToLower(v) { - case "1", "true", "yes", "y", "on": - return true - case "0", "false", "no", "n", "off": - return false - default: - log.Fatalf("invalid bool for %s: %s", key, v) - return def - } -} - -func getenvFloat(key string, def float64) float64 { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - - f, err := strconv.ParseFloat(v, 64) - if err != nil { - log.Fatalf("invalid float for %s: %v", key, err) - } - - return f -} - -func (s *server) handleHealthz(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - ingestRejectedTotal.WithLabelValues("method_not_allowed").Inc() - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - writeJSON(w, http.StatusOK, map[string]any{ - "status": "ok", - "uptime_sec": int(time.Since(s.startTime).Seconds()), - }) -} - -func (s *server) handleReadyz(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second) - defer cancel() - if err := s.db.PingContext(ctx); err != nil { - writeError(w, http.StatusServiceUnavailable, "database not ready") - return - } - writeJSON(w, http.StatusOK, map[string]string{"status": "ready"}) -} - -func (s *server) handleIngest(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - ingestRejectedTotal.WithLabelValues("method_not_allowed").Inc() - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - apiKey := strings.TrimSpace(r.Header.Get("X-API-Key")) - enrollmentKey := strings.TrimSpace(r.Header.Get("X-Enrollment-Key")) - if apiKey == "" { - ingestRejectedTotal.WithLabelValues("missing_api_key").Inc() - writeError(w, http.StatusUnauthorized, "missing api key") - return - } - - r.Body = http.MaxBytesReader(w, r.Body, s.cfg.MaxBodyBytes) - defer r.Body.Close() - - dec := json.NewDecoder(r.Body) - dec.DisallowUnknownFields() - - var batch []LogPayload - if err := dec.Decode(&batch); err != nil { - ingestRejectedTotal.WithLabelValues("invalid_json").Inc() - writeError(w, http.StatusBadRequest, "invalid json") - return - } - - if len(batch) == 0 { - ingestRejectedTotal.WithLabelValues("empty_batch").Inc() - writeError(w, http.StatusBadRequest, "empty batch") - return - } - if len(batch) > 1000 { - ingestRejectedTotal.WithLabelValues("batch_too_large").Inc() - writeError(w, http.StatusBadRequest, "batch too large") - return - } - - for i := range batch { - if err := validatePayload(&batch[i]); err != nil { - ingestRejectedTotal.WithLabelValues("invalid_payload").Inc() - writeError(w, http.StatusBadRequest, fmt.Sprintf("invalid payload at index %d: %v", i, err)) - return - } - } - - hostname := batch[0].Hostname - for i := 1; i < len(batch); i++ { - if batch[i].Hostname != hostname { - ingestRejectedTotal.WithLabelValues("mixed_hostnames").Inc() - writeError(w, http.StatusBadRequest, "all events in a batch must use the same hostname") - return - } - } - - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - agentID, err := s.authenticateTouchOrEnrollAgent( - ctx, - hostname, - apiKey, - enrollmentKey, - clientIP(r.RemoteAddr), - ) - if err != nil { - if errors.Is(err, errUnauthorized) { - ingestRejectedTotal.WithLabelValues("unauthorized").Inc() - writeError(w, http.StatusUnauthorized, "invalid api key or hostname") - return - } - ingestRejectedTotal.WithLabelValues("auth_error").Inc() - s.logger.Printf("authenticate agent: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - if err := s.insertBatch(ctx, agentID, batch); err != nil { - dbInsertFailuresTotal.Inc() - s.logger.Printf("insert batch: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - ingestBatchesTotal.Inc() - for _, item := range batch { - ingestEventsTotal.WithLabelValues(item.Channel, strconv.FormatUint(uint64(item.EventID), 10)).Inc() - } - - writeJSON(w, http.StatusAccepted, ingestResponse{Accepted: len(batch)}) -} - -func (s *server) handleDetections(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { - writeError(w, http.StatusMethodNotAllowed, "method not allowed") - return - } - - limit := s.cfg.DetectionsLimit - if v := strings.TrimSpace(r.URL.Query().Get("limit")); v != "" { - if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 500 { - limit = n - } - } - - host := strings.TrimSpace(r.URL.Query().Get("host")) - rule := strings.TrimSpace(r.URL.Query().Get("rule")) - severity := strings.TrimSpace(r.URL.Query().Get("severity")) - status := strings.TrimSpace(r.URL.Query().Get("status")) - ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) - defer cancel() - - items, err := s.listDetections(ctx, host, rule, severity, status, limit) - if err != nil { - s.logger.Printf("list detections: %v", err) - writeError(w, http.StatusInternalServerError, "internal error") - return - } - - writeJSON(w, http.StatusOK, items) -} - -func (s *server) authenticateTouchOrEnrollAgent(ctx context.Context, hostname, apiKey, enrollmentKey, remoteIP string) (uint64, error) { - const q = ` -SELECT id, api_key_hash, is_enabled -FROM agents -WHERE hostname = ? -LIMIT 1 -` - - var id uint64 - var storedHash string - var enabled bool - - err := s.db.QueryRowContext(ctx, q, hostname).Scan(&id, &storedHash, &enabled) - if err != nil { - if errors.Is(err, sql.ErrNoRows) { - if enrollmentKey == "" { - return 0, errUnauthorized - } - if !constantTimeEqual(sha256Hex(enrollmentKey), sha256Hex(s.cfg.EnrollmentKey)) { - return 0, errUnauthorized - } - return s.enrollNewAgent(ctx, hostname, apiKey, remoteIP) - } - return 0, err - } - - if !enabled { - return 0, errUnauthorized - } - - if !constantTimeEqual(strings.ToLower(storedHash), sha256Hex(apiKey)) { - return 0, errUnauthorized - } - - const upd = ` -UPDATE agents -SET last_seen = UTC_TIMESTAMP(6), last_ip = ? -WHERE id = ? -` - if _, err := s.db.ExecContext(ctx, upd, remoteIP, id); err != nil { - return 0, err - } - - return id, nil -} - -func (s *server) enrollNewAgent(ctx context.Context, hostname, apiKey, remoteIP string) (uint64, error) { - if strings.TrimSpace(hostname) == "" { - return 0, errUnauthorized - } - if strings.TrimSpace(apiKey) == "" { - return 0, errUnauthorized - } - - tx, err := s.db.BeginTx(ctx, nil) - if err != nil { - return 0, err - } - defer func() { _ = tx.Rollback() }() - - const ins = ` -INSERT INTO agents (hostname, api_key_hash, first_seen, last_seen, last_ip, is_enabled) -VALUES (?, ?, UTC_TIMESTAMP(6), UTC_TIMESTAMP(6), ?, 1) -` - res, err := tx.ExecContext(ctx, ins, hostname, sha256Hex(apiKey), remoteIP) - if err != nil { - return 0, err - } - - id, err := res.LastInsertId() - if err != nil { - return 0, err - } - - if err := tx.Commit(); err != nil { - return 0, err - } - - s.logger.Printf("agent auto-enrolled: host=%s ip=%s", hostname, remoteIP) - return uint64(id), nil -} - -var errUnauthorized = errors.New("unauthorized") - -func shouldStoreMetadataContext(cfg Config, channel string, eventID uint32) bool { - if len(cfg.MetadataContextEventIDs) == 0 { - return false - } - _, ok := cfg.MetadataContextEventIDs[eventID] - if !ok { - return false - } - // Context extraction is currently designed around Windows Security semantics. - // Event 7045 and other non-Security IDs can still be counted universally via - // event_count_buckets without creating a high-cardinality context row. - return strings.EqualFold(strings.TrimSpace(channel), "Security") -} - -func compactMetadataContext(eventID uint32, norm NormalizedEvent) (targetUser, subjectUser, srcIP, workstation, logonType, statusText, failureReason string) { - targetUser = normalizeUsername(norm.TargetUser) - subjectUser = normalizeUsername(norm.SubjectUser) - srcIP = strings.TrimSpace(norm.SrcIP) - workstation = strings.TrimSpace(norm.Workstation) - logonType = strings.TrimSpace(norm.LogonType) - statusText = strings.TrimSpace(norm.StatusText) - failureReason = strings.TrimSpace(norm.FailureReason) - - // Keep only dimensions that are actually useful for the event family. This is - // the key protection against event_occurrences degenerating into one row/event. - switch eventID { - case 4624, 4625: // successful / failed logon - subjectUser = "" - case 4672: // special privileges assigned - targetUser, srcIP, workstation, logonType, statusText, failureReason = "", "", "", "", "", "" - case 4740: // account locked out - subjectUser, srcIP, logonType, statusText, failureReason = "", "", "", "", "" - case 4768, 4769, 4771, 4776: // Kerberos / credential validation - subjectUser = "" - default: - // Account/group-management events benefit mostly from actor + target. - srcIP, logonType, statusText, failureReason = "", "", "", "" - } - - if len(failureReason) > 255 { - failureReason = failureReason[:255] - } - return -} - -func retainCompactContext(eventID uint32, targetUser, subjectUser string) bool { - switch eventID { - case 4624, 4625, 4740, 4768, 4769, 4771, 4776: - return !isNoiseAccount(targetUser) - case 4672: - return !isNoiseAccount(subjectUser) - default: - return !isNoiseAccount(targetUser) || !isNoiseAccount(subjectUser) - } -} - -func (s *server) insertBatch(ctx context.Context, agentID uint64, batch []LogPayload) error { - start := time.Now() - - tx, err := s.db.BeginTx(ctx, nil) - if err != nil { - return err - } - defer func() { _ = tx.Rollback() }() - - // Full per-event rows are now strictly opt-in. STORE_RAW_XML implies an event - // row because event_log_raw references event_log_id. - storeEventRows := s.cfg.StoreEventRows || s.cfg.StoreRawXML - var sb strings.Builder - args := make([]any, 0, len(batch)*30) - if storeEventRows { - sb.WriteString(` -INSERT INTO event_logs ( - agent_id, hostname, channel_name, event_id, source, computer, provider_name, - level_value, task_value, opcode_value, keywords, - target_user, target_user_norm, target_domain, - subject_user, subject_user_norm, subject_domain, - workstation, src_ip, src_port, logon_type, process_name, - authentication_package, logon_process, status_text, sub_status_text, - failure_reason, ts, received_at, msg, msg_sha256 -) VALUES -`) - } - - bucketAgg := make(map[string]*EventCountBucketAgg) - occurrenceAgg := make(map[string]*MetadataOccurrenceAgg) - rawEvents := make([]RawEventInsert, 0, len(batch)) - privilegedLogonMetric := make(map[string]int) - privilegedFailureMetric := make(map[string]int) - privilegedGrantMetric := make(map[string]int) - - for i, item := range batch { - msgHash := payloadFingerprint(item) - norm := NormalizeEventXML(item.Message) - if item.Metadata != nil { - norm = overlayMetadata(norm, *item.Metadata) - } - realHost := firstNonEmpty(norm.Computer, item.Hostname) - targetUserNorm := normalizeUsername(norm.TargetUser) - subjectUserNorm := normalizeUsername(norm.SubjectUser) - providerName := firstNonEmpty(norm.ProviderName, item.Source) - eventTS := item.Time.UTC() - - if storeEventRows { - if i > 0 { - sb.WriteString(",") - } - sb.WriteString("(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,UTC_TIMESTAMP(6),'',?)") - args = append(args, - agentID, realHost, item.Channel, item.EventID, item.Source, norm.Computer, providerName, - norm.LevelValue, norm.TaskValue, norm.OpcodeValue, norm.Keywords, - norm.TargetUser, targetUserNorm, norm.TargetDomain, - norm.SubjectUser, subjectUserNorm, norm.SubjectDomain, - norm.Workstation, norm.SrcIP, norm.SrcPort, norm.LogonType, norm.ProcessName, - norm.AuthenticationPackage, norm.LogonProcess, norm.StatusText, norm.SubStatusText, - norm.FailureReason, eventTS, msgHash, - ) - } - - if s.cfg.StoreRawXML && strings.TrimSpace(item.Message) != "" { - rawEvents = append(rawEvents, RawEventInsert{ - EventOffset: i, - Message: item.Message, - SHA256: msgHash, - Time: eventTS, - }) - } - - // Universal low-cardinality counter: one row per host/channel/event ID and - // baseline bucket, regardless of how many individual events arrive. - baselineBucketSize := s.cfg.BaselineWindow - if baselineBucketSize <= 0 { - baselineBucketSize = 5 * time.Minute - } - bs := bucketStart(eventTS, baselineBucketSize) - be := bs.Add(baselineBucketSize) - bucketKey := fmt.Sprintf("%s|%s|%s|%d", bs.Format(time.RFC3339Nano), realHost, item.Channel, item.EventID) - agg := bucketAgg[bucketKey] - if agg == nil { - agg = &EventCountBucketAgg{ - BucketStart: bs, BucketEnd: be, Hostname: realHost, Channel: item.Channel, - EventID: item.EventID, FirstTS: eventTS, LastTS: eventTS, - } - bucketAgg[bucketKey] = agg - } - agg.Count++ - if eventTS.Before(agg.FirstTS) { - agg.FirstTS = eventTS - } - if eventTS.After(agg.LastTS) { - agg.LastTS = eventTS - } - - // Context rows are intentionally restricted to a small allow-list of - // security events. All other events exist only in event_count_buckets and - // event_catalog, preventing metadata storage from mirroring raw cardinality. - if shouldStoreMetadataContext(s.cfg, item.Channel, item.EventID) { - metadataBucketSize := s.cfg.MetadataBucket - if metadataBucketSize <= 0 { - metadataBucketSize = 5 * time.Minute - } - mbs := bucketStart(eventTS, metadataBucketSize) - mbe := mbs.Add(metadataBucketSize) - target, subject, srcIP, workstation, logonType, statusText, failureReason := compactMetadataContext(item.EventID, norm) - if retainCompactContext(item.EventID, target, subject) { - dimKey := metadataDimensionKey( - realHost, item.Channel, strconv.FormatUint(uint64(item.EventID), 10), - target, subject, srcIP, workstation, logonType, statusText, - ) - occKey := mbs.Format(time.RFC3339Nano) + "|" + hex.EncodeToString(dimKey) - occ := occurrenceAgg[occKey] - if occ == nil { - occ = &MetadataOccurrenceAgg{ - BucketStart: mbs, BucketEnd: mbe, DimensionKey: dimKey, - Hostname: realHost, Channel: item.Channel, EventID: item.EventID, - ProviderName: providerName, TargetUser: target, SubjectUser: subject, - SrcIP: srcIP, Workstation: workstation, LogonType: logonType, - StatusText: statusText, FailureReason: failureReason, - FirstTS: eventTS, LastTS: eventTS, - } - occurrenceAgg[occKey] = occ - } - occ.Count++ - if eventTS.Before(occ.FirstTS) { - occ.FirstTS = eventTS - } - if eventTS.After(occ.LastTS) { - occ.LastTS = eventTS - // Preserve the most recent descriptive values without making them - // part of the uniqueness key. - occ.ProviderName = providerName - occ.FailureReason = failureReason - } - } - } - - if item.Channel == "Security" && item.EventID == 4624 && targetUserNorm != "" { - privilegedLogonMetric[targetUserNorm+"\x00"+realHost]++ - } - if item.Channel == "Security" && item.EventID == 4625 && targetUserNorm != "" { - privilegedFailureMetric[targetUserNorm+"\x00"+realHost]++ - } - if item.Channel == "Security" && item.EventID == 4672 && subjectUserNorm != "" { - privilegedGrantMetric[subjectUserNorm+"\x00"+realHost]++ - } - } - - var firstID int64 - if storeEventRows { - res, err := tx.ExecContext(ctx, sb.String(), args...) - if err != nil { - return err - } - firstID, err = res.LastInsertId() - if err != nil { - return fmt.Errorf("event_logs LastInsertId: %w", err) - } - affected, err := res.RowsAffected() - if err != nil { - return fmt.Errorf("event_logs RowsAffected: %w", err) - } - if affected != int64(len(batch)) { - return fmt.Errorf("event_logs insert affected %d rows, expected %d", affected, len(batch)) - } - } - - if s.cfg.StoreRawXML { - if err := insertRawEventsTx(ctx, tx, uint64(firstID), rawEvents); err != nil { - return err - } - } - if err := upsertEventCountBucketsTx(ctx, tx, bucketAgg); err != nil { - return err - } - if err := upsertEventCatalogTx(ctx, tx, bucketAgg); err != nil { - return err - } - if err := upsertMetadataOccurrencesTx(ctx, tx, occurrenceAgg); err != nil { - return err - } - if err := tx.Commit(); err != nil { - return err - } - - // Privileged-user metrics are intentionally evaluated after the transaction and - // only once per unique user. Older builds performed a DB lookup per event. - privilegedCache := make(map[string]bool) - checkPrivileged := func(user string) bool { - if v, ok := privilegedCache[user]; ok { - return v - } - v, err := s.detector.isPrivilegedUser(ctx, user) - if err != nil { - s.logger.Printf("privileged user check failed for %s: %v", user, err) - privilegedCache[user] = false - return false - } - privilegedCache[user] = v - return v - } - emitMetric := func(values map[string]int, counter *prometheus.CounterVec) { - for key, count := range values { - parts := strings.SplitN(key, "\x00", 2) - if len(parts) != 2 || !checkPrivileged(parts[0]) { - continue - } - counter.WithLabelValues(parts[0], parts[1]).Add(float64(count)) - } - } - emitMetric(privilegedLogonMetric, s.detector.privilegedLogonsTotal) - emitMetric(privilegedGrantMetric, s.detector.privilegedLogonsTotal) - emitMetric(privilegedFailureMetric, s.detector.privilegedLogonFailuresTotal) - - dbBatchSizeHist.Observe(float64(len(batch))) - dbInsertEventsTotal.Add(float64(len(batch))) - dbTxDurationHist.Observe(time.Since(start).Seconds()) - return nil -} - -func (d *detector) finalizeEventCountBuckets(ctx context.Context) error { - _, err := d.db.ExecContext(ctx, ` -UPDATE event_count_buckets -SET finalized = 1, - updated_at = UTC_TIMESTAMP(6) -WHERE finalized = 0 - AND bucket_end < UTC_TIMESTAMP(6) - INTERVAL 2 MINUTE -`) - return err -} - -func (d *detector) markBucketAnomalyChecked(ctx context.Context, bucketStart time.Time, hostname, channel string, eventID uint32) error { - _, err := d.db.ExecContext(ctx, ` -UPDATE event_count_buckets -SET anomaly_checked_at = UTC_TIMESTAMP(6) -WHERE bucket_start = ? - AND hostname = ? - AND channel_name = ? - AND event_id = ? -`, bucketStart.UTC(), hostname, channel, eventID) - - return err -} - -func (d *detector) markBucketLearned(ctx context.Context, bucketStart time.Time, hostname, channel string, eventID uint32) error { - _, err := d.db.ExecContext(ctx, ` -UPDATE event_count_buckets -SET learned_at = UTC_TIMESTAMP(6) -WHERE bucket_start = ? - AND hostname = ? - AND channel_name = ? - AND event_id = ? -`, bucketStart.UTC(), hostname, channel, eventID) - - return err -} - -func insertRawEventsTx(ctx context.Context, tx *sql.Tx, firstEventID uint64, rawEvents []RawEventInsert) error { - if len(rawEvents) == 0 { - return nil - } - - var sb strings.Builder - args := make([]any, 0, len(rawEvents)*4) - - sb.WriteString(` -INSERT INTO event_log_raw -(event_log_id, ts, msg, msg_sha256, created_at) -VALUES -`) - - for i, raw := range rawEvents { - if raw.Time.IsZero() { - return fmt.Errorf("raw event at index %d has zero timestamp", i) - } - if strings.TrimSpace(raw.Message) == "" { - return fmt.Errorf("raw event at index %d has empty message", i) - } - if strings.TrimSpace(raw.SHA256) == "" { - return fmt.Errorf("raw event at index %d has empty sha256", i) - } - - if i > 0 { - sb.WriteString(",") - } - - sb.WriteString("(?,?,?,?,UTC_TIMESTAMP(6))") - - args = append(args, - firstEventID+uint64(raw.EventOffset), - raw.Time.UTC(), - raw.Message, - raw.SHA256, - ) - } - - _, err := tx.ExecContext(ctx, sb.String(), args...) - if err != nil { - return fmt.Errorf("insert event_log_raw: %w", err) - } - - return nil -} - -func (s *server) listDetections(ctx context.Context, host, rule, severity, status string, limit int) ([]Detection, error) { - base := ` -SELECT id, rule_name, severity, hostname, channel_name, event_id, score, - window_start, window_end, summary, details_json, created_at, - status, - COALESCE(analyst_note, ''), - COALESCE(reviewed_by, ''), - reviewed_at, - is_false_positive, - is_legitimate -FROM detections -WHERE 1=1 -` - args := make([]any, 0, 4) - - if host != "" { - base += " AND hostname = ?" - args = append(args, host) - } - if rule != "" { - base += " AND rule_name = ?" - args = append(args, rule) - } - if severity != "" { - base += " AND severity = ?" - args = append(args, severity) - } - if status != "" { - base += " AND status = ?" - args = append(args, status) - } - - base += " ORDER BY created_at DESC LIMIT ?" - args = append(args, limit) - - rows, err := s.db.QueryContext(ctx, base, args...) - if err != nil { - return nil, err - } - defer rows.Close() - - var out []Detection - for rows.Next() { - var d Detection - if err := rows.Scan( - &d.ID, - &d.RuleName, - &d.Severity, - &d.Hostname, - &d.Channel, - &d.EventID, - &d.Score, - &d.WindowStart, - &d.WindowEnd, - &d.Summary, - &d.Details, - &d.CreatedAt, - &d.Status, - &d.AnalystNote, - &d.ReviewedBy, - &d.ReviewedAt, - &d.IsFalsePositive, - &d.IsLegitimate, - ); err != nil { - return nil, err - } - out = append(out, d) - } - return out, rows.Err() -} - -func (s *server) runDetectionLoop() { - ticker := time.NewTicker(s.cfg.DetectionInterval) - defer ticker.Stop() - - s.runDetectionsOnce() - - for range ticker.C { - s.runDetectionsOnce() - } -} - -func (d *detector) runBaselineUpdate(ctx context.Context) error { - if !d.cfg.BaselineEnabled { - return nil - } - - rows, err := d.db.QueryContext(ctx, ` -SELECT - bucket_start, - bucket_end, - hostname, - channel_name, - event_id, - cnt -FROM event_count_buckets -WHERE finalized = 1 - AND learned_at IS NULL -ORDER BY bucket_start ASC -LIMIT 5000 -`) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var bucketStart time.Time - var bucketEnd time.Time - var b BaselineBucket - - if err := rows.Scan( - &bucketStart, - &bucketEnd, - &b.Hostname, - &b.Channel, - &b.EventID, - &b.Count, - ); err != nil { - return err - } - - bucketStart = bucketStart.UTC() - bucketEnd = bucketEnd.UTC() - - b.Hour = bucketStart.Hour() - b.DayOfWeek = mysqlWeekday(bucketStart) - - excluded, err := d.isBaselineExcluded(ctx, b.Hostname, b.Channel, b.EventID) - if err != nil { - return err - } - - if !excluded { - incident, err := d.hasConfirmedIncidentInWindow(ctx, b.Hostname, b.Channel, b.EventID, bucketStart, bucketEnd) - if err != nil { - return err - } - - if !incident { - if err := d.updateBaselineBucket(ctx, b); err != nil { - return err - } - } - } - - if err := d.markBucketLearned(ctx, bucketStart, b.Hostname, b.Channel, b.EventID); err != nil { - return err - } - } - - return rows.Err() -} - -func mysqlWeekday(t time.Time) int { - // MySQL WEEKDAY(): Monday=0 ... Sunday=6 - // Go Weekday(): Sunday=0 ... Saturday=6 - return (int(t.UTC().Weekday()) + 6) % 7 -} - -func (d *detector) hasConfirmedIncidentInWindow(ctx context.Context, hostname, channel string, eventID uint32, windowStart, windowEnd time.Time) (bool, error) { - var count int - - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM detections -WHERE status = 'confirmed_incident' - AND hostname = ? - AND channel_name = ? - AND event_id = ? - AND window_start < ? - AND window_end > ? -`, - hostname, - channel, - eventID, - windowEnd, - windowStart, - ).Scan(&count) - - if err != nil { - return false, err - } - - return count > 0, nil -} - -func (d *detector) updateBaselineBucket(ctx context.Context, b BaselineBucket) error { - tx, err := d.db.BeginTx(ctx, nil) - if err != nil { - return err - } - defer func() { _ = tx.Rollback() }() - - var stat BaselineStat - - err = tx.QueryRowContext(ctx, ` -SELECT avg_count, m2_count, stddev_count, sample_count -FROM baseline_event_stats -WHERE hostname = ? - AND channel_name = ? - AND event_id = ? - AND hour_of_day = ? - AND day_of_week = ? -FOR UPDATE -`, - b.Hostname, - b.Channel, - b.EventID, - b.Hour, - b.DayOfWeek, - ).Scan( - &stat.AvgCount, - &stat.M2Count, - &stat.StddevCount, - &stat.SampleCount, - ) - - if err != nil && !errors.Is(err, sql.ErrNoRows) { - return err - } - - x := float64(b.Count) - - if errors.Is(err, sql.ErrNoRows) { - _, err := tx.ExecContext(ctx, ` -INSERT INTO baseline_event_stats -(hostname, channel_name, event_id, hour_of_day, day_of_week, - avg_count, m2_count, stddev_count, sample_count) -VALUES (?, ?, ?, ?, ?, ?, 0, 0, 1) -`, - b.Hostname, - b.Channel, - b.EventID, - b.Hour, - b.DayOfWeek, - x, - ) - if err != nil { - return err - } - - return tx.Commit() - } - - newSamples := stat.SampleCount + 1 - delta := x - stat.AvgCount - newAvg := stat.AvgCount + delta/float64(newSamples) - delta2 := x - newAvg - newM2 := stat.M2Count + delta*delta2 - - newStddev := 0.0 - if newSamples > 1 { - newStddev = math.Sqrt(newM2 / float64(newSamples-1)) - } - - _, err = tx.ExecContext(ctx, ` -UPDATE baseline_event_stats -SET avg_count = ?, - m2_count = ?, - stddev_count = ?, - sample_count = ?, - last_updated = UTC_TIMESTAMP(6) -WHERE hostname = ? - AND channel_name = ? - AND event_id = ? - AND hour_of_day = ? - AND day_of_week = ? -`, - newAvg, - newM2, - newStddev, - newSamples, - b.Hostname, - b.Channel, - b.EventID, - b.Hour, - b.DayOfWeek, - ) - if err != nil { - return err - } - - return tx.Commit() -} - -func (d *detector) runBaselineAnomalyRule(ctx context.Context) error { - if !d.cfg.BaselineEnabled { - return nil - } - - rows, err := d.db.QueryContext(ctx, ` -SELECT - e.bucket_start, - e.bucket_end, - e.hostname, - e.channel_name, - e.event_id, - e.cnt, - COALESCE(b.avg_count, 0), - COALESCE(b.stddev_count, 0), - COALESCE(b.sample_count, 0) -FROM event_count_buckets e -LEFT JOIN baseline_event_stats b - ON b.hostname = e.hostname - AND b.channel_name = e.channel_name - AND b.event_id = e.event_id - AND b.hour_of_day = HOUR(e.bucket_start) - AND b.day_of_week = WEEKDAY(e.bucket_start) -WHERE e.finalized = 1 - AND e.anomaly_checked_at IS NULL -ORDER BY e.bucket_start ASC -LIMIT 5000 -`) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var bucketStart time.Time - var bucketEnd time.Time - var host string - var channel string - var eventID uint32 - var count int - var avg float64 - var stddev float64 - var samples int - - if err := rows.Scan( - &bucketStart, - &bucketEnd, - &host, - &channel, - &eventID, - &count, - &avg, - &stddev, - &samples, - ); err != nil { - return err - } - - bucketStart = bucketStart.UTC() - bucketEnd = bucketEnd.UTC() - - eventIDStr := strconv.Itoa(int(eventID)) - - d.baselineCurrentCountGauge.WithLabelValues(host, channel, eventIDStr).Set(float64(count)) - d.baselineAverageGauge.WithLabelValues(host, channel, eventIDStr).Set(avg) - d.baselineStddevGauge.WithLabelValues(host, channel, eventIDStr).Set(stddev) - d.baselineSamplesGauge.WithLabelValues(host, channel, eventIDStr).Set(float64(samples)) - - if samples >= d.cfg.BaselineMinSamples && - count >= d.cfg.BaselineMinCount && - stddev > 0 { - - z := (float64(count) - avg) / stddev - - if z >= d.cfg.BaselineMediumZScore { - severity := "medium" - if z >= d.cfg.BaselineHighZScore { - severity = "high" - } - - suppressed, err := d.isBaselineSuppressed(ctx, host, channel, eventID, bucketEnd) - if err != nil { - return err - } - - if !suppressed { - created, err := d.insertDetection(ctx, Detection{ - RuleName: "baseline_event_rate_anomaly", - Severity: severity, - Hostname: host, - Channel: channel, - EventID: eventID, - Score: z, - WindowStart: bucketStart, - WindowEnd: bucketEnd, - Summary: fmt.Sprintf( - "Baseline-Anomalie auf %s: %s EventID %d kam %d-mal im Bucket %s bis %s, normal %.2f ± %.2f, z=%.2f", - host, - channel, - eventID, - count, - bucketStart.Format(time.RFC3339), - bucketEnd.Format(time.RFC3339), - avg, - stddev, - z, - ), - Details: mustJSON(map[string]any{ - "hostname": host, - "channel": channel, - "event_id": eventID, - "count": count, - "avg_count": avg, - "stddev_count": stddev, - "z_score": z, - "sample_count": samples, - "hour_of_day": bucketStart.Hour(), - "day_of_week": mysqlWeekday(bucketStart), - "bucket_start": bucketStart.Format(time.RFC3339Nano), - "bucket_end": bucketEnd.Format(time.RFC3339Nano), - "window_minutes": int(d.cfg.BaselineWindow.Minutes()), - "min_samples": d.cfg.BaselineMinSamples, - "medium_z": d.cfg.BaselineMediumZScore, - "high_z": d.cfg.BaselineHighZScore, - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("baseline_event_rate_anomaly", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "baseline_event_rate_anomaly").Set(z) - } - } - } - } - - if err := d.markBucketAnomalyChecked(ctx, bucketStart, host, channel, eventID); err != nil { - return err - } - } - - return rows.Err() -} - -func (d *detector) isBaselineSuppressed(ctx context.Context, hostname, channel string, eventID uint32, now time.Time) (bool, error) { - if d.cfg.BaselineSuppressFor <= 0 { - return false, nil - } - - since := now.UTC().Add(-d.cfg.BaselineSuppressFor) - - var count int - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM detections -WHERE rule_name = 'baseline_event_rate_anomaly' - AND hostname = ? - AND channel_name = ? - AND event_id = ? - AND created_at >= ? -`, - hostname, - channel, - eventID, - since, - ).Scan(&count) - if err != nil { - return false, err - } - - return count > 0, nil -} - -func (d *detector) runDynamicRules(ctx context.Context) error { - rows, err := d.db.QueryContext(ctx, ` -SELECT id, name, description, severity, channel, event_ids, - match_field, match_operator, match_value, - threshold_count, threshold_window_seconds, suppress_for_seconds -FROM detection_rules -WHERE enabled = 1 -`) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var r DynamicRule - if err := rows.Scan( - &r.ID, - &r.Name, - &r.Description, - &r.Severity, - &r.Channel, - &r.EventIDs, - &r.MatchField, - &r.MatchOperator, - &r.MatchValue, - &r.ThresholdCount, - &r.ThresholdWindowSeconds, - &r.SuppressForSeconds, - ); err != nil { - return err - } - - if err := d.evaluateDynamicRule(ctx, r); err != nil { - d.logger.Printf("dynamic rule %s error: %v", r.Name, err) - d.ruleErrorsTotal.WithLabelValues("dynamic_" + r.Name).Inc() - continue - } - - d.ruleLastRunGauge.WithLabelValues("dynamic_" + r.Name).Set(float64(time.Now().Unix())) - } - - return rows.Err() -} - -func (d *detector) evaluateDynamicRule(ctx context.Context, r DynamicRule) error { - eventIDs := parseCSVUint32(r.EventIDs) - channels := parseCSVStrings(r.Channel) - - if len(eventIDs) == 0 || len(channels) == 0 { - return nil - } - - if r.ThresholdCount <= 1 || r.ThresholdWindowSeconds <= 0 { - return d.evaluateSingleEventRule(ctx, r, channels, eventIDs) - } - - return d.evaluateThresholdRule(ctx, r, channels, eventIDs) -} - -func dynamicRuleNeedsContext(r DynamicRule) bool { - switch strings.ToLower(strings.TrimSpace(r.MatchField)) { - case "target_user", "subject_user", "src_ip", "workstation": - return true - default: - return false - } -} - -func dynamicRuleNeedsFullEvent(r DynamicRule) bool { - switch strings.ToLower(strings.TrimSpace(r.MatchField)) { - case "process_name", "msg": - return true - default: - return false - } -} - -func metadataRuleMatchCondition(r DynamicRule, context bool) (string, []any) { - if r.MatchField == "" || r.MatchOperator == "" || r.MatchValue == "" { - return "", nil - } - fieldMap := map[string]string{ - "hostname": "hostname", - "channel": "channel_name", - "event_id": "event_id", - } - if context { - fieldMap["target_user"] = "target_user" - fieldMap["subject_user"] = "subject_user" - fieldMap["src_ip"] = "src_ip" - fieldMap["workstation"] = "workstation" - } - col, ok := fieldMap[strings.ToLower(strings.TrimSpace(r.MatchField))] - if !ok { - return "", nil - } - val := strings.TrimSpace(r.MatchValue) - switch strings.ToLower(strings.TrimSpace(r.MatchOperator)) { - case "eq": - return col + " = ?", []any{val} - case "contains": - return col + " LIKE ?", []any{"%" + val + "%"} - case "in": - values := parseCSVStrings(val) - if len(values) == 0 { - return "", nil - } - return buildInClause(strings.TrimSpace(col), len(values))[5:], stringSliceAny(values) - default: - return "", nil - } -} - -func stringSliceAny(in []string) []any { - out := make([]any, 0, len(in)) - for _, v := range in { - out = append(out, v) - } - return out -} - -func (d *detector) evaluateSingleEventRule(ctx context.Context, r DynamicRule, channels []string, eventIDs []uint32) error { - if dynamicRuleNeedsFullEvent(r) { - if !d.cfg.StoreEventRows { - d.logger.Printf("dynamic rule %s skipped: match_field=%s requires STORE_EVENT_ROWS=true", r.Name, r.MatchField) - return nil - } - return d.evaluateSingleEventRuleLegacy(ctx, r, channels, eventIDs) - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.DetectionInterval) - useContext := dynamicRuleNeedsContext(r) - table := "event_count_buckets" - selectFields := "hostname, channel_name, event_id, '', '', '', '', SUM(cnt), MIN(first_event_ts), MAX(last_event_ts)" - groupFields := "hostname, channel_name, event_id" - bucketSize := d.cfg.BaselineWindow - if useContext { - table = "event_occurrences" - selectFields = "hostname, channel_name, event_id, target_user, subject_user, src_ip, workstation, SUM(cnt), MIN(first_event_ts), MAX(last_event_ts)" - groupFields = "hostname, channel_name, event_id, target_user, subject_user, src_ip, workstation" - bucketSize = d.cfg.MetadataBucket - } - - query := "SELECT " + selectFields + " FROM " + table + " WHERE bucket_start >= ? AND bucket_start < ?" - args := []any{bucketStart(windowStart, bucketSize), windowEnd} - query += buildInClause("channel_name", len(channels)) - for _, ch := range channels { - args = append(args, ch) - } - query += buildInClause("event_id", len(eventIDs)) - for _, id := range eventIDs { - args = append(args, id) - } - if cond, condArgs := metadataRuleMatchCondition(r, useContext); cond != "" { - query += " AND " + cond - args = append(args, condArgs...) - } - query += " GROUP BY " + groupFields + " ORDER BY MAX(last_event_ts) DESC LIMIT 500" - - rows, err := d.db.QueryContext(ctx, query, args...) - if err != nil { - return err - } - defer rows.Close() - for rows.Next() { - var host, channel, target, subject, srcIP, workstation string - var eventID uint32 - var count int - var firstSeen, lastSeen time.Time - if err := rows.Scan(&host, &channel, &eventID, &target, &subject, &srcIP, &workstation, &count, &firstSeen, &lastSeen); err != nil { - return err - } - if suppressed, err := d.isDynamicRuleSuppressed(ctx, r, host, lastSeen); err != nil { - return err - } else if suppressed { - continue - } - created, err := d.insertDetection(ctx, Detection{ - RuleName: "dynamic_" + r.Name, Severity: r.Severity, Hostname: host, Channel: channel, EventID: eventID, - Score: float64(maxInt(count, 1)), WindowStart: firstSeen, WindowEnd: lastSeen, - Summary: fmt.Sprintf("Dynamic Rule %s auf %s ausgelöst: EventID %d (%d Vorkommen)", r.Name, host, eventID, count), - Details: mustJSON(map[string]any{"rule_id": r.ID, "rule_name": r.Name, "description": r.Description, "count": count, "target_user": target, "subject_user": subject, "src_ip": srcIP, "workstation": workstation, "storage": table}), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("dynamic_"+r.Name, r.Severity).Inc() - } - } - return rows.Err() -} - -func (d *detector) evaluateThresholdRule(ctx context.Context, r DynamicRule, channels []string, eventIDs []uint32) error { - if dynamicRuleNeedsFullEvent(r) { - if !d.cfg.StoreEventRows { - d.logger.Printf("dynamic rule %s skipped: match_field=%s requires STORE_EVENT_ROWS=true", r.Name, r.MatchField) - return nil - } - return d.evaluateThresholdRuleLegacy(ctx, r, channels, eventIDs) - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(time.Duration(-r.ThresholdWindowSeconds) * time.Second) - useContext := dynamicRuleNeedsContext(r) - table := "event_count_buckets" - bucketSize := d.cfg.BaselineWindow - if useContext { - table = "event_occurrences" - bucketSize = d.cfg.MetadataBucket - } - query := "SELECT hostname, SUM(cnt) AS cnt, MIN(first_event_ts), MAX(last_event_ts) FROM " + table + " WHERE bucket_start >= ? AND bucket_start < ?" - args := []any{bucketStart(windowStart, bucketSize), windowEnd} - query += buildInClause("channel_name", len(channels)) - for _, ch := range channels { - args = append(args, ch) - } - query += buildInClause("event_id", len(eventIDs)) - for _, id := range eventIDs { - args = append(args, id) - } - if cond, condArgs := metadataRuleMatchCondition(r, useContext); cond != "" { - query += " AND " + cond - args = append(args, condArgs...) - } - query += " GROUP BY hostname HAVING SUM(cnt) >= ?" - args = append(args, r.ThresholdCount) - - rows, err := d.db.QueryContext(ctx, query, args...) - if err != nil { - return err - } - defer rows.Close() - for rows.Next() { - var host string - var count int - var firstSeen, lastSeen time.Time - if err := rows.Scan(&host, &count, &firstSeen, &lastSeen); err != nil { - return err - } - if suppressed, err := d.isDynamicRuleSuppressed(ctx, r, host, windowEnd); err != nil { - return err - } else if suppressed { - continue - } - score := float64(count) / float64(r.ThresholdCount) - created, err := d.insertDetection(ctx, Detection{ - RuleName: "dynamic_" + r.Name, Severity: r.Severity, Hostname: host, Score: score, - WindowStart: windowStart, WindowEnd: windowEnd, - Summary: fmt.Sprintf("Dynamic Rule %s auf %s: %d Events in %d Sekunden", r.Name, host, count, r.ThresholdWindowSeconds), - Details: mustJSON(map[string]any{"rule_id": r.ID, "rule_name": r.Name, "description": r.Description, "count": count, "threshold_count": r.ThresholdCount, "threshold_window_sec": r.ThresholdWindowSeconds, "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), "last_seen": lastSeen.UTC().Format(time.RFC3339Nano), "event_ids": r.EventIDs, "channels": r.Channel, "match_field": r.MatchField, "match_operator": r.MatchOperator, "match_value": r.MatchValue, "storage": table}), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("dynamic_"+r.Name, r.Severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "dynamic_"+r.Name).Set(score) - } - } - return rows.Err() -} - -func (d *detector) evaluateSingleEventRuleLegacy(ctx context.Context, r DynamicRule, channels []string, eventIDs []uint32) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.DetectionInterval) - - query := ` -SELECT id, hostname, channel_name, event_id, target_user, subject_user, src_ip, workstation, process_name, msg, ts -FROM event_logs -WHERE ts >= ? AND ts < ? -` - args := []any{windowStart, windowEnd} - - query += buildInClause("channel_name", len(channels)) - for _, ch := range channels { - args = append(args, ch) - } - - query += buildInClause("event_id", len(eventIDs)) - for _, id := range eventIDs { - args = append(args, id) - } - - query += ` ORDER BY ts DESC LIMIT 500` - - rows, err := d.db.QueryContext(ctx, query, args...) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var ev struct { - ID uint64 - Hostname string - Channel string - EventID uint32 - TargetUser string - SubjectUser string - SrcIP string - Workstation string - ProcessName string - Message string - Time time.Time - } - - if err := rows.Scan( - &ev.ID, - &ev.Hostname, - &ev.Channel, - &ev.EventID, - &ev.TargetUser, - &ev.SubjectUser, - &ev.SrcIP, - &ev.Workstation, - &ev.ProcessName, - &ev.Message, - &ev.Time, - ); err != nil { - return err - } - - fields := map[string]string{ - "hostname": ev.Hostname, - "channel": ev.Channel, - "event_id": strconv.Itoa(int(ev.EventID)), - "target_user": ev.TargetUser, - "subject_user": ev.SubjectUser, - "src_ip": ev.SrcIP, - "workstation": ev.Workstation, - "process_name": ev.ProcessName, - "msg": ev.Message, - } - - if !dynamicRuleMatches(r, fields) { - continue - } - - if suppressed, err := d.isDynamicRuleSuppressed(ctx, r, ev.Hostname, ev.Time); err != nil { - return err - } else if suppressed { - continue - } - - summary := fmt.Sprintf("Dynamic Rule %s auf %s ausgelöst: EventID %d", r.Name, ev.Hostname, ev.EventID) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "dynamic_" + r.Name, - Severity: r.Severity, - Hostname: ev.Hostname, - Channel: ev.Channel, - EventID: ev.EventID, - Score: 1.0, - WindowStart: ev.Time, - WindowEnd: ev.Time, - Summary: summary, - Details: mustJSON(map[string]any{ - "rule_id": r.ID, - "rule_name": r.Name, - "description": r.Description, - "event_log_id": ev.ID, - "target_user": ev.TargetUser, - "subject_user": ev.SubjectUser, - "src_ip": ev.SrcIP, - "workstation": ev.Workstation, - "process_name": ev.ProcessName, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("dynamic_"+r.Name, r.Severity).Inc() - } - } - - return rows.Err() -} - -func (d *detector) evaluateThresholdRuleLegacy(ctx context.Context, r DynamicRule, channels []string, eventIDs []uint32) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(time.Duration(-r.ThresholdWindowSeconds) * time.Second) - - query := ` -SELECT hostname, COUNT(*) AS cnt, MIN(ts), MAX(ts) -FROM event_logs -WHERE ts >= ? AND ts < ? -` - args := []any{windowStart, windowEnd} - - query += buildInClause("channel_name", len(channels)) - for _, ch := range channels { - args = append(args, ch) - } - - query += buildInClause("event_id", len(eventIDs)) - for _, id := range eventIDs { - args = append(args, id) - } - - if r.MatchField != "" && r.MatchOperator != "" && r.MatchValue != "" { - sqlCond, sqlArgs := buildSQLMatchCondition(r) - if sqlCond != "" { - query += " AND " + sqlCond - args = append(args, sqlArgs...) - } - } - - query += ` -GROUP BY hostname -HAVING COUNT(*) >= ? -` - args = append(args, r.ThresholdCount) - - rows, err := d.db.QueryContext(ctx, query, args...) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host string - var count int - var firstSeen time.Time - var lastSeen time.Time - - if err := rows.Scan(&host, &count, &firstSeen, &lastSeen); err != nil { - return err - } - - if suppressed, err := d.isDynamicRuleSuppressed(ctx, r, host, windowEnd); err != nil { - return err - } else if suppressed { - continue - } - - score := float64(count) / float64(r.ThresholdCount) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "dynamic_" + r.Name, - Severity: r.Severity, - Hostname: host, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf( - "Dynamic Rule %s auf %s: %d Events in %d Sekunden", - r.Name, - host, - count, - r.ThresholdWindowSeconds, - ), - Details: mustJSON(map[string]any{ - "rule_id": r.ID, - "rule_name": r.Name, - "description": r.Description, - "count": count, - "threshold_count": r.ThresholdCount, - "threshold_window_sec": r.ThresholdWindowSeconds, - "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), - "last_seen": lastSeen.UTC().Format(time.RFC3339Nano), - "event_ids": r.EventIDs, - "channels": r.Channel, - "match_field": r.MatchField, - "match_operator": r.MatchOperator, - "match_value": r.MatchValue, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("dynamic_"+r.Name, r.Severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "dynamic_"+r.Name).Set(score) - } - } - - return rows.Err() -} - -func (d *detector) isDynamicRuleSuppressed(ctx context.Context, r DynamicRule, hostname string, now time.Time) (bool, error) { - if r.SuppressForSeconds <= 0 { - return false, nil - } - - since := now.UTC().Add(time.Duration(-r.SuppressForSeconds) * time.Second) - - var count int - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM detections -WHERE rule_name = ? - AND hostname = ? - AND created_at >= ? -`, "dynamic_"+r.Name, hostname, since).Scan(&count) - if err != nil { - return false, err - } - - return count > 0, nil -} - -func parseCSVStrings(v string) []string { - parts := strings.Split(v, ",") - out := make([]string, 0, len(parts)) - - for _, p := range parts { - p = strings.TrimSpace(p) - if p != "" { - out = append(out, p) - } - } - - return out -} - -func parseCSVUint32(v string) []uint32 { - parts := strings.Split(v, ",") - out := make([]uint32, 0, len(parts)) - - for _, p := range parts { - p = strings.TrimSpace(p) - if p == "" { - continue - } - - n, err := strconv.ParseUint(p, 10, 32) - if err != nil { - continue - } - - out = append(out, uint32(n)) - } - - return out -} - -func buildInClause(field string, count int) string { - if count <= 0 { - return "" - } - - var sb strings.Builder - sb.WriteString(" AND ") - sb.WriteString(field) - sb.WriteString(" IN (") - - for i := 0; i < count; i++ { - if i > 0 { - sb.WriteString(",") - } - sb.WriteString("?") - } - - sb.WriteString(")") - return sb.String() -} - -func dynamicRuleMatches(r DynamicRule, fields map[string]string) bool { - if r.MatchField == "" || r.MatchOperator == "" || r.MatchValue == "" { - return true - } - - actual := strings.TrimSpace(fields[r.MatchField]) - expected := strings.TrimSpace(r.MatchValue) - - switch strings.ToLower(r.MatchOperator) { - case "eq": - return strings.EqualFold(actual, expected) - - case "contains": - return strings.Contains( - strings.ToLower(actual), - strings.ToLower(expected), - ) - - case "in": - values := parseCSVStrings(expected) - for _, v := range values { - if strings.EqualFold(actual, v) { - return true - } - } - return false - - default: - return false - } -} - -func buildSQLMatchCondition(r DynamicRule) (string, []any) { - fieldMap := map[string]string{ - "hostname": "hostname", - "channel": "channel_name", - "event_id": "event_id", - "target_user": "target_user", - "subject_user": "subject_user", - "src_ip": "src_ip", - "workstation": "workstation", - "process_name": "process_name", - "msg": "msg", - } - - col, ok := fieldMap[strings.ToLower(strings.TrimSpace(r.MatchField))] - if !ok { - return "", nil - } - - op := strings.ToLower(strings.TrimSpace(r.MatchOperator)) - val := strings.TrimSpace(r.MatchValue) - - switch op { - case "eq": - return col + " = ?", []any{val} - - case "contains": - return col + " LIKE ?", []any{"%" + val + "%"} - - case "in": - values := parseCSVStrings(val) - if len(values) == 0 { - return "", nil - } - - var sb strings.Builder - sb.WriteString(col) - sb.WriteString(" IN (") - - args := make([]any, 0, len(values)) - for i, v := range values { - if i > 0 { - sb.WriteString(",") - } - sb.WriteString("?") - args = append(args, v) - } - - sb.WriteString(")") - return sb.String(), args - } - - return "", nil -} - -func (s *server) runDetectionsOnce() { - ctx, cancel := context.WithTimeout(context.Background(), s.cfg.DetectionInterval) - defer cancel() - - if err := s.detector.updateAgentMetrics(ctx); err != nil { - s.logger.Printf("update agent metrics: %v", err) - } - - type ruleSpec struct { - name string - timeout time.Duration - fn func(context.Context) error - } - - rules := []ruleSpec{ - {"agent_offline", 15 * time.Second, s.detector.runAgentOfflineRule}, - {"failed_logon_spike", 45 * time.Second, s.detector.runFailedLogonSpikeRule}, - {"reboot_spike", 30 * time.Second, s.detector.runRebootSpikeRule}, - {"new_event_id", 60 * time.Second, s.detector.runNewEventIDRule}, - {"password_spray", 60 * time.Second, s.detector.runPasswordSprayRule}, - {"success_after_failures", 90 * time.Second, s.detector.runSuccessAfterFailuresRule}, - {"new_source_ip_for_user", 90 * time.Second, s.detector.runNewSourceIPForUserRule}, - {"dynamic_rules", 60 * time.Second, s.detector.runDynamicRules}, - - {"ueba_admin_new_host", 90 * time.Second, s.detector.runAdminNewHostRule}, - {"ueba_offhours_login", 60 * time.Second, s.detector.runOffHoursLoginRule}, - {"ueba_first_privileged_use", 60 * time.Second, s.detector.runFirstTimePrivilegedRule}, - - // Diese Regel ist bei dir aktuell der Problemfall. - {"ueba_new_user_context", 180 * time.Second, s.detector.runUEBANewUserContextRule}, - - {"ueba_update", 120 * time.Second, s.detector.runUEBABaselineUpdate}, - } - - for _, rule := range rules { - start := time.Now() - - ctx, cancel := context.WithTimeout(context.Background(), rule.timeout) - err := rule.fn(ctx) - cancel() - - dur := time.Since(start) - - if err != nil { - s.logger.Printf("rule %s error after %s: %v", rule.name, dur, err) - s.detector.ruleErrorsTotal.WithLabelValues(rule.name).Inc() - continue - } - - s.logger.Printf("rule %s completed in %s", rule.name, dur) - - s.detector.ruleLastRunGauge.WithLabelValues(rule.name).Set(float64(time.Now().Unix())) - s.detector.ruleRuntimeHist.WithLabelValues(rule.name).Observe(dur.Seconds()) - } -} - -func (d *detector) runUEBABaselineUpdate(ctx context.Context) error { - if !d.cfg.UEBAEnabled { - return nil - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.UEBANewContextWindow) - - rows, err := d.db.QueryContext(ctx, ` -SELECT target_user, hostname, src_ip, workstation, SUM(cnt) -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4624 - AND bucket_start >= ? AND bucket_start < ? - AND target_user <> '' - AND target_user NOT LIKE '%$' -GROUP BY target_user, hostname, src_ip, workstation -`, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var user, host, srcIP, workstation string - var count int - - if err := rows.Scan(&user, &host, &srcIP, &workstation, &count); err != nil { - return err - } - - _, err := d.db.ExecContext(ctx, ` -INSERT INTO ueba_user_baseline -(username, hostname, src_ip, workstation, first_seen, last_seen, seen_count) -VALUES (?, ?, ?, ?, UTC_TIMESTAMP(6), UTC_TIMESTAMP(6), ?) -ON DUPLICATE KEY UPDATE -last_seen = UTC_TIMESTAMP(6), -seen_count = seen_count + VALUES(seen_count) -`, - user, - host, - srcIP, - workstation, - count, - ) - if err != nil { - return err - } - } - - return rows.Err() -} - -func (d *detector) runUEBANewUserContextRule(ctx context.Context) error { - if !d.cfg.UEBAEnabled { - return nil - } - - start := time.Now() - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.UEBANewContextWindow) - - const q = ` -SELECT - e.hostname, - e.target_user, - e.src_ip, - e.workstation, - MIN(e.first_event_ts) AS first_seen, - SUM(e.cnt) AS cnt -FROM event_occurrences e -WHERE e.channel_name = 'Security' - AND e.event_id = 4624 - AND e.bucket_start >= ? AND e.bucket_start < ? - AND e.target_user <> '' - AND e.target_user NOT LIKE '%$' -GROUP BY e.hostname, e.target_user, e.src_ip, e.workstation -` - - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd) - if err != nil { - return fmt.Errorf("query ueba_new_user_context after %s: %w", time.Since(start), err) - } - defer rows.Close() - - rowCount := 0 - newCount := 0 - - for rows.Next() { - rowCount++ - - var host, user, srcIP, workstation string - var firstSeen time.Time - var count int - - if err := rows.Scan(&host, &user, &srcIP, &workstation, &firstSeen, &count); err != nil { - return err - } - - user = normalizeUsername(user) - if isNoiseAccount(user) { - continue - } - - isNew, err := d.touchUEBAUserContext(ctx, user, host, srcIP, workstation, firstSeen, count) - if err != nil { - return err - } - - if !isNew { - continue - } - - newCount++ - - score := 1.0 - severity := "low" - - if count >= 5 { - score = 4.0 - severity = "high" - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "ueba_new_user_context", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf( - "UEBA: Benutzer %s meldet sich in neuem Kontext an: Host=%s IP=%s Workstation=%s", - user, - host, - srcIP, - workstation, - ), - Details: mustJSON(map[string]any{ - "user": user, - "src_ip": srcIP, - "workstation": workstation, - "host": host, - "count": count, - "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), - "window": d.cfg.UEBANewContextWindow.String(), - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("ueba_new_user_context", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "ueba_new_user_context").Set(score) - } - } else { - created, err := d.insertDetection(ctx, Detection{ - RuleName: "ueba_new_user_context", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - Status: "plausible", - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf( - "UEBA: Benutzer %s meldet sich in neuem Kontext an: Host=%s IP=%s Workstation=%s", - user, - host, - srcIP, - workstation, - ), - Details: mustJSON(map[string]any{ - "user": user, - "src_ip": srcIP, - "workstation": workstation, - "host": host, - "count": count, - "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), - "window": d.cfg.UEBANewContextWindow.String(), - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("ueba_new_user_context", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "ueba_new_user_context").Set(score) - } - } - - } - - if err := rows.Err(); err != nil { - return err - } - - d.logger.Printf( - "ueba_new_user_context processed rows=%d new_contexts=%d window=%s duration=%s", - rowCount, - newCount, - d.cfg.UEBANewContextWindow, - time.Since(start), - ) - - return nil -} -func riskWeight(severity string) float64 { - switch severity { - case "critical": - return 25 - case "high": - return 10 - case "medium": - return 2 - case "low": - return 0.3 - case "info": - return 0.05 - default: - return 0.2 - } -} - -func severityFromRisk(score float64) string { - switch { - case score >= 120: - return "critical" - case score >= 60: - return "high" - case score >= 20: - return "medium" - case score >= 5: - return "low" - default: - return "info" - } -} - -func (d *detector) runHostRiskScoreUpdate(ctx context.Context) error { - windowStart := time.Now().UTC().Add(-d.cfg.RiskScoreWindow) - - if _, err := d.db.ExecContext(ctx, ` -UPDATE host_risk_scores -SET risk_score = 0, - severity = 'info', - open_detections = 0, - high_detections = 0, - critical_detections = 0, - confirmed_incidents = 0, - updated_at = UTC_TIMESTAMP(6) -`); err != nil { - return err - } - - rows, err := d.db.QueryContext(ctx, ` -SELECT - hostname, - severity, - status, - COUNT(*) AS cnt, - MAX(created_at) AS last_detection_at -FROM detections -WHERE created_at >= ? - AND status NOT IN ('false_positive', 'suppressed', 'legitimate', 'resolved', 'plausible') -GROUP BY hostname, severity, status -`, windowStart) - if err != nil { - return err - } - defer rows.Close() - - type agg struct { - score float64 - open int - high int - critical int - confirmedIncidents int - last time.Time - } - - stats := map[string]*agg{} - - for rows.Next() { - var host, sev, status string - var count int - var last time.Time - - if err := rows.Scan(&host, &sev, &status, &count, &last); err != nil { - return err - } - - a := stats[host] - if a == nil { - a = &agg{} - stats[host] = a - } - - w := riskWeight(sev) - - switch status { - case "confirmed_incident": - w += 75 - a.confirmedIncidents += count - case "investigating": - w *= 2 - a.open += count - case "acknowledged": - w *= 0.5 - a.open += count - case "open": - w *= 0.35 - a.open += count - default: - a.open += count - } - - if sev == "high" { - a.high += count - } - if sev == "critical" { - a.critical += count - } - - // Dämpfung: 100 gleiche offene Events sollen nicht 100x hart zählen. - a.score += w * math.Sqrt(float64(count)) - - if last.After(a.last) { - a.last = last - } - } - - if err := rows.Err(); err != nil { - return err - } - - for host, a := range stats { - sev := severityFromRisk(a.score) - - _, err := d.db.ExecContext(ctx, ` -INSERT INTO host_risk_scores -(hostname, risk_score, severity, open_detections, high_detections, critical_detections, confirmed_incidents, last_detection_at) -VALUES (?, ?, ?, ?, ?, ?, ?, ?) -ON DUPLICATE KEY UPDATE -risk_score = VALUES(risk_score), -severity = VALUES(severity), -open_detections = VALUES(open_detections), -high_detections = VALUES(high_detections), -critical_detections = VALUES(critical_detections), -confirmed_incidents = VALUES(confirmed_incidents), -last_detection_at = VALUES(last_detection_at), -updated_at = UTC_TIMESTAMP(6) -`, - host, - a.score, - sev, - a.open, - a.high, - a.critical, - a.confirmedIncidents, - a.last, - ) - if err != nil { - return err - } - - if d.hostRiskScoreGauge != nil { - d.hostRiskScoreGauge.WithLabelValues(host, sev).Set(a.score) - } - } - - return nil -} - -func (d *detector) updateAgentMetrics(ctx context.Context) error { - const q = ` -SELECT hostname, last_seen -FROM agents -WHERE is_enabled = 1 -` - - rows, err := d.db.QueryContext(ctx, q) - if err != nil { - return err - } - defer rows.Close() - - active := 0 - now := time.Now().UTC() - - for rows.Next() { - var host string - var lastSeen time.Time - - if err := rows.Scan(&host, &lastSeen); err != nil { - return err - } - - lastSeenUTC := lastSeen.UTC() - - d.lastSeenGauge.WithLabelValues(host).Set(float64(lastSeenUTC.Unix())) - - if now.Sub(lastSeenUTC) <= d.cfg.OfflineAfter { - active++ - } - } - - if err := rows.Err(); err != nil { - return err - } - - d.activeAgentsGauge.Set(float64(active)) - return nil -} - -func (d *detector) runAgentOfflineRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - - offlineAfterTime := windowEnd.Add(-d.cfg.OfflineAfter) - maxOfflineTime := windowEnd.Add(-d.cfg.OfflineAlertMax) - - const q = ` -SELECT hostname, last_seen -FROM agents -WHERE is_enabled = 1 - AND last_seen < ? - AND last_seen >= ? -` - rows, err := d.db.QueryContext(ctx, q, offlineAfterTime, maxOfflineTime) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host string - var lastSeen time.Time - if err := rows.Scan(&host, &lastSeen); err != nil { - return err - } - - minutes := int(windowEnd.Sub(lastSeen.UTC()).Minutes()) - - score := math.Min(1.0, math.Max(0.1, float64(minutes)/float64(int(d.cfg.OfflineAfter.Minutes())))) - severity := "low" - - d.anomalyScoreGauge.WithLabelValues(host, "agent_offline").Set(score) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "agent_offline", - Severity: severity, - Hostname: host, - Score: score, - WindowStart: offlineAfterTime, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Agent %s ist seit %d Minuten offline", host, minutes), - Details: mustJSON(map[string]any{ - "last_seen": lastSeen.UTC().Format(time.RFC3339Nano), - "offline_minutes": minutes, - "offline_after_min": int(d.cfg.OfflineAfter.Minutes()), - "offline_alert_max_min": int(d.cfg.OfflineAlertMax.Minutes()), - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("agent_offline", severity).Inc() - } - } - - return rows.Err() -} - -func (d *detector) runFailedLogonSpikeRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.FailedLogonWindow) - - const q = ` -SELECT hostname, SUM(cnt) AS cnt -FROM event_count_buckets -WHERE channel_name = 'Security' - AND event_id = 4625 - AND bucket_start >= ? AND bucket_start < ? -GROUP BY hostname -HAVING SUM(cnt) >= ? -` - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.BaselineWindow), windowEnd, d.cfg.FailedLogonThreshold) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host string - var count int - if err := rows.Scan(&host, &count); err != nil { - return err - } - - score := float64(count) / float64(d.cfg.FailedLogonThreshold) - severity := severityFromScore(score, 1.0, 2.0, 4.0, 8.0) - d.anomalyScoreGauge.WithLabelValues(host, "failed_logon_spike").Set(score) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "failed_logon_spike", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4625, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Host %s hatte %d fehlgeschlagene Logons in %d Minuten", host, count, int(d.cfg.FailedLogonWindow.Minutes())), - Details: mustJSON(map[string]any{ - "count": count, - "threshold": d.cfg.FailedLogonThreshold, - "window_minutes": int(d.cfg.FailedLogonWindow.Minutes()), - "event_id": 4625, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("failed_logon_spike", severity).Inc() - } - } - return rows.Err() -} - -func (d *detector) runRebootSpikeRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.RebootWindow) - - const q = ` -SELECT hostname, SUM(cnt) AS cnt -FROM event_count_buckets -WHERE channel_name = 'System' - AND event_id IN (1074, 6005, 6006) - AND bucket_start >= ? AND bucket_start < ? -GROUP BY hostname -HAVING SUM(cnt) >= ? -` - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.BaselineWindow), windowEnd, d.cfg.RebootThreshold) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host string - var count int - if err := rows.Scan(&host, &count); err != nil { - return err - } - - score := float64(count) / float64(d.cfg.RebootThreshold) - severity := severityFromScore(score, 1.0, 2.0, 4.0, 8.0) - d.anomalyScoreGauge.WithLabelValues(host, "reboot_spike").Set(score) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "reboot_spike", - Severity: severity, - Hostname: host, - Channel: "System", - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Host %s hatte %d Reboot-/Shutdown-Events in %d Minuten", host, count, int(d.cfg.RebootWindow.Minutes())), - Details: mustJSON(map[string]any{ - "count": count, - "threshold": d.cfg.RebootThreshold, - "window_minutes": int(d.cfg.RebootWindow.Minutes()), - "event_ids": []int{1074, 6005, 6006}, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("reboot_spike", severity).Inc() - } - } - return rows.Err() -} - -func (d *detector) runNewEventIDRule(ctx context.Context) error { - mode := strings.ToLower(strings.TrimSpace(d.cfg.NewEventIDMode)) - if mode == "" || mode == "off" || mode == "disabled" { - return nil - } - if mode != "inventory" && mode != "alert" { - return fmt.Errorf("invalid NEW_EVENT_ID_MODE %q (expected off, inventory or alert)", d.cfg.NewEventIDMode) - } - - windowEnd := time.Now().UTC() - confirmWindow := d.cfg.NewEventIDConfirmWindow - if confirmWindow <= 0 { - confirmWindow = 15 * time.Minute - } - windowStart := windowEnd.Add(-confirmWindow) - - const q = ` -SELECT ec.hostname, ec.channel_name, ec.event_id, ec.total_count, - ec.first_seen, ec.last_seen, - COALESCE(a.first_seen, TIMESTAMP('1970-01-01 00:00:00')) AS agent_first_seen -FROM event_catalog ec -LEFT JOIN agents a ON a.hostname = ec.hostname -WHERE ec.first_seen >= ? AND ec.first_seen < ? - AND ec.total_count >= ? -ORDER BY ec.first_seen ASC -` - rows, err := d.db.QueryContext(ctx, q, windowStart, windowEnd, maxInt(d.cfg.NewEventIDMinCount, 1)) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, channel string - var eventID uint32 - var count int - var eventFirstSeen, eventLastSeen, agentFirstSeen time.Time - if err := rows.Scan(&host, &channel, &eventID, &count, &eventFirstSeen, &eventLastSeen, &agentFirstSeen); err != nil { - return err - } - - assessment := assessNewEventID(d.cfg, channel, eventID, count, agentFirstSeen, eventFirstSeen) - if !assessment.Emit { - continue - } - - d.anomalyScoreGauge.WithLabelValues(host, "new_event_id").Set(assessment.Score) - - summary := fmt.Sprintf("Neue Event-ID im Inventar: Host %s, Event-ID %d, Channel %s", host, eventID, channel) - if assessment.Status == "open" { - summary = fmt.Sprintf("Ungewohnte Event-ID: Host %s, Event-ID %d, Channel %s", host, eventID, channel) - } - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "new_event_id", - Severity: assessment.Severity, - Status: assessment.Status, - Hostname: host, - Channel: channel, - EventID: eventID, - Score: assessment.Score, - // Stable per Event-ID. This prevents a new duplicate on every rule run. - WindowStart: eventFirstSeen.UTC(), - WindowEnd: eventFirstSeen.UTC().Add(confirmWindow), - Summary: summary, - Details: mustJSON(map[string]any{ - "classification": assessment.Reason, - "mode": mode, - "count": count, - "channel": channel, - "event_id": eventID, - "event_first_seen": eventFirstSeen.UTC(), - "event_last_seen": eventLastSeen.UTC(), - "agent_first_seen": agentFirstSeen.UTC(), - "agent_age_at_event_h": eventFirstSeen.Sub(agentFirstSeen).Hours(), - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("new_event_id", assessment.Severity).Inc() - } - } - return rows.Err() -} - -type newEventIDAssessment struct { - Emit bool - Severity string - Status string - Score float64 - Reason string -} - -func assessNewEventID(cfg Config, channel string, eventID uint32, count int, agentFirstSeen, eventFirstSeen time.Time) newEventIDAssessment { - mode := strings.ToLower(strings.TrimSpace(cfg.NewEventIDMode)) - if mode == "" || mode == "off" || mode == "disabled" { - return newEventIDAssessment{Reason: "rule_disabled"} - } - if count < maxInt(cfg.NewEventIDMinCount, 1) { - return newEventIDAssessment{Reason: "below_confirmation_count"} - } - if cfg.NewEventIDLearningPeriod > 0 && eventFirstSeen.Sub(agentFirstSeen) < cfg.NewEventIDLearningPeriod { - return newEventIDAssessment{Reason: "agent_learning_period"} - } - if containsFold(cfg.NewEventIDIgnoreChannels, channel) { - return newEventIDAssessment{Reason: "ignored_operational_channel"} - } - - score := 0.25 + math.Log10(float64(count)+1)/2 - if score > 5 { - score = 5 - } - - if mode == "inventory" { - return newEventIDAssessment{true, "info", "plausible", score, "inventory_only"} - } - - _, highRisk := cfg.NewEventIDHighRiskIDs[eventID] - if highRisk { - severity := "medium" - if count >= 10 { - severity = "high" - } - return newEventIDAssessment{true, severity, "open", score + 1, "configured_high_risk_event_id"} - } - if containsFold(cfg.NewEventIDAlertChannels, channel) { - severity := "low" - if count >= 20 { - severity = "medium" - } - return newEventIDAssessment{true, severity, "open", score, "new_id_in_alert_channel"} - } - return newEventIDAssessment{true, "info", "plausible", score, "new_id_in_non_alert_channel"} -} - -func (d *detector) runPasswordSprayRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.PasswordSprayWindow) - - const q = ` -SELECT hostname, src_ip, SUM(cnt) AS attempts, COUNT(DISTINCT target_user) AS users -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4625 - AND bucket_start >= ? AND bucket_start < ? - AND src_ip <> '' AND src_ip <> '-' AND src_ip <> '::1' AND src_ip <> '127.0.0.1' - AND target_user <> '' AND target_user <> '-' -GROUP BY hostname, src_ip -HAVING SUM(cnt) >= ? AND COUNT(DISTINCT target_user) >= ? -` - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd, d.cfg.PasswordSprayMinAttempts, d.cfg.PasswordSprayMinUsers) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, srcIP string - var attempts, users int - if err := rows.Scan(&host, &srcIP, &attempts, &users); err != nil { - return err - } - - score := math.Max(float64(attempts)/float64(d.cfg.PasswordSprayMinAttempts), float64(users)/float64(d.cfg.PasswordSprayMinUsers)) - severity := severityFromScore(score, 1.0, 2.0, 4.0, 8.0) - d.anomalyScoreGauge.WithLabelValues(host, "password_spray").Set(score) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "password_spray", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4625, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Möglicher Password-Spray auf %s von %s: %d Fehlversuche gegen %d Benutzer", host, srcIP, attempts, users), - Details: mustJSON(map[string]any{ - "src_ip": srcIP, - "attempts": attempts, - "distinct_users": users, - "window_minutes": int(d.cfg.PasswordSprayWindow.Minutes()), - "event_id": 4625, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("password_spray", severity).Inc() - } - } - return rows.Err() -} - -func (d *detector) runSuccessAfterFailuresRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.SuccessAfterFailureWindow) - - const q = ` -SELECT s.hostname, s.target_user, s.src_ip, SUM(s.cnt) AS success_count -FROM event_occurrences s -WHERE s.channel_name = 'Security' - AND s.event_id = 4624 - AND s.bucket_start >= ? AND s.bucket_start < ? - AND s.target_user <> '' - AND EXISTS ( - SELECT 1 - FROM event_occurrences f - WHERE f.hostname = s.hostname - AND f.channel_name = 'Security' - AND f.event_id = 4625 - AND f.target_user = s.target_user - AND ( - (f.src_ip = s.src_ip AND s.src_ip <> '' AND s.src_ip <> '-') - OR (s.src_ip = '' OR s.src_ip = '-' OR f.src_ip = '' OR f.src_ip = '-') - ) - AND f.last_event_ts >= DATE_SUB(s.first_event_ts, INTERVAL ? SECOND) - AND f.first_event_ts < s.last_event_ts - ) -GROUP BY s.hostname, s.target_user, s.src_ip -` - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd, int(d.cfg.SuccessAfterFailureWindow.Seconds())) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, user, srcIP string - var successCount int - if err := rows.Scan(&host, &user, &srcIP, &successCount); err != nil { - return err - } - - score := 2.0 + math.Log10(float64(successCount)+1) - severity := "high" - d.anomalyScoreGauge.WithLabelValues(host, "success_after_failures").Set(score) - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "success_after_failures", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Erfolgreicher Logon nach Fehlversuchen auf %s für Benutzer %s", host, user), - Details: mustJSON(map[string]any{ - "user": user, - "src_ip": srcIP, - "success_count": successCount, - "window_minutes": int(d.cfg.SuccessAfterFailureWindow.Minutes()), - "success_event": 4624, - "failure_event": 4625, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("success_after_failures", severity).Inc() - } - } - return rows.Err() -} - -func (d *detector) runNewSourceIPForUserRule(ctx context.Context) error { - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.NewSourceIPWindow) - - const q = ` -SELECT e.hostname, e.target_user, e.src_ip, MIN(e.first_event_ts) AS first_seen, SUM(e.cnt) AS cnt -FROM event_occurrences e -WHERE e.channel_name = 'Security' - AND e.event_id = 4624 - AND e.bucket_start >= ? AND e.bucket_start < ? - AND e.target_user <> '' - AND e.target_user NOT LIKE '%$' - AND e.src_ip <> '' - AND e.src_ip <> '-' - AND e.src_ip <> '::1' - AND e.src_ip <> '127.0.0.1' - AND e.target_user NOT IN ( - 'system', - 'localsystem', - 'local service', - 'network service', - 'anonymous logon' - ) -GROUP BY e.hostname, e.target_user, e.src_ip -` - - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, user, srcIP string - var firstSeen time.Time - var cnt int - - if err := rows.Scan(&host, &user, &srcIP, &firstSeen, &cnt); err != nil { - return err - } - - user = normalizeUsername(user) - if isNoiseAccount(user) { - continue - } - - isNew, err := d.touchUserSourceIPSeen(ctx, user, srcIP, host, firstSeen, cnt) - if err != nil { - return err - } - - if !isNew { - continue - } - - score := 1.0 + math.Log10(float64(cnt)+1) - - severity := "low" - if cnt >= 5 { - severity = "medium" - created, err := d.insertDetection(ctx, Detection{ - RuleName: "new_source_ip_for_user", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("Benutzer %s meldet sich auf %s erstmals von Quell-IP %s an", user, host, srcIP), - Details: mustJSON(map[string]any{ - "user": user, - "src_ip": srcIP, - "host": host, - "count": cnt, - "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), - "window_minutes": int(d.cfg.NewSourceIPWindow.Minutes()), - "event_id": 4624, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("new_source_ip_for_user", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "new_source_ip_for_user").Set(score) - } - } else { - created, err := d.insertDetection(ctx, Detection{ - RuleName: "new_source_ip_for_user", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Status: "plausible", - Summary: fmt.Sprintf("Benutzer %s meldet sich auf %s erstmals von Quell-IP %s an", user, host, srcIP), - Details: mustJSON(map[string]any{ - "user": user, - "src_ip": srcIP, - "host": host, - "count": cnt, - "first_seen": firstSeen.UTC().Format(time.RFC3339Nano), - "window_minutes": int(d.cfg.NewSourceIPWindow.Minutes()), - "event_id": 4624, - }), - }) - if err != nil { - return err - } - if created { - d.detectionHitsTotal.WithLabelValues("new_source_ip_for_user", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "new_source_ip_for_user").Set(score) - } - } - - } - - return rows.Err() -} - -func (d *detector) touchUEBAUserContext(ctx context.Context, username, hostname, srcIP, workstation string, firstSeen time.Time, count int) (bool, error) { - username = normalizeUsername(username) - hostname = strings.TrimSpace(hostname) - srcIP = strings.TrimSpace(srcIP) - workstation = strings.TrimSpace(workstation) - - if username == "" || hostname == "" { - return false, nil - } - - if srcIP == "" { - srcIP = "-" - } - if workstation == "" { - workstation = "-" - } - - res, err := d.db.ExecContext(ctx, ` -INSERT INTO ueba_user_baseline -(username, hostname, src_ip, workstation, first_seen, last_seen, seen_count) -VALUES (?, ?, ?, ?, ?, ?, ?) -ON DUPLICATE KEY UPDATE -last_seen = VALUES(last_seen), -seen_count = seen_count + VALUES(seen_count) -`, - username, - hostname, - srcIP, - workstation, - firstSeen.UTC(), - firstSeen.UTC(), - count, - ) - if err != nil { - return false, err - } - - affected, err := res.RowsAffected() - if err != nil { - return false, err - } - - // MySQL: - // 1 = neuer Insert - // 2 = Update wegen ON DUPLICATE KEY - // 0 = kein effektives Update, abhängig von Client/Settings möglich - return affected == 1, nil -} - -func (d *detector) isDetectionSuppressed(ctx context.Context, det Detection) (bool, error) { - var count int - - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM detection_suppressions -WHERE enabled = 1 - AND rule_name = ? - AND (hostname = '' OR hostname = ?) - AND (channel_name = '' OR channel_name = ?) - AND (event_id = 0 OR event_id = ?) - AND (expires_at IS NULL OR expires_at > UTC_TIMESTAMP(6)) -`, - det.RuleName, - det.Hostname, - det.Channel, - det.EventID, - ).Scan(&count) - - if err != nil { - return false, err - } - - return count > 0, nil -} - -func (d *detector) insertDetection(ctx context.Context, det Detection) (bool, error) { - suppressed, err := d.isDetectionSuppressed(ctx, det) - if err != nil { - return false, err - } - if suppressed { - return false, nil - } - var status0 string - if det.Status == "plausible" { - status0 = "plausible" - } else { - status0 = "open" - } - - const q = ` -INSERT IGNORE INTO detections -(rule_name, severity, hostname, channel_name, event_id, score, window_start, window_end, summary, details_json, created_at, status) -VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, UTC_TIMESTAMP(6), ?) -` - - res, err := d.db.ExecContext(ctx, q, - det.RuleName, - det.Severity, - det.Hostname, - det.Channel, - det.EventID, - det.Score, - det.WindowStart.UTC(), - det.WindowEnd.UTC(), - det.Summary, - string(det.Details), - status0, - ) - if err != nil { - return false, err - } - - affected, err := res.RowsAffected() - if err != nil { - return false, err - } - - return affected > 0, nil -} - -func payloadFingerprint(p LogPayload) string { - if strings.TrimSpace(p.Message) != "" { - return sha256Hex(p.Message) - } - b, err := json.Marshal(struct { - Host string `json:"host"` - Channel string `json:"channel"` - ID uint32 `json:"id"` - Source string `json:"source"` - Time time.Time `json:"ts"` - Meta *EventMetadataPayload `json:"meta"` - }{p.Hostname, p.Channel, p.EventID, p.Source, p.Time.UTC(), p.Metadata}) - if err != nil { - return sha256Hex(fmt.Sprintf("%s|%s|%d|%s", p.Hostname, p.Channel, p.EventID, p.Time.UTC().Format(time.RFC3339Nano))) - } - return sha256Hex(string(b)) -} - -func overlayMetadata(base NormalizedEvent, m EventMetadataPayload) NormalizedEvent { - base.Computer = firstNonEmpty(m.Computer, base.Computer) - base.ProviderName = firstNonEmpty(m.ProviderName, base.ProviderName) - base.TargetUser = firstNonEmpty(m.TargetUser, base.TargetUser) - base.TargetDomain = firstNonEmpty(m.TargetDomain, base.TargetDomain) - base.SubjectUser = firstNonEmpty(m.SubjectUser, base.SubjectUser) - base.SubjectDomain = firstNonEmpty(m.SubjectDomain, base.SubjectDomain) - base.Workstation = firstNonEmpty(m.Workstation, m.Device, base.Workstation) - base.SrcIP = firstNonEmpty(m.SrcIP, base.SrcIP) - base.SrcPort = firstNonEmpty(m.SrcPort, base.SrcPort) - base.LogonType = firstNonEmpty(m.LogonType, base.LogonType) - base.ProcessName = firstNonEmpty(m.ProcessName, base.ProcessName) - base.AuthenticationPackage = firstNonEmpty(m.AuthenticationPackage, base.AuthenticationPackage) - base.LogonProcess = firstNonEmpty(m.LogonProcess, base.LogonProcess) - base.StatusText = firstNonEmpty(m.StatusText, base.StatusText) - base.SubStatusText = firstNonEmpty(m.SubStatusText, base.SubStatusText) - base.FailureReason = firstNonEmpty(m.FailureReason, base.FailureReason) - return base -} - -func NormalizeEventXML(xmlStr string) NormalizedEvent { - var out NormalizedEvent - if strings.TrimSpace(xmlStr) == "" { - return out - } - - dec := xml.NewDecoder(strings.NewReader(xmlStr)) - - var path []string - var currentDataName string - - for { - tok, err := dec.Token() - if err != nil { - if err == io.EOF { - break - } - return out - } - - switch t := tok.(type) { - case xml.StartElement: - path = append(path, t.Name.Local) - - switch t.Name.Local { - case "Provider": - for _, a := range t.Attr { - if a.Name.Local == "Name" { - out.ProviderName = strings.TrimSpace(a.Value) - } - } - case "Data": - currentDataName = "" - for _, a := range t.Attr { - if a.Name.Local == "Name" { - currentDataName = strings.TrimSpace(a.Value) - break - } - } - } - - case xml.EndElement: - if len(path) > 0 { - path = path[:len(path)-1] - } - if t.Name.Local == "Data" { - currentDataName = "" - } - - case xml.CharData: - v := strings.TrimSpace(string(t)) - if v == "" { - continue - } - - if endsWithPath(path, "System", "Computer") { - out.Computer = v - continue - } - if endsWithPath(path, "System", "Level") { - out.LevelValue = parseUint32(v) - continue - } - if endsWithPath(path, "System", "Task") { - out.TaskValue = parseUint32(v) - continue - } - if endsWithPath(path, "System", "Opcode") { - out.OpcodeValue = parseUint32(v) - continue - } - if endsWithPath(path, "System", "Keywords") { - out.Keywords = v - continue - } - - if currentDataName != "" { - switch currentDataName { - case "TargetUserName": - out.TargetUser = v - case "TargetDomainName": - out.TargetDomain = v - case "SubjectUserName": - out.SubjectUser = v - case "SubjectDomainName": - out.SubjectDomain = v - case "WorkstationName", "CallerComputerName": - out.Workstation = v - case "IpAddress": - out.SrcIP = v - case "IpPort": - out.SrcPort = v - case "LogonType": - out.LogonType = v - case "ProcessName": - out.ProcessName = v - case "AuthenticationPackageName": - out.AuthenticationPackage = v - case "LogonProcessName": - out.LogonProcess = v - case "Status": - out.StatusText = v - case "SubStatus": - out.SubStatusText = v - case "FailureReason": - out.FailureReason = v - case "MemberName": - out.MemberName = v - } - } - } - } - - out.TargetUser = cleanupWinField(out.TargetUser) - out.TargetDomain = cleanupWinField(out.TargetDomain) - out.SubjectUser = cleanupWinField(out.SubjectUser) - out.SubjectDomain = cleanupWinField(out.SubjectDomain) - out.Workstation = cleanupWinField(out.Workstation) - out.SrcIP = cleanupWinField(out.SrcIP) - out.SrcPort = cleanupWinField(out.SrcPort) - out.LogonType = cleanupWinField(out.LogonType) - out.ProcessName = cleanupWinField(out.ProcessName) - out.AuthenticationPackage = cleanupWinField(out.AuthenticationPackage) - out.LogonProcess = cleanupWinField(out.LogonProcess) - out.StatusText = cleanupWinField(out.StatusText) - out.SubStatusText = cleanupWinField(out.SubStatusText) - out.FailureReason = cleanupWinField(out.FailureReason) - out.Computer = cleanupWinField(out.Computer) - out.ProviderName = cleanupWinField(out.ProviderName) - - return out -} - -func endsWithPath(path []string, parts ...string) bool { - if len(path) < len(parts) { - return false - } - start := len(path) - len(parts) - for i := range parts { - if path[start+i] != parts[i] { - return false - } - } - return true -} - -func parseUint32(v string) uint32 { - n, err := strconv.ParseUint(strings.TrimSpace(v), 10, 32) - if err != nil { - return 0 - } - return uint32(n) -} - -func cleanupWinField(v string) string { - v = strings.TrimSpace(v) - if v == "" { - return "" - } - if v == "-" { - return "" - } - return v -} - -func firstNonEmpty(v ...string) string { - for _, s := range v { - if strings.TrimSpace(s) != "" { - return strings.TrimSpace(s) - } - } - return "" -} - -func validatePayload(p *LogPayload) error { - p.Hostname = strings.TrimSpace(p.Hostname) - p.Channel = strings.TrimSpace(p.Channel) - p.Source = strings.TrimSpace(p.Source) - - if p.Hostname == "" { - return errors.New("host is required") - } - if len(p.Hostname) > 255 { - return errors.New("host too long") - } - if p.Channel == "" { - return errors.New("channel is required") - } - if len(p.Channel) > 128 { - return errors.New("channel too long") - } - if p.Source == "" { - return errors.New("source is required") - } - if len(p.Source) > 255 { - return errors.New("source too long") - } - if p.EventID == 0 { - return errors.New("event id is required") - } - if strings.TrimSpace(p.Message) == "" && p.Metadata == nil { - return errors.New("either msg or meta is required") - } - if len(p.Message) > 2*1024*1024 { - return errors.New("msg too large") - } - if p.Time.IsZero() { - return errors.New("ts is required") - } - return nil -} - -func severityFromScore(score, low, medium, high, critical float64) string { - switch { - case score >= critical: - return "critical" - case score >= high: - return "high" - case score >= medium: - return "medium" - case score >= low: - return "low" - default: - return "info" - } -} - -func mustJSON(v any) json.RawMessage { - b, _ := json.Marshal(v) - return b -} - -func metricsMiddleware(logger *log.Logger, next http.Handler) http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - rw := &statusRecorder{ResponseWriter: w, status: 200} - start := time.Now() - next.ServeHTTP(rw, r) - - status := strconv.Itoa(rw.status) - path := r.URL.Path - - httpRequestsTotal.WithLabelValues(path, r.Method, status).Inc() - httpRequestDuration.WithLabelValues(path, r.Method, status).Observe(time.Since(start).Seconds()) - logger.Printf("%s %s remote=%s status=%s dur=%s", r.Method, r.URL.Path, r.RemoteAddr, status, time.Since(start)) - }) -} - -type statusRecorder struct { - http.ResponseWriter - status int -} - -func (r *statusRecorder) WriteHeader(status int) { - r.status = status - r.ResponseWriter.WriteHeader(status) -} - -func recoveryMiddleware(next http.Handler) http.Handler { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - defer func() { - if recover() != nil { - writeError(w, http.StatusInternalServerError, "internal error") - } - }() - next.ServeHTTP(w, r) - }) -} - -func writeJSON(w http.ResponseWriter, status int, v any) { - w.Header().Set("Content-Type", "application/json; charset=utf-8") - w.WriteHeader(status) - _ = json.NewEncoder(w).Encode(v) -} - -func writeError(w http.ResponseWriter, status int, msg string) { - writeJSON(w, status, map[string]string{"error": msg}) -} - -func sha256Hex(s string) string { - sum := sha256.Sum256([]byte(s)) - return hex.EncodeToString(sum[:]) -} - -func constantTimeEqual(a, b string) bool { - if len(a) != len(b) { - return false - } - var v byte - for i := 0; i < len(a); i++ { - v |= a[i] ^ b[i] - } - return v == 0 -} - -func clientIP(remoteAddr string) string { - host, _, err := net.SplitHostPort(remoteAddr) - if err != nil { - return remoteAddr - } - return host -} - -func getenv(key, def string) string { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - return v -} - -func mustGetenv(key string) string { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - log.Fatalf("missing required environment variable: %s", key) - } - return v -} - -func getenvInt(key string, def int) int { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - n, err := strconv.Atoi(v) - if err != nil { - log.Fatalf("invalid int for %s: %v", key, err) - } - return n -} - -func getenvInt64(key string, def int64) int64 { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - n, err := strconv.ParseInt(v, 10, 64) - if err != nil { - log.Fatalf("invalid int64 for %s: %v", key, err) - } - return n -} - -func getenvDuration(key string, def time.Duration) time.Duration { - v := strings.TrimSpace(os.Getenv(key)) - if v == "" { - return def - } - d, err := time.ParseDuration(v) - if err != nil { - log.Fatalf("invalid duration for %s: %v", key, err) - } - return d -} - -func (d *detector) runOffHoursLoginRule(ctx context.Context) error { - if !d.cfg.UEBAEnabled { - return nil - } - - nowLocal := time.Now().Local() - hour := nowLocal.Hour() - - // Arbeitszeit: 6–20 Uhr - if hour >= 6 && hour <= 20 { - return nil - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-5 * time.Minute) - - const q = ` -SELECT hostname, target_user, SUM(cnt) AS cnt -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4624 - AND bucket_start >= ? AND bucket_start < ? - AND target_user <> '' - AND target_user NOT LIKE '%$' - AND logon_type IN ('2', '7', '10', '11') - AND target_user NOT IN ( - 'system', - 'localsystem', - 'local service', - 'network service', - 'anonymous logon' - ) -GROUP BY hostname, target_user -` - - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, user string - var count int - - if err := rows.Scan(&host, &user, &count); err != nil { - return err - } - - user = normalizeUsername(user) - if isNoiseAccount(user) { - continue - } - - score := 2.0 - severity := "medium" - - if count >= 5 { - score = 4.0 - severity = "high" - } - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "ueba_offhours_login", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - Summary: fmt.Sprintf("UEBA: Login außerhalb der Arbeitszeit: Benutzer %s auf Host %s", user, host), - WindowStart: windowStart, - WindowEnd: windowEnd, - Details: mustJSON(map[string]any{ - "user": user, - "host": host, - "local_hour": hour, - "count": count, - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("ueba_offhours_login", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "ueba_offhours_login").Set(score) - } - } - - return rows.Err() -} - -func (d *detector) touchUserSourceIPSeen(ctx context.Context, username, srcIP, hostname string, firstSeen time.Time, count int) (bool, error) { - username = normalizeUsername(username) - hostname = strings.TrimSpace(hostname) - srcIP = strings.TrimSpace(srcIP) - - if username == "" || srcIP == "" || hostname == "" { - return false, nil - } - - res, err := d.db.ExecContext(ctx, ` -INSERT INTO user_source_ip_seen -(username, src_ip, hostname, first_seen, last_seen, seen_count) -VALUES (?, ?, ?, ?, ?, ?) -ON DUPLICATE KEY UPDATE -last_seen = VALUES(last_seen), -seen_count = seen_count + VALUES(seen_count) -`, - username, - srcIP, - hostname, - firstSeen.UTC(), - firstSeen.UTC(), - count, - ) - if err != nil { - return false, err - } - - affected, err := res.RowsAffected() - if err != nil { - return false, err - } - - // MySQL: - // 1 = Insert - // 2 = Update durch ON DUPLICATE KEY UPDATE - // 0 = kein effektives Update, je nach Client/Settings möglich - return affected == 1, nil -} - -func (d *detector) runFirstTimePrivilegedRule(ctx context.Context) error { - if !d.cfg.UEBAEnabled { - return nil - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-10 * time.Minute) - - const q = ` -SELECT hostname, subject_user, SUM(cnt) AS cnt -FROM event_occurrences -WHERE channel_name = 'Security' - AND event_id = 4672 - AND bucket_start >= ? AND bucket_start < ? - AND subject_user <> '' - AND subject_user NOT LIKE '%$' -GROUP BY hostname, subject_user -` - - rows, err := d.db.QueryContext(ctx, q, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, user string - var count int - - if err := rows.Scan(&host, &user, &count); err != nil { - return err - } - - user = normalizeUsername(user) - if user == "" || isMachineAccount(user) { - continue - } - - var exists int - err := d.db.QueryRowContext(ctx, ` -SELECT 1 -FROM user_privilege_baseline -WHERE username = ? -LIMIT 1 -`, user).Scan(&exists) - - if err != nil && !errors.Is(err, sql.ErrNoRows) { - return err - } - - if errors.Is(err, sql.ErrNoRows) { - created, err := d.insertDetection(ctx, Detection{ - RuleName: "ueba_first_privileged_use", - Severity: "high", - Hostname: host, - Channel: "Security", - EventID: 4672, - Score: 6.0, - Summary: fmt.Sprintf("UEBA: Benutzer %s nutzt erstmals privilegierte Rechte auf Host %s", user, host), - WindowStart: windowStart, - WindowEnd: windowEnd, - Details: mustJSON(map[string]any{ - "user": user, - "host": host, - "count": count, - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("ueba_first_privileged_use", "high").Inc() - d.anomalyScoreGauge.WithLabelValues(host, "ueba_first_privileged_use").Set(6.0) - } - - _, err = d.db.ExecContext(ctx, ` -INSERT INTO user_privilege_baseline -(username, first_seen, last_seen, seen_count) -VALUES (?, UTC_TIMESTAMP(6), UTC_TIMESTAMP(6), ?) -ON DUPLICATE KEY UPDATE -last_seen = UTC_TIMESTAMP(6), -seen_count = seen_count + VALUES(seen_count) -`, user, count) - if err != nil { - return err - } - - continue - } - - _, err = d.db.ExecContext(ctx, ` -UPDATE user_privilege_baseline -SET last_seen = UTC_TIMESTAMP(6), - seen_count = seen_count + ? -WHERE username = ? -`, count, user) - if err != nil { - return err - } - } - - return rows.Err() -} - -func (d *detector) isPrivilegedUser(ctx context.Context, username string) (bool, error) { - u := normalizeUsername(username) - if isMachineAccount(u) { - return false, nil - } - - if strings.HasPrefix(u, "adm-") || strings.HasSuffix(u, "-adm") || strings.HasSuffix(u, ".adm") { - return true, nil - } - - var count int - err := d.db.QueryRowContext(ctx, ` -SELECT COUNT(*) -FROM privileged_users -WHERE enabled = 1 - AND LOWER(username) = ? -`, u).Scan(&count) - - if err != nil { - return false, err - } - - return count > 0, nil -} - -func normalizeUsername(username string) string { - u := strings.ToLower(strings.TrimSpace(username)) - - if strings.Contains(u, `\`) { - parts := strings.Split(u, `\`) - u = parts[len(parts)-1] - } - - if strings.Contains(u, "@") { - parts := strings.Split(u, "@") - u = parts[0] - } - - return u -} - -func isMachineAccount(username string) bool { - u := normalizeUsername(username) - return u == "" || strings.HasSuffix(u, "$") -} - -func (d *detector) runAdminNewHostRule(ctx context.Context) error { - if !d.cfg.UEBAEnabled { - return nil - } - - windowEnd := time.Now().UTC() - windowStart := windowEnd.Add(-d.cfg.UEBANewContextWindow) - lookbackStart := windowEnd.Add(-d.cfg.UEBALookback) - - rows, err := d.db.QueryContext(ctx, ` -SELECT e.hostname, e.target_user, SUM(e.cnt) AS cnt -FROM event_occurrences e -WHERE e.channel_name = 'Security' - AND e.event_id = 4624 - AND e.bucket_start >= ? AND e.bucket_start < ? - AND e.target_user <> '' - AND e.target_user NOT LIKE '%$' - AND NOT EXISTS ( - SELECT 1 - FROM ueba_user_baseline b - WHERE b.username = e.target_user - AND b.hostname = e.hostname - AND b.last_seen >= ? - AND b.first_seen < ? - ) -GROUP BY e.hostname, e.target_user -`, bucketStart(windowStart, d.cfg.MetadataBucket), windowEnd, lookbackStart, windowStart) - if err != nil { - return err - } - defer rows.Close() - - for rows.Next() { - var host, user string - var count int - - if err := rows.Scan(&host, &user, &count); err != nil { - return err - } - - privileged, err := d.isPrivilegedUser(ctx, user) - if err != nil { - return err - } - if !privileged { - continue - } - - score := 6.0 - severity := "high" - - if count >= 3 { - score = 9.0 - severity = "critical" - } - - created, err := d.insertDetection(ctx, Detection{ - RuleName: "ueba_admin_new_host", - Severity: severity, - Hostname: host, - Channel: "Security", - EventID: 4624, - Score: score, - WindowStart: windowStart, - WindowEnd: windowEnd, - Summary: fmt.Sprintf("UEBA: Privilegierter Benutzer %s meldet sich erstmals auf Host %s an", user, host), - Details: mustJSON(map[string]any{ - "user": user, - "host": host, - "count": count, - "lookback": d.cfg.UEBALookback.String(), - "window": d.cfg.UEBANewContextWindow.String(), - }), - }) - if err != nil { - return err - } - - if created { - d.detectionHitsTotal.WithLabelValues("ueba_admin_new_host", severity).Inc() - d.anomalyScoreGauge.WithLabelValues(host, "ueba_admin_new_host").Set(score) - if d.privilegedNewHostTotal != nil { - d.privilegedNewHostTotal.WithLabelValues(normalizeUsername(user), host).Inc() - } - } - } - - return rows.Err() -} - -func isNoiseAccount(username string) bool { - u := normalizeUsername(username) - - if u == "" || isMachineAccount(u) { - return true - } - - switch u { - case "system", - "localsystem", - "local service", - "network service", - "anonymous logon", - "dwm-1", - "dwm-2", - "dwm-3", - "umfd-0", - "umfd-1", - "umfd-2", - "umfd-3": - return true - } - - return false -} diff --git a/main_test.go b/main_test.go deleted file mode 100644 index f5636d6..0000000 --- a/main_test.go +++ /dev/null @@ -1,131 +0,0 @@ -package main - -import ( - "testing" - "time" -) - -func TestNormalizeLockoutEvent4740(t *testing.T) { - xml := `DC01.example.localaliceEXAMPLEDC01$CLIENT-42` - got := NormalizeEventXML(xml) - if got.TargetUser != "alice" { - t.Fatalf("TargetUser=%q", got.TargetUser) - } - if got.Workstation != "CLIENT-42" { - t.Fatalf("Workstation=%q", got.Workstation) - } - if got.Computer != "DC01.example.local" { - t.Fatalf("Computer=%q", got.Computer) - } -} - -func TestMetadataOnlyOverlay(t *testing.T) { - got := overlayMetadata(NormalizedEvent{}, EventMetadataPayload{ - Computer: "DC01", - TargetUser: "EXAMPLE\\alice", - Device: "CLIENT-42", - SrcIP: "10.0.0.42", - }) - if got.Workstation != "CLIENT-42" { - t.Fatalf("Workstation=%q", got.Workstation) - } - if normalizeUsername(got.TargetUser) != "alice" { - t.Fatalf("normalized user=%q", normalizeUsername(got.TargetUser)) - } -} - -func TestBucketStart(t *testing.T) { - ts := time.Date(2026, 7, 18, 12, 34, 56, 0, time.UTC) - got := bucketStart(ts, time.Minute) - want := time.Date(2026, 7, 18, 12, 34, 0, 0, time.UTC) - if !got.Equal(want) { - t.Fatalf("got %s want %s", got, want) - } -} - -func TestNewEventIDIgnoresWMIActivityOperational(t *testing.T) { - cfg := Config{ - NewEventIDMode: "inventory", - NewEventIDMinCount: 3, - NewEventIDLearningPeriod: 24 * time.Hour, - NewEventIDIgnoreChannels: []string{"Microsoft-Windows-WMI-Activity/Operational"}, - } - agentFirst := time.Date(2026, 7, 1, 10, 0, 0, 0, time.UTC) - eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC) - got := assessNewEventID(cfg, "Microsoft-Windows-WMI-Activity/Operational", 5857, 12, agentFirst, eventFirst) - if got.Emit { - t.Fatalf("WMI operational event should be ignored: %+v", got) - } - if got.Reason != "ignored_operational_channel" { - t.Fatalf("reason=%q", got.Reason) - } -} - -func TestNewEventIDInventoryIsInformationalAndPlausible(t *testing.T) { - cfg := Config{ - NewEventIDMode: "inventory", - NewEventIDMinCount: 3, - NewEventIDLearningPeriod: 24 * time.Hour, - } - agentFirst := time.Date(2026, 7, 1, 10, 0, 0, 0, time.UTC) - eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC) - got := assessNewEventID(cfg, "Application", 1000, 3, agentFirst, eventFirst) - if !got.Emit || got.Severity != "info" || got.Status != "plausible" { - t.Fatalf("unexpected assessment: %+v", got) - } -} - -func TestNewEventIDSkipsAgentLearningPeriod(t *testing.T) { - cfg := Config{ - NewEventIDMode: "alert", - NewEventIDMinCount: 1, - NewEventIDLearningPeriod: 24 * time.Hour, - } - agentFirst := time.Date(2026, 7, 18, 17, 30, 0, 0, time.UTC) - eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC) - got := assessNewEventID(cfg, "Security", 4719, 4, agentFirst, eventFirst) - if got.Emit || got.Reason != "agent_learning_period" { - t.Fatalf("unexpected assessment: %+v", got) - } -} - -func TestMetadataContextAllowlist(t *testing.T) { - cfg := Config{MetadataContextEventIDs: map[uint32]struct{}{4624: {}, 4740: {}}} - if !shouldStoreMetadataContext(cfg, "Security", 4740) { - t.Fatal("Security/4740 should retain compact context") - } - if shouldStoreMetadataContext(cfg, "Microsoft-Windows-WMI-Activity/Operational", 4740) { - t.Fatal("non-Security channel must not create context rows") - } - if shouldStoreMetadataContext(cfg, "Security", 4688) { - t.Fatal("event outside allowlist must stay count-only") - } -} - -func TestCompactLockoutContextDropsNoise(t *testing.T) { - norm := NormalizedEvent{ - TargetUser: "EXAMPLE\\Alice", - SubjectUser: "DC01$", - Workstation: "CLIENT-42", - SrcIP: "10.0.0.10", - LogonType: "3", - StatusText: "status-varies", - FailureReason: "large noisy reason", - } - target, subject, srcIP, workstation, logonType, statusText, failureReason := compactMetadataContext(4740, norm) - if target != "alice" || workstation != "CLIENT-42" { - t.Fatalf("unexpected lockout identity target=%q workstation=%q", target, workstation) - } - if subject != "" || srcIP != "" || logonType != "" || statusText != "" || failureReason != "" { - t.Fatalf("lockout context retained high-cardinality noise: subject=%q src=%q logon=%q status=%q reason=%q", subject, srcIP, logonType, statusText, failureReason) - } -} - -func TestRetainCompactContextDropsMachineLogons(t *testing.T) { - if retainCompactContext(4624, "server01$", "") { - t.Fatal("machine-account logon context should not be retained") - } - if !retainCompactContext(4624, "alice", "") { - t.Fatal("human logon context should be retained") - } -} diff --git a/raw-event.sh b/raw-event.sh new file mode 100755 index 0000000..ae42008 --- /dev/null +++ b/raw-event.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env sh +set -eu +cd "$(dirname "$0")" +if [ "$#" -lt 1 ]; then echo "usage: $0 [raw_index]" >&2; exit 2; fi +key="$1"; idx="${2:-}" +[ -f .env ] && { set -a; . ./.env; set +a; } +if [ -n "$idx" ] && command -v python3 >/dev/null 2>&1; then + docker compose exec -T archive-uploader rclone cat "garage:${GARAGE_BUCKET:-siem-raw}/$key" | gzip -dc | python3 -c 'import json,sys; d=json.load(sys.stdin); i=int(sys.argv[1]); print(json.dumps(d["events"][i], ensure_ascii=False, indent=2))' "$idx" +else + docker compose exec -T archive-uploader rclone cat "garage:${GARAGE_BUCKET:-siem-raw}/$key" | gzip -dc +fi diff --git a/reset-DANGEROUS.sh b/reset-DANGEROUS.sh new file mode 100755 index 0000000..d7af9f0 --- /dev/null +++ b/reset-DANGEROUS.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env sh +set -eu +cd "$(dirname "$0")" +printf 'ACHTUNG: Alle SIEM-Daten löschen. Tippe DELETE: ' +read x +[ "$x" = "DELETE" ] || exit 1 +docker compose down -v --remove-orphans +rm -f deploy/garage/garage.toml diff --git a/schema.legacy.sql b/schema.legacy.sql deleted file mode 100644 index 64cba70..0000000 --- a/schema.legacy.sql +++ /dev/null @@ -1,101 +0,0 @@ -CREATE DATABASE IF NOT EXISTS eventcollector - CHARACTER SET utf8mb4 - COLLATE utf8mb4_unicode_ci; - -USE eventcollector; - -CREATE TABLE IF NOT EXISTS agents ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - hostname VARCHAR(255) NOT NULL, - api_key_hash CHAR(64) NOT NULL, - first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - last_ip VARCHAR(64) NOT NULL DEFAULT '', - is_enabled TINYINT(1) NOT NULL DEFAULT 1, - PRIMARY KEY (id), - UNIQUE KEY ux_agents_hostname (hostname), - KEY ix_agents_last_seen (last_seen) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE IF NOT EXISTS event_logs ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - agent_id BIGINT UNSIGNED NOT NULL, - hostname VARCHAR(255) NOT NULL, - channel_name VARCHAR(128) NOT NULL, - event_id INT UNSIGNED NOT NULL, - source VARCHAR(255) NOT NULL, - ts DATETIME(6) NOT NULL, - received_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - msg LONGTEXT NOT NULL, - msg_sha256 CHAR(64) NOT NULL, - PRIMARY KEY (id), - KEY ix_event_logs_ts (ts), - KEY ix_event_logs_received_at (received_at), - KEY ix_event_logs_agent_ts (agent_id, ts), - KEY ix_event_logs_eventid_ts (event_id, ts), - KEY ix_event_logs_hostname_ts (hostname, ts), - KEY ix_event_logs_channel_event_ts (channel_name, event_id, ts), - CONSTRAINT fk_event_logs_agent - FOREIGN KEY (agent_id) REFERENCES agents(id) - ON DELETE RESTRICT - ON UPDATE RESTRICT -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -CREATE TABLE IF NOT EXISTS detections ( - id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, - rule_name VARCHAR(128) NOT NULL, - severity VARCHAR(32) NOT NULL, - hostname VARCHAR(255) NOT NULL, - channel_name VARCHAR(128) NOT NULL DEFAULT '', - event_id INT UNSIGNED NOT NULL DEFAULT 0, - score DOUBLE NOT NULL DEFAULT 0, - window_start DATETIME(6) NOT NULL, - window_end DATETIME(6) NOT NULL, - summary VARCHAR(512) NOT NULL, - details_json JSON NOT NULL, - created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), - PRIMARY KEY (id), - UNIQUE KEY ux_detection_dedupe (rule_name, hostname, channel_name, event_id, window_start, window_end), - KEY ix_detections_created (created_at), - KEY ix_detections_rule_host_time (rule_name, hostname, created_at), - KEY ix_detections_severity_time (severity, created_at) -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; - -USE eventcollector; - -INSERT INTO agents (hostname, api_key_hash) -VALUES - ('client01.domain.local', SHA2('SUPER-LANGER-AGENT-KEY-01', 256)), - ('client02.domain.local', SHA2('SUPER-LANGER-AGENT-KEY-02', 256)); - - #V2 - - ALTER TABLE event_logs - ADD COLUMN computer VARCHAR(255) NOT NULL DEFAULT '' AFTER source, - ADD COLUMN provider_name VARCHAR(255) NOT NULL DEFAULT '' AFTER computer, - ADD COLUMN level_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER provider_name, - ADD COLUMN task_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER level_value, - ADD COLUMN opcode_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER task_value, - ADD COLUMN keywords VARCHAR(255) NOT NULL DEFAULT '' AFTER opcode_value, - ADD COLUMN target_user VARCHAR(255) NOT NULL DEFAULT '' AFTER keywords, - ADD COLUMN target_domain VARCHAR(255) NOT NULL DEFAULT '' AFTER target_user, - ADD COLUMN subject_user VARCHAR(255) NOT NULL DEFAULT '' AFTER target_domain, - ADD COLUMN subject_domain VARCHAR(255) NOT NULL DEFAULT '' AFTER subject_user, - ADD COLUMN workstation VARCHAR(255) NOT NULL DEFAULT '' AFTER subject_domain, - ADD COLUMN src_ip VARCHAR(64) NOT NULL DEFAULT '' AFTER workstation, - ADD COLUMN src_port VARCHAR(32) NOT NULL DEFAULT '' AFTER src_ip, - ADD COLUMN logon_type VARCHAR(32) NOT NULL DEFAULT '' AFTER src_port, - ADD COLUMN process_name VARCHAR(512) NOT NULL DEFAULT '' AFTER logon_type, - ADD COLUMN authentication_package VARCHAR(128) NOT NULL DEFAULT '' AFTER process_name, - ADD COLUMN logon_process VARCHAR(128) NOT NULL DEFAULT '' AFTER authentication_package, - ADD COLUMN status_text VARCHAR(64) NOT NULL DEFAULT '' AFTER logon_process, - ADD COLUMN sub_status_text VARCHAR(64) NOT NULL DEFAULT '' AFTER status_text, - ADD COLUMN failure_reason VARCHAR(512) NOT NULL DEFAULT '' AFTER sub_status_text; - -ALTER TABLE event_logs - ADD KEY ix_event_logs_target_user_ts (target_user, ts), - ADD KEY ix_event_logs_src_ip_ts (src_ip, ts), - ADD KEY ix_event_logs_target_user_src_ip_ts (target_user, src_ip, ts), - ADD KEY ix_event_logs_eventid_srcip_ts (event_id, src_ip, ts), - ADD KEY ix_event_logs_eventid_targetuser_ts (event_id, target_user, ts), - ADD KEY ix_event_logs_eventid_logontype_ts (event_id, logon_type, ts); \ No newline at end of file diff --git a/schema.sql b/schema.sql deleted file mode 100644 index f007780..0000000 --- a/schema.sql +++ /dev/null @@ -1,11 +0,0 @@ --- Canonical schema notice --- --- New empty database: --- deploy/mariadb/init/001-schema.sql --- --- Existing database: --- deploy/mariadb/migrations/002-metadata-first.sql --- --- The former schema.sql was an obsolete, non-idempotent development schema and --- has been retained as schema.legacy.sql only for historical reference. Do not --- run schema.legacy.sql against a production database. diff --git a/status.sh b/status.sh new file mode 100755 index 0000000..82d0dd2 --- /dev/null +++ b/status.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env sh +set -eu +cd "$(dirname "$0")" +docker compose ps +printf '\nIngress readiness:\n' +curl -fsS http://127.0.0.1:$(grep '^INGRESS_PORT=' .env | cut -d= -f2-)/readyz || true +printf '\n\nRecent errors:\n' +docker compose logs --since=10m ingress processor detector api 2>&1 | grep -Ei 'error|fatal|failed|panic' | tail -50 || true diff --git a/stop.sh b/stop.sh new file mode 100755 index 0000000..794c40e --- /dev/null +++ b/stop.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env sh +set -eu +cd "$(dirname "$0")" +docker compose down diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000..98b194c --- /dev/null +++ b/web/index.html @@ -0,0 +1,22 @@ + +Greenfield SIEM + +
Greenfield SIEMTenant: {{.Tenant}}
+
Events / 24h
–
Aktive Hosts / 24h
–
Offene High/Critical
–
Pipeline
OK
+
+
ZeitHostEventAktionUserSource-IPWorkstationProzess
+ +
+