sendnrw/siem-backend
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Vollständiges Redesign
Some checks failed
release-tag / release-image (push) Failing after 5m54s
Details

Browse Source
This commit is contained in:
Jan
2026-07-23 14:45:35 +02:00
parent 42db7d18f6
commit c49bfa1c34
58 changed files with 2781 additions and 18124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
ANALYSE-COMPACT-STORAGE.md
View File

@@ -1,59 +0,0 @@
# Analyse: Warum 19,2 Mio. Zeilen nach zwei Tagen entstanden
## Befund
Die vorige Version war noch nicht wirklich „Metadata-first“:
1. `insertBatch()` führte für **jedes** eingehende Event weiterhin ein `INSERT INTO event_logs` aus.
2. Danach wurden zusätzlich `event_count_buckets`, `event_catalog` und `event_occurrences` geschrieben.
3. `event_occurrences` verwendete zu viele Dimensionen für die Eindeutigkeit. Provider, Benutzer, IP, Workstation, LogonType, Status und FailureReason konnten die Aggregation stark fragmentieren.
4. Mehrere UEBA-/Detection-Regeln lasen weiterhin aus `event_logs`.
5. `/ui` las die letzten Events und den 24h-Eventzähler aus `event_occurrences`. Bei hoher Kardinalität wurde damit bereits die Startseite teuer.
19,2 Mio. Events in 48 Stunden entsprechen im Mittel ungefähr 111 Events/s. Diese Rate ist für kleine aggregierte Upserts nicht das Kernproblem. Eine breite Einzelzeile mit mehreren Sekundärindizes pro Event dagegen erzeugt sehr viel Daten- und Index-I/O.
## Korrektur
### Universal: `event_count_buckets`
Für jedes Event wird nur noch ein Counter auf
`5-Minuten-Bucket × Host × Channel × Event-ID`
hochgezählt. Ein Host mit 200 verschiedenen Event-IDs erzeugt damit theoretisch höchstens etwa 57.600 Counter-Zeilen pro Tag, selbst wenn dahinter Millionen reale Events liegen (200 IDs × 288 Buckets), praktisch meist deutlich weniger.
### Kontext nur bei Bedarf
`event_occurrences` wird nur für eine konfigurierbare Allowlist von Security-IDs geschrieben. Der Dimensionsschlüssel enthält nur stabile, detection-relevante Werte. Beispiel 4740: Zieluser und CallerComputer/Workstation; nicht FailureReason, Status, IP oder Subject-Machine-Account.
### Keine Einzelzeilen im Normalbetrieb
`STORE_EVENT_ROWS=false` ist der neue Standard. `event_logs` wird dann überhaupt nicht beschrieben. `STORE_RAW_XML=true` aktiviert implizit wieder Full-Event-Zeilen und ist deshalb nur für kurze, gezielte Forensik gedacht.
### UI
`/ui` verwendet nur `event_count_buckets`. Die Eventliste hat ohne explizites `from` automatisch ein 24h-Fenster. User-/IP-Suchen wechseln gezielt auf die kleine Kontexttabelle.
## Sofortmaßnahme für die bestehende Datenbank
Nach Deployment des neuen Backends:
```sql
SOURCE deploy/mariadb/migrations/004-compact-storage.sql;
SOURCE deploy/mariadb/emergency-compact-cleanup.sql;
SOURCE deploy/mariadb/cardinality-diagnostics.sql;
```
Der Emergency-Cleanup löscht absichtlich die redundante alte Full-Event-/Kontext-Historie, nicht jedoch `event_count_buckets`, `event_catalog`, Detections oder Baselines.
## Retention
Die detaillierten Zeit-Buckets werden standardmäßig 30 Tage gehalten. `event_catalog` bleibt dauerhaft und behält First/Last-Seen plus Gesamtzähler. Damit wächst die Zeitreihenhistorie nicht unbegrenzt.
## Erwartetes Monitoring
Der wichtigste Wert ist künftig nicht `COUNT(*)` allein, sondern das Verhältnis:
`SUM(cnt) / COUNT(*)`
in den Aggregattabellen. Je höher dieser Wert, desto mehr reale Events werden durch eine Datenbankzeile repräsentiert. `cardinality-diagnostics.sql` zeigt diesen Wert für die letzten 24 Stunden.

104
ANALYSE-METADATA-FIRST.md
View File

@@ -1,104 +0,0 @@
> **Überholt:** Dieses Dokument beschreibt die erste Metadata-first-Iteration.
> Für den produktiven Stand nach dem 19,2-Mio.-Zeilen-Befund gilt
> `ANALYSE-COMPACT-STORAGE.md`. Insbesondere werden `event_logs` jetzt
> standardmäßig gar nicht mehr geschrieben und `event_occurrences` ist nur noch
> eine selektive Security-Kontexttabelle.
# Untersuchung und Anpassung: Metadata-first
## Befund
Die Performanceprobleme kommen nicht nur von „zu viel XML“, sondern aus mehreren sich verstärkenden Ursachen:
1. Jedes Event wurde breit in `event_logs` und zusätzlich vollständig in `event_log_raw` gespeichert.
2. Die Partitionswartung verwendete `event_logs_raw`, die Tabelle heißt jedoch `event_log_raw`. Der Lauf brach damit ab; alte Partitionen wurden nicht gelöscht.
3. `event_count_buckets` und `ueba_context_buckets` waren nicht partitioniert und wuchsen unbegrenzt.
4. `target_user_norm` und `subject_user_norm` waren im Schema und in Indizes vorgesehen, wurden beim Insert aber nicht befüllt.
5. Dashboard und Event-UI lasen direkt aus der größten Einzeltabelle. Die Event-Filter enthielten zusätzlich eine korrelierte `EXISTS`-Abfrage auf Detections.
6. Die „neue Event-ID“-Regel verwendete einen teuren `NOT EXISTS`-Vergleich gegen die gesamte Eventhistorie.
7. Zählerregeln wie Failed-Logon-Spike, Reboot-Spike und Password-Spray zählten Einzelevents, obwohl dafür Aggregate ausreichen.
Ohne Zugriff auf die produktive Datenbank konnte ich keine realen `EXPLAIN ANALYZE`-Pläne oder Größenverteilungen messen. Die oben genannten Punkte ergeben sich direkt aus Schema und Abfragepfaden im Projekt.
## Umgesetztes Zielmodell
### Hot Data
`event_logs` enthält weiterhin kompakte Einzelevent-Metadaten, weil einige Regeln die genaue Reihenfolge benötigen, etwa „erfolgreicher Login nach Fehlversuchen“. Die Standardaufbewahrung beträgt 72 Stunden.
### Langfristige Metadaten
`event_occurrences` speichert pro Zeit-Bucket und Dimensionskombination:
- Host, Channel, Event-ID und Provider
- Target-/Subject-User
- Source-IP und Workstation/Gerät
- Logon-Type, Status und Fehlergrund
- Anzahl sowie erstes und letztes Auftreten
Die UI liest Bucket-Zeilen direkt mit `ORDER BY bucket_start DESC LIMIT ...`; dadurch bleibt die Abfrage index- und partitionsfreundlich und benötigt keinen großen `GROUP BY`.
### Event-Katalog
`event_catalog` enthält pro Host/Channel/Event-ID nur erstes Auftreten, letztes Auftreten und Gesamtzahl. Die „neue Event-ID“-Regel benötigt dadurch keinen Anti-Join mehr.
### Raw XML
Raw-XML ist mit `STORE_RAW_XML=false` standardmäßig deaktiviert. Bei temporärem forensischem Bedarf kann es aktiviert und mit eigener kurzer `RAW_RETENTION` betrieben werden.
### Direkter Metadaten-Ingest
Collector können weiterhin XML in `msg` senden. Neu ist ein `meta`-Objekt, das ohne XML auskommt. Für Event 4740 wird `CallerComputerName` als Workstation/Gerät verarbeitet.
## Erwartete Wirkung
- Wegfall der dauerhaften XML-Doppelhaltung.
- Garantierte Aufbewahrungsgrenzen durch korrigierte und erweiterte Partitionswartung.
- Deutlich weniger Zeilen in langfristigen Abfragen durch Zeit-Buckets.
- Schnellere Dashboard-/Event-Seiten, da sie nicht mehr `event_logs` durchsuchen.
- Schnellere Zählerregeln durch `SUM(cnt)` auf `event_occurrences`.
- Kleine, konstante Abfrage für neue Event-IDs über `event_catalog`.
- Nutzbare User-Indizes, da normalisierte User-Spalten nun tatsächlich befüllt und in wichtigen Regeln verwendet werden.
Die tatsächliche Kompressionswirkung hängt von der Kardinalität ab. Viele identische Events innerhalb einer Minute werden stark verdichtet. Einzigartige Kombinationen aus User, IP und Workstation reduzieren sich weniger, bleiben aber deutlich kleiner als Voll-XML plus breite Einzeleventzeile.
## Lockout-Abfrage
```sql
SELECT
first_event_ts,
last_event_ts,
cnt,
hostname AS reporting_host,
target_user,
workstation AS caller_device,
status_text,
failure_reason
FROM event_occurrences
WHERE channel_name = 'Security'
AND event_id = 4740
AND bucket_start >= UTC_TIMESTAMP() - INTERVAL 7 DAY
ORDER BY bucket_start DESC;
```
Bei einem Domain-Lockout ist `hostname` typischerweise der meldende Domain Controller; `workstation` enthält – sofern im Event vorhanden – `CallerComputerName` und damit das verursachende Gerät.
## Empfohlener Rollout
1. Datenbank sichern und Ingest stoppen.
2. Tabellen- und Partitionsgrößen mit `deploy/mariadb/diagnostics.sql` erfassen.
3. `deploy/mariadb/migrations/002-metadata-first.sql` in einem Wartungsfenster ausführen.
4. `.env.example` nach `.env` kopieren beziehungsweise bestehende Secrets übernehmen und die neuen Variablen ergänzen.
5. Backend deployen, zunächst `STORE_RAW_XML=false`, `EVENT_RETENTION=72h`, `METADATA_BUCKET=1m`.
6. Partition-Maintenance-Logs und Datenbankgrößen kontrollieren.
7. Nach 3–7 Tagen Schwellenwerte und Bucketgröße anhand der realen Eventrate nachjustieren.
## Wann ein Wechsel der Datenbank sinnvoll wäre
MariaDB ist für dieses Metadatenmodell weiterhin geeignet. Ein Wechsel zu ClickHouse wäre interessant, wenn dauerhaft sehr hohe Raten und große analytische Zeitreihen über Jahre benötigt werden. OpenSearch wäre sinnvoll, wenn Volltextsuche im Rohinhalt eine Kernanforderung bleibt. Für „welches Event, wann, auf welchem Gerät, für welchen User und wie oft“ ist ein Engine-Wechsel derzeit nicht erforderlich.
## Ergänzung: `new_event_id` ist Inventar, nicht automatisch Incident
Die bisherige Regel hat jedes erste Auftreten pauschal als offenen Medium-Alarm behandelt. Das überschätzt insbesondere diagnostische und operative Channels. Die neue Standardklassifikation ist `inventory`: nach Lernphase und Mindestanzahl entsteht höchstens eine plausible Info-Meldung. `Microsoft-Windows-WMI-Activity/Operational` wird standardmäßig ignoriert. Nur explizit konfigurierte High-Risk-IDs oder der Betriebsmodus `alert` können eine offene Detection erzeugen.
Für reproduzierbare Lasttests liegt unter `cmd/siem-stress-agent` ein begrenzter Metadaten-Agent. Er markiert sämtliche Daten eindeutig, benötigt `--confirm-load-test` und besitzt harte Maximalwerte für Rate, Laufzeit, Worker und Eventzahl.

42
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,42 @@
# Architekturentscheidungen
## 1. MariaDB ist kein Event Store mehr
PostgreSQL ist ausschließlich Control Plane. ClickHouse ist ausschließlich Event-/Analytics-Plane. Dadurch konkurrieren Agent-Updates, Incident-Status und Benutzeraktionen nicht mit Milliarden append-only Events.
## 2. Queue vor Datenbank
Der Ingress quittiert erst, nachdem Redpanda den Batch bestätigt hat. ClickHouse-Ausfälle führen damit nicht zu HTTP-Timeout-Kaskaden bei den Collectoren. Der Processor kann nachholen, solange die Queue-Retention nicht überschritten wird.
## 3. Ein kanonisches Event
Es gibt keine parallelen `event_logs`, `event_occurrences`, `event_count_buckets`, `raw_event`- und Rule-Helper-Kopien mehr. Die einzige langfristige Eventtabelle ist `siem.events`. Das Dashboard-Rollup ist eine explizite, kleine Materialized View.
## 4. Raw getrennt von Analytics
Raw-Payloads sind für Parserfehler und Forensik wertvoll, aber ungeeignet als primäre Analytics-Zeile. Sie werden gzip-komprimiert in S3-kompatiblen Object Storage verschoben. ClickHouse speichert nur den Object-Key und den Index im Batch.
## 5. Kein Graph als Primärspeicher
Graph-Sichten können später aus ClickHouse abgeleitet werden, z. B. `(User)-[:LOGGED_ON_TO]->(Host)` mit `first_seen`, `last_seen`, `count`. Jedes einzelne Event als Graph-Knoten würde die gleiche Explosion nur in einer anderen Engine wiederholen.
## 6. Keine TSDB für Security-Events
Prometheus überwacht die Pipeline, speichert aber keine Benutzer-/Host-/IP-Eventdimensionen. Hochkardinale Security-Attribute gehören nach ClickHouse.
## 7. Idempotenz
Redpanda/Kafka-Consumer liefern at-least-once und auch ein Collector kann nach einem HTTP-Timeout denselben Batch erneut senden. Deshalb berechnet der Ingress `batch_uid = SHA-256(agent_id + Batch-Inhalt)` und der Processor verwendet `event_uid = batch_uid:index`. ReplacingMergeTree und `uniqExact`-States verhindern damit Zählfehler bei Queue- und identischen HTTP-Retries.
## 8. UI ist kein Compute-Job
Die Startseite liest `events_5m` und PostgreSQL-Findings. Sie startet keine Detection-Regeln. Die Event-Timeline hat immer ein begrenztes Zeitfenster und Limit.
## 9. Retention nach Datenklasse
- normalisierte Events: 90 Tage
- Raw: 30 Tage
- Rollups: 730 Tage
- Queue: 24 Stunden
Diese Werte sind Defaults, keine Compliance-Aussage. Rechtliche und organisatorische Anforderungen haben Vorrang.

37
Dockerfile
View File

@@ -1,32 +1,15 @@
FROM golang:1.26 AS builder
FROM golang:1.26.5-alpine AS build
RUN apk add --no-cache ca-certificates
WORKDIR /src
COPY go.mod go.sum ./
COPY go.mod ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/siem ./cmd/siem
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build -trimpath -ldflags="-s -w" -o /out/eventcollector .
FROM debian:trixie-slim
ENV LISTEN_ADDR=:8080
ENV TZ=Europe/Berlin
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates tzdata wget \
&& rm -rf /var/lib/apt/lists/* \
&& groupadd --system --gid 10001 app \
&& useradd --system --uid 10001 --gid 10001 --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin app
FROM scratch
WORKDIR /app
COPY --from=builder /out/eventcollector /app/eventcollector
USER 10001:10001
EXPOSE 8080
ENTRYPOINT ["/app/eventcollector"]
COPY --from=build /out/siem /siem
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY web /app/web
USER 65532:65532
ENTRYPOINT ["/siem"]

10
Dockerfile.stress
View File

@@ -1,10 +0,0 @@
FROM golang:1.26 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/siem-stress-agent ./cmd/siem-stress-agent
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o /out/siem-stress-agent ./cmd/siem-stress-agent
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /out/siem-stress-agent /siem-stress-agent
ENTRYPOINT ["/siem-stress-agent"]

4882
README.md
View File

File diff suppressed because it is too large Load Diff

24
RELEASE-CHECKS.md Normal file
View File

@@ -0,0 +1,24 @@
# Release checks
Stand: 2026-07-23
Durchgeführt:
- `go test ./...` gegen lokale Interface-Stubs für pgx/kafka-go mit der vorhandenen Go-1.23.2-Toolchain: erfolgreich.
- `go vet ./...` unter denselben isolierten Bedingungen: erfolgreich.
- Ingress-Tests: Batch-UID/Retry-Idempotenz und Validierung: erfolgreich.
- Normalizer-Tests: Windows-Metadaten/XML und Lockout-Workstation: erfolgreich.
- `compose.yml` und Prometheus-YAML syntaktisch geparst: erfolgreich.
- ClickHouse-Prometheus-XML geparst: erfolgreich.
- POSIX-Shellskripte mit `sh -n` geprüft: erfolgreich.
- `deploy.sh` mit simuliertem Docker/Healthcheck end-to-end ausgeführt: Secret-Erzeugung, bestehender Enrollment-Key, Template-Rendering und Compose-Aufrufe erfolgreich.
- ZIP-Integrität mit `unzip -t`: erfolgreich.
Nicht in dieser Ausführungsumgebung möglich:
- echter Docker-Compose-Start der Datenbanken/Broker/Object-Storage-Komponenten;
- Integrationstest gegen reale ClickHouse-, PostgreSQL-, Redpanda- und Garage-Container;
- Ausführung von `deploy.ps1`, da PowerShell hier nicht installiert ist;
- Build mit Go 1.26.5, da die lokale Umgebung keinen Netzwerkzugriff für die Toolchain/Module besitzt. Das Dockerfile verwendet Go 1.26.5.
Der erste reale Deployment-Lauf sollte daher anschließend mit `./doctor.sh` und einem begrenzten Stress-Test geprüft werden.

476
cmd/siem-stress-agent/main.go
View File

@@ -1,476 +0,0 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"math"
"net/http"
"net/url"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
)
const (
maxRate = 100_000
maxWorkers = 64
maxDuration = time.Hour
maxEvents = 5_000_000
)
type config struct {
Endpoint string
Hostname string
APIKey string
EnrollmentKey string
Scenario string
Rate int
BatchSize int
Workers int
Duration time.Duration
EventLimit int64
Timeout time.Duration
Confirm bool
}
type event struct {
Hostname string `json:"host"`
Channel string `json:"channel"`
EventID uint32 `json:"id"`
Source string `json:"source"`
Time time.Time `json:"ts"`
Metadata metadata `json:"meta"`
}
type metadata struct {
Computer string `json:"computer,omitempty"`
ProviderName string `json:"provider,omitempty"`
TargetUser string `json:"target_user,omitempty"`
TargetDomain string `json:"target_domain,omitempty"`
SubjectUser string `json:"subject_user,omitempty"`
Workstation string `json:"workstation,omitempty"`
Device string `json:"device,omitempty"`
SrcIP string `json:"src_ip,omitempty"`
SrcPort string `json:"src_port,omitempty"`
LogonType string `json:"logon_type,omitempty"`
ProcessName string `json:"process_name,omitempty"`
StatusText string `json:"status,omitempty"`
SubStatus string `json:"sub_status,omitempty"`
Failure string `json:"failure_reason,omitempty"`
}
type result struct {
Events int
Duration time.Duration
Status int
Err error
}
type stats struct {
requests atomic.Int64
accepted atomic.Int64
failed atomic.Int64
bytesSent atomic.Int64
mu sync.Mutex
latencies []time.Duration
statuses map[int]int64
errors map[string]int64
}
func main() {
cfg := parseFlags()
if err := validateConfig(cfg); err != nil {
fmt.Fprintln(os.Stderr, "Konfigurationsfehler:", err)
os.Exit(2)
}
runCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
produceCtx, cancelProduction := context.WithTimeout(runCtx, cfg.Duration)
defer cancelProduction()
transport := &http.Transport{
MaxIdleConns: cfg.Workers * 2,
MaxIdleConnsPerHost: cfg.Workers,
IdleConnTimeout: 30 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: cfg.Timeout}
defer transport.CloseIdleConnections()
jobs := make(chan []event, cfg.Workers*2)
results := make(chan result, cfg.Workers*2)
st := &stats{statuses: make(map[int]int64), errors: make(map[string]int64)}
var workers sync.WaitGroup
for i := 0; i < cfg.Workers; i++ {
workers.Add(1)
go func() {
defer workers.Done()
for batch := range jobs {
results <- sendBatch(runCtx, client, cfg, batch)
}
}()
}
collectorDone := make(chan struct{})
go func() {
defer close(collectorDone)
for r := range results {
st.requests.Add(1)
st.mu.Lock()
st.latencies = append(st.latencies, r.Duration)
if r.Status != 0 {
st.statuses[r.Status]++
}
if r.Err != nil {
st.failed.Add(int64(r.Events))
st.errors[shortError(r.Err)]++
} else if r.Status >= 200 && r.Status < 300 {
st.accepted.Add(int64(r.Events))
} else {
st.failed.Add(int64(r.Events))
}
st.mu.Unlock()
}
}()
fmt.Printf("SIEM-Stresstest startet: endpoint=%s host=%s scenario=%s rate=%d/s batch=%d workers=%d duration=%s limit=%d\n",
cfg.Endpoint, cfg.Hostname, cfg.Scenario, cfg.Rate, cfg.BatchSize, cfg.Workers, cfg.Duration, cfg.EventLimit)
started := time.Now()
produced := produce(produceCtx, cfg, jobs, st)
close(jobs)
workers.Wait()
close(results)
<-collectorDone
printReport(st, produced, time.Since(started))
if st.failed.Load() > 0 {
os.Exit(1)
}
}
func parseFlags() config {
var cfg config
flag.StringVar(&cfg.Endpoint, "url", getenv("SIEM_STRESS_URL", "http://127.0.0.1:8090/ingest"), "vollständige Ingest-URL")
flag.StringVar(&cfg.Hostname, "host", getenv("SIEM_STRESS_HOST", "SIEM-STRESS-01"), "Test-Hostname; alle Events eines Batches verwenden diesen Host")
flag.StringVar(&cfg.APIKey, "api-key", os.Getenv("SIEM_STRESS_API_KEY"), "API-Key des Test-Agenten")
flag.StringVar(&cfg.EnrollmentKey, "enrollment-key", os.Getenv("SIEM_STRESS_ENROLLMENT_KEY"), "Enrollment-Key für die erste Registrierung")
flag.StringVar(&cfg.Scenario, "scenario", getenv("SIEM_STRESS_SCENARIO", "mixed"), "mixed, failed-logon, lockout, normal oder catalog")
flag.IntVar(&cfg.Rate, "rate", getenvInt("SIEM_STRESS_RATE", 200), "Zielrate in Events pro Sekunde")
flag.IntVar(&cfg.BatchSize, "batch", getenvInt("SIEM_STRESS_BATCH", 100), "Events pro HTTP-Request, maximal 1000")
flag.IntVar(&cfg.Workers, "workers", getenvInt("SIEM_STRESS_WORKERS", 4), "parallele HTTP-Worker")
flag.DurationVar(&cfg.Duration, "duration", getenvDuration("SIEM_STRESS_DURATION", 30*time.Second), "maximale Laufzeit")
flag.Int64Var(&cfg.EventLimit, "max-events", getenvInt64("SIEM_STRESS_MAX_EVENTS", 100_000), "zusätzliche harte Obergrenze der erzeugten Events")
flag.DurationVar(&cfg.Timeout, "timeout", getenvDuration("SIEM_STRESS_TIMEOUT", 20*time.Second), "HTTP-Timeout pro Request")
flag.BoolVar(&cfg.Confirm, "confirm-load-test", getenvBool("SIEM_STRESS_CONFIRM", false), "bestätigt, dass das Zielsystem für diesen Lasttest autorisiert ist")
flag.Parse()
return cfg
}
func validateConfig(cfg config) error {
if !cfg.Confirm {
return errors.New("--confirm-load-test fehlt")
}
u, err := url.Parse(cfg.Endpoint)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return errors.New("--url muss eine gültige HTTP(S)-Ingest-URL sein")
}
if strings.TrimSpace(cfg.Hostname) == "" || len(cfg.Hostname) > 191 {
return errors.New("--host ist leer oder länger als 191 Zeichen")
}
if strings.TrimSpace(cfg.APIKey) == "" {
return errors.New("--api-key ist erforderlich")
}
if cfg.Rate < 1 || cfg.Rate > maxRate {
return fmt.Errorf("--rate muss zwischen 1 und %d liegen", maxRate)
}
if cfg.BatchSize < 1 || cfg.BatchSize > 1000 {
return errors.New("--batch muss zwischen 1 und 1000 liegen")
}
if cfg.Workers < 1 || cfg.Workers > maxWorkers {
return fmt.Errorf("--workers muss zwischen 1 und %d liegen", maxWorkers)
}
if cfg.Duration <= 0 || cfg.Duration > maxDuration {
return fmt.Errorf("--duration muss größer 0 und höchstens %s sein", maxDuration)
}
if cfg.EventLimit < 1 || cfg.EventLimit > maxEvents {
return fmt.Errorf("--max-events muss zwischen 1 und %d liegen", maxEvents)
}
if cfg.Timeout <= 0 || cfg.Timeout > 2*time.Minute {
return errors.New("--timeout muss größer 0 und höchstens 2m sein")
}
switch strings.ToLower(cfg.Scenario) {
case "mixed", "failed-logon", "lockout", "normal", "catalog":
default:
return errors.New("unbekanntes --scenario")
}
return nil
}
func produce(ctx context.Context, cfg config, jobs chan<- []event, st *stats) int64 {
batchInterval := time.Duration(float64(time.Second) * float64(cfg.BatchSize) / float64(cfg.Rate))
if batchInterval < time.Millisecond {
batchInterval = time.Millisecond
}
ticker := time.NewTicker(batchInterval)
defer ticker.Stop()
var produced int64
var sequence int64
for produced < cfg.EventLimit {
remaining := cfg.EventLimit - produced
n := cfg.BatchSize
if int64(n) > remaining {
n = int(remaining)
}
batch := make([]event, n)
now := time.Now().UTC()
for i := range batch {
sequence++
batch[i] = generateEvent(cfg, sequence, now.Add(time.Duration(i)*time.Microsecond))
}
encoded, _ := json.Marshal(batch)
st.bytesSent.Add(int64(len(encoded)))
select {
case jobs <- batch:
produced += int64(n)
case <-ctx.Done():
return produced
}
select {
case <-ticker.C:
case <-ctx.Done():
return produced
}
}
return produced
}
func generateEvent(cfg config, seq int64, ts time.Time) event {
base := event{
Hostname: cfg.Hostname,
Source: "SIEM-Stress-Agent",
Time: ts,
Metadata: metadata{
Computer: cfg.Hostname,
ProviderName: "SIEM-Stress-Agent",
TargetDomain: "STRESS",
SubjectUser: "stress-agent$",
Workstation: fmt.Sprintf("STRESS-CLIENT-%03d", seq%25),
Device: fmt.Sprintf("STRESS-CLIENT-%03d", seq%25),
SrcIP: fmt.Sprintf("198.18.%d.%d", (seq/250)%250, seq%250+1),
SrcPort: strconv.FormatInt(40000+seq%20000, 10),
ProcessName: `C:\\Program Files\\SIEM-Stress-Agent\\stress.exe`,
},
}
scenario := strings.ToLower(cfg.Scenario)
if scenario == "mixed" {
switch seq % 10 {
case 0, 1, 2, 3:
scenario = "failed-logon"
case 4:
scenario = "lockout"
case 5, 6, 7:
scenario = "normal"
case 8:
base.Channel, base.EventID = "System", 1074
return base
default:
base.Channel, base.EventID = "Microsoft-Windows-WMI-Activity/Operational", 5857
return base
}
}
switch scenario {
case "failed-logon":
base.Channel, base.EventID = "Security", 4625
base.Metadata.TargetUser = fmt.Sprintf("stress-user-%03d", seq%100)
base.Metadata.LogonType = "3"
base.Metadata.StatusText = "0xC000006D"
base.Metadata.SubStatus = "0xC000006A"
base.Metadata.Failure = "Unknown user name or bad password"
case "lockout":
base.Channel, base.EventID = "Security", 4740
base.Metadata.TargetUser = fmt.Sprintf("stress-lockout-%03d", seq%20)
case "normal":
base.Channel, base.EventID = "Security", 4624
base.Metadata.TargetUser = fmt.Sprintf("stress-user-%03d", seq%100)
base.Metadata.LogonType = "3"
case "catalog":
base.Channel = "SIEM-Stress-Agent/Operational"
base.EventID = uint32(50_000 + seq%5_000)
base.Metadata.TargetUser = fmt.Sprintf("catalog-user-%03d", seq%50)
}
return base
}
func sendBatch(ctx context.Context, client *http.Client, cfg config, batch []event) result {
body, err := json.Marshal(batch)
if err != nil {
return result{Events: len(batch), Err: err}
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, cfg.Endpoint, bytes.NewReader(body))
if err != nil {
return result{Events: len(batch), Err: err}
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-API-Key", cfg.APIKey)
if cfg.EnrollmentKey != "" {
req.Header.Set("X-Enrollment-Key", cfg.EnrollmentKey)
}
req.Header.Set("User-Agent", "siem-stress-agent/1.0")
started := time.Now()
resp, err := client.Do(req)
duration := time.Since(started)
if err != nil {
return result{Events: len(batch), Duration: duration, Err: err}
}
defer resp.Body.Close()
limited, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
if readErr != nil {
return result{Events: len(batch), Duration: duration, Status: resp.StatusCode, Err: readErr}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return result{Events: len(batch), Duration: duration, Status: resp.StatusCode, Err: fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(limited)))}
}
return result{Events: len(batch), Duration: duration, Status: resp.StatusCode}
}
func printReport(st *stats, produced int64, elapsed time.Duration) {
st.mu.Lock()
latencies := append([]time.Duration(nil), st.latencies...)
statuses := cloneMap(st.statuses)
errorsByText := cloneStringMap(st.errors)
st.mu.Unlock()
sort.Slice(latencies, func(i, j int) bool { return latencies[i] < latencies[j] })
accepted := st.accepted.Load()
failed := st.failed.Load()
throughput := float64(accepted) / math.Max(elapsed.Seconds(), 0.001)
fmt.Println("\n--- Ergebnis ---")
fmt.Printf("Laufzeit: %s\n", elapsed.Round(time.Millisecond))
fmt.Printf("Erzeugte Events: %d\n", produced)
fmt.Printf("Akzeptierte Events:%d\n", accepted)
fmt.Printf("Fehlgeschlagen: %d\n", failed)
fmt.Printf("HTTP-Requests: %d\n", st.requests.Load())
fmt.Printf("Durchsatz: %.1f Events/s\n", throughput)
fmt.Printf("JSON gesendet: %.2f MiB\n", float64(st.bytesSent.Load())/(1024*1024))
if len(latencies) > 0 {
fmt.Printf("Latenz p50/p95/p99:%s / %s / %s\n", percentile(latencies, 0.50), percentile(latencies, 0.95), percentile(latencies, 0.99))
}
fmt.Printf("HTTP-Status: %v\n", statuses)
if len(errorsByText) > 0 {
fmt.Printf("Fehler: %v\n", errorsByText)
}
}
func percentile(values []time.Duration, p float64) time.Duration {
if len(values) == 0 {
return 0
}
idx := int(math.Ceil(float64(len(values))*p)) - 1
if idx < 0 {
idx = 0
}
if idx >= len(values) {
idx = len(values) - 1
}
return values[idx].Round(time.Millisecond)
}
func shortError(err error) string {
text := strings.TrimSpace(err.Error())
if len(text) > 180 {
return text[:180] + "..."
}
return text
}
func getenv(key, fallback string) string {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
return value
}
func getenvInt(key string, fallback int) int {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
n, err := strconv.Atoi(value)
if err != nil {
fmt.Fprintf(os.Stderr, "ungültiger Integer in %s: %v\n", key, err)
os.Exit(2)
}
return n
}
func getenvInt64(key string, fallback int64) int64 {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
n, err := strconv.ParseInt(value, 10, 64)
if err != nil {
fmt.Fprintf(os.Stderr, "ungültiger Integer in %s: %v\n", key, err)
os.Exit(2)
}
return n
}
func getenvDuration(key string, fallback time.Duration) time.Duration {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
d, err := time.ParseDuration(value)
if err != nil {
fmt.Fprintf(os.Stderr, "ungültige Dauer in %s: %v\n", key, err)
os.Exit(2)
}
return d
}
func getenvBool(key string, fallback bool) bool {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.ParseBool(value)
if err != nil {
fmt.Fprintf(os.Stderr, "ungültiger Boolean in %s: %v\n", key, err)
os.Exit(2)
}
return parsed
}
func cloneMap(in map[int]int64) map[int]int64 {
out := make(map[int]int64, len(in))
for k, v := range in {
out[k] = v
}
return out
}
func cloneStringMap(in map[string]int64) map[string]int64 {
out := make(map[string]int64, len(in))
for k, v := range in {
out[k] = v
}
return out
}

52
cmd/siem-stress-agent/main_test.go
View File

@@ -1,52 +0,0 @@
package main
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestSendBatch(t *testing.T) {
var got []event
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("X-API-Key") != "test-key" {
t.Fatalf("unexpected api key")
}
if err := json.NewDecoder(r.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
w.WriteHeader(http.StatusAccepted)
}))
defer server.Close()
cfg := config{Endpoint: server.URL, APIKey: "test-key", Timeout: time.Second}
batch := []event{generateEvent(config{Hostname: "SIEM-STRESS-01", Scenario: "lockout"}, 1, time.Now().UTC())}
res := sendBatch(context.Background(), server.Client(), cfg, batch)
if res.Err != nil || res.Status != http.StatusAccepted {
t.Fatalf("unexpected result: %+v", res)
}
if len(got) != 1 || got[0].EventID != 4740 || got[0].Source != "SIEM-Stress-Agent" {
t.Fatalf("unexpected payload: %+v", got)
}
}
func TestValidateConfigRequiresConfirmation(t *testing.T) {
cfg := config{
Endpoint: "http://127.0.0.1:8080/ingest",
Hostname: "SIEM-STRESS-01",
APIKey: "x",
Scenario: "mixed",
Rate: 1,
BatchSize: 1,
Workers: 1,
Duration: time.Second,
EventLimit: 1,
Timeout: time.Second,
}
if err := validateConfig(cfg); err == nil {
t.Fatal("expected missing confirmation to fail")
}
}

47
cmd/siem/main.go Normal file
View File

@@ -0,0 +1,47 @@
package main
import (
"context"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"example.com/siem-greenfield/internal/api"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/detector"
"example.com/siem-greenfield/internal/ingress"
"example.com/siem-greenfield/internal/processor"
"example.com/siem-greenfield/internal/stress"
)
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, "usage: siem <ingress|processor|detector|api|stress-agent>")
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
cfg := config.Load()
var err error
switch os.Args[1] {
case "ingress":
err = ingress.Run(ctx, cfg)
case "processor":
err = processor.Run(ctx, cfg)
case "detector":
err = detector.Run(ctx, cfg)
case "api":
err = api.Run(ctx, cfg)
case "stress-agent":
err = stress.Run(ctx, cfg, os.Args[2:])
default:
err = fmt.Errorf("unknown command %q", os.Args[1])
}
if err != nil {
log.Printf("fatal: %v", err)
os.Exit(1)
}
}

401
compose.yml
View File

@@ -1,179 +1,256 @@
name: greenfield-siem
services:
mariadb:
image: mariadb:11.8
container_name: siem-mariadb
restart: unless-stopped
ports:
- 3307:3306
env_file:
- .env
postgres:
image: postgres:18.4-alpine
environment:
MARIADB_DATABASE: ${MARIADB_DATABASE}
MARIADB_USER: ${MARIADB_USER}
MARIADB_PASSWORD: ${MARIADB_PASSWORD}
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD}
TZ: ${TZ}
command:
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_unicode_ci
- --innodb-buffer-pool-size=512M
- --max-connections=300
POSTGRES_DB: siem
POSTGRES_USER: siem
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- mariadb_data:/var/lib/mysql
- ./deploy/mariadb/init:/docker-entrypoint-initdb.d:ro
- postgres_data:/var/lib/postgresql
- ./deploy/postgres/init.sql:/docker-entrypoint-initdb.d/001-init.sql:ro
healthcheck:
test:
- CMD-SHELL
- mariadb-admin ping -h 127.0.0.1 -u root -p$$MARIADB_ROOT_PASSWORD
--silent
interval: 20s
timeout: 5s
retries: 10
start_period: 30s
networks:
- dockge_default
siem-backend:
image: git.send.nrw/sendnrw/siem-backend:latest
container_name: siem-backend
test: ["CMD-SHELL", "pg_isready -U siem -d siem"]
interval: 5s
timeout: 3s
retries: 30
restart: unless-stopped
env_file:
- .env
environment:
LISTEN_ADDR: ${LISTEN_ADDR}
DB_DSN: ${DB_DSN}
DB_MAX_OPEN_CONNS: ${DB_MAX_OPEN_CONNS}
DB_MAX_IDLE_CONNS: ${DB_MAX_IDLE_CONNS}
DB_CONN_MAX_LIFETIME: ${DB_CONN_MAX_LIFETIME}
DB_CONN_MAX_IDLE_TIME: ${DB_CONN_MAX_IDLE_TIME}
MAX_BODY_BYTES: ${MAX_BODY_BYTES}
HTTP_READ_TIMEOUT: ${HTTP_READ_TIMEOUT}
HTTP_WRITE_TIMEOUT: ${HTTP_WRITE_TIMEOUT}
HTTP_IDLE_TIMEOUT: ${HTTP_IDLE_TIMEOUT}
DETECTION_INTERVAL: ${DETECTION_INTERVAL}
OFFLINE_AFTER: ${OFFLINE_AFTER}
FAILED_LOGON_WINDOW: ${FAILED_LOGON_WINDOW}
FAILED_LOGON_THRESHOLD: ${FAILED_LOGON_THRESHOLD}
REBOOT_WINDOW: ${REBOOT_WINDOW}
REBOOT_THRESHOLD: ${REBOOT_THRESHOLD}
PASSWORD_SPRAY_WINDOW: ${PASSWORD_SPRAY_WINDOW}
PASSWORD_SPRAY_MIN_USERS: ${PASSWORD_SPRAY_MIN_USERS}
PASSWORD_SPRAY_MIN_ATTEMPTS: ${PASSWORD_SPRAY_MIN_ATTEMPTS}
SUCCESS_AFTER_FAILURE_WINDOW: ${SUCCESS_AFTER_FAILURE_WINDOW}
NEW_SOURCE_IP_LOOKBACK: ${NEW_SOURCE_IP_LOOKBACK}
NEW_SOURCE_IP_WINDOW: ${NEW_SOURCE_IP_WINDOW}
DETECTIONS_LIMIT: ${DETECTIONS_LIMIT}
NEW_EVENT_ID_MODE: ${NEW_EVENT_ID_MODE}
NEW_EVENT_ID_LEARNING_PERIOD: ${NEW_EVENT_ID_LEARNING_PERIOD}
NEW_EVENT_ID_CONFIRM_WINDOW: ${NEW_EVENT_ID_CONFIRM_WINDOW}
NEW_EVENT_ID_MIN_COUNT: ${NEW_EVENT_ID_MIN_COUNT}
NEW_EVENT_ID_IGNORE_CHANNELS: ${NEW_EVENT_ID_IGNORE_CHANNELS}
NEW_EVENT_ID_ALERT_CHANNELS: ${NEW_EVENT_ID_ALERT_CHANNELS}
NEW_EVENT_ID_HIGH_RISK_IDS: ${NEW_EVENT_ID_HIGH_RISK_IDS}
STORE_EVENT_ROWS: ${STORE_EVENT_ROWS}
STORE_RAW_XML: ${STORE_RAW_XML}
METADATA_CONTEXT_EVENT_IDS: ${METADATA_CONTEXT_EVENT_IDS}
METADATA_BUCKET: ${METADATA_BUCKET}
EVENT_RETENTION: ${EVENT_RETENTION}
RAW_RETENTION: ${RAW_RETENTION}
METADATA_RETENTION: ${METADATA_RETENTION}
PARTITION_MAINTENANCE_ENABLED: ${PARTITION_MAINTENANCE_ENABLED}
PARTITION_MAINTENANCE_INTERVAL: ${PARTITION_MAINTENANCE_INTERVAL}
PARTITION_INTERVAL: ${PARTITION_INTERVAL}
PARTITION_AHEAD: ${PARTITION_AHEAD}
PARTITION_BEHIND: ${PARTITION_BEHIND}
PARTITION_RETENTION: ${PARTITION_RETENTION}
TZ: ${TZ}
depends_on:
mariadb:
condition: service_healthy
ports:
- 8090:8080
healthcheck:
test:
- CMD-SHELL
- wget -qO- http://127.0.0.1:8080/healthz >/dev/null 2>&1 || exit 1
interval: 30s
timeout: 5s
retries: 5
start_period: 20s
networks:
- dockge_default
stress-agent:
profiles: ["stress"]
build:
context: .
dockerfile: Dockerfile.stress
container_name: siem-stress-agent
restart: "no"
- "127.0.0.1:${POSTGRES_PORT:-5432}:5432"
clickhouse:
image: clickhouse/clickhouse-server:26.6.2.81
environment:
SIEM_STRESS_URL: ${SIEM_STRESS_URL}
SIEM_STRESS_HOST: ${SIEM_STRESS_HOST}
SIEM_STRESS_API_KEY: ${SIEM_STRESS_API_KEY}
SIEM_STRESS_ENROLLMENT_KEY: ${ENROLLMENT_KEY}
SIEM_STRESS_SCENARIO: ${SIEM_STRESS_SCENARIO}
SIEM_STRESS_RATE: ${SIEM_STRESS_RATE}
SIEM_STRESS_BATCH: ${SIEM_STRESS_BATCH}
SIEM_STRESS_WORKERS: ${SIEM_STRESS_WORKERS}
SIEM_STRESS_DURATION: ${SIEM_STRESS_DURATION}
SIEM_STRESS_MAX_EVENTS: ${SIEM_STRESS_MAX_EVENTS}
SIEM_STRESS_TIMEOUT: ${SIEM_STRESS_TIMEOUT}
SIEM_STRESS_CONFIRM: ${SIEM_STRESS_CONFIRM}
CLICKHOUSE_DB: siem
CLICKHOUSE_USER: siem
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1"
volumes:
- clickhouse_data:/var/lib/clickhouse
- clickhouse_logs:/var/log/clickhouse-server
- ./deploy/clickhouse/init.sql:/docker-entrypoint-initdb.d/001-init.sql:ro
- ./deploy/clickhouse/prometheus.xml:/etc/clickhouse-server/config.d/prometheus.xml:ro
ulimits:
nofile:
soft: 262144
hard: 262144
healthcheck:
test: ["CMD-SHELL", "clickhouse-client --user siem --password $$CLICKHOUSE_PASSWORD --query 'SELECT 1' >/dev/null"]
interval: 5s
timeout: 5s
retries: 30
restart: unless-stopped
ports:
- "127.0.0.1:${CLICKHOUSE_HTTP_PORT:-8123}:8123"
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v26.1.13
command:
- redpanda
- start
- --kafka-addr
- internal://0.0.0.0:9092,external://0.0.0.0:19092
- --advertise-kafka-addr
- internal://redpanda:9092,external://localhost:19092
- --pandaproxy-addr
- internal://0.0.0.0:8082,external://0.0.0.0:18082
- --advertise-pandaproxy-addr
- internal://redpanda:8082,external://localhost:18082
- --schema-registry-addr
- internal://0.0.0.0:8081,external://0.0.0.0:18081
- --rpc-addr
- redpanda:33145
- --advertise-rpc-addr
- redpanda:33145
- --mode
- dev-container
- --smp
- "2"
- --default-log-level=info
volumes:
- redpanda_data:/var/lib/redpanda/data
healthcheck:
test: ["CMD-SHELL", "RPK_BROKERS=localhost:9092 rpk cluster health | grep -q 'Healthy:.*true'"]
interval: 5s
timeout: 5s
retries: 40
restart: unless-stopped
ports:
- "127.0.0.1:${REDPANDA_KAFKA_PORT:-19092}:19092"
- "127.0.0.1:${REDPANDA_ADMIN_PORT:-19644}:9644"
redpanda-init:
image: docker.redpanda.com/redpandadata/redpanda:v26.1.13
depends_on:
siem-backend:
redpanda:
condition: service_healthy
networks:
- dockge_default
entrypoint: ["/bin/sh", "-ec"]
environment:
RPK_BROKERS: redpanda:9092
command: >-
rpk topic create --if-not-exists -p ${KAFKA_PARTITIONS:-6} -r 1
-c retention.ms=${KAFKA_RETENTION_MS:-86400000}
siem-events
restart: "no"
redpanda-console:
image: docker.redpanda.com/redpandadata/console:v3.8.0
depends_on:
redpanda:
condition: service_healthy
entrypoint: /bin/sh
command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
environment:
CONFIG_FILEPATH: /tmp/config.yml
CONSOLE_CONFIG_FILE: |
kafka:
brokers: ["redpanda:9092"]
schemaRegistry:
enabled: true
urls: ["http://redpanda:8081"]
redpanda:
adminApi:
enabled: true
urls: ["http://redpanda:9644"]
restart: unless-stopped
ports:
- "127.0.0.1:${REDPANDA_CONSOLE_PORT:-8081}:8080"
garage:
image: dxflrs/garage:v2.3.0
command: ["/garage", "server", "--single-node", "--default-bucket"]
environment:
GARAGE_DEFAULT_ACCESS_KEY: ${GARAGE_ACCESS_KEY}
GARAGE_DEFAULT_SECRET_KEY: ${GARAGE_SECRET_KEY}
GARAGE_DEFAULT_BUCKET: ${GARAGE_BUCKET:-siem-raw}
volumes:
- garage_meta:/var/lib/garage/meta
- garage_data:/var/lib/garage/data
- ./deploy/garage/garage.toml:/etc/garage.toml:ro
restart: unless-stopped
ports:
- "127.0.0.1:${GARAGE_S3_PORT:-3900}:3900"
- "127.0.0.1:${GARAGE_ADMIN_PORT:-3903}:3903"
ingress:
image: greenfield-siem-app:local
build: .
command: ["ingress"]
environment: &app-env
SERVICE_ADDR: :8080
POSTGRES_URL: postgres://siem:${POSTGRES_PASSWORD}@postgres:5432/siem?sslmode=disable
CLICKHOUSE_URL: http://clickhouse:8123
CLICKHOUSE_USER: siem
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
KAFKA_BROKERS: redpanda:9092
KAFKA_TOPIC: siem-events
TENANT_ID: ${TENANT_ID:-default}
UI_USERNAME: ${UI_USERNAME:-admin}
UI_PASSWORD: ${UI_PASSWORD}
ENROLLMENT_KEY: ${ENROLLMENT_KEY}
RAW_ARCHIVE_ENABLED: ${RAW_ARCHIVE_ENABLED:-true}
RAW_SPOOL_DIR: /var/spool/siem-raw
RAW_RETENTION: ${RAW_RETENTION:-720h}
depends_on:
postgres:
condition: service_healthy
redpanda-init:
condition: service_completed_successfully
restart: unless-stopped
ports:
- "${INGRESS_BIND:-0.0.0.0}:${INGRESS_PORT:-8090}:8080"
processor:
image: greenfield-siem-app:local
command: ["processor"]
user: "0:0"
environment:
<<: *app-env
KAFKA_GROUP: siem-processor-v1
volumes:
- raw_spool:/var/spool/siem-raw
depends_on:
clickhouse:
condition: service_healthy
redpanda-init:
condition: service_completed_successfully
restart: unless-stopped
detector:
image: greenfield-siem-app:local
command: ["detector"]
environment:
<<: *app-env
DETECTOR_INTERVAL: ${DETECTOR_INTERVAL:-30s}
DETECTOR_LOOKBACK: ${DETECTOR_LOOKBACK:-20m}
depends_on:
clickhouse:
condition: service_healthy
postgres:
condition: service_healthy
processor:
condition: service_started
restart: unless-stopped
api:
image: greenfield-siem-app:local
command: ["api"]
environment:
<<: *app-env
UI_QUERY_LIMIT: ${UI_QUERY_LIMIT:-500}
depends_on:
clickhouse:
condition: service_healthy
postgres:
condition: service_healthy
restart: unless-stopped
ports:
- "${UI_BIND:-0.0.0.0}:${UI_PORT:-8080}:8080"
archive-uploader:
image: rclone/rclone:1.74.4
entrypoint: ["/bin/sh", "-ec"]
command: |
while true; do
rclone move /spool garage:${GARAGE_BUCKET:-siem-raw} --exclude '*.tmp' --create-empty-src-dirs --transfers 8 --checkers 8 --retries 5 || true
rclone delete garage:${GARAGE_BUCKET:-siem-raw} --min-age ${RAW_RETENTION:-720h} --retries 5 || true
sleep 30
done
environment:
RCLONE_CONFIG_GARAGE_TYPE: s3
RCLONE_CONFIG_GARAGE_PROVIDER: Other
RCLONE_CONFIG_GARAGE_ACCESS_KEY_ID: ${GARAGE_ACCESS_KEY}
RCLONE_CONFIG_GARAGE_SECRET_ACCESS_KEY: ${GARAGE_SECRET_KEY}
RCLONE_CONFIG_GARAGE_ENDPOINT: http://garage:3900
RCLONE_CONFIG_GARAGE_REGION: garage
RCLONE_CONFIG_GARAGE_FORCE_PATH_STYLE: "true"
volumes:
- raw_spool:/spool
depends_on:
- garage
- processor
restart: unless-stopped
prometheus:
image: prom/prometheus:latest
container_name: siem-prometheus
restart: unless-stopped
env_file:
- .env
environment:
TZ: ${TZ}
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=30d
- --web.enable-lifecycle
depends_on:
siem-backend:
condition: service_healthy
image: prom/prometheus:v3.13.1
command: ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.retention.time=30d"]
volumes:
- ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./deploy/prometheus/rules:/etc/prometheus/rules:ro
- prometheus_data:/prometheus
ports:
- 9090:9090
networks:
- dockge_default
grafana:
image: grafana/grafana:latest
container_name: siem-grafana
restart: unless-stopped
env_file:
- .env
environment:
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
GF_USERS_ALLOW_SIGN_UP: "false"
GF_SERVER_ROOT_URL: http://localhost:3000
TZ: ${TZ}
depends_on:
- prometheus
volumes:
- grafana_data:/var/lib/grafana
- ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
- ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro
- ingress
- redpanda
restart: unless-stopped
ports:
- 3090:3000
networks:
- dockge_default
- "127.0.0.1:${PROMETHEUS_PORT:-9090}:9090"
volumes:
mariadb_data: null
prometheus_data: null
grafana_data: null
networks:
dockge_default:
external: true
postgres_data:
clickhouse_data:
clickhouse_logs:
redpanda_data:
garage_meta:
garage_data:
raw_spool:
prometheus_data:

70
deploy.ps1 Normal file
View File

@@ -0,0 +1,70 @@
$ErrorActionPreference = "Stop"
Set-Location $PSScriptRoot
if (-not (Get-Command docker -ErrorAction SilentlyContinue)) { throw "Docker fehlt." }
docker compose version | Out-Null
function Hex([int]$bytes) { $b = New-Object byte[] $bytes; [Security.Cryptography.RandomNumberGenerator]::Fill($b); return ([BitConverter]::ToString($b)).Replace("-","").ToLower() }
if (-not (Test-Path .env)) {
$presetEnrollment = $env:ENROLLMENT_KEY
$enrollment = if ([string]::IsNullOrWhiteSpace($presetEnrollment)) { Hex 32 } else { $presetEnrollment }
$vals = @{
POSTGRES_PASSWORD=Hex 24; CLICKHOUSE_PASSWORD=Hex 24; ENROLLMENT_KEY=$enrollment; UI_PASSWORD=Hex 18;
GARAGE_ACCESS_KEY="GK$(Hex 16)"; GARAGE_SECRET_KEY=Hex 32; GARAGE_RPC_SECRET=Hex 32;
GARAGE_ADMIN_TOKEN=Hex 32; GARAGE_METRICS_TOKEN=Hex 32
}
@"
INGRESS_BIND=0.0.0.0
INGRESS_PORT=8090
UI_BIND=0.0.0.0
UI_PORT=8080
TENANT_ID=default
UI_USERNAME=admin
UI_PASSWORD=$($vals.UI_PASSWORD)
POSTGRES_PASSWORD=$($vals.POSTGRES_PASSWORD)
CLICKHOUSE_PASSWORD=$($vals.CLICKHOUSE_PASSWORD)
ENROLLMENT_KEY=$($vals.ENROLLMENT_KEY)
GARAGE_ACCESS_KEY=$($vals.GARAGE_ACCESS_KEY)
GARAGE_SECRET_KEY=$($vals.GARAGE_SECRET_KEY)
GARAGE_RPC_SECRET=$($vals.GARAGE_RPC_SECRET)
GARAGE_ADMIN_TOKEN=$($vals.GARAGE_ADMIN_TOKEN)
GARAGE_METRICS_TOKEN=$($vals.GARAGE_METRICS_TOKEN)
GARAGE_BUCKET=siem-raw
EVENT_RETENTION_DAYS=90
ROLLUP_RETENTION_DAYS=730
RAW_RETENTION=720h
RAW_ARCHIVE_ENABLED=true
KAFKA_RETENTION_MS=86400000
KAFKA_PARTITIONS=6
DETECTOR_INTERVAL=30s
DETECTOR_LOOKBACK=20m
UI_QUERY_LIMIT=500
"@ | Set-Content .env
}
$envMap=@{}; Get-Content .env | ForEach-Object { if ($_ -match '^([^#=]+)=(.*)$') {$envMap[$matches[1]]=$matches[2]} }
if (-not $envMap.ContainsKey('UI_USERNAME')) { $envMap['UI_USERNAME']='admin'; Add-Content .env "UI_USERNAME=admin" }
if (-not $envMap.ContainsKey('UI_PASSWORD') -or [string]::IsNullOrWhiteSpace($envMap['UI_PASSWORD'])) { $envMap['UI_PASSWORD']=Hex 18; Add-Content .env "UI_PASSWORD=$($envMap.UI_PASSWORD)" }
(Get-Content deploy/garage/garage.toml.template -Raw).Replace('__GARAGE_RPC_SECRET__',$envMap.GARAGE_RPC_SECRET).Replace('__GARAGE_ADMIN_TOKEN__',$envMap.GARAGE_ADMIN_TOKEN).Replace('__GARAGE_METRICS_TOKEN__',$envMap.GARAGE_METRICS_TOKEN) | Set-Content deploy/garage/garage.toml
(Get-Content deploy/clickhouse/init.sql.template -Raw).Replace('__EVENT_RETENTION_DAYS__',$envMap.EVENT_RETENTION_DAYS).Replace('__ROLLUP_RETENTION_DAYS__',$envMap.ROLLUP_RETENTION_DAYS) | Set-Content deploy/clickhouse/init.sql
Write-Host "Baue und starte Greenfield SIEM ..."
docker compose build --pull
docker compose up -d --remove-orphans
Write-Host "Pruefe Dienste ..."
$healthy = $false
for ($i = 0; $i -lt 60; $i++) {
try {
Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 "http://127.0.0.1:$($envMap.UI_PORT)/readyz" | Out-Null
Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 "http://127.0.0.1:$($envMap.INGRESS_PORT)/readyz" | Out-Null
$healthy = $true
break
} catch {
Start-Sleep -Seconds 2
}
}
if (-not $healthy) { throw "Deployment gestartet, Healthcheck noch nicht gruen. Pruefe: docker compose ps; docker compose logs" }
Write-Host ""
Write-Host "Greenfield SIEM laeuft."
Write-Host "UI: http://127.0.0.1:$($envMap.UI_PORT)/ui"
Write-Host "Ingress: http://127.0.0.1:$($envMap.INGRESS_PORT)/ingest"
Write-Host "UI-Login: $($envMap.UI_USERNAME) / $($envMap.UI_PASSWORD)"
Write-Host "Enrollment-Key: $($envMap.ENROLLMENT_KEY)"
Write-Host "Secrets stehen in .env (Datei schuetzen und sichern)."

88
deploy.sh Executable file
View File

@@ -0,0 +1,88 @@
#!/usr/bin/env sh
set -eu
cd "$(dirname "$0")"
command -v docker >/dev/null 2>&1 || { echo "Docker fehlt." >&2; exit 1; }
docker compose version >/dev/null 2>&1 || { echo "Docker Compose Plugin fehlt." >&2; exit 1; }
PRESET_ENROLLMENT_KEY="${ENROLLMENT_KEY:-}"
randhex(){ n="$1"; if command -v openssl >/dev/null 2>&1; then openssl rand -hex "$n"; else dd if=/dev/urandom bs="$n" count=1 2>/dev/null | od -An -tx1 | tr -d ' \n'; fi; }
http_ok(){
url="$1"
if command -v curl >/dev/null 2>&1; then curl -fsS "$url" >/dev/null 2>&1; return $?; fi
if command -v wget >/dev/null 2>&1; then wget -q -O /dev/null "$url" >/dev/null 2>&1; return $?; fi
return 2
}
if [ ! -f .env ]; then
POSTGRES_PASSWORD="$(randhex 24)"
CLICKHOUSE_PASSWORD="$(randhex 24)"
ENROLLMENT_KEY="${PRESET_ENROLLMENT_KEY:-$(randhex 32)}"
UI_PASSWORD="$(randhex 18)"
GARAGE_ACCESS_KEY="GK$(randhex 16)"
GARAGE_SECRET_KEY="$(randhex 32)"
GARAGE_RPC_SECRET="$(randhex 32)"
GARAGE_ADMIN_TOKEN="$(randhex 32)"
GARAGE_METRICS_TOKEN="$(randhex 32)"
cat > .env <<EOT
INGRESS_BIND=0.0.0.0
INGRESS_PORT=8090
UI_BIND=0.0.0.0
UI_PORT=8080
TENANT_ID=default
UI_USERNAME=admin
UI_PASSWORD=$UI_PASSWORD
POSTGRES_PASSWORD=$POSTGRES_PASSWORD
CLICKHOUSE_PASSWORD=$CLICKHOUSE_PASSWORD
ENROLLMENT_KEY=$ENROLLMENT_KEY
GARAGE_ACCESS_KEY=$GARAGE_ACCESS_KEY
GARAGE_SECRET_KEY=$GARAGE_SECRET_KEY
GARAGE_RPC_SECRET=$GARAGE_RPC_SECRET
GARAGE_ADMIN_TOKEN=$GARAGE_ADMIN_TOKEN
GARAGE_METRICS_TOKEN=$GARAGE_METRICS_TOKEN
GARAGE_BUCKET=siem-raw
EVENT_RETENTION_DAYS=90
ROLLUP_RETENTION_DAYS=730
RAW_RETENTION=720h
RAW_ARCHIVE_ENABLED=true
KAFKA_RETENTION_MS=86400000
KAFKA_PARTITIONS=6
DETECTOR_INTERVAL=30s
DETECTOR_LOOKBACK=20m
UI_QUERY_LIMIT=500
EOT
chmod 600 .env 2>/dev/null || true
fi
set -a; . ./.env; set +a
UI_USERNAME="${UI_USERNAME:-admin}"
if [ -z "${UI_PASSWORD:-}" ]; then
UI_PASSWORD="$(randhex 18)"
printf '\nUI_USERNAME=%s\nUI_PASSWORD=%s\n' "$UI_USERNAME" "$UI_PASSWORD" >> .env
export UI_USERNAME UI_PASSWORD
fi
sed -e "s/__GARAGE_RPC_SECRET__/$GARAGE_RPC_SECRET/g" -e "s/__GARAGE_ADMIN_TOKEN__/$GARAGE_ADMIN_TOKEN/g" -e "s/__GARAGE_METRICS_TOKEN__/$GARAGE_METRICS_TOKEN/g" deploy/garage/garage.toml.template > deploy/garage/garage.toml
sed -e "s/__EVENT_RETENTION_DAYS__/${EVENT_RETENTION_DAYS:-90}/g" -e "s/__ROLLUP_RETENTION_DAYS__/${ROLLUP_RETENTION_DAYS:-730}/g" deploy/clickhouse/init.sql.template > deploy/clickhouse/init.sql
chmod 600 deploy/garage/garage.toml 2>/dev/null || true
echo "Baue und starte Greenfield SIEM ..."
docker compose build --pull
docker compose up -d --remove-orphans
echo "Prüfe Dienste ..."
if command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1; then
i=0
while [ "$i" -lt 60 ]; do
if http_ok "http://127.0.0.1:${UI_PORT:-8080}/readyz" && http_ok "http://127.0.0.1:${INGRESS_PORT:-8090}/readyz"; then break; fi
i=$((i+1)); sleep 2
done
if [ "$i" -ge 60 ]; then echo "Deployment gestartet, Healthcheck noch nicht grün. Prüfe: docker compose ps && docker compose logs" >&2; exit 1; fi
else
echo "Hinweis: curl/wget fehlt; HTTP-Healthcheck wird übersprungen."
docker compose ps
fi
echo
echo "Greenfield SIEM läuft."
echo "UI: http://127.0.0.1:${UI_PORT:-8080}/ui"
echo "Ingress: http://127.0.0.1:${INGRESS_PORT:-8090}/ingest"
echo "UI-Login: $UI_USERNAME / $UI_PASSWORD"
echo "Enrollment-Key: $ENROLLMENT_KEY"
echo "Secrets stehen in .env (Datei schützen und sichern)."

82
deploy/clickhouse/init.sql Normal file
View File

@@ -0,0 +1,82 @@
CREATE DATABASE IF NOT EXISTS siem;
CREATE TABLE IF NOT EXISTS siem.events
(
event_uid String,
queue_partition Int32,
queue_offset Int64,
tenant_id LowCardinality(String),
event_time DateTime64(3, 'UTC'),
ingest_time DateTime64(3, 'UTC'),
event_date Date MATERIALIZED toDate(event_time),
agent_id String,
host_name LowCardinality(String),
source_type LowCardinality(String),
channel LowCardinality(String),
provider LowCardinality(String),
event_code UInt32,
category LowCardinality(String),
action LowCardinality(String),
outcome LowCardinality(String),
severity UInt8,
user_name String,
user_domain LowCardinality(String),
subject_user String,
subject_domain LowCardinality(String),
target_user String,
target_domain LowCardinality(String),
source_ip String,
source_port UInt16,
destination_ip String,
destination_port UInt16,
workstation String,
logon_type LowCardinality(String),
authentication_package LowCardinality(String),
logon_process LowCardinality(String),
status_code LowCardinality(String),
sub_status_code LowCardinality(String),
failure_reason LowCardinality(String),
process_path String,
parent_process_path String,
command_line String CODEC(ZSTD(3)),
message String CODEC(ZSTD(3)),
attributes Map(String, String) CODEC(ZSTD(3)),
raw_object_key String,
raw_index UInt32,
payload_hash FixedString(64),
schema_version UInt16,
parser_version UInt16,
ingest_delay_ms Int64,
INDEX idx_host host_name TYPE set(10000) GRANULARITY 4,
INDEX idx_event_code event_code TYPE set(256) GRANULARITY 4,
INDEX idx_user user_name TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_target target_user TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_source_ip source_ip TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_process process_path TYPE bloom_filter(0.01) GRANULARITY 8
)
ENGINE = ReplacingMergeTree(ingest_time)
PARTITION BY toYYYYMM(event_time)
ORDER BY (tenant_id, event_time, host_name, event_code, event_uid)
TTL event_time + INTERVAL 90 DAY DELETE
SETTINGS index_granularity = 8192;
CREATE TABLE IF NOT EXISTS siem.events_5m
(
tenant_id LowCardinality(String),
bucket DateTime('UTC'),
host_name LowCardinality(String),
event_code UInt32,
category LowCardinality(String),
action LowCardinality(String),
outcome LowCardinality(String),
cnt_state AggregateFunction(uniqExact, String)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(bucket)
ORDER BY (tenant_id, bucket, host_name, event_code, category, action, outcome)
TTL bucket + INTERVAL 730 DAY DELETE;
CREATE MATERIALIZED VIEW IF NOT EXISTS siem.events_5m_mv TO siem.events_5m AS
SELECT tenant_id, toStartOfInterval(event_time, INTERVAL 5 MINUTE) AS bucket, host_name, event_code, category, action, outcome, uniqExactState(event_uid) AS cnt_state
FROM siem.events
GROUP BY tenant_id, bucket, host_name, event_code, category, action, outcome;

82
deploy/clickhouse/init.sql.template Normal file
View File

@@ -0,0 +1,82 @@
CREATE DATABASE IF NOT EXISTS siem;
CREATE TABLE IF NOT EXISTS siem.events
(
event_uid String,
queue_partition Int32,
queue_offset Int64,
tenant_id LowCardinality(String),
event_time DateTime64(3, 'UTC'),
ingest_time DateTime64(3, 'UTC'),
event_date Date MATERIALIZED toDate(event_time),
agent_id String,
host_name LowCardinality(String),
source_type LowCardinality(String),
channel LowCardinality(String),
provider LowCardinality(String),
event_code UInt32,
category LowCardinality(String),
action LowCardinality(String),
outcome LowCardinality(String),
severity UInt8,
user_name String,
user_domain LowCardinality(String),
subject_user String,
subject_domain LowCardinality(String),
target_user String,
target_domain LowCardinality(String),
source_ip String,
source_port UInt16,
destination_ip String,
destination_port UInt16,
workstation String,
logon_type LowCardinality(String),
authentication_package LowCardinality(String),
logon_process LowCardinality(String),
status_code LowCardinality(String),
sub_status_code LowCardinality(String),
failure_reason LowCardinality(String),
process_path String,
parent_process_path String,
command_line String CODEC(ZSTD(3)),
message String CODEC(ZSTD(3)),
attributes Map(String, String) CODEC(ZSTD(3)),
raw_object_key String,
raw_index UInt32,
payload_hash FixedString(64),
schema_version UInt16,
parser_version UInt16,
ingest_delay_ms Int64,
INDEX idx_host host_name TYPE set(10000) GRANULARITY 4,
INDEX idx_event_code event_code TYPE set(256) GRANULARITY 4,
INDEX idx_user user_name TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_target target_user TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_source_ip source_ip TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_process process_path TYPE bloom_filter(0.01) GRANULARITY 8
)
ENGINE = ReplacingMergeTree(ingest_time)
PARTITION BY toYYYYMM(event_time)
ORDER BY (tenant_id, event_time, host_name, event_code, event_uid)
TTL event_time + INTERVAL __EVENT_RETENTION_DAYS__ DAY DELETE
SETTINGS index_granularity = 8192;
CREATE TABLE IF NOT EXISTS siem.events_5m
(
tenant_id LowCardinality(String),
bucket DateTime('UTC'),
host_name LowCardinality(String),
event_code UInt32,
category LowCardinality(String),
action LowCardinality(String),
outcome LowCardinality(String),
cnt_state AggregateFunction(uniqExact, String)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(bucket)
ORDER BY (tenant_id, bucket, host_name, event_code, category, action, outcome)
TTL bucket + INTERVAL __ROLLUP_RETENTION_DAYS__ DAY DELETE;
CREATE MATERIALIZED VIEW IF NOT EXISTS siem.events_5m_mv TO siem.events_5m AS
SELECT tenant_id, toStartOfInterval(event_time, INTERVAL 5 MINUTE) AS bucket, host_name, event_code, category, action, outcome, uniqExactState(event_uid) AS cnt_state
FROM siem.events
GROUP BY tenant_id, bucket, host_name, event_code, category, action, outcome;

10
deploy/clickhouse/prometheus.xml Normal file
View File

@@ -0,0 +1,10 @@
<clickhouse>
<prometheus>
<endpoint>/metrics</endpoint>
<port>9363</port>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>true</asynchronous_metrics>
<status_info>true</status_info>
</prometheus>
</clickhouse>

17
deploy/garage/garage.toml.template Normal file
View File

@@ -0,0 +1,17 @@
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "garage:3901"
rpc_secret = "__GARAGE_RPC_SECRET__"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.garage.localhost"
[admin]
api_bind_addr = "[::]:3903"
admin_token = "__GARAGE_ADMIN_TOKEN__"
metrics_token = "__GARAGE_METRICS_TOKEN__"

12
deploy/grafana/provisioning/dashboards/dashboards.yml
View File

@@ -1,12 +0,0 @@
apiVersion: 1
providers:
- name: SIEM Dashboards
orgId: 1
folder: SIEM
type: file
disableDeletion: false
editable: true
updateIntervalSeconds: 30
options:
path: /var/lib/grafana/dashboards

745
deploy/grafana/provisioning/dashboards/siem-overview.json
View File

@@ -1,745 +0,0 @@
{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"links": [],
"liveNow": false,
"panels": [
{
"type": "stat",
"title": "Active Agents",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
"targets": [
{
"expr": "eventcollector_active_agents",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "green", "value": 1 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"orientation": "auto",
"textMode": "auto",
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto"
}
},
{
"type": "stat",
"title": "Events/s",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
"targets": [
{
"expr": "sum(rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m]))",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "eps",
"decimals": 2
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "auto"
}
},
{
"type": "stat",
"title": "High Detections 5m",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 8, "y": 0 },
"targets": [
{
"expr": "sum(increase(eventcollector_detection_hits_total{severity=\"high\",rule=~\"$rule\"}[5m]))",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "red", "value": 1 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "auto"
}
},
{
"type": "stat",
"title": "Baseline Max Z-Score",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 12, "y": 0 },
"targets": [
{
"expr": "max(eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"decimals": 2,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 3 },
{ "color": "red", "value": 5 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "auto"
}
},
{
"type": "stat",
"title": "Rule Errors 5m",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 16, "y": 0 },
"targets": [
{
"expr": "sum(increase(eventcollector_rule_errors_total{rule=~\"$rule\"}[5m]))",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "red", "value": 1 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "auto"
}
},
{
"type": "stat",
"title": "DB Insert Failures 5m",
"datasource": "$datasource",
"gridPos": { "h": 4, "w": 4, "x": 20, "y": 0 },
"targets": [
{
"expr": "increase(eventcollector_db_insert_failures_total[5m])",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "red", "value": 1 }
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"textMode": "auto"
}
},
{
"type": "timeseries",
"title": "Ingested Events / Second by Channel",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"targets": [
{
"expr": "sum by (channel) (rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m]))",
"legendFormat": "{{channel}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "eps",
"decimals": 2
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "Detection Hits by Rule / Severity",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"targets": [
{
"expr": "sum by (rule,severity) (increase(eventcollector_detection_hits_total{rule=~\"$rule\",severity=~\"$severity\"}[5m]))",
"legendFormat": "{{rule}} / {{severity}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "Baseline: Current Count vs Average",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"targets": [
{
"expr": "eventcollector_baseline_current_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}",
"legendFormat": "current {{host}} {{channel}} {{event_id}}",
"refId": "A"
},
{
"expr": "eventcollector_baseline_avg_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}",
"legendFormat": "avg {{host}} {{channel}} {{event_id}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 2
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "Baseline Z-Score",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
"targets": [
{
"expr": "eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"}",
"legendFormat": "{{host}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"decimals": 2,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 3 },
{ "color": "red", "value": 5 }
]
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "bargauge",
"title": "Top Baseline Z-Scores",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 20 },
"targets": [
{
"expr": "topk(10, eventcollector_anomaly_score{host=~\"$host\",rule=\"baseline_event_rate_anomaly\"})",
"legendFormat": "{{host}}",
"refId": "A",
"instant": true
}
],
"fieldConfig": {
"defaults": {
"decimals": 2,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 3 },
{ "color": "red", "value": 5 }
]
}
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showUnfilled": true
}
},
{
"type": "bargauge",
"title": "Top EventIDs by Ingest Rate",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 20 },
"targets": [
{
"expr": "topk(15, sum by (channel,event_id) (rate(eventcollector_ingest_events_total{channel=~\"$channel\",event_id=~\"$event_id\"}[5m])))",
"legendFormat": "{{channel}} / {{event_id}}",
"refId": "A",
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "eps",
"decimals": 2
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showUnfilled": true
}
},
{
"type": "bargauge",
"title": "Top Detection Rules 1h",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 20 },
"targets": [
{
"expr": "topk(15, sum by (rule,severity) (increase(eventcollector_detection_hits_total{rule=~\"$rule\",severity=~\"$severity\"}[1h])))",
"legendFormat": "{{rule}} / {{severity}}",
"refId": "A",
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showUnfilled": true
}
},
{
"type": "timeseries",
"title": "HTTP Requests by Path / Status",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
"targets": [
{
"expr": "sum by (path,status) (rate(eventcollector_http_requests_total[5m]))",
"legendFormat": "{{path}} {{status}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"decimals": 2
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "HTTP Latency p95",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
"targets": [
{
"expr": "histogram_quantile(0.95, sum by (le,path) (rate(eventcollector_http_request_duration_seconds_bucket[5m])))",
"legendFormat": "{{path}} p95",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 3
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "DB Insert Transaction Latency p95",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
"targets": [
{
"expr": "histogram_quantile(0.95, sum by (le) (rate(eventcollector_db_tx_duration_seconds_bucket[5m])))",
"legendFormat": "db tx p95",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 3
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "timeseries",
"title": "DB Batch Size p95",
"datasource": "$datasource",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
"targets": [
{
"expr": "histogram_quantile(0.95, sum by (le) (rate(eventcollector_db_batch_size_bucket[5m])))",
"legendFormat": "batch size p95",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"type": "table",
"title": "Agent Last Seen",
"datasource": "$datasource",
"gridPos": { "h": 10, "w": 12, "x": 0, "y": 44 },
"targets": [
{
"expr": "time() - eventcollector_agent_last_seen_unixtime{host=~\"$host\"}",
"legendFormat": "{{host}}",
"refId": "A",
"instant": true,
"format": "table"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 0
},
"overrides": []
},
"options": {
"showHeader": true
}
},
{
"type": "table",
"title": "Baseline Samples",
"datasource": "$datasource",
"gridPos": { "h": 10, "w": 12, "x": 12, "y": 44 },
"targets": [
{
"expr": "eventcollector_baseline_sample_count{host=~\"$host\",channel=~\"$channel\",event_id=~\"$event_id\"}",
"legendFormat": "{{host}} {{channel}} {{event_id}}",
"refId": "A",
"instant": true,
"format": "table"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0
},
"overrides": []
},
"options": {
"showHeader": true
}
}
],
"refresh": "30s",
"schemaVersion": 39,
"style": "dark",
"tags": ["siem", "baseline", "ad"],
"templating": {
"list": [
{
"name": "datasource",
"type": "datasource",
"query": "prometheus",
"current": {},
"hide": 0,
"label": "Datasource"
},
{
"name": "host",
"type": "query",
"datasource": "$datasource",
"query": "label_values(eventcollector_agent_last_seen_unixtime, host)",
"refresh": 1,
"includeAll": true,
"multi": true,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
},
"label": "Host"
},
{
"name": "channel",
"type": "query",
"datasource": "$datasource",
"query": "label_values(eventcollector_ingest_events_total, channel)",
"refresh": 1,
"includeAll": true,
"multi": true,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
},
"label": "Channel"
},
{
"name": "event_id",
"type": "query",
"datasource": "$datasource",
"query": "label_values(eventcollector_ingest_events_total, event_id)",
"refresh": 1,
"includeAll": true,
"multi": true,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
},
"label": "Event ID"
},
{
"name": "rule",
"type": "query",
"datasource": "$datasource",
"query": "label_values(eventcollector_detection_hits_total, rule)",
"refresh": 1,
"includeAll": true,
"multi": true,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
},
"label": "Rule"
},
{
"name": "severity",
"type": "custom",
"query": "low,medium,high",
"includeAll": true,
"multi": true,
"allValue": ".*",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
},
"label": "Severity"
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timezone": "browser",
"title": "SIEM Overview Extended",
"uid": "siem-overview-extended",
"version": 1
}

1006
deploy/grafana/provisioning/dashboards/siem-soc-privileged.json
View File

File diff suppressed because it is too large Load Diff

10
deploy/grafana/provisioning/datasources/datasource.yml
View File

@@ -1,10 +0,0 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: true

35
deploy/mariadb/cardinality-diagnostics.sql
View File

@@ -1,35 +0,0 @@
-- Shows where rows are growing and how effective aggregation currently is.
SET time_zone = '+00:00';
SELECT table_name,
table_rows,
ROUND(data_length/1024/1024,1) AS data_mb,
ROUND(index_length/1024/1024,1) AS index_mb,
ROUND((data_length+index_length)/1024/1024,1) AS total_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY data_length + index_length DESC;
SELECT 'event_count_buckets' AS source,
COUNT(*) AS rows_24h,
COALESCE(SUM(cnt),0) AS represented_events_24h,
ROUND(COALESCE(SUM(cnt),0) / NULLIF(COUNT(*),0), 1) AS events_per_row
FROM event_count_buckets
WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR
UNION ALL
SELECT 'event_occurrences',
COUNT(*),
COALESCE(SUM(cnt),0),
ROUND(COALESCE(SUM(cnt),0) / NULLIF(COUNT(*),0), 1)
FROM event_occurrences
WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR;
SELECT channel_name, event_id,
COUNT(*) AS context_rows_24h,
SUM(cnt) AS represented_events_24h,
ROUND(SUM(cnt)/NULLIF(COUNT(*),0),1) AS events_per_row
FROM event_occurrences
WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR
GROUP BY channel_name, event_id
ORDER BY context_rows_24h DESC
LIMIT 50;

52
deploy/mariadb/diagnostics.sql
View File

@@ -1,52 +0,0 @@
-- Read-only diagnostics for sizing and partition verification.
SET time_zone = '+00:00';
SELECT
TABLE_NAME,
ENGINE,
TABLE_ROWS,
ROUND(DATA_LENGTH / 1024 / 1024, 1) AS data_mb,
ROUND(INDEX_LENGTH / 1024 / 1024, 1) AS index_mb,
ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024, 1) AS total_mb
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE()
ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC;
SELECT
TABLE_NAME,
COUNT(*) AS partitions,
MIN(PARTITION_NAME) AS oldest_partition,
MAX(PARTITION_NAME) AS newest_partition,
ROUND(SUM(DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024, 1) AS total_mb
FROM information_schema.PARTITIONS
WHERE TABLE_SCHEMA = DATABASE()
AND PARTITION_NAME IS NOT NULL
GROUP BY TABLE_NAME
ORDER BY total_mb DESC;
SELECT
channel_name,
event_id,
SUM(cnt) AS events_24h,
COUNT(*) AS metadata_rows_24h,
ROUND(SUM(cnt) / NULLIF(COUNT(*), 0), 2) AS compression_factor
FROM event_occurrences
WHERE bucket_start >= UTC_TIMESTAMP() - INTERVAL 24 HOUR
GROUP BY channel_name, event_id
ORDER BY events_24h DESC
LIMIT 50;
SELECT
first_event_ts,
last_event_ts,
cnt,
hostname,
target_user,
workstation,
src_ip
FROM event_occurrences
WHERE channel_name = 'Security'
AND event_id = 4740
AND bucket_start >= UTC_TIMESTAMP() - INTERVAL 7 DAY
ORDER BY bucket_start DESC
LIMIT 200;

43
deploy/mariadb/emergency-compact-cleanup.sql
View File

@@ -1,43 +0,0 @@
-- EMERGENCY CLEANUP FOR THE PREVIOUS METADATA-FIRST BUILD
--
-- Run ONLY AFTER the new backend is deployed with:
-- STORE_EVENT_ROWS=false
-- STORE_RAW_XML=false
--
-- This intentionally removes redundant per-event/full-context history generated
-- by older builds. Aggregate event history remains in event_count_buckets and
-- event_catalog. Existing detections and UEBA baselines are not deleted.
--
-- Take a database backup/snapshot first if old raw/event detail may still matter.
SET NAMES utf8mb4;
SET time_zone = '+00:00';
SELECT 'before' AS phase,
table_name,
table_rows,
ROUND((data_length + index_length) / 1024 / 1024, 1) AS size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
AND table_name IN ('event_logs','event_log_raw','event_occurrences','event_count_buckets','event_catalog','detections')
ORDER BY size_mb DESC;
-- Raw XML must be cleared before its logical parent event rows.
TRUNCATE TABLE event_log_raw;
TRUNCATE TABLE event_logs;
-- Old versions used too many dimensions and could make this table nearly as
-- large as event_logs. The new backend will repopulate only selected Security
-- contexts in 5-minute buckets.
TRUNCATE TABLE event_occurrences;
ANALYZE TABLE event_count_buckets, event_catalog, detections;
SELECT 'after' AS phase,
table_name,
table_rows,
ROUND((data_length + index_length) / 1024 / 1024, 1) AS size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
AND table_name IN ('event_logs','event_log_raw','event_occurrences','event_count_buckets','event_catalog','detections')
ORDER BY size_mb DESC;

2239
deploy/mariadb/init/001-schema.sql
View File

File diff suppressed because it is too large Load Diff

98
deploy/mariadb/migrations/002-metadata-first.sql
View File

@@ -1,98 +0,0 @@
-- Metadata-first migration for an existing SIEM-lite database.
-- IMPORTANT: ALTER TABLE ... PARTITION BY can rebuild and lock large tables.
-- Run during a maintenance window and take a backup first.
SET NAMES utf8mb4;
SET time_zone = '+00:00';
CREATE TABLE IF NOT EXISTS event_catalog (
hostname VARCHAR(191) NOT NULL,
channel_name VARCHAR(128) NOT NULL,
event_id INT UNSIGNED NOT NULL,
first_seen DATETIME(6) NOT NULL,
last_seen DATETIME(6) NOT NULL,
total_count BIGINT UNSIGNED NOT NULL DEFAULT 0,
updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
PRIMARY KEY (hostname, channel_name, event_id),
KEY idx_event_catalog_first_seen (first_seen),
KEY idx_event_catalog_last_seen (last_seen)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- Warm the catalog from the much smaller baseline buckets so the
-- new_event_id rule does not treat every known event type as new after migration.
INSERT INTO event_catalog
(hostname, channel_name, event_id, first_seen, last_seen, total_count, updated_at)
SELECT hostname,
channel_name,
event_id,
MIN(COALESCE(first_event_ts, bucket_start)),
MAX(COALESCE(last_event_ts, bucket_end)),
SUM(cnt),
UTC_TIMESTAMP(6)
FROM event_count_buckets
GROUP BY hostname, channel_name, event_id
ON DUPLICATE KEY UPDATE
first_seen = LEAST(first_seen, VALUES(first_seen)),
last_seen = GREATEST(last_seen, VALUES(last_seen)),
total_count = GREATEST(total_count, VALUES(total_count)),
updated_at = UTC_TIMESTAMP(6);
CREATE TABLE IF NOT EXISTS event_occurrences (
bucket_start DATETIME(6) NOT NULL,
bucket_end DATETIME(6) NOT NULL,
dimension_key BINARY(16) NOT NULL,
hostname VARCHAR(191) NOT NULL,
channel_name VARCHAR(128) NOT NULL,
event_id INT UNSIGNED NOT NULL,
provider_name VARCHAR(191) NOT NULL DEFAULT '',
target_user VARCHAR(191) NOT NULL DEFAULT '',
subject_user VARCHAR(191) NOT NULL DEFAULT '',
src_ip VARCHAR(64) NOT NULL DEFAULT '',
workstation VARCHAR(191) NOT NULL DEFAULT '',
logon_type VARCHAR(32) NOT NULL DEFAULT '',
status_text VARCHAR(128) NOT NULL DEFAULT '',
failure_reason VARCHAR(255) NOT NULL DEFAULT '',
cnt BIGINT UNSIGNED NOT NULL DEFAULT 0,
first_event_ts DATETIME(6) NOT NULL,
last_event_ts DATETIME(6) NOT NULL,
updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
PRIMARY KEY (bucket_start, dimension_key),
KEY idx_occurrences_time_host_event (bucket_start, hostname, channel_name, event_id),
KEY idx_occurrences_host_event_time (hostname, channel_name, event_id, bucket_start),
KEY idx_occurrences_target_user_time (target_user, bucket_start),
KEY idx_occurrences_subject_user_time (subject_user, bucket_start),
KEY idx_occurrences_src_ip_time (src_ip, bucket_start)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE COLUMNS(bucket_start) (
PARTITION pmax VALUES LESS THAN (MAXVALUE)
);
-- Partition bucket tables only when needed, so this migration can be rerun.
DELIMITER $$
CREATE PROCEDURE partition_bucket_table_if_needed(IN p_table VARCHAR(64))
BEGIN
DECLARE v_partition_count INT DEFAULT 0;
SELECT COUNT(*) INTO v_partition_count
FROM information_schema.PARTITIONS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = p_table
AND PARTITION_NAME IS NOT NULL;
IF v_partition_count = 0 THEN
SET @partition_sql = CONCAT(
'ALTER TABLE `', p_table, '` ',
'PARTITION BY RANGE COLUMNS(bucket_start) ',
'(PARTITION pmax VALUES LESS THAN (MAXVALUE))'
);
PREPARE partition_stmt FROM @partition_sql;
EXECUTE partition_stmt;
DEALLOCATE PREPARE partition_stmt;
END IF;
END$$
DELIMITER ;
CALL partition_bucket_table_if_needed('event_occurrences');
CALL partition_bucket_table_if_needed('event_count_buckets');
CALL partition_bucket_table_if_needed('ueba_context_buckets');
DROP PROCEDURE partition_bucket_table_if_needed;

27
deploy/mariadb/migrations/003-new-event-id-classification.sql
View File

@@ -1,27 +0,0 @@
-- Metadata-first SIEM: realistischere Einordnung der Regel new_event_id
-- Diese Migration verändert kein Schema. Sie bereinigt nur bereits offene
-- WMI-Operational-Inventarmeldungen, die von der alten Regel als Medium-Alarm
-- angelegt wurden. Vor produktiver Ausführung wie üblich ein Backup erstellen.
START TRANSACTION;
UPDATE detections
SET severity = 'info',
status = 'legitimate',
is_legitimate = 1,
reviewed_by = 'system:migration-003',
reviewed_at = UTC_TIMESTAMP(6),
analyst_note = CASE
WHEN analyst_note IS NULL OR TRIM(analyst_note) = '' THEN
'Automatisch neu klassifiziert: neue Event-IDs im WMI-Activity/Operational-Channel sind Inventar-/Diagnoseinformationen und allein kein Incident.'
ELSE CONCAT(
analyst_note,
'\nAutomatisch neu klassifiziert: neue Event-IDs im WMI-Activity/Operational-Channel sind Inventar-/Diagnoseinformationen und allein kein Incident.'
)
END
WHERE rule_name = 'new_event_id'
AND channel_name = 'Microsoft-Windows-WMI-Activity/Operational'
AND status IN ('open', 'acknowledged', 'investigating', 'plausible')
AND is_legitimate = 0;
COMMIT;

21
deploy/mariadb/migrations/004-compact-storage.sql
View File

@@ -1,21 +0,0 @@
-- Compact-storage migration.
-- Safe to apply without deleting event history. The destructive cleanup of old
-- per-event rows is intentionally kept in ../emergency-compact-cleanup.sql.
SET NAMES utf8mb4;
SET time_zone = '+00:00';
CREATE INDEX IF NOT EXISTS idx_event_count_buckets_event_time
ON event_count_buckets (channel_name, event_id, bucket_start, hostname);
CREATE INDEX IF NOT EXISTS idx_occurrences_security_event_time
ON event_occurrences (
channel_name,
event_id,
bucket_start,
hostname,
target_user,
src_ip
);
ANALYZE TABLE event_count_buckets, event_occurrences, event_catalog, detections;

38
deploy/postgres/init.sql Normal file
View File

@@ -0,0 +1,38 @@
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE TABLE IF NOT EXISTS agents (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id text NOT NULL,
hostname text NOT NULL,
api_key_hash char(64) NOT NULL,
enabled boolean NOT NULL DEFAULT true,
last_ip text NOT NULL DEFAULT '',
first_seen timestamptz NOT NULL DEFAULT now(),
last_seen timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, hostname)
);
CREATE INDEX IF NOT EXISTS agents_last_seen_idx ON agents(tenant_id,last_seen DESC);
CREATE TABLE IF NOT EXISTS detections (
id bigserial PRIMARY KEY,
tenant_id text NOT NULL,
fingerprint char(64) NOT NULL,
rule_name text NOT NULL,
severity text NOT NULL,
status text NOT NULL DEFAULT 'open',
hostname text NOT NULL DEFAULT '',
user_name text NOT NULL DEFAULT '',
source_ip text NOT NULL DEFAULT '',
workstation text NOT NULL DEFAULT '',
event_code integer NOT NULL DEFAULT 0,
score double precision NOT NULL DEFAULT 0,
window_start timestamptz NOT NULL,
window_end timestamptz NOT NULL,
summary text NOT NULL,
hit_count bigint NOT NULL DEFAULT 1,
first_seen timestamptz NOT NULL DEFAULT now(),
last_seen timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, fingerprint)
);
CREATE INDEX IF NOT EXISTS detections_open_idx ON detections(tenant_id,status,last_seen DESC);
CREATE INDEX IF NOT EXISTS detections_rule_idx ON detections(tenant_id,rule_name,last_seen DESC);

21
deploy/prometheus/prometheus.yml
View File

@@ -1,18 +1,11 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: siem-backend
- job_name: siem-ingress
static_configs: [{targets: ['ingress:8080']}]
- job_name: redpanda
metrics_path: /public_metrics
static_configs: [{targets: ['redpanda:9644']}]
- job_name: clickhouse
metrics_path: /metrics
static_configs:
- targets:
- siem-backend:8080
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
static_configs: [{targets: ['clickhouse:9363']}]

132
deploy/prometheus/rules/siem-alerts.yml
View File

@@ -1,132 +0,0 @@
groups:
- name: siem-backend-availability
rules:
- alert: SiemBackendDown
expr: up{job="siem-backend"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "SIEM backend nicht erreichbar"
description: "Prometheus kann das SIEM-Backend seit mindestens 2 Minuten nicht scrapen."
- alert: SiemNoIngestEvents
expr: sum(rate(eventcollector_ingest_events_total[15m])) == 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keine eingehenden SIEM Events"
description: "Seit mindestens 15 Minuten wurden keine Events mehr ingestiert."
- alert: SiemTooFewActiveAgents
expr: eventcollector_active_agents < 1
for: 5m
labels:
severity: warning
annotations:
summary: "Zu wenige aktive Agents"
description: "Es wurden weniger aktive Agents erkannt als erwartet."
- alert: SiemCriticalDetections
expr: increase(eventcollector_detection_hits_total{severity="critical"}[5m]) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Neue Critical Detection"
description: "Es wurde mindestens eine Critical-Detection erzeugt."
- alert: SiemHighDetections
expr: increase(eventcollector_detection_hits_total{severity="high"}[5m]) > 0
for: 1m
labels:
severity: high
annotations:
summary: "Neue High-Severity Detection"
description: "Es wurde mindestens eine High-Severity-Detection erzeugt."
- alert: SiemManyMediumDetections
expr: sum(increase(eventcollector_detection_hits_total{severity="medium"}[15m])) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Viele Medium-Detections"
description: "Es wurden mehr als 10 Medium-Detections in 15 Minuten erzeugt."
- alert: SiemBaselineHighAnomaly
expr: eventcollector_anomaly_score{rule="baseline_event_rate_anomaly"} >= 5
for: 2m
labels:
severity: high
annotations:
summary: "Hohe Baseline-Anomalie"
description: "Host {{ $labels.host }} hat einen hohen Baseline-Z-Score: {{ $value }}."
- alert: SiemBaselineMediumAnomaly
expr: eventcollector_anomaly_score{rule="baseline_event_rate_anomaly"} >= 3
for: 5m
labels:
severity: warning
annotations:
summary: "Baseline-Anomalie"
description: "Host {{ $labels.host }} hat einen erhöhten Baseline-Z-Score: {{ $value }}."
- alert: SiemRuleErrors
expr: increase(eventcollector_rule_errors_total[5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Fehler in Detection-Regeln"
description: "Mindestens eine Detection-Regel hat in den letzten 5 Minuten einen Fehler erzeugt."
- name: siem-backend-ingest
rules:
- alert: SiemIngestRejected
expr: sum(increase(eventcollector_ingest_rejected_total[5m])) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Ingest Requests abgelehnt"
description: "In den letzten 5 Minuten wurden Ingest Requests abgelehnt."
- alert: SiemDBInsertFailures
expr: increase(eventcollector_db_insert_failures_total[5m]) > 0
for: 1m
labels:
severity: high
annotations:
summary: "DB Insert Fehler"
description: "Das SIEM-Backend konnte Events nicht in die Datenbank schreiben."
- alert: SiemHighIngestRate
expr: sum(rate(eventcollector_ingest_events_total[5m])) > 500
for: 5m
labels:
severity: warning
annotations:
summary: "Sehr hohe Eventrate"
description: "Die Eventrate liegt seit 5 Minuten über 500 Events/s."
- name: siem-backend-baseline
rules:
- alert: SiemBaselineNotEnoughSamples
expr: eventcollector_baseline_sample_count > 0 and eventcollector_baseline_sample_count < 24
for: 30m
labels:
severity: info
annotations:
summary: "Baseline lernt noch"
description: "Für {{ $labels.host }} / {{ $labels.channel }} / {{ $labels.event_id }} gibt es erst {{ $value }} Samples."
- alert: SiemBaselineCurrentFarAboveAverage
expr: eventcollector_baseline_avg_count > 0 and (eventcollector_baseline_current_count / eventcollector_baseline_avg_count) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "Eventrate deutlich über Baseline"
description: "{{ $labels.host }} / {{ $labels.channel }} / {{ $labels.event_id }} liegt mehr als 10x über Durchschnitt."

35
doctor.sh Executable file
View File

@@ -0,0 +1,35 @@
#!/usr/bin/env sh
set -u
cd "$(dirname "$0")"
[ -f .env ] && { set -a; . ./.env; set +a; }
echo "== Docker =="
docker compose ps || true
echo "\n== Host filesystem =="
df -h . 2>/dev/null || true
echo "\n== HTTP readiness =="
if command -v curl >/dev/null 2>&1; then
printf "Ingress: "; curl -fsS "http://127.0.0.1:${INGRESS_PORT:-8090}/readyz" || true; echo
printf "API: "; curl -fsS "http://127.0.0.1:${UI_PORT:-8080}/readyz" || true; echo
else
echo "curl fehlt; HTTP-Prüfung übersprungen."
fi
echo "\n== Redpanda =="
docker compose exec -T redpanda rpk cluster health 2>/dev/null || true
docker compose exec -T redpanda rpk group describe siem-processor-v1 2>/dev/null || true
echo "\n== ClickHouse events =="
docker compose exec -T clickhouse clickhouse-client --user siem --password "${CLICKHOUSE_PASSWORD:-}" --query "SELECT formatReadableQuantity(sum(rows)) AS rows, formatReadableSize(sum(bytes_on_disk)) AS disk, count() AS active_parts FROM system.parts WHERE database='siem' AND table='events' AND active" 2>/dev/null || true
docker compose exec -T clickhouse clickhouse-client --user siem --password "${CLICKHOUSE_PASSWORD:-}" --query "SELECT max(event_time) AS latest_event, max(ingest_time) AS latest_ingest, quantile(0.95)(abs(ingest_delay_ms))/1000 AS p95_ingest_delay_seconds FROM siem.events WHERE event_time >= now() - INTERVAL 1 DAY" 2>/dev/null || true
echo "\n== PostgreSQL control plane =="
docker compose exec -T postgres psql -U siem -d siem -Atc "SELECT 'agents='||count(*) FROM agents; SELECT 'open_detections='||count(*) FROM detections WHERE status IN ('open','investigating');" 2>/dev/null || true
echo "\n== Raw spool =="
docker compose exec -T archive-uploader sh -c 'du -sh /spool 2>/dev/null || true; find /spool -type f 2>/dev/null | wc -l' 2>/dev/null || true
echo "\n== Recent errors =="
docker compose logs --since=15m ingress processor detector api archive-uploader 2>&1 | grep -Ei 'error|fatal|failed|panic|not_ready' | tail -80 || true

93
dot_env
View File

@@ -1,93 +0,0 @@
TZ=Europe/Berlin
LISTEN_ADDR=:8080
DB_DSN=eventuser:DEINPASSWORT@tcp(mariadb:3306)/eventcollector?parseTime=true&charset=utf8mb4,utf8&collation=utf8mb4_unicode_ci&loc=UTC
DB_MAX_OPEN_CONNS=50
DB_MAX_IDLE_CONNS=25
DB_CONN_MAX_LIFETIME=3m
DB_CONN_MAX_IDLE_TIME=1m
MAX_BODY_BYTES=10485760
HTTP_READ_TIMEOUT=15s
HTTP_WRITE_TIMEOUT=30s
HTTP_IDLE_TIMEOUT=60s
DETECTION_INTERVAL=1m
OFFLINE_AFTER=10m
OFFLINE_ALERT_MAX=120m
FAILED_LOGON_WINDOW=5m
FAILED_LOGON_THRESHOLD=25
REBOOT_WINDOW=15m
REBOOT_THRESHOLD=3
PASSWORD_SPRAY_WINDOW=5m
PASSWORD_SPRAY_MIN_USERS=5
PASSWORD_SPRAY_MIN_ATTEMPTS=15
SUCCESS_AFTER_FAILURE_WINDOW=10m
NEW_SOURCE_IP_LOOKBACK=720h
NEW_SOURCE_IP_WINDOW=10m
DETECTIONS_LIMIT=100
MARIADB_DATABASE=eventcollector
MARIADB_USER=eventuser
MARIADB_PASSWORD=DEINPASSWORT
MARIADB_ROOT_PASSWORD=ROOTPASSWORT
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=admin
ENROLLMENT_KEY=BITTE_SEHR_LANG_UND_ZUFAELLIG
BASELINE_ENABLED=true
BASELINE_WINDOW=5m
BASELINE_MIN_SAMPLES=24
BASELINE_MIN_COUNT=10
BASELINE_MEDIUM_Z=2.5
BASELINE_HIGH_Z=4.0
BASELINE_SUPPRESS_FOR=1h
#BASELINE_MIN_SAMPLES=84
#BASELINE_MEDIUM_Z=3.0
#BASELINE_HIGH_Z=5.0
#BASELINE_MIN_COUNT=20
# Metadata-first storage
# Raw XML is disabled by default. Enable only for short forensic retention.
STORE_EVENT_ROWS=false
STORE_RAW_XML=false
METADATA_CONTEXT_EVENT_IDS=4624,4625,4648,4672,4720,4726,4728,4732,4740,4756,4768,4769,4771,4776
METADATA_BUCKET=5m
EVENT_RETENTION=6h
RAW_RETENTION=6h
METADATA_RETENTION=720h
# Partition maintenance
PARTITION_MAINTENANCE_ENABLED=true
PARTITION_MAINTENANCE_INTERVAL=15m
PARTITION_INTERVAL=3h
PARTITION_AHEAD=24h
PARTITION_BEHIND=6h
# Compatibility fallback; table-specific retention values above take precedence.
PARTITION_RETENTION=720h
# New Event-ID classification
NEW_EVENT_ID_MODE=inventory
NEW_EVENT_ID_LEARNING_PERIOD=24h
NEW_EVENT_ID_CONFIRM_WINDOW=15m
NEW_EVENT_ID_MIN_COUNT=3
NEW_EVENT_ID_IGNORE_CHANNELS=Microsoft-Windows-WMI-Activity/Operational
NEW_EVENT_ID_ALERT_CHANNELS=Security,System
NEW_EVENT_ID_HIGH_RISK_IDS=1102,4697,4719,7045
# Optional stress-agent profile
SIEM_STRESS_URL=http://siem-backend:8080/ingest
SIEM_STRESS_HOST=SIEM-STRESS-01
SIEM_STRESS_API_KEY=CHANGE-ME-STRESS-API-KEY
SIEM_STRESS_SCENARIO=mixed
SIEM_STRESS_RATE=200
SIEM_STRESS_BATCH=100
SIEM_STRESS_WORKERS=4
SIEM_STRESS_DURATION=30s
SIEM_STRESS_MAX_EVENTS=100000
SIEM_STRESS_TIMEOUT=20s
SIEM_STRESS_CONFIRM=true

22
go.mod
View File

@@ -1,22 +1,8 @@
module git.send.nrw/sendnrw/siem-backend
module example.com/siem-greenfield
go 1.26.1
require github.com/go-sql-driver/mysql v1.9.3
go 1.26
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/sys v0.35.0 // indirect
google.golang.org/protobuf v1.36.8 // indirect
)
require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/prometheus/client_golang v1.23.2
github.com/jackc/pgx/v5 v5.9.2
github.com/segmentio/kafka-go v0.4.51
)

25
go.sum
View File

@@ -1,25 +0,0 @@
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

187
internal/api/server.go Normal file
View File

@@ -0,0 +1,187 @@
package api
import (
"context"
"crypto/subtle"
"encoding/json"
"fmt"
"html/template"
"log"
"net/http"
"strconv"
"strings"
"time"
"example.com/siem-greenfield/internal/clickhouse"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/postgres"
)
func Run(ctx context.Context, cfg config.Config) error {
pg, e := postgres.Open(ctx, cfg.PostgresURL)
if e != nil {
return e
}
defer pg.Close()
ch := clickhouse.New(cfg)
tpl, e := template.ParseFiles("/app/web/index.html")
if e != nil {
return e
}
mux := http.NewServeMux()
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { j(w, 200, map[string]string{"status": "ok"}) })
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
cctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
if e := pg.Pool.Ping(cctx); e != nil {
j(w, 503, map[string]string{"status": "not_ready", "component": "postgres"})
return
}
if e := ch.Exec(cctx, "SELECT 1"); e != nil {
j(w, 503, map[string]string{"status": "not_ready", "component": "clickhouse"})
return
}
j(w, 200, map[string]string{"status": "ready"})
})
mux.HandleFunc("/ui", func(w http.ResponseWriter, r *http.Request) {
_ = tpl.Execute(w, map[string]string{"Tenant": cfg.TenantID})
})
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/" {
http.Redirect(w, r, "/ui", http.StatusFound)
return
}
http.NotFound(w, r)
})
mux.HandleFunc("/api/summary", func(w http.ResponseWriter, r *http.Request) { summary(w, r, cfg, pg, ch) })
mux.HandleFunc("/api/events", func(w http.ResponseWriter, r *http.Request) { events(w, r, cfg, ch) })
mux.HandleFunc("/api/detections", func(w http.ResponseWriter, r *http.Request) { detections(w, r, cfg, pg) })
mux.HandleFunc("/api/detections/status", func(w http.ResponseWriter, r *http.Request) { detectionStatus(w, r, cfg, pg) })
mux.HandleFunc("/api/agents", func(w http.ResponseWriter, r *http.Request) {
x, e := pg.ListAgents(r.Context(), cfg.TenantID)
if e != nil {
j(w, 500, map[string]string{"error": e.Error()})
return
}
j(w, 200, x)
})
srv := &http.Server{Addr: cfg.ServiceAddr, Handler: security(basicAuth(mux, cfg.UIUsername, cfg.UIPassword)), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 30 * time.Second, IdleTimeout: 60 * time.Second}
go func() {
<-ctx.Done()
c, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(c)
}()
log.Printf("api/ui listening on %s", cfg.ServiceAddr)
e = srv.ListenAndServe()
if e == http.ErrServerClosed {
return nil
}
return e
}
func summary(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store, ch *clickhouse.Client) {
ctx, cancel := context.WithTimeout(r.Context(), 8*time.Second)
defer cancel()
q := fmt.Sprintf(`SELECT uniqExactMerge(cnt_state) events_24h, uniqExact(host_name) active_hosts FROM siem.events_5m WHERE tenant_id=%s AND bucket>=now()-INTERVAL 24 HOUR`, clickhouse.Q(cfg.TenantID))
rows, e := ch.QueryJSON(ctx, q)
if e != nil {
j(w, 500, map[string]string{"error": e.Error()})
return
}
out := map[string]any{"events_24h": 0, "active_hosts": 0}
if len(rows) > 0 {
for k, v := range rows[0] {
out[k] = v
}
}
dc, _ := pg.DetectionCounts(ctx, cfg.TenantID)
out["detections"] = dc
j(w, 200, out)
}
func events(w http.ResponseWriter, r *http.Request, cfg config.Config, ch *clickhouse.Client) {
limit := cfg.UIQueryLimit
if n, e := strconv.Atoi(r.URL.Query().Get("limit")); e == nil && n > 0 && n <= 2000 {
limit = n
}
hours := 24
if n, e := strconv.Atoi(r.URL.Query().Get("hours")); e == nil && n > 0 && n <= 2160 {
hours = n
}
where := []string{"tenant_id=" + clickhouse.Q(cfg.TenantID), fmt.Sprintf("event_time>=now()-INTERVAL %d HOUR", hours)}
for key, col := range map[string]string{"host": "host_name", "user": "user_name", "ip": "source_ip", "channel": "channel", "action": "action"} {
if v := strings.TrimSpace(r.URL.Query().Get(key)); v != "" {
where = append(where, col+"="+clickhouse.Q(v))
}
}
if v := strings.TrimSpace(r.URL.Query().Get("event_code")); v != "" {
if _, e := strconv.ParseUint(v, 10, 32); e == nil {
where = append(where, "event_code="+v)
}
}
q := fmt.Sprintf(`SELECT event_uid,event_time,host_name,channel,event_code,category,action,outcome,severity,user_name,subject_user,target_user,source_ip,workstation,process_path,message,raw_object_key,raw_index FROM siem.events WHERE %s ORDER BY event_time DESC, ingest_time DESC LIMIT 1 BY event_uid LIMIT %d`, strings.Join(where, " AND "), limit)
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
defer cancel()
rows, e := ch.QueryJSON(ctx, q)
if e != nil {
j(w, 500, map[string]string{"error": e.Error()})
return
}
j(w, 200, rows)
}
func detections(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store) {
x, e := pg.ListDetections(r.Context(), cfg.TenantID, 500)
if e != nil {
j(w, 500, map[string]string{"error": e.Error()})
return
}
j(w, 200, x)
}
func detectionStatus(w http.ResponseWriter, r *http.Request, cfg config.Config, pg *postgres.Store) {
if r.Method != http.MethodPost {
j(w, 405, map[string]string{"error": "method not allowed"})
return
}
var v struct {
ID int64 `json:"id"`
Status string `json:"status"`
}
if e := json.NewDecoder(r.Body).Decode(&v); e != nil {
j(w, 400, map[string]string{"error": "invalid json"})
return
}
if e := pg.UpdateDetectionStatus(r.Context(), cfg.TenantID, v.ID, v.Status); e != nil {
j(w, 400, map[string]string{"error": e.Error()})
return
}
j(w, 200, map[string]string{"status": "ok"})
}
func j(w http.ResponseWriter, s int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(s)
_ = json.NewEncoder(w).Encode(v)
}
func basicAuth(next http.Handler, user, pass string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/healthz" || r.URL.Path == "/readyz" {
next.ServeHTTP(w, r)
return
}
u, p, ok := r.BasicAuth()
if !ok || subtle.ConstantTimeCompare([]byte(u), []byte(user)) != 1 || subtle.ConstantTimeCompare([]byte(p), []byte(pass)) != 1 {
w.Header().Set("WWW-Authenticate", `Basic realm="Greenfield SIEM", charset="UTF-8"`)
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
next.ServeHTTP(w, r)
})
}
func security(n http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("X-Frame-Options", "DENY")
w.Header().Set("Referrer-Policy", "no-referrer")
w.Header().Set("Content-Security-Policy", "default-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'")
n.ServeHTTP(w, r)
})
}

98
internal/clickhouse/client.go Normal file
View File

@@ -0,0 +1,98 @@
package clickhouse
import (
"bufio"
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/contracts"
)
type Client struct {
base, user, pass string
hc *http.Client
}
func New(cfg config.Config) *Client {
return &Client{base: cfg.ClickHouseURL, user: cfg.ClickHouseUser, pass: cfg.ClickHousePassword, hc: &http.Client{Timeout: 30 * time.Second}}
}
func (c *Client) request(ctx context.Context, query string, body io.Reader) (*http.Response, error) {
u := c.base + "/?query=" + url.QueryEscape(query)
req, e := http.NewRequestWithContext(ctx, http.MethodPost, u, body)
if e != nil {
return nil, e
}
req.SetBasicAuth(c.user, c.pass)
return c.hc.Do(req)
}
func (c *Client) Exec(ctx context.Context, q string) error {
r, e := c.request(ctx, q, nil)
if e != nil {
return e
}
defer r.Body.Close()
if r.StatusCode/100 != 2 {
b, _ := io.ReadAll(io.LimitReader(r.Body, 8192))
return fmt.Errorf("clickhouse %s: %s", r.Status, string(b))
}
return nil
}
func (c *Client) InsertEvents(ctx context.Context, events []contracts.CanonicalEvent) error {
if len(events) == 0 {
return nil
}
var b bytes.Buffer
enc := json.NewEncoder(&b)
enc.SetEscapeHTML(false)
for i := range events {
if e := enc.Encode(events[i]); e != nil {
return e
}
}
q := `INSERT INTO siem.events FORMAT JSONEachRow`
r, e := c.request(ctx, q, &b)
if e != nil {
return e
}
defer r.Body.Close()
if r.StatusCode/100 != 2 {
x, _ := io.ReadAll(io.LimitReader(r.Body, 16384))
return fmt.Errorf("clickhouse insert %s: %s", r.Status, string(x))
}
return nil
}
func (c *Client) QueryJSON(ctx context.Context, q string) ([]map[string]any, error) {
if !strings.Contains(strings.ToUpper(q), "FORMAT") {
q += " FORMAT JSONEachRow"
}
r, e := c.request(ctx, q, nil)
if e != nil {
return nil, e
}
defer r.Body.Close()
if r.StatusCode/100 != 2 {
x, _ := io.ReadAll(io.LimitReader(r.Body, 16384))
return nil, fmt.Errorf("clickhouse query %s: %s", r.Status, string(x))
}
var out []map[string]any
s := bufio.NewScanner(r.Body)
buf := make([]byte, 0, 64*1024)
s.Buffer(buf, 4*1024*1024)
for s.Scan() {
var m map[string]any
if e := json.Unmarshal(s.Bytes(), &m); e != nil {
return nil, e
}
out = append(out, m)
}
return out, s.Err()
}
func Q(s string) string { return "'" + strings.ReplaceAll(s, "'", "''") + "'" }

103
internal/config/config.go Normal file
View File

@@ -0,0 +1,103 @@
package config
import (
"os"
"strconv"
"strings"
"time"
)
type Config struct {
ServiceAddr string
PostgresURL string
ClickHouseURL string
ClickHouseUser string
ClickHousePassword string
KafkaBrokers []string
KafkaTopic string
KafkaGroup string
EnrollmentKey string
TenantID string
MaxBodyBytes int64
MaxBatchEvents int
RawArchiveEnabled bool
RawSpoolDir string
RawRetention time.Duration
EventRetention time.Duration
DetectorInterval time.Duration
DetectorLookback time.Duration
UIQueryLimit int
UIUsername string
UIPassword string
}
func Load() Config {
return Config{
ServiceAddr: env("SERVICE_ADDR", ":8080"),
PostgresURL: env("POSTGRES_URL", "postgres://siem:siem@postgres:5432/siem?sslmode=disable"),
ClickHouseURL: strings.TrimRight(env("CLICKHOUSE_URL", "http://clickhouse:8123"), "/"),
ClickHouseUser: env("CLICKHOUSE_USER", "siem"),
ClickHousePassword: env("CLICKHOUSE_PASSWORD", "siem"),
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "redpanda:9092")),
KafkaTopic: env("KAFKA_TOPIC", "siem-events"),
KafkaGroup: env("KAFKA_GROUP", "siem-processor-v1"),
EnrollmentKey: env("ENROLLMENT_KEY", "change-me"),
TenantID: env("TENANT_ID", "default"),
MaxBodyBytes: int64(envInt("MAX_BODY_BYTES", 8*1024*1024)),
MaxBatchEvents: envInt("MAX_BATCH_EVENTS", 1000),
RawArchiveEnabled: envBool("RAW_ARCHIVE_ENABLED", true),
RawSpoolDir: env("RAW_SPOOL_DIR", "/var/spool/siem-raw"),
RawRetention: envDuration("RAW_RETENTION", 30*24*time.Hour),
EventRetention: envDuration("EVENT_RETENTION", 90*24*time.Hour),
DetectorInterval: envDuration("DETECTOR_INTERVAL", 30*time.Second),
DetectorLookback: envDuration("DETECTOR_LOOKBACK", 20*time.Minute),
UIQueryLimit: envInt("UI_QUERY_LIMIT", 500),
UIUsername: env("UI_USERNAME", "admin"),
UIPassword: env("UI_PASSWORD", "change-me"),
}
}
func env(k, d string) string {
if v := strings.TrimSpace(os.Getenv(k)); v != "" {
return v
}
return d
}
func envInt(k string, d int) int {
v := strings.TrimSpace(os.Getenv(k))
if v == "" {
return d
}
n, e := strconv.Atoi(v)
if e != nil {
return d
}
return n
}
func envBool(k string, d bool) bool {
v := strings.ToLower(strings.TrimSpace(os.Getenv(k)))
if v == "" {
return d
}
return v == "1" || v == "true" || v == "yes" || v == "on"
}
func envDuration(k string, d time.Duration) time.Duration {
v := strings.TrimSpace(os.Getenv(k))
if v == "" {
return d
}
x, e := time.ParseDuration(v)
if e != nil {
return d
}
return x
}
func splitCSV(v string) []string {
var out []string
for _, p := range strings.Split(v, ",") {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}

94
internal/contracts/event.go Normal file
View File

@@ -0,0 +1,94 @@
package contracts
import "time"
type EventMetadataPayload struct {
Computer string `json:"computer,omitempty"`
ProviderName string `json:"provider,omitempty"`
TargetUser string `json:"target_user,omitempty"`
TargetDomain string `json:"target_domain,omitempty"`
SubjectUser string `json:"subject_user,omitempty"`
SubjectDomain string `json:"subject_domain,omitempty"`
Workstation string `json:"workstation,omitempty"`
Device string `json:"device,omitempty"`
SrcIP string `json:"src_ip,omitempty"`
SrcPort string `json:"src_port,omitempty"`
DstIP string `json:"dst_ip,omitempty"`
DstPort string `json:"dst_port,omitempty"`
LogonType string `json:"logon_type,omitempty"`
ProcessName string `json:"process_name,omitempty"`
ParentProcessName string `json:"parent_process_name,omitempty"`
CommandLine string `json:"command_line,omitempty"`
AuthenticationPackage string `json:"authentication_package,omitempty"`
LogonProcess string `json:"logon_process,omitempty"`
StatusText string `json:"status,omitempty"`
SubStatusText string `json:"sub_status,omitempty"`
FailureReason string `json:"failure_reason,omitempty"`
}
type LogPayload struct {
Hostname string `json:"host"`
Channel string `json:"channel"`
EventID uint32 `json:"id"`
Source string `json:"source"`
Time time.Time `json:"ts"`
Message string `json:"msg,omitempty"`
Metadata *EventMetadataPayload `json:"meta,omitempty"`
}
type IngestEnvelope struct {
Version int `json:"version"`
TenantID string `json:"tenant_id"`
AgentID string `json:"agent_id"`
BatchUID string `json:"batch_uid"`
RemoteIP string `json:"remote_ip"`
ReceivedAt time.Time `json:"received_at"`
Events []LogPayload `json:"events"`
}
type CanonicalEvent struct {
EventUID string `json:"event_uid"`
QueuePartition int32 `json:"queue_partition"`
QueueOffset int64 `json:"queue_offset"`
TenantID string `json:"tenant_id"`
EventTime string `json:"event_time"`
IngestTime string `json:"ingest_time"`
AgentID string `json:"agent_id"`
HostName string `json:"host_name"`
SourceType string `json:"source_type"`
Channel string `json:"channel"`
Provider string `json:"provider"`
EventCode uint32 `json:"event_code"`
Category string `json:"category"`
Action string `json:"action"`
Outcome string `json:"outcome"`
Severity uint8 `json:"severity"`
UserName string `json:"user_name"`
UserDomain string `json:"user_domain"`
SubjectUser string `json:"subject_user"`
SubjectDomain string `json:"subject_domain"`
TargetUser string `json:"target_user"`
TargetDomain string `json:"target_domain"`
SourceIP string `json:"source_ip"`
SourcePort uint16 `json:"source_port"`
DestinationIP string `json:"destination_ip"`
DestinationPort uint16 `json:"destination_port"`
Workstation string `json:"workstation"`
LogonType string `json:"logon_type"`
AuthenticationPackage string `json:"authentication_package"`
LogonProcess string `json:"logon_process"`
StatusCode string `json:"status_code"`
SubStatusCode string `json:"sub_status_code"`
FailureReason string `json:"failure_reason"`
ProcessPath string `json:"process_path"`
ParentProcessPath string `json:"parent_process_path"`
CommandLine string `json:"command_line"`
Message string `json:"message"`
Attributes map[string]string `json:"attributes"`
RawObjectKey string `json:"raw_object_key"`
RawIndex uint32 `json:"raw_index"`
PayloadHash string `json:"payload_hash"`
SchemaVersion uint16 `json:"schema_version"`
ParserVersion uint16 `json:"parser_version"`
IngestDelayMS int64 `json:"ingest_delay_ms"`
}

152
internal/detector/detector.go Normal file
View File

@@ -0,0 +1,152 @@
package detector
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"log"
"strconv"
"strings"
"time"
"example.com/siem-greenfield/internal/clickhouse"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/postgres"
)
type rule struct {
name, severity string
eventCode uint32
score float64
query func(time.Time, time.Time, string) string
summary func(map[string]any) string
}
func Run(ctx context.Context, cfg config.Config) error {
pg, e := postgres.Open(ctx, cfg.PostgresURL)
if e != nil {
return e
}
defer pg.Close()
ch := clickhouse.New(cfg)
ticker := time.NewTicker(cfg.DetectorInterval)
defer ticker.Stop()
run := func() {
if e := runAll(ctx, cfg, pg, ch); e != nil {
log.Printf("detector cycle: %v", e)
}
}
run()
for {
select {
case <-ctx.Done():
return nil
case <-ticker.C:
run()
}
}
}
func runAll(ctx context.Context, cfg config.Config, pg *postgres.Store, ch *clickhouse.Client) error {
end := time.Now().UTC()
start := end.Add(-cfg.DetectorLookback)
for _, r := range rules() {
q := r.query(start, end, cfg.TenantID)
rows, e := ch.QueryJSON(ctx, q)
if e != nil {
return fmt.Errorf("%s: %w", r.name, e)
}
for _, row := range rows {
host := str(row["host_name"])
user := str(row["user_name"])
ip := str(row["source_ip"])
workstation := str(row["workstation"])
count := int64(num(row["cnt"]))
ws := timeVal(row["window_start"], start)
we := timeVal(row["window_end"], end)
fp := fingerprint(r.name, host, user, ip, workstation, strconv.FormatInt(ws.Unix()/300, 10))
d := postgres.Detection{Fingerprint: fp, RuleName: r.name, Severity: r.severity, Hostname: host, UserName: user, SourceIP: ip, Workstation: workstation, EventCode: r.eventCode, Score: r.score, WindowStart: ws, WindowEnd: we, Summary: r.summary(row), Count: max64(1, count)}
if e := pg.UpsertDetection(ctx, d, cfg.TenantID); e != nil {
return e
}
}
}
return nil
}
func rules() []rule {
return []rule{
{name: "audit_log_cleared", severity: "critical", eventCode: 1102, score: 9.8, query: simpleEvent(1102, 1), summary: func(m map[string]any) string {
return fmt.Sprintf("Audit-Log auf %s wurde gelöscht", str(m["host_name"]))
}},
{name: "service_installed", severity: "high", eventCode: 7045, score: 8.0, query: simpleEvent(7045, 1), summary: func(m map[string]any) string {
return fmt.Sprintf("Neuer Dienst auf %s installiert", str(m["host_name"]))
}},
{name: "account_lockout", severity: "medium", eventCode: 4740, score: 5.5, query: func(s, e time.Time, t string) string {
return fmt.Sprintf(`SELECT host_name, target_user AS user_name, '' AS source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4740 GROUP BY host_name,user_name,workstation HAVING cnt>=1`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)))
}, summary: func(m map[string]any) string {
return fmt.Sprintf("Account-Lockout: %s; Caller %s; DC/Host %s (%d×)", str(m["user_name"]), fallback(str(m["workstation"]), "unbekannt"), str(m["host_name"]), int64(num(m["cnt"])))
}},
{name: "failed_logon_burst", severity: "high", eventCode: 4625, score: 7.5, query: func(s, e time.Time, t string) string {
return fmt.Sprintf(`SELECT host_name, target_user AS user_name, source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4625 AND target_user!='' GROUP BY host_name,user_name,source_ip,workstation HAVING cnt>=20`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)))
}, summary: func(m map[string]any) string {
return fmt.Sprintf("%d fehlgeschlagene Logons für %s auf %s", int64(num(m["cnt"])), str(m["user_name"]), str(m["host_name"]))
}},
{name: "password_spray", severity: "high", eventCode: 4625, score: 8.5, query: func(s, e time.Time, t string) string {
return fmt.Sprintf(`SELECT '' AS host_name, '' AS user_name, source_ip, '' AS workstation, uniqExact(target_user) users, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=4625 AND source_ip!='' AND target_user!='' GROUP BY source_ip HAVING users>=10 AND cnt>=20`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)))
}, summary: func(m map[string]any) string {
return fmt.Sprintf("Password-Spray von %s gegen %.0f Benutzer (%d Versuche)", str(m["source_ip"]), num(m["users"]), int64(num(m["cnt"])))
}},
{name: "privileged_group_change", severity: "critical", eventCode: 4728, score: 9.2, query: func(s, e time.Time, t string) string {
return fmt.Sprintf(`SELECT host_name, target_user AS user_name, '' AS source_ip, workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code IN (4728,4732,4756) GROUP BY host_name,user_name,workstation HAVING cnt>=1`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)))
}, summary: func(m map[string]any) string {
return fmt.Sprintf("Privilegierte Gruppenmitgliedschaft geändert: %s auf %s", str(m["user_name"]), str(m["host_name"]))
}},
}
}
func simpleEvent(id uint32, min int) func(time.Time, time.Time, string) string {
return func(s, e time.Time, t string) string {
return fmt.Sprintf(`SELECT host_name, '' AS user_name, '' AS source_ip, '' AS workstation, uniqExact(event_uid) cnt, min(event_time) window_start, max(event_time) window_end FROM siem.events WHERE tenant_id=%s AND event_time>=%s AND event_time<%s AND event_code=%d GROUP BY host_name HAVING cnt>=%d`, clickhouse.Q(t), clickhouse.Q(ts(s)), clickhouse.Q(ts(e)), id, min)
}
}
func ts(t time.Time) string { return t.UTC().Format("2006-01-02 15:04:05.000") }
func str(v any) string {
if v == nil {
return ""
}
return fmt.Sprint(v)
}
func num(v any) float64 {
switch x := v.(type) {
case float64:
return x
case jsonNumber:
return x.Float()
default:
f, _ := strconv.ParseFloat(fmt.Sprint(v), 64)
return f
}
}
type jsonNumber string
func (n jsonNumber) Float() float64 { f, _ := strconv.ParseFloat(string(n), 64); return f }
func timeVal(v any, d time.Time) time.Time {
s := str(v)
for _, layout := range []string{"2006-01-02 15:04:05.999999", "2006-01-02 15:04:05", "2006-01-02T15:04:05Z07:00"} {
if t, e := time.Parse(layout, s); e == nil {
return t.UTC()
}
}
return d
}
func fingerprint(v ...string) string {
h := sha256.Sum256([]byte(strings.Join(v, "|")))
return hex.EncodeToString(h[:])
}
func fallback(v, d string) string { if strings.TrimSpace(v) == "" { return d }; return v }
func max64(a, b int64) int64 {
if a > b {
return a
}
return b
}

178
internal/ingress/server.go Normal file
View File

@@ -0,0 +1,178 @@
package ingress
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"strings"
"time"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/contracts"
"example.com/siem-greenfield/internal/metrics"
"example.com/siem-greenfield/internal/postgres"
"example.com/siem-greenfield/internal/queue"
)
func Run(ctx context.Context, cfg config.Config) error {
pg, err := postgres.Open(ctx, cfg.PostgresURL)
if err != nil {
return fmt.Errorf("postgres: %w", err)
}
defer pg.Close()
prod := queue.NewProducer(cfg.KafkaBrokers, cfg.KafkaTopic)
defer prod.Close()
m := metrics.New()
accepted := m.Counter("siem_ingress_events_accepted_total")
rejected := m.Counter("siem_ingress_requests_rejected_total")
batches := m.Counter("siem_ingress_batches_total")
mux := http.NewServeMux()
mux.Handle("/metrics", m.Handler())
mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, 200, map[string]string{"status": "ok"}) })
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
cctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
if e := pg.Pool.Ping(cctx); e != nil {
writeJSON(w, 503, map[string]string{"status": "not_ready"})
return
}
writeJSON(w, 200, map[string]string{"status": "ready"})
})
mux.HandleFunc("/ingest", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
rejected.Add(1)
writeJSON(w, 405, map[string]string{"error": "method not allowed"})
return
}
apiKey := strings.TrimSpace(r.Header.Get("X-API-Key"))
if apiKey == "" {
rejected.Add(1)
writeJSON(w, 401, map[string]string{"error": "missing api key"})
return
}
r.Body = http.MaxBytesReader(w, r.Body, cfg.MaxBodyBytes)
defer r.Body.Close()
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
var batch []contracts.LogPayload
if e := dec.Decode(&batch); e != nil || (func() bool { var x any; return dec.Decode(&x) != io.EOF })() {
rejected.Add(1)
writeJSON(w, 400, map[string]string{"error": "invalid json"})
return
}
if len(batch) == 0 || len(batch) > cfg.MaxBatchEvents {
rejected.Add(1)
writeJSON(w, 400, map[string]string{"error": "invalid batch size"})
return
}
host := strings.TrimSpace(batch[0].Hostname)
for i := range batch {
if e := validate(&batch[i]); e != nil {
rejected.Add(1)
writeJSON(w, 400, map[string]string{"error": fmt.Sprintf("invalid payload at index %d: %v", i, e)})
return
}
if batch[i].Hostname != host {
rejected.Add(1)
writeJSON(w, 400, map[string]string{"error": "all events in a batch must use the same hostname"})
return
}
}
cctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
agentID, e := pg.AuthenticateOrEnroll(cctx, cfg.TenantID, host, apiKey, strings.TrimSpace(r.Header.Get("X-Enrollment-Key")), cfg.EnrollmentKey, postgres.ClientIP(r.RemoteAddr))
if e != nil {
rejected.Add(1)
if errors.Is(e, postgres.ErrUnauthorized) {
writeJSON(w, 401, map[string]string{"error": "invalid api key or hostname"})
} else {
log.Printf("auth: %v", e)
writeJSON(w, 503, map[string]string{"error": "control plane unavailable"})
}
return
}
env := contracts.IngestEnvelope{Version: 1, TenantID: cfg.TenantID, AgentID: agentID, BatchUID: batchUID(agentID, batch), RemoteIP: postgres.ClientIP(r.RemoteAddr), ReceivedAt: time.Now().UTC(), Events: batch}
payload, e := json.Marshal(env)
if e != nil {
writeJSON(w, 500, map[string]string{"error": "internal error"})
return
}
qctx, qcancel := context.WithTimeout(r.Context(), 8*time.Second)
defer qcancel()
if e = prod.Write(qctx, agentID, payload); e != nil {
log.Printf("queue write: %v", e)
writeJSON(w, 503, map[string]string{"error": "ingest queue unavailable"})
return
}
accepted.Add(uint64(len(batch)))
batches.Add(1)
writeJSON(w, 202, map[string]int{"accepted": len(batch)})
})
srv := &http.Server{Addr: cfg.ServiceAddr, Handler: withLimits(mux), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second}
go func() {
<-ctx.Done()
cctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(cctx)
}()
log.Printf("ingress listening on %s", cfg.ServiceAddr)
e := srv.ListenAndServe()
if e == http.ErrServerClosed {
return nil
}
return e
}
func batchUID(agentID string, batch []contracts.LogPayload) string {
b, _ := json.Marshal(batch)
h := sha256.New()
_, _ = h.Write([]byte(agentID))
_, _ = h.Write([]byte{0})
_, _ = h.Write(b)
return hex.EncodeToString(h.Sum(nil))
}
func validate(p *contracts.LogPayload) error {
p.Hostname = strings.TrimSpace(p.Hostname)
p.Channel = strings.TrimSpace(p.Channel)
p.Source = strings.TrimSpace(p.Source)
if p.Hostname == "" || len(p.Hostname) > 255 {
return fmt.Errorf("invalid host")
}
if p.Channel == "" || len(p.Channel) > 128 {
return fmt.Errorf("invalid channel")
}
if p.Source == "" || len(p.Source) > 255 {
return fmt.Errorf("invalid source")
}
if p.EventID == 0 {
return fmt.Errorf("event id required")
}
if p.Time.IsZero() {
return fmt.Errorf("ts required")
}
if strings.TrimSpace(p.Message) == "" && p.Metadata == nil {
return fmt.Errorf("either msg or meta required")
}
if len(p.Message) > 2*1024*1024 {
return fmt.Errorf("msg too large")
}
return nil
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
func withLimits(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Cache-Control", "no-store")
next.ServeHTTP(w, r)
})
}

29
internal/ingress/server_test.go Normal file
View File

@@ -0,0 +1,29 @@
package ingress
import (
"testing"
"time"
"example.com/siem-greenfield/internal/contracts"
)
func TestValidateCompatiblePayload(t *testing.T) {
p := contracts.LogPayload{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Now().UTC(), Metadata: &contracts.EventMetadataPayload{TargetUser: "alice"}}
if err := validate(&p); err != nil {
t.Fatalf("valid payload rejected: %v", err)
}
}
func TestValidateRequiresMessageOrMetadata(t *testing.T) {
p := contracts.LogPayload{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Now().UTC()}
if err := validate(&p); err == nil {
t.Fatal("expected validation error")
}
}
func TestBatchUIDIsDeterministic(t *testing.T) {
b := []contracts.LogPayload{{Hostname: "PC01", Channel: "Security", EventID: 4625, Source: "agent", Time: time.Date(2026,7,23,12,0,0,0,time.UTC), Metadata: &contracts.EventMetadataPayload{TargetUser: "alice"}}}
a := batchUID("agent-1", b)
if a == "" || a != batchUID("agent-1", b) { t.Fatalf("batch uid is not deterministic") }
if a == batchUID("agent-2", b) { t.Fatalf("batch uid must be scoped to agent") }
}

67
internal/metrics/metrics.go Normal file
View File

@@ -0,0 +1,67 @@
package metrics
import (
"fmt"
"net/http"
"sort"
"strings"
"sync"
"sync/atomic"
)
type Registry struct {
mu sync.RWMutex
counters map[string]*atomic.Uint64
gauges map[string]*atomic.Int64
}
func New() *Registry {
return &Registry{counters: map[string]*atomic.Uint64{}, gauges: map[string]*atomic.Int64{}}
}
func (r *Registry) Counter(name string) *atomic.Uint64 {
r.mu.Lock()
defer r.mu.Unlock()
if c := r.counters[name]; c != nil {
return c
}
c := &atomic.Uint64{}
r.counters[name] = c
return c
}
func (r *Registry) Gauge(name string) *atomic.Int64 {
r.mu.Lock()
defer r.mu.Unlock()
if g := r.gauges[name]; g != nil {
return g
}
g := &atomic.Int64{}
r.gauges[name] = g
return g
}
func (r *Registry) Handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/plain; version=0.0.4")
r.mu.RLock()
defer r.mu.RUnlock()
var lines []string
for n, c := range r.counters {
lines = append(lines, fmt.Sprintf("%s %d", sanitize(n), c.Load()))
}
for n, g := range r.gauges {
lines = append(lines, fmt.Sprintf("%s %d", sanitize(n), g.Load()))
}
sort.Strings(lines)
fmt.Fprintln(w, strings.Join(lines, "\n"))
})
}
func sanitize(s string) string {
var b strings.Builder
for i, r := range s {
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9' && i > 0) || r == '_' || r == ':' {
b.WriteRune(r)
} else {
b.WriteByte('_')
}
}
return b.String()
}

297
internal/normalize/normalize.go Normal file
View File

@@ -0,0 +1,297 @@
package normalize
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"encoding/xml"
"io"
"net"
"strconv"
"strings"
"time"
"example.com/siem-greenfield/internal/contracts"
)
type Parsed struct {
Computer, Provider, TargetUser, TargetDomain, SubjectUser, SubjectDomain, Workstation, SrcIP, SrcPort, DstIP, DstPort, LogonType, ProcessName, ParentProcessName, CommandLine, AuthPackage, LogonProcess, Status, SubStatus, FailureReason string
Attributes map[string]string
}
func Event(env contracts.IngestEnvelope, p contracts.LogPayload, rawKey string, partition int, offset int64, idx int) contracts.CanonicalEvent {
parsed := parseXML(p.Message)
if p.Metadata != nil {
overlay(&parsed, *p.Metadata)
}
host := first(parsed.Computer, p.Hostname)
provider := first(parsed.Provider, p.Source)
cat, action, outcome, sev := classify(p.Channel, p.EventID)
user := first(parsed.TargetUser, parsed.SubjectUser)
msg := strings.TrimSpace(p.Message)
if strings.HasPrefix(msg, "<") {
msg = ""
}
if len(msg) > 4096 {
msg = msg[:4096]
}
attrs := parsed.Attributes
if attrs == nil {
attrs = map[string]string{}
}
eventUID := env.BatchUID
if eventUID == "" {
eventUID = strconv.Itoa(partition) + ":" + strconv.FormatInt(offset, 10)
}
eventUID += ":" + strconv.Itoa(idx)
return contracts.CanonicalEvent{
EventUID: eventUID, QueuePartition: int32(partition), QueueOffset: offset,
TenantID: env.TenantID, EventTime: p.Time.UTC().Format("2006-01-02 15:04:05.000"), IngestTime: env.ReceivedAt.UTC().Format("2006-01-02 15:04:05.000"), AgentID: env.AgentID, HostName: host, SourceType: p.Source, Channel: p.Channel, Provider: provider, EventCode: p.EventID, Category: cat, Action: action, Outcome: outcome, Severity: sev,
UserName: cleanUser(user), UserDomain: first(parsed.TargetDomain, parsed.SubjectDomain), SubjectUser: cleanUser(parsed.SubjectUser), SubjectDomain: parsed.SubjectDomain, TargetUser: cleanUser(parsed.TargetUser), TargetDomain: parsed.TargetDomain, SourceIP: normalizeIP(parsed.SrcIP), SourcePort: parsePort(parsed.SrcPort), DestinationIP: normalizeIP(parsed.DstIP), DestinationPort: parsePort(parsed.DstPort), Workstation: parsed.Workstation, LogonType: parsed.LogonType, AuthenticationPackage: parsed.AuthPackage, LogonProcess: parsed.LogonProcess, StatusCode: parsed.Status, SubStatusCode: parsed.SubStatus, FailureReason: parsed.FailureReason, ProcessPath: parsed.ProcessName, ParentProcessPath: parsed.ParentProcessName, CommandLine: truncate(parsed.CommandLine, 8192), Message: msg, Attributes: attrs, RawObjectKey: rawKey, RawIndex: uint32(idx), PayloadHash: fingerprint(p), SchemaVersion: 1, ParserVersion: 1, IngestDelayMS: env.ReceivedAt.Sub(p.Time).Milliseconds(),
}
}
func overlay(p *Parsed, m contracts.EventMetadataPayload) {
p.Computer = first(m.Computer, p.Computer)
p.Provider = first(m.ProviderName, p.Provider)
p.TargetUser = first(m.TargetUser, p.TargetUser)
p.TargetDomain = first(m.TargetDomain, p.TargetDomain)
p.SubjectUser = first(m.SubjectUser, p.SubjectUser)
p.SubjectDomain = first(m.SubjectDomain, p.SubjectDomain)
p.Workstation = first(m.Workstation, m.Device, p.Workstation)
p.SrcIP = first(m.SrcIP, p.SrcIP)
p.SrcPort = first(m.SrcPort, p.SrcPort)
p.DstIP = first(m.DstIP, p.DstIP)
p.DstPort = first(m.DstPort, p.DstPort)
p.LogonType = first(m.LogonType, p.LogonType)
p.ProcessName = first(m.ProcessName, p.ProcessName)
p.ParentProcessName = first(m.ParentProcessName, p.ParentProcessName)
p.CommandLine = first(m.CommandLine, p.CommandLine)
p.AuthPackage = first(m.AuthenticationPackage, p.AuthPackage)
p.LogonProcess = first(m.LogonProcess, p.LogonProcess)
p.Status = first(m.StatusText, p.Status)
p.SubStatus = first(m.SubStatusText, p.SubStatus)
p.FailureReason = first(m.FailureReason, p.FailureReason)
}
func parseXML(s string) Parsed {
out := Parsed{Attributes: map[string]string{}}
if !strings.HasPrefix(strings.TrimSpace(s), "<") {
return out
}
dec := xml.NewDecoder(strings.NewReader(s))
var path []string
dataName := ""
for {
tok, e := dec.Token()
if e == io.EOF {
break
}
if e != nil {
return out
}
switch t := tok.(type) {
case xml.StartElement:
path = append(path, t.Name.Local)
if t.Name.Local == "Provider" {
for _, a := range t.Attr {
if a.Name.Local == "Name" {
out.Provider = strings.TrimSpace(a.Value)
}
}
}
if t.Name.Local == "Data" {
dataName = ""
for _, a := range t.Attr {
if a.Name.Local == "Name" {
dataName = strings.TrimSpace(a.Value)
}
}
}
case xml.EndElement:
if len(path) > 0 {
path = path[:len(path)-1]
}
if t.Name.Local == "Data" {
dataName = ""
}
case xml.CharData:
v := strings.TrimSpace(string(t))
if v == "" {
continue
}
if ends(path, "System", "Computer") {
out.Computer = v
continue
}
if dataName != "" {
if !promotedField(dataName) && len(out.Attributes) < 32 {
out.Attributes[dataName] = truncate(v, 1024)
}
switch dataName {
case "TargetUserName":
out.TargetUser = v
case "TargetDomainName":
out.TargetDomain = v
case "SubjectUserName":
out.SubjectUser = v
case "SubjectDomainName":
out.SubjectDomain = v
case "WorkstationName", "CallerComputerName":
out.Workstation = v
case "IpAddress", "SourceAddress":
out.SrcIP = v
case "IpPort", "SourcePort":
out.SrcPort = v
case "DestinationAddress", "DestAddress":
out.DstIP = v
case "DestinationPort", "DestPort":
out.DstPort = v
case "LogonType":
out.LogonType = v
case "ProcessName", "NewProcessName":
out.ProcessName = v
case "ParentProcessName":
out.ParentProcessName = v
case "CommandLine", "ProcessCommandLine":
out.CommandLine = v
case "AuthenticationPackageName":
out.AuthPackage = v
case "LogonProcessName":
out.LogonProcess = v
case "Status":
out.Status = v
case "SubStatus":
out.SubStatus = v
case "FailureReason":
out.FailureReason = v
}
}
}
}
return out
}
func promotedField(name string) bool {
switch name {
case "TargetUserName", "TargetDomainName", "SubjectUserName", "SubjectDomainName",
"WorkstationName", "CallerComputerName", "IpAddress", "SourceAddress", "IpPort", "SourcePort",
"DestinationAddress", "DestAddress", "DestinationPort", "DestPort", "LogonType",
"ProcessName", "NewProcessName", "ParentProcessName", "CommandLine", "ProcessCommandLine",
"AuthenticationPackageName", "LogonProcessName", "Status", "SubStatus", "FailureReason":
return true
default:
return false
}
}
func ends(path []string, p ...string) bool {
if len(path) < len(p) {
return false
}
o := len(path) - len(p)
for i := range p {
if path[o+i] != p[i] {
return false
}
}
return true
}
func first(v ...string) string {
for _, s := range v {
if s = strings.TrimSpace(s); s != "" && s != "-" {
return s
}
}
return ""
}
func cleanUser(v string) string {
v = strings.TrimSpace(v)
if v == "-" {
return ""
}
return v
}
func normalizeIP(v string) string {
v = strings.TrimSpace(v)
if v == "" || v == "-" || v == "::1" || v == "127.0.0.1" {
return ""
}
ip := net.ParseIP(v)
if ip == nil {
return ""
}
if x := ip.To4(); x != nil {
return x.String()
}
return ip.String()
}
func parsePort(v string) uint16 {
n, e := strconv.ParseUint(strings.TrimSpace(v), 10, 16)
if e != nil {
return 0
}
return uint16(n)
}
func truncate(v string, n int) string {
if len(v) > n {
return v[:n]
}
return v
}
func fingerprint(p contracts.LogPayload) string {
b, _ := json.Marshal(p)
h := sha256.Sum256(b)
return hex.EncodeToString(h[:])
}
func classify(channel string, id uint32) (string, string, string, uint8) {
if strings.EqualFold(channel, "Security") {
switch id {
case 4624:
return "authentication", "logon", "success", 2
case 4625:
return "authentication", "logon", "failure", 3
case 4648:
return "authentication", "explicit_credentials", "unknown", 3
case 4672:
return "iam", "special_privileges", "success", 3
case 4720:
return "iam", "user_created", "success", 4
case 4726:
return "iam", "user_deleted", "success", 4
case 4728, 4732, 4756:
return "iam", "privileged_group_membership_changed", "success", 5
case 4740:
return "authentication", "account_locked", "failure", 4
case 4768:
return "authentication", "kerberos_tgt", "unknown", 2
case 4769:
return "authentication", "kerberos_service_ticket", "unknown", 2
case 4771:
return "authentication", "kerberos_preauth", "failure", 3
case 4776:
return "authentication", "credential_validation", "unknown", 2
case 1102:
return "audit", "audit_log_cleared", "success", 5
}
}
switch id {
case 7045:
return "persistence", "service_installed", "success", 5
case 1074:
return "host", "shutdown_requested", "success", 2
case 6005:
return "host", "eventlog_started", "success", 1
case 6006:
return "host", "eventlog_stopped", "success", 2
}
return "event", "observed", "unknown", 1
}
func ArchiveKey(t time.Time, agent, batchUID string, partition int, offset int64) string {
id := strings.TrimSpace(batchUID)
if id == "" {
id = "p" + strconv.Itoa(partition) + "-o" + strconv.FormatInt(offset, 10)
}
return t.UTC().Format("2006/01/02/15") + "/" + strings.ReplaceAll(agent, "/", "_") + "-" + id + ".json.gz"
}

37
internal/normalize/normalize_test.go Normal file
View File

@@ -0,0 +1,37 @@
package normalize
import (
"testing"
"time"
"example.com/siem-greenfield/internal/contracts"
)
func TestEvent4740ExtractsCallerComputer(t *testing.T) {
xml := `<Event><System><Provider Name="Microsoft-Windows-Security-Auditing"/><Computer>DC01.example.local</Computer></System><EventData><Data Name="TargetUserName">alice</Data><Data Name="CallerComputerName">CLIENT-42</Data></EventData></Event>`
env := contracts.IngestEnvelope{TenantID: "default", AgentID: "agent-1", BatchUID: "batch-abc", ReceivedAt: time.Date(2026, 7, 23, 12, 0, 1, 0, time.UTC)}
p := contracts.LogPayload{Hostname: "DC01", Channel: "Security", EventID: 4740, Source: "windows-agent", Time: time.Date(2026, 7, 23, 12, 0, 0, 0, time.UTC), Message: xml}
e := Event(env, p, "2026/07/raw.json.gz", 3, 99, 4)
if e.EventUID != "batch-abc:4" {
t.Fatalf("event uid = %q", e.EventUID)
}
if e.TargetUser != "alice" || e.Workstation != "CLIENT-42" {
t.Fatalf("unexpected normalized event: %#v", e)
}
if e.Action != "account_locked" || e.Severity != 4 {
t.Fatalf("unexpected classification: action=%s severity=%d", e.Action, e.Severity)
}
if e.Message != "" {
t.Fatalf("raw XML must not be stored as message")
}
if len(e.Attributes) != 0 {
t.Fatalf("promoted fields must not be duplicated in attributes: %#v", e.Attributes)
}
}
func TestWMI5857IsGenericNotDetectionClass(t *testing.T) {
cat, action, _, sev := classify("Microsoft-Windows-WMI-Activity/Operational", 5857)
if cat != "event" || action != "observed" || sev != 1 {
t.Fatalf("unexpected WMI classification: %s %s %d", cat, action, sev)
}
}

184
internal/postgres/store.go Normal file
View File

@@ -0,0 +1,184 @@
package postgres
import (
"context"
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"errors"
"fmt"
"net"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
var ErrUnauthorized = errors.New("unauthorized")
type Store struct{ Pool *pgxpool.Pool }
type Agent struct {
ID string `json:"id"`
Hostname string `json:"hostname"`
LastIP string `json:"last_ip"`
Enabled bool `json:"enabled"`
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
}
type Detection struct {
ID int64 `json:"id"`
Fingerprint string `json:"fingerprint"`
RuleName string `json:"rule_name"`
Severity string `json:"severity"`
Status string `json:"status"`
Hostname string `json:"hostname"`
UserName string `json:"user_name"`
SourceIP string `json:"source_ip"`
Workstation string `json:"workstation"`
Summary string `json:"summary"`
EventCode uint32 `json:"event_code"`
Score float64 `json:"score"`
WindowStart time.Time `json:"window_start"`
WindowEnd time.Time `json:"window_end"`
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
Count int64 `json:"count"`
}
func Open(ctx context.Context, url string) (*Store, error) {
p, e := pgxpool.New(ctx, url)
if e != nil {
return nil, e
}
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if e = p.Ping(cctx); e != nil {
p.Close()
return nil, e
}
return &Store{Pool: p}, nil
}
func (s *Store) Close() { s.Pool.Close() }
func (s *Store) AuthenticateOrEnroll(ctx context.Context, tenant, hostname, apiKey, enrollmentKey, expectedEnrollment, remoteIP string) (string, error) {
hostname = strings.TrimSpace(hostname)
apiKey = strings.TrimSpace(apiKey)
var id, hash string
var enabled bool
err := s.Pool.QueryRow(ctx, `SELECT id::text, api_key_hash, enabled FROM agents WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname).Scan(&id, &hash, &enabled)
if err != nil {
if !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
if enrollmentKey == "" || !secureEqual(hashHex(enrollmentKey), hashHex(expectedEnrollment)) {
return "", ErrUnauthorized
}
err = s.Pool.QueryRow(ctx, `INSERT INTO agents(tenant_id,hostname,api_key_hash,last_ip) VALUES($1,$2,$3,$4) ON CONFLICT(tenant_id,hostname) DO NOTHING RETURNING id::text`, tenant, hostname, hashHex(apiKey), remoteIP).Scan(&id)
if err == nil {
return id, nil
}
if !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
// A concurrent first request may have enrolled the host between SELECT and INSERT.
err = s.Pool.QueryRow(ctx, `SELECT id::text, api_key_hash, enabled FROM agents WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname).Scan(&id, &hash, &enabled)
if err != nil || !enabled || !secureEqual(strings.ToLower(hash), hashHex(apiKey)) {
return "", ErrUnauthorized
}
return id, nil
}
if !enabled || !secureEqual(strings.ToLower(hash), hashHex(apiKey)) {
return "", ErrUnauthorized
}
_, err = s.Pool.Exec(ctx, `UPDATE agents SET last_seen=now(), last_ip=$3 WHERE tenant_id=$1 AND hostname=$2`, tenant, hostname, remoteIP)
return id, err
}
func (s *Store) ListAgents(ctx context.Context, tenant string) ([]Agent, error) {
rows, e := s.Pool.Query(ctx, `SELECT id::text,hostname,enabled,COALESCE(last_ip,''),first_seen,last_seen FROM agents WHERE tenant_id=$1 ORDER BY last_seen DESC`, tenant)
if e != nil {
return nil, e
}
defer rows.Close()
var out []Agent
for rows.Next() {
var a Agent
if e := rows.Scan(&a.ID, &a.Hostname, &a.Enabled, &a.LastIP, &a.FirstSeen, &a.LastSeen); e != nil {
return nil, e
}
out = append(out, a)
}
return out, rows.Err()
}
func (s *Store) UpsertDetection(ctx context.Context, d Detection, tenant string) error {
_, e := s.Pool.Exec(ctx, `INSERT INTO detections(tenant_id,fingerprint,rule_name,severity,status,hostname,user_name,source_ip,workstation,event_code,score,window_start,window_end,summary,hit_count,first_seen,last_seen)
VALUES($1,$2,$3,$4,'open',$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$11,$12)
ON CONFLICT(tenant_id,fingerprint) DO UPDATE SET
first_seen=LEAST(detections.first_seen,EXCLUDED.first_seen),
last_seen=GREATEST(detections.last_seen,EXCLUDED.last_seen),
window_start=LEAST(detections.window_start,EXCLUDED.window_start),
window_end=GREATEST(detections.window_end,EXCLUDED.window_end),
hit_count=GREATEST(detections.hit_count,EXCLUDED.hit_count),
score=GREATEST(detections.score,EXCLUDED.score),
summary=EXCLUDED.summary, workstation=EXCLUDED.workstation, updated_at=now()`, tenant, d.Fingerprint, d.RuleName, d.Severity, d.Hostname, d.UserName, d.SourceIP, d.Workstation, d.EventCode, d.Score, d.WindowStart, d.WindowEnd, d.Summary, d.Count)
return e
}
func (s *Store) ListDetections(ctx context.Context, tenant string, limit int) ([]Detection, error) {
rows, e := s.Pool.Query(ctx, `SELECT id,fingerprint,rule_name,severity,status,hostname,user_name,source_ip,workstation,event_code,score,window_start,window_end,summary,hit_count,first_seen,last_seen FROM detections WHERE tenant_id=$1 ORDER BY last_seen DESC LIMIT $2`, tenant, limit)
if e != nil {
return nil, e
}
defer rows.Close()
var out []Detection
for rows.Next() {
var d Detection
if e := rows.Scan(&d.ID, &d.Fingerprint, &d.RuleName, &d.Severity, &d.Status, &d.Hostname, &d.UserName, &d.SourceIP, &d.Workstation, &d.EventCode, &d.Score, &d.WindowStart, &d.WindowEnd, &d.Summary, &d.Count, &d.FirstSeen, &d.LastSeen); e != nil {
return nil, e
}
out = append(out, d)
}
return out, rows.Err()
}
func (s *Store) UpdateDetectionStatus(ctx context.Context, tenant string, id int64, status string) error {
if status != "open" && status != "investigating" && status != "closed" && status != "false_positive" {
return fmt.Errorf("invalid status")
}
_, e := s.Pool.Exec(ctx, `UPDATE detections SET status=$3, updated_at=now() WHERE tenant_id=$1 AND id=$2`, tenant, id, status)
return e
}
func (s *Store) DetectionCounts(ctx context.Context, tenant string) (map[string]int64, error) {
rows, e := s.Pool.Query(ctx, `SELECT severity,count(*) FROM detections WHERE tenant_id=$1 AND status IN ('open','investigating') GROUP BY severity`, tenant)
if e != nil {
return nil, e
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var k string
var n int64
if e := rows.Scan(&k, &n); e != nil {
return nil, e
}
out[k] = n
}
return out, rows.Err()
}
func hashHex(v string) string { h := sha256.Sum256([]byte(v)); return hex.EncodeToString(h[:]) }
func secureEqual(a, b string) bool {
if len(a) != len(b) {
return false
}
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
func ClientIP(remote string) string {
h, _, e := net.SplitHostPort(remote)
if e == nil {
return h
}
return remote
}

88
internal/processor/processor.go Normal file
View File

@@ -0,0 +1,88 @@
package processor
import (
"compress/gzip"
"context"
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"time"
"example.com/siem-greenfield/internal/clickhouse"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/contracts"
"example.com/siem-greenfield/internal/normalize"
"example.com/siem-greenfield/internal/queue"
)
func Run(ctx context.Context, cfg config.Config) error {
ch := clickhouse.New(cfg)
con := queue.NewConsumer(cfg.KafkaBrokers, cfg.KafkaTopic, cfg.KafkaGroup)
defer con.Close()
log.Printf("processor started topic=%s group=%s", cfg.KafkaTopic, cfg.KafkaGroup)
for {
m, e := con.Fetch(ctx)
if e != nil {
if ctx.Err() != nil {
return nil
}
return e
}
var env contracts.IngestEnvelope
if e = json.Unmarshal(m.Value, &env); e != nil {
log.Printf("dropping malformed queue message offset=%d: %v", m.Offset, e)
_ = con.Commit(ctx, m)
continue
}
rawKey := ""
if cfg.RawArchiveEnabled {
rawKey = normalize.ArchiveKey(env.ReceivedAt, env.AgentID, env.BatchUID, m.Partition, m.Offset)
if e = spool(cfg, rawKey, m.Value); e != nil {
log.Printf("raw spool failed offset=%d (message will be retried): %v", m.Offset, e)
time.Sleep(time.Second)
continue
}
}
events := make([]contracts.CanonicalEvent, 0, len(env.Events))
for i, p := range env.Events {
events = append(events, normalize.Event(env, p, rawKey, m.Partition, m.Offset, i))
}
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
e = ch.InsertEvents(cctx, events)
cancel()
if e != nil {
log.Printf("clickhouse insert failed offset=%d: %v", m.Offset, e)
time.Sleep(time.Second)
continue
}
if e = con.Commit(ctx, m); e != nil {
return fmt.Errorf("commit queue offset: %w", e)
}
}
}
func spool(cfg config.Config, key string, payload []byte) error {
path := filepath.Join(cfg.RawSpoolDir, filepath.FromSlash(key))
if e := os.MkdirAll(filepath.Dir(path), 0750); e != nil {
return e
}
tmp := path + ".tmp"
f, e := os.Create(tmp)
if e != nil {
return e
}
gz := gzip.NewWriter(f)
_, e = gz.Write(payload)
if ce := gz.Close(); e == nil {
e = ce
}
if ce := f.Close(); e == nil {
e = ce
}
if e != nil {
_ = os.Remove(tmp)
return e
}
return os.Rename(tmp, path)
}

29
internal/queue/queue.go Normal file
View File

@@ -0,0 +1,29 @@
package queue
import (
"context"
"time"
"github.com/segmentio/kafka-go"
)
type Producer struct{ w *kafka.Writer }
func NewProducer(brokers []string, topic string) *Producer {
return &Producer{w: &kafka.Writer{Addr: kafka.TCP(brokers...), Topic: topic, Balancer: &kafka.Hash{}, BatchSize: 100, BatchTimeout: 10 * time.Millisecond, RequiredAcks: kafka.RequireAll, Async: false}}
}
func (p *Producer) Write(ctx context.Context, key string, value []byte) error {
return p.w.WriteMessages(ctx, kafka.Message{Key: []byte(key), Value: value, Time: time.Now().UTC()})
}
func (p *Producer) Close() error { return p.w.Close() }
type Consumer struct{ r *kafka.Reader }
func NewConsumer(brokers []string, topic, group string) *Consumer {
return &Consumer{r: kafka.NewReader(kafka.ReaderConfig{Brokers: brokers, Topic: topic, GroupID: group, MinBytes: 1e3, MaxBytes: 16e6, MaxWait: 500 * time.Millisecond, CommitInterval: 0})}
}
func (c *Consumer) Fetch(ctx context.Context) (kafka.Message, error) { return c.r.FetchMessage(ctx) }
func (c *Consumer) Commit(ctx context.Context, m kafka.Message) error {
return c.r.CommitMessages(ctx, m)
}
func (c *Consumer) Close() error { return c.r.Close() }

132
internal/stress/stress.go Normal file
View File

@@ -0,0 +1,132 @@
package stress
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"math/rand/v2"
"net/http"
"strconv"
"sync"
"sync/atomic"
"time"
"example.com/siem-greenfield/internal/config"
"example.com/siem-greenfield/internal/contracts"
)
func Run(ctx context.Context, _ config.Config, args []string) error {
fs := flag.NewFlagSet("stress-agent", flag.ContinueOnError)
url := fs.String("url", "http://127.0.0.1:8090/ingest", "")
api := fs.String("api-key", "stress-agent-key", "")
enroll := fs.String("enrollment-key", "", "")
host := fs.String("host", "SIEM-STRESS-01", "")
rate := fs.Int("rate", 1000, "events per second")
batch := fs.Int("batch", 100, "events per request")
workers := fs.Int("workers", 8, "max concurrent requests")
duration := fs.Duration("duration", time.Minute, "")
maxEvents := fs.Int64("max-events", 5_000_000, "hard event limit")
scenario := fs.String("scenario", "mixed", "")
confirm := fs.Bool("confirm-load-test", false, "")
if e := fs.Parse(args); e != nil {
return e
}
if !*confirm {
return fmt.Errorf("refusing to start without --confirm-load-test")
}
if *rate < 1 || *rate > 100000 || *batch < 1 || *batch > 1000 || *workers < 1 || *workers > 64 || *duration <= 0 || *duration > time.Hour || *maxEvents < 1 || *maxEvents > 50_000_000 {
return fmt.Errorf("unsafe load-test parameters")
}
client := &http.Client{Timeout: 10 * time.Second}
end := time.Now().Add(*duration)
var sent, failed, scheduled atomic.Uint64
sem := make(chan struct{}, *workers)
var wg sync.WaitGroup
for time.Now().Before(end) && int64(scheduled.Load()) < *maxEvents {
select {
case <-ctx.Done():
wg.Wait()
return nil
default:
}
remaining := *maxEvents - int64(scheduled.Load())
n := min(*batch, *rate)
if int64(n) > remaining {
n = int(remaining)
}
if n <= 0 {
break
}
events := make([]contracts.LogPayload, 0, n)
for i := 0; i < n; i++ {
events = append(events, makeEvent(*host, *scenario))
}
scheduled.Add(uint64(n))
sem <- struct{}{}
wg.Add(1)
go func(b []contracts.LogPayload) {
defer wg.Done()
defer func() { <-sem }()
data, _ := json.Marshal(b)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, *url, bytes.NewReader(data))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-API-Key", *api)
if *enroll != "" {
req.Header.Set("X-Enrollment-Key", *enroll)
}
resp, e := client.Do(req)
if e != nil {
failed.Add(uint64(len(b)))
return
}
resp.Body.Close()
if resp.StatusCode/100 != 2 {
failed.Add(uint64(len(b)))
return
}
sent.Add(uint64(len(b)))
}(events)
// Pace by event count, not requests. This remains correct when rate < batch.
time.Sleep(time.Duration(float64(time.Second) * float64(n) / float64(*rate)))
}
wg.Wait()
fmt.Printf("stress test finished: accepted=%d failed=%d scheduled=%d duration=%s\n", sent.Load(), failed.Load(), scheduled.Load(), *duration)
return nil
}
func makeEvent(host, scenario string) contracts.LogPayload {
id := uint32(4624)
switch scenario {
case "failed-logon":
id = 4625
case "lockout":
id = 4740
case "catalog":
id = uint32(10000 + rand.IntN(30000))
case "mixed":
ids := []uint32{4624, 4625, 4740, 4768, 4769, 7045, 5857}
id = ids[rand.IntN(len(ids))]
}
u := "user" + strconv.Itoa(rand.IntN(500))
ip := fmt.Sprintf("10.%d.%d.%d", rand.IntN(250)+1, rand.IntN(250)+1, rand.IntN(250)+1)
return contracts.LogPayload{Hostname: host, Channel: func() string {
if id == 7045 {
return "System"
}
if id == 5857 {
return "Microsoft-Windows-WMI-Activity/Operational"
}
return "Security"
}(), EventID: id, Source: "SIEM-Stress-Agent", Time: time.Now().UTC(), Metadata: &contracts.EventMetadataPayload{ProviderName: "SIEM-Stress-Agent", TargetUser: u, Workstation: "STRESS-CLIENT-" + strconv.Itoa(rand.IntN(50)), SrcIP: ip, LogonType: "3"}}
}
func min(a, b int) int {
if a < b {
return a
}
return b
}

7797
main.go
View File

File diff suppressed because it is too large Load Diff

131
main_test.go
View File

@@ -1,131 +0,0 @@
package main
import (
"testing"
"time"
)
func TestNormalizeLockoutEvent4740(t *testing.T) {
xml := `<Event><System><Provider Name="Microsoft-Windows-Security-Auditing"/><Computer>DC01.example.local</Computer></System><EventData><Data Name="TargetUserName">alice</Data><Data Name="TargetDomainName">EXAMPLE</Data><Data Name="SubjectUserName">DC01$</Data><Data Name="CallerComputerName">CLIENT-42</Data></EventData></Event>`
got := NormalizeEventXML(xml)
if got.TargetUser != "alice" {
t.Fatalf("TargetUser=%q", got.TargetUser)
}
if got.Workstation != "CLIENT-42" {
t.Fatalf("Workstation=%q", got.Workstation)
}
if got.Computer != "DC01.example.local" {
t.Fatalf("Computer=%q", got.Computer)
}
}
func TestMetadataOnlyOverlay(t *testing.T) {
got := overlayMetadata(NormalizedEvent{}, EventMetadataPayload{
Computer: "DC01",
TargetUser: "EXAMPLE\\alice",
Device: "CLIENT-42",
SrcIP: "10.0.0.42",
})
if got.Workstation != "CLIENT-42" {
t.Fatalf("Workstation=%q", got.Workstation)
}
if normalizeUsername(got.TargetUser) != "alice" {
t.Fatalf("normalized user=%q", normalizeUsername(got.TargetUser))
}
}
func TestBucketStart(t *testing.T) {
ts := time.Date(2026, 7, 18, 12, 34, 56, 0, time.UTC)
got := bucketStart(ts, time.Minute)
want := time.Date(2026, 7, 18, 12, 34, 0, 0, time.UTC)
if !got.Equal(want) {
t.Fatalf("got %s want %s", got, want)
}
}
func TestNewEventIDIgnoresWMIActivityOperational(t *testing.T) {
cfg := Config{
NewEventIDMode: "inventory",
NewEventIDMinCount: 3,
NewEventIDLearningPeriod: 24 * time.Hour,
NewEventIDIgnoreChannels: []string{"Microsoft-Windows-WMI-Activity/Operational"},
}
agentFirst := time.Date(2026, 7, 1, 10, 0, 0, 0, time.UTC)
eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC)
got := assessNewEventID(cfg, "Microsoft-Windows-WMI-Activity/Operational", 5857, 12, agentFirst, eventFirst)
if got.Emit {
t.Fatalf("WMI operational event should be ignored: %+v", got)
}
if got.Reason != "ignored_operational_channel" {
t.Fatalf("reason=%q", got.Reason)
}
}
func TestNewEventIDInventoryIsInformationalAndPlausible(t *testing.T) {
cfg := Config{
NewEventIDMode: "inventory",
NewEventIDMinCount: 3,
NewEventIDLearningPeriod: 24 * time.Hour,
}
agentFirst := time.Date(2026, 7, 1, 10, 0, 0, 0, time.UTC)
eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC)
got := assessNewEventID(cfg, "Application", 1000, 3, agentFirst, eventFirst)
if !got.Emit || got.Severity != "info" || got.Status != "plausible" {
t.Fatalf("unexpected assessment: %+v", got)
}
}
func TestNewEventIDSkipsAgentLearningPeriod(t *testing.T) {
cfg := Config{
NewEventIDMode: "alert",
NewEventIDMinCount: 1,
NewEventIDLearningPeriod: 24 * time.Hour,
}
agentFirst := time.Date(2026, 7, 18, 17, 30, 0, 0, time.UTC)
eventFirst := time.Date(2026, 7, 18, 18, 0, 0, 0, time.UTC)
got := assessNewEventID(cfg, "Security", 4719, 4, agentFirst, eventFirst)
if got.Emit || got.Reason != "agent_learning_period" {
t.Fatalf("unexpected assessment: %+v", got)
}
}
func TestMetadataContextAllowlist(t *testing.T) {
cfg := Config{MetadataContextEventIDs: map[uint32]struct{}{4624: {}, 4740: {}}}
if !shouldStoreMetadataContext(cfg, "Security", 4740) {
t.Fatal("Security/4740 should retain compact context")
}
if shouldStoreMetadataContext(cfg, "Microsoft-Windows-WMI-Activity/Operational", 4740) {
t.Fatal("non-Security channel must not create context rows")
}
if shouldStoreMetadataContext(cfg, "Security", 4688) {
t.Fatal("event outside allowlist must stay count-only")
}
}
func TestCompactLockoutContextDropsNoise(t *testing.T) {
norm := NormalizedEvent{
TargetUser: "EXAMPLE\\Alice",
SubjectUser: "DC01$",
Workstation: "CLIENT-42",
SrcIP: "10.0.0.10",
LogonType: "3",
StatusText: "status-varies",
FailureReason: "large noisy reason",
}
target, subject, srcIP, workstation, logonType, statusText, failureReason := compactMetadataContext(4740, norm)
if target != "alice" || workstation != "CLIENT-42" {
t.Fatalf("unexpected lockout identity target=%q workstation=%q", target, workstation)
}
if subject != "" || srcIP != "" || logonType != "" || statusText != "" || failureReason != "" {
t.Fatalf("lockout context retained high-cardinality noise: subject=%q src=%q logon=%q status=%q reason=%q", subject, srcIP, logonType, statusText, failureReason)
}
}
func TestRetainCompactContextDropsMachineLogons(t *testing.T) {
if retainCompactContext(4624, "server01$", "") {
t.Fatal("machine-account logon context should not be retained")
}
if !retainCompactContext(4624, "alice", "") {
t.Fatal("human logon context should be retained")
}
}

11
raw-event.sh Executable file
View File

@@ -0,0 +1,11 @@
#!/usr/bin/env sh
set -eu
cd "$(dirname "$0")"
if [ "$#" -lt 1 ]; then echo "usage: $0 <raw_object_key> [raw_index]" >&2; exit 2; fi
key="$1"; idx="${2:-}"
[ -f .env ] && { set -a; . ./.env; set +a; }
if [ -n "$idx" ] && command -v python3 >/dev/null 2>&1; then
docker compose exec -T archive-uploader rclone cat "garage:${GARAGE_BUCKET:-siem-raw}/$key" | gzip -dc | python3 -c 'import json,sys; d=json.load(sys.stdin); i=int(sys.argv[1]); print(json.dumps(d["events"][i], ensure_ascii=False, indent=2))' "$idx"
else
docker compose exec -T archive-uploader rclone cat "garage:${GARAGE_BUCKET:-siem-raw}/$key" | gzip -dc
fi

8
reset-DANGEROUS.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/usr/bin/env sh
set -eu
cd "$(dirname "$0")"
printf 'ACHTUNG: Alle SIEM-Daten löschen. Tippe DELETE: '
read x
[ "$x" = "DELETE" ] || exit 1
docker compose down -v --remove-orphans
rm -f deploy/garage/garage.toml

101
schema.legacy.sql
View File

@@ -1,101 +0,0 @@
CREATE DATABASE IF NOT EXISTS eventcollector
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
USE eventcollector;
CREATE TABLE IF NOT EXISTS agents (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
hostname VARCHAR(255) NOT NULL,
api_key_hash CHAR(64) NOT NULL,
first_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
last_seen DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
last_ip VARCHAR(64) NOT NULL DEFAULT '',
is_enabled TINYINT(1) NOT NULL DEFAULT 1,
PRIMARY KEY (id),
UNIQUE KEY ux_agents_hostname (hostname),
KEY ix_agents_last_seen (last_seen)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CREATE TABLE IF NOT EXISTS event_logs (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
agent_id BIGINT UNSIGNED NOT NULL,
hostname VARCHAR(255) NOT NULL,
channel_name VARCHAR(128) NOT NULL,
event_id INT UNSIGNED NOT NULL,
source VARCHAR(255) NOT NULL,
ts DATETIME(6) NOT NULL,
received_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
msg LONGTEXT NOT NULL,
msg_sha256 CHAR(64) NOT NULL,
PRIMARY KEY (id),
KEY ix_event_logs_ts (ts),
KEY ix_event_logs_received_at (received_at),
KEY ix_event_logs_agent_ts (agent_id, ts),
KEY ix_event_logs_eventid_ts (event_id, ts),
KEY ix_event_logs_hostname_ts (hostname, ts),
KEY ix_event_logs_channel_event_ts (channel_name, event_id, ts),
CONSTRAINT fk_event_logs_agent
FOREIGN KEY (agent_id) REFERENCES agents(id)
ON DELETE RESTRICT
ON UPDATE RESTRICT
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CREATE TABLE IF NOT EXISTS detections (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
rule_name VARCHAR(128) NOT NULL,
severity VARCHAR(32) NOT NULL,
hostname VARCHAR(255) NOT NULL,
channel_name VARCHAR(128) NOT NULL DEFAULT '',
event_id INT UNSIGNED NOT NULL DEFAULT 0,
score DOUBLE NOT NULL DEFAULT 0,
window_start DATETIME(6) NOT NULL,
window_end DATETIME(6) NOT NULL,
summary VARCHAR(512) NOT NULL,
details_json JSON NOT NULL,
created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
PRIMARY KEY (id),
UNIQUE KEY ux_detection_dedupe (rule_name, hostname, channel_name, event_id, window_start, window_end),
KEY ix_detections_created (created_at),
KEY ix_detections_rule_host_time (rule_name, hostname, created_at),
KEY ix_detections_severity_time (severity, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
USE eventcollector;
INSERT INTO agents (hostname, api_key_hash)
VALUES
('client01.domain.local', SHA2('SUPER-LANGER-AGENT-KEY-01', 256)),
('client02.domain.local', SHA2('SUPER-LANGER-AGENT-KEY-02', 256));
#V2
ALTER TABLE event_logs
ADD COLUMN computer VARCHAR(255) NOT NULL DEFAULT '' AFTER source,
ADD COLUMN provider_name VARCHAR(255) NOT NULL DEFAULT '' AFTER computer,
ADD COLUMN level_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER provider_name,
ADD COLUMN task_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER level_value,
ADD COLUMN opcode_value INT UNSIGNED NOT NULL DEFAULT 0 AFTER task_value,
ADD COLUMN keywords VARCHAR(255) NOT NULL DEFAULT '' AFTER opcode_value,
ADD COLUMN target_user VARCHAR(255) NOT NULL DEFAULT '' AFTER keywords,
ADD COLUMN target_domain VARCHAR(255) NOT NULL DEFAULT '' AFTER target_user,
ADD COLUMN subject_user VARCHAR(255) NOT NULL DEFAULT '' AFTER target_domain,
ADD COLUMN subject_domain VARCHAR(255) NOT NULL DEFAULT '' AFTER subject_user,
ADD COLUMN workstation VARCHAR(255) NOT NULL DEFAULT '' AFTER subject_domain,
ADD COLUMN src_ip VARCHAR(64) NOT NULL DEFAULT '' AFTER workstation,
ADD COLUMN src_port VARCHAR(32) NOT NULL DEFAULT '' AFTER src_ip,
ADD COLUMN logon_type VARCHAR(32) NOT NULL DEFAULT '' AFTER src_port,
ADD COLUMN process_name VARCHAR(512) NOT NULL DEFAULT '' AFTER logon_type,
ADD COLUMN authentication_package VARCHAR(128) NOT NULL DEFAULT '' AFTER process_name,
ADD COLUMN logon_process VARCHAR(128) NOT NULL DEFAULT '' AFTER authentication_package,
ADD COLUMN status_text VARCHAR(64) NOT NULL DEFAULT '' AFTER logon_process,
ADD COLUMN sub_status_text VARCHAR(64) NOT NULL DEFAULT '' AFTER status_text,
ADD COLUMN failure_reason VARCHAR(512) NOT NULL DEFAULT '' AFTER sub_status_text;
ALTER TABLE event_logs
ADD KEY ix_event_logs_target_user_ts (target_user, ts),
ADD KEY ix_event_logs_src_ip_ts (src_ip, ts),
ADD KEY ix_event_logs_target_user_src_ip_ts (target_user, src_ip, ts),
ADD KEY ix_event_logs_eventid_srcip_ts (event_id, src_ip, ts),
ADD KEY ix_event_logs_eventid_targetuser_ts (event_id, target_user, ts),
ADD KEY ix_event_logs_eventid_logontype_ts (event_id, logon_type, ts);

11
schema.sql
View File

@@ -1,11 +0,0 @@
-- Canonical schema notice
--
-- New empty database:
-- deploy/mariadb/init/001-schema.sql
--
-- Existing database:
-- deploy/mariadb/migrations/002-metadata-first.sql
--
-- The former schema.sql was an obsolete, non-idempotent development schema and
-- has been retained as schema.legacy.sql only for historical reference. Do not
-- run schema.legacy.sql against a production database.

8
status.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/usr/bin/env sh
set -eu
cd "$(dirname "$0")"
docker compose ps
printf '\nIngress readiness:\n'
curl -fsS http://127.0.0.1:$(grep '^INGRESS_PORT=' .env | cut -d= -f2-)/readyz || true
printf '\n\nRecent errors:\n'
docker compose logs --since=10m ingress processor detector api 2>&1 | grep -Ei 'error|fatal|failed|panic' | tail -50 || true

4
stop.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/usr/bin/env sh
set -eu
cd "$(dirname "$0")"
docker compose down

22
web/index.html Normal file
View File

@@ -0,0 +1,22 @@
<!doctype html>
<html lang="de"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Greenfield SIEM</title>
<style>
:root{font-family:Inter,Segoe UI,Arial,sans-serif;color:#172033;background:#f4f7fb}*{box-sizing:border-box}body{margin:0}header{background:#101828;color:white;padding:16px 24px;display:flex;justify-content:space-between;align-items:center}header b{font-size:18px}main{padding:22px;max-width:1800px;margin:auto}.cards{display:grid;grid-template-columns:repeat(4,minmax(170px,1fr));gap:12px}.card,.panel{background:white;border:1px solid #e4e7ec;border-radius:10px;padding:16px;box-shadow:0 1px 2px #1018280f}.value{font-size:28px;font-weight:700;margin-top:8px}.muted{color:#667085;font-size:13px}.tabs{display:flex;gap:8px;margin:18px 0}.tabs button,button{border:1px solid #d0d5dd;background:white;border-radius:7px;padding:8px 12px;cursor:pointer}.tabs button.active,button.primary{background:#175cd3;color:white;border-color:#175cd3}.filters{display:grid;grid-template-columns:repeat(6,minmax(120px,1fr));gap:8px;margin-bottom:10px}input,select{width:100%;padding:8px;border:1px solid #d0d5dd;border-radius:7px}table{width:100%;border-collapse:collapse;font-size:13px}th,td{text-align:left;border-bottom:1px solid #eaecf0;padding:8px;vertical-align:top}th{color:#475467;background:#f9fafb;position:sticky;top:0}.tablewrap{max-height:65vh;overflow:auto}.sev-critical{color:#b42318;font-weight:700}.sev-high{color:#c4320a;font-weight:700}.sev-medium{color:#b54708;font-weight:700}.pill{display:inline-block;padding:2px 7px;border-radius:999px;background:#eef2f6;font-size:12px}.hidden{display:none}.error{background:#fef3f2;color:#b42318;padding:10px;border-radius:6px;margin:10px 0}@media(max-width:900px){.cards{grid-template-columns:1fr 1fr}.filters{grid-template-columns:1fr 1fr}}
</style></head><body>
<header><b>Greenfield SIEM</b><span class="muted" style="color:#d0d5dd">Tenant: {{.Tenant}}</span></header>
<main><div id="err"></div><div class="cards"><div class="card"><div class="muted">Events / 24h</div><div id="events24" class="value">–</div></div><div class="card"><div class="muted">Aktive Hosts / 24h</div><div id="hosts24" class="value">–</div></div><div class="card"><div class="muted">Offene High/Critical</div><div id="highdet" class="value">–</div></div><div class="card"><div class="muted">Pipeline</div><div id="pipeline" class="value">OK</div></div></div>
<div class="tabs"><button class="active" data-tab="events">Events</button><button data-tab="detections">Detections</button><button data-tab="agents">Agents</button></div>
<section id="events" class="panel"><div class="filters"><input id="fhost" placeholder="Host"><input id="fuser" placeholder="User"><input id="fip" placeholder="Source-IP"><input id="fcode" placeholder="Event-ID"><input id="fchannel" placeholder="Channel"><select id="fhours"><option value="1">1 Stunde</option><option value="24" selected>24 Stunden</option><option value="168">7 Tage</option><option value="720">30 Tage</option><option value="2160">90 Tage</option></select></div><button class="primary" onclick="loadEvents()">Suchen</button><div class="tablewrap"><table><thead><tr><th>Zeit</th><th>Host</th><th>Event</th><th>Aktion</th><th>User</th><th>Source-IP</th><th>Workstation</th><th>Prozess</th></tr></thead><tbody id="eventrows"></tbody></table></div></section>
<section id="detections" class="panel hidden"><div class="tablewrap"><table><thead><tr><th>Zuletzt</th><th>Schwere</th><th>Regel</th><th>Host/User</th><th>Zusammenfassung</th><th>Status</th></tr></thead><tbody id="detrows"></tbody></table></div></section>
<section id="agents" class="panel hidden"><div class="tablewrap"><table><thead><tr><th>Host</th><th>Enabled</th><th>IP</th><th>First seen</th><th>Last seen</th></tr></thead><tbody id="agentrows"></tbody></table></div></section></main>
<script>
const esc=s=>String(s??'').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
async function get(url,opt){const r=await fetch(url,opt);const d=await r.json();if(!r.ok)throw new Error(d.error||r.statusText);return d}
function fail(e){document.getElementById('err').innerHTML='<div class="error">'+esc(e.message)+'</div>';document.getElementById('pipeline').textContent='Fehler'}
async function summary(){try{const d=await get('/api/summary');events24.textContent=Number(d.events_24h||0).toLocaleString('de-DE');hosts24.textContent=Number(d.active_hosts||0).toLocaleString('de-DE');const x=d.detections||{};highdet.textContent=Number((x.high||0)+(x.critical||0)).toLocaleString('de-DE')}catch(e){fail(e)}}
async function loadEvents(){try{const q=new URLSearchParams({hours:fhours.value,limit:'500'});[['host',fhost],['user',fuser],['ip',fip],['event_code',fcode],['channel',fchannel]].forEach(([k,x])=>{if(x.value.trim())q.set(k,x.value.trim())});const d=await get('/api/events?'+q);eventrows.innerHTML=d.map(x=>`<tr><td>${esc(x.event_time)}</td><td>${esc(x.host_name)}</td><td><b>${esc(x.event_code)}</b><br><span class="pill">${esc(x.channel)}</span></td><td>${esc(x.category)} / ${esc(x.action)}<br><span class="muted">${esc(x.outcome)}</span></td><td>${esc(x.user_name||x.target_user||x.subject_user)}</td><td>${esc(x.source_ip)}</td><td>${esc(x.workstation)}</td><td>${esc(x.process_path||x.message)}</td></tr>`).join('')}catch(e){fail(e)}}
async function loadDetections(){try{const d=await get('/api/detections');detrows.innerHTML=d.map(x=>`<tr><td>${esc(x.LastSeen||x.last_seen)}</td><td class="sev-${esc(x.Severity||x.severity)}">${esc(x.Severity||x.severity)}</td><td>${esc(x.RuleName||x.rule_name)}</td><td>${esc(x.Hostname||x.hostname)}<br>${esc(x.UserName||x.user_name)}<br><span class="muted">${esc(x.Workstation||x.workstation||x.SourceIP||x.source_ip)}</span></td><td>${esc(x.Summary||x.summary)}</td><td><select onchange="setStatus(${Number(x.ID||x.id)},this.value)">${['open','investigating','closed','false_positive'].map(s=>`<option ${s==(x.Status||x.status)?'selected':''}>${s}</option>`).join('')}</select></td></tr>`).join('')}catch(e){fail(e)}}
async function setStatus(id,status){try{await get('/api/detections/status',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({id,status})});summary()}catch(e){fail(e)}}
async function loadAgents(){try{const d=await get('/api/agents');agentrows.innerHTML=d.map(x=>`<tr><td>${esc(x.hostname||x.Hostname)}</td><td>${(x.enabled??x.Enabled)?'ja':'nein'}</td><td>${esc(x.last_ip||x.LastIP)}</td><td>${esc(x.first_seen||x.FirstSeen)}</td><td>${esc(x.last_seen||x.LastSeen)}</td></tr>`).join('')}catch(e){fail(e)}}
document.querySelectorAll('.tabs button').forEach(b=>b.onclick=()=>{document.querySelectorAll('.tabs button').forEach(x=>x.classList.remove('active'));b.classList.add('active');document.querySelectorAll('main section').forEach(x=>x.classList.add('hidden'));document.getElementById(b.dataset.tab).classList.remove('hidden');if(b.dataset.tab==='detections')loadDetections();if(b.dataset.tab==='agents')loadAgents()});summary();loadEvents();setInterval(summary,15000);
</script></body></html>