diff --git a/docs.json b/docs.json index fe240e3..566e7b9 100644 --- a/docs.json +++ b/docs.json @@ -133,7 +133,13 @@ "manage/analytics/access", "manage/analytics/connection", "manage/analytics/action", - "manage/analytics/streaming" + { + "group": "Event Streaming", + "pages": [ + "manage/analytics/streaming", + "manage/analytics/streaming/http" + ] + } ] }, { diff --git a/manage/analytics/streaming.mdx b/manage/analytics/streaming.mdx index 8e51636..d587ae3 100644 --- a/manage/analytics/streaming.mdx +++ b/manage/analytics/streaming.mdx @@ -1,5 +1,5 @@ --- -title: "Log Streaming" +title: "Event Streaming" description: "Stream Pangolin log events to external collectors and SIEM tools" --- @@ -7,55 +7,36 @@ import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx"; -Log streaming sends your organization’s log events to third-party data collectors such as Datadog, Splunk, or Microsoft Sentinel—often used for SIEM-style monitoring and analysis. You define a destination, a delivery method (for example HTTP, S3, or a vendor-specific integration), and which Pangolin log types to forward: access logs, action logs, connection logs, or request logs. Pangolin pushes events to your external service as they are generated. +Log streaming forwards your organization's audit logs to external data collectors such as Datadog, Splunk, Microsoft Sentinel, Elastic, or any HTTP endpoint you operate. You add a **destination** (how events are delivered), choose which **log types** to include, and Pangolin pushes new events as they are recorded. -Log streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition). +Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition). -## Event Streaming in the dashboard +## In the dashboard -In the dashboard, this feature appears under Organization → Logs & Analytics → Streaming as Event Streaming. From there you add destinations and configure how events are delivered. +Open **Organization → Logs & Analytics → Streaming** to add destinations and monitor delivery status. Each destination has its own connection settings, optional body customization (where supported), and log-type selection. -## HTTP destination (example) +## Log types -The steps below use an HTTP webhook only as an example. Other destination types (object storage, vendor APIs, and so on) follow the same general idea—pick a destination, configure connection details, and choose log types—but the exact fields and options differ by implementation. - -### Choose a destination type - -Open Add Destination and select how events should be delivered. HTTP webhook is one option; additional destination types may appear over time. - - - - - -### Configure the connection - -On the Settings tab, set a name, the endpoint URL, and authentication (none, bearer token, basic auth, or a custom header). Requests use JSON by default unless you change it elsewhere. - - - - - -### Headers, body, and log types - -- **Headers** — Optional custom headers on every request (for example static API keys or a non-default `Content-Type`). By default, `Content-Type: application/json` is sent. -- **Body** — Optionally use a custom JSON body template with variables; you can also choose how batched events are serialized (for example a JSON array versus newline-delimited JSON for tools that expect that format). - - - - - - - - - -On the Logs tab, choose which log categories are forwarded to this destination. Only log types that are enabled for your organization can be streamed. +You choose which categories each destination receives. Only log types enabled for your organization can be streamed. -## Vendor-specific setups +## Destination types -For Amazon S3, Datadog, Microsoft Sentinel, or other provider-specific implementations and guidance, contact [sales@pangolin.net](mailto:sales@pangolin.net). +Each destination type has its own configuration and payload behavior. Select **Add destination** and pick a delivery method. + + + + + +### Other destinations + +## General behavior + +- **No backfill:** New destinations start from the current log cursor. Historical logs already in Pangolin are not replayed. +- **Per-log-type cursors:** Each enabled log type on a destination is tracked independently. +- **Errors in the UI:** When delivery fails, the destination's last error is shown in the dashboard so you can fix configuration or endpoint issues. \ No newline at end of file diff --git a/manage/analytics/streaming/http.mdx b/manage/analytics/streaming/http.mdx new file mode 100644 index 0000000..61c6914 --- /dev/null +++ b/manage/analytics/streaming/http.mdx @@ -0,0 +1,203 @@ +--- +title: "HTTP webhook" +description: "Forward audit logs to any HTTP endpoint with optional custom body templates" +--- + +import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx"; + + + +HTTP destinations POST your organization’s audit logs to a URL you control. Use them for generic webhooks, Splunk HEC, Elastic or OpenSearch ingest, Grafana Loki push endpoints, or any receiver that accepts JSON over HTTP. + + +Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition). + + +## Overview + +An HTTP destination sends **POST** requests to your endpoint. Configure: + +1. **Settings:** Name, URL, and authentication. +2. **Headers:** Optional static headers on every request. +3. **Body:** Default JSON shape or a custom body template, plus payload format (how batches are packaged). +4. **Logs:** Which log types (`request`, `access`, `action`, `connection`) are forwarded. + +Enable **Custom body template** when your receiver expects a different JSON layout than Pangolin’s default. Leave it off to send the standard `{ event, timestamp, data }` object per log record. + +## Configure the connection + +On the **Settings** tab, set a display name, the endpoint URL, and authentication: + +| Auth type | Behavior | +| --- | --- | +| None | No `Authorization` header | +| Bearer token | `Authorization: Bearer ` | +| Basic auth | `Authorization: Basic ` | +| Custom header | A single header name and value (for example an API key header) | + + + + + +All delivery uses **POST**. Requests time out after 30 seconds. + +## Authentication and headers + +On the **Headers** tab, add optional static headers sent with every request, for example a vendor-specific API key or a non-default `Content-Type`. When you do not override it, Pangolin sends `Content-Type: application/json` (or `application/x-ndjson` when using the NDJSON payload format). + + + + + +## Default payload (template off) + +When custom body template is disabled, each log event is serialized as: + +```json +{ + "event": "request", + "timestamp": "2025-06-15T12:34:56.789Z", + "data": { + "timestamp": 1718454896, + "action": true, + "method": "GET", + "path": "/api/health" + } +} +``` + +| Field | Meaning | +| --- | --- | +| `event` | Log type: `request`, `access`, `action`, or `connection` | +| `timestamp` | Event time as ISO-8601 UTC | +| `data` | The **complete stored log row** for that record, not a curated subset | + +The field set inside `data` depends on the log type. The same destination can stream multiple types; batches may contain heterogeneous `data` shapes. See [Log type reference](#log-type-reference) below and the dedicated log docs for full field lists. + + +Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them on the receiver if you need structured fields. + + +## Custom body template + +On the **Body** tab, enable **Custom body template** and provide a JSON template string. Pangolin performs simple placeholder substitution, **not** a full templating language like Handlebars. + + + + + +### Template variables + +Only these three placeholders are supported: + +| Variable | Source | How to use in the template | +| --- | --- | --- | +| `{{event}}` | Log type (`request`, `access`, `action`, `connection`) | Inside JSON **string quotes** | +| `{{timestamp}}` | Event time (ISO-8601 UTC) | Inside JSON **string quotes** | +| `{{data}}` | Full log row as JSON | **Never wrap in quotes**; inlined as raw JSON | + +**Canonical example** (equivalent to the default payload): + +```json +{ + "event": "{{event}}", + "timestamp": "{{timestamp}}", + "data": {{data}} +} +``` + +**Remapping property names** for a downstream schema: + +```json +{ + "type": "{{event}}", + "ts": "{{timestamp}}", + "payload": {{data}} +} +``` + +You may use the same token multiple times and nest placeholders at any depth in your JSON structure. Nested objects and arrays **inside** the substituted `{{data}}` value are preserved from the log row. + +### Rules and constraints + +- **Simple substitution only:** No conditionals, loops, filters, or expressions. +- **No field paths:** Placeholders like `{{data.orgId}}`, `{{orgId}}`, or `{{ip}}` do **not** work. To use a single field, read it from the full `data` object on the receiver or transform after ingest. +- **Quote `{{data}}` correctly:** `"field": {{data}}` is valid; `"field": "{{data}}"` stringifies the object incorrectly and produces invalid or useless JSON. +- **One template per destination:** The same template applies to every log type enabled on that destination. You cannot define different templates per log type on one HTTP destination. +- **String escaping:** `{{event}}` and `{{timestamp}}` are JSON-escaped for safe use inside quoted strings. +- **Invalid JSON:** Pangolin does not validate templates at save time. If the rendered body is not valid JSON, delivery may still occur but your receiver may reject it. Validate templates with a JSON linter before saving. +- **Not available on other destination types:** Body templates apply to HTTP streaming only, not S3 or Datadog destinations. + +## Payload format + +Payload format is separate from the body template. The template defines the shape of **one event**; payload format controls **how many events** are sent per HTTP request. + +| Format | HTTP body | Content-Type | +| --- | --- | --- | +| **JSON array** (default) | One POST per batch: `[{…}, {…}, …]` | `application/json` | +| **NDJSON** | One JSON object per line, no outer array | `application/x-ndjson` | +| **One event per request** | Separate POST for each event | `application/json` | + +The template is applied once per event, then results are batched into an array, joined as NDJSON lines, or sent individually, depending on the format you select. + +Choose **NDJSON** for aggregators that expect newline-delimited ingest (Splunk HEC, Elastic/OpenSearch bulk-style HTTP inputs, Loki). Choose **one event per request** when the endpoint cannot accept batches. + +## Log type reference + +The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape. + +## Integration examples + +### Generic webhook (default shape, JSON array) + +Leave custom body template disabled. Select **JSON array** payload format. Point the destination at your webhook URL with bearer or custom-header auth. + +Each batch POST body looks like: + +```json +[ + { + "event": "action", + "timestamp": "2025-06-15T12:34:56.789Z", + "data": { "action": "updateUser", "actor": "admin@example.com" } + } +] +``` + +### Log aggregator (NDJSON, minimal template) + +Enable a custom template and select **NDJSON**: + +```json +{ + "type": "{{event}}", + "ts": "{{timestamp}}", + "payload": {{data}} +} +``` + +Each line in the POST body is one rendered event. Set any vendor-required headers on the **Headers** tab. + +### Vendor schema remapping + +If a tool expects your log row under a specific key, wrap `{{data}}` without quotes: + +```json +{ + "source": "pangolin", + "sourcetype": "_json", + "time": "{{timestamp}}", + "event": {{data}} +} +``` + +Adjust property names to match the vendor; field extraction beyond the three template variables happens on the receiver. + +## Limitations and troubleshooting + +- **Field selection:** Cannot pick individual columns in the template. Use full `{{data}}` or transform after delivery. +- **Mixed log types:** Enabling multiple log types on one destination produces heterogeneous `data` in the same batch. Enable one type per destination if your pipeline expects a uniform schema. +- **Historical logs:** New destinations do not backfill. Only events recorded after the destination is created are streamed. +- **Delivery errors:** Check the destination’s **last error** in the dashboard. Common causes: wrong URL, auth failure, TLS issues, or receiver rejecting malformed JSON. +- **Quoting `{{data}}`:** `"payload": "{{data}}"` treats the entire row as a string, which is almost always wrong. Use `"payload": {{data}}`. +- **Splunk field extraction:** Pangolin does not emit Splunk-style indexed fields in the template. Parse `data` or use a receiver-side pipeline. \ No newline at end of file