add http seim destination docs

docs.json

@@ -133,7 +133,13 @@
                                     "manage/analytics/access",
                                     "manage/analytics/connection",
                                     "manage/analytics/action",
-                                    "manage/analytics/streaming"
+                                    {
+                                        "group": "Event Streaming",
+                                        "pages": [
+                                            "manage/analytics/streaming",
+                                            "manage/analytics/streaming/http"
+                                        ]
+                                    }
                                 ]
                             },
                             {

manage/analytics/streaming.mdx

@@ -1,5 +1,5 @@
 ---
-title: "Log Streaming"
+title: "Event Streaming"
 description: "Stream Pangolin log events to external collectors and SIEM tools"
 ---
 
@@ -7,55 +7,36 @@ import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx";
 
 <PangolinCloudTocCta />
 
-Log streaming sends your organization’s log events to third-party data collectors such as Datadog, Splunk, or Microsoft Sentinel—often used for SIEM-style monitoring and analysis. You define a destination, a delivery method (for example HTTP, S3, or a vendor-specific integration), and which Pangolin log types to forward: access logs, action logs, connection logs, or request logs. Pangolin pushes events to your external service as they are generated.
+Log streaming forwards your organization's audit logs to external data collectors such as Datadog, Splunk, Microsoft Sentinel, Elastic, or any HTTP endpoint you operate. You add a **destination** (how events are delivered), choose which **log types** to include, and Pangolin pushes new events as they are recorded.
 
 <Note>
-Log streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
+Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
 </Note>
 
-## Event Streaming in the dashboard
+## In the dashboard
 
-In the dashboard, this feature appears under Organization → Logs & Analytics → Streaming as Event Streaming. From there you add destinations and configure how events are delivered.
+Open **Organization → Logs & Analytics → Streaming** to add destinations and monitor delivery status. Each destination has its own connection settings, optional body customization (where supported), and log-type selection.
 
-## HTTP destination (example)
+## Log types
 
-The steps below use an HTTP webhook only as an example. Other destination types (object storage, vendor APIs, and so on) follow the same general idea—pick a destination, configure connection details, and choose log types—but the exact fields and options differ by implementation.
+You choose which categories each destination receives. Only log types enabled for your organization can be streamed.
-
-### Choose a destination type
-
-Open Add Destination and select how events should be delivered. HTTP webhook is one option; additional destination types may appear over time.
-
-<Frame>
-  <img src="/images/streaming-add-destination.png" centered />
-</Frame>
-
-### Configure the connection
-
-On the Settings tab, set a name, the endpoint URL, and authentication (none, bearer token, basic auth, or a custom header). Requests use JSON by default unless you change it elsewhere.
-
-<Frame>
-  <img src="/images/streaming-http-settings.png" centered />
-</Frame>
-
-### Headers, body, and log types
-
-- **Headers** — Optional custom headers on every request (for example static API keys or a non-default `Content-Type`). By default, `Content-Type: application/json` is sent.
-- **Body** — Optionally use a custom JSON body template with variables; you can also choose how batched events are serialized (for example a JSON array versus newline-delimited JSON for tools that expect that format).
-
-<Frame>
-  <img src="/images/streaming-http-headers.png" centered />
-</Frame>
-
-<Frame>
-  <img src="/images/streaming-http-body.png" centered />
-</Frame>
-
-On the Logs tab, choose which log categories are forwarded to this destination. Only log types that are enabled for your organization can be streamed.
 
 <Frame>
   <img src="/images/streaming-log-types.png" centered />
 </Frame>
 
-## Vendor-specific setups
+## Destination types
 
-For Amazon S3, Datadog, Microsoft Sentinel, or other provider-specific implementations and guidance, contact [sales@pangolin.net](mailto:sales@pangolin.net).
+Each destination type has its own configuration and payload behavior. Select **Add destination** and pick a delivery method.
+
+<Frame>
+  <img src="/images/streaming-add-destination.png" centered />
+</Frame>
+
+### Other destinations
+
+## General behavior
+
+- **No backfill:** New destinations start from the current log cursor. Historical logs already in Pangolin are not replayed.
+- **Per-log-type cursors:** Each enabled log type on a destination is tracked independently.
+- **Errors in the UI:** When delivery fails, the destination's last error is shown in the dashboard so you can fix configuration or endpoint issues.

manage/analytics/streaming/http.mdx

@@ -0,0 +1,203 @@
+---
+title: "HTTP webhook"
+description: "Forward audit logs to any HTTP endpoint with optional custom body templates"
+---
+
+import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx";
+
+<PangolinCloudTocCta />
+
+HTTP destinations POST your organization’s audit logs to a URL you control. Use them for generic webhooks, Splunk HEC, Elastic or OpenSearch ingest, Grafana Loki push endpoints, or any receiver that accepts JSON over HTTP.
+
+<Note>
+Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
+</Note>
+
+## Overview
+
+An HTTP destination sends **POST** requests to your endpoint. Configure:
+
+1. **Settings:** Name, URL, and authentication.
+2. **Headers:** Optional static headers on every request.
+3. **Body:** Default JSON shape or a custom body template, plus payload format (how batches are packaged).
+4. **Logs:** Which log types (`request`, `access`, `action`, `connection`) are forwarded.
+
+Enable **Custom body template** when your receiver expects a different JSON layout than Pangolin’s default. Leave it off to send the standard `{ event, timestamp, data }` object per log record.
+
+## Configure the connection
+
+On the **Settings** tab, set a display name, the endpoint URL, and authentication:
+
+| Auth type | Behavior |
+| --- | --- |
+| None | No `Authorization` header |
+| Bearer token | `Authorization: Bearer <token>` |
+| Basic auth | `Authorization: Basic <base64(user:password)>` |
+| Custom header | A single header name and value (for example an API key header) |
+
+<Frame>
+  <img src="/images/streaming-http-settings.png" centered />
+</Frame>
+
+All delivery uses **POST**. Requests time out after 30 seconds.
+
+## Authentication and headers
+
+On the **Headers** tab, add optional static headers sent with every request, for example a vendor-specific API key or a non-default `Content-Type`. When you do not override it, Pangolin sends `Content-Type: application/json` (or `application/x-ndjson` when using the NDJSON payload format).
+
+<Frame>
+  <img src="/images/streaming-http-headers.png" centered />
+</Frame>
+
+## Default payload (template off)
+
+When custom body template is disabled, each log event is serialized as:
+
+```json
+{
+  "event": "request",
+  "timestamp": "2025-06-15T12:34:56.789Z",
+  "data": {
+    "timestamp": 1718454896,
+    "action": true,
+    "method": "GET",
+    "path": "/api/health"
+  }
+}
+```
+
+| Field | Meaning |
+| --- | --- |
+| `event` | Log type: `request`, `access`, `action`, or `connection` |
+| `timestamp` | Event time as ISO-8601 UTC |
+| `data` | The **complete stored log row** for that record, not a curated subset |
+
+The field set inside `data` depends on the log type. The same destination can stream multiple types; batches may contain heterogeneous `data` shapes. See [Log type reference](#log-type-reference) below and the dedicated log docs for full field lists.
+
+<Warning>
+Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them on the receiver if you need structured fields.
+</Warning>
+
+## Custom body template
+
+On the **Body** tab, enable **Custom body template** and provide a JSON template string. Pangolin performs simple placeholder substitution, **not** a full templating language like Handlebars.
+
+<Frame>
+  <img src="/images/streaming-http-body.png" centered />
+</Frame>
+
+### Template variables
+
+Only these three placeholders are supported:
+
+| Variable | Source | How to use in the template |
+| --- | --- | --- |
+| `{{event}}` | Log type (`request`, `access`, `action`, `connection`) | Inside JSON **string quotes** |
+| `{{timestamp}}` | Event time (ISO-8601 UTC) | Inside JSON **string quotes** |
+| `{{data}}` | Full log row as JSON | **Never wrap in quotes**; inlined as raw JSON |
+
+**Canonical example** (equivalent to the default payload):
+
+```json
+{
+  "event": "{{event}}",
+  "timestamp": "{{timestamp}}",
+  "data": {{data}}
+}
+```
+
+**Remapping property names** for a downstream schema:
+
+```json
+{
+  "type": "{{event}}",
+  "ts": "{{timestamp}}",
+  "payload": {{data}}
+}
+```
+
+You may use the same token multiple times and nest placeholders at any depth in your JSON structure. Nested objects and arrays **inside** the substituted `{{data}}` value are preserved from the log row.
+
+### Rules and constraints
+
+- **Simple substitution only:** No conditionals, loops, filters, or expressions.
+- **No field paths:** Placeholders like `{{data.orgId}}`, `{{orgId}}`, or `{{ip}}` do **not** work. To use a single field, read it from the full `data` object on the receiver or transform after ingest.
+- **Quote `{{data}}` correctly:** `"field": {{data}}` is valid; `"field": "{{data}}"` stringifies the object incorrectly and produces invalid or useless JSON.
+- **One template per destination:** The same template applies to every log type enabled on that destination. You cannot define different templates per log type on one HTTP destination.
+- **String escaping:** `{{event}}` and `{{timestamp}}` are JSON-escaped for safe use inside quoted strings.
+- **Invalid JSON:** Pangolin does not validate templates at save time. If the rendered body is not valid JSON, delivery may still occur but your receiver may reject it. Validate templates with a JSON linter before saving.
+- **Not available on other destination types:** Body templates apply to HTTP streaming only, not S3 or Datadog destinations.
+
+## Payload format
+
+Payload format is separate from the body template. The template defines the shape of **one event**; payload format controls **how many events** are sent per HTTP request.
+
+| Format | HTTP body | Content-Type |
+| --- | --- | --- |
+| **JSON array** (default) | One POST per batch: `[{…}, {…}, …]` | `application/json` |
+| **NDJSON** | One JSON object per line, no outer array | `application/x-ndjson` |
+| **One event per request** | Separate POST for each event | `application/json` |
+
+The template is applied once per event, then results are batched into an array, joined as NDJSON lines, or sent individually, depending on the format you select.
+
+Choose **NDJSON** for aggregators that expect newline-delimited ingest (Splunk HEC, Elastic/OpenSearch bulk-style HTTP inputs, Loki). Choose **one event per request** when the endpoint cannot accept batches.
+
+## Log type reference
+
+The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
+
+## Integration examples
+
+### Generic webhook (default shape, JSON array)
+
+Leave custom body template disabled. Select **JSON array** payload format. Point the destination at your webhook URL with bearer or custom-header auth.
+
+Each batch POST body looks like:
+
+```json
+[
+  {
+    "event": "action",
+    "timestamp": "2025-06-15T12:34:56.789Z",
+    "data": { "action": "updateUser", "actor": "admin@example.com" }
+  }
+]
+```
+
+### Log aggregator (NDJSON, minimal template)
+
+Enable a custom template and select **NDJSON**:
+
+```json
+{
+  "type": "{{event}}",
+  "ts": "{{timestamp}}",
+  "payload": {{data}}
+}
+```
+
+Each line in the POST body is one rendered event. Set any vendor-required headers on the **Headers** tab.
+
+### Vendor schema remapping
+
+If a tool expects your log row under a specific key, wrap `{{data}}` without quotes:
+
+```json
+{
+  "source": "pangolin",
+  "sourcetype": "_json",
+  "time": "{{timestamp}}",
+  "event": {{data}}
+}
+```
+
+Adjust property names to match the vendor; field extraction beyond the three template variables happens on the receiver.
+
+## Limitations and troubleshooting
+
+- **Field selection:** Cannot pick individual columns in the template. Use full `{{data}}` or transform after delivery.
+- **Mixed log types:** Enabling multiple log types on one destination produces heterogeneous `data` in the same batch. Enable one type per destination if your pipeline expects a uniform schema.
+- **Historical logs:** New destinations do not backfill. Only events recorded after the destination is created are streamed.
+- **Delivery errors:** Check the destination’s **last error** in the dashboard. Common causes: wrong URL, auth failure, TLS issues, or receiver rejecting malformed JSON.
+- **Quoting `{{data}}`:** `"payload": "{{data}}"` treats the entire row as a string, which is almost always wrong. Use `"payload": {{data}}`.
+- **Splunk field extraction:** Pangolin does not emit Splunk-style indexed fields in the template. Parse `data` or use a receiver-side pipeline.