Files
docs-v2/manage/analytics/streaming/http.mdx
anjan713 5d6448aafa docs: add missing alt attributes to documentation images
Add context-aware alt text to 36 images across 20 MDX files flagged
by mint a11y. Descriptions are derived from each image's caption and
surrounding step text. Part of fosrl/docs-v2#116.
2026-07-12 20:21:01 -07:00

203 lines
8.5 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "HTTP webhook"
description: "Forward audit logs to any HTTP endpoint with optional custom body templates"
---
import PangolinCloudTocCta from "/snippets/pangolin-cloud-toc-cta.mdx";
<PangolinCloudTocCta />
HTTP destinations POST your organizations audit logs to a URL you control. Use them for generic webhooks, Splunk HEC, Elastic or OpenSearch ingest, Grafana Loki push endpoints, or any receiver that accepts JSON over HTTP.
<Note>
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
</Note>
## Overview
An HTTP destination sends **POST** requests to your endpoint. Configure:
1. **Settings:** Name, URL, and authentication.
2. **Headers:** Optional static headers on every request.
3. **Body:** Default JSON shape or a custom body template, plus payload format (how batches are packaged).
4. **Logs:** Which log types are forwarded.
Enable **Custom body template** when your receiver expects a different JSON layout than Pangolins default. Leave it off to send the standard `{ event, timestamp, data }` object per log record.
## Configure the connection
On the **Settings** tab, set a display name, the endpoint URL, and authentication:
| Auth type | Behavior |
| --- | --- |
| None | No `Authorization` header |
| Bearer token | `Authorization: Bearer <token>` |
| Basic auth | `Authorization: Basic <base64(user:password)>` |
| Custom header | A single header name and value (for example an API key header) |
<Frame>
<img src="/images/streaming-http-settings.png" alt="HTTP destination settings with URL and authentication options" centered />
</Frame>
All delivery uses **POST**. Requests time out after 30 seconds.
## Authentication and headers
On the **Headers** tab, add optional static headers sent with every request, for example a vendor-specific API key or a non-default `Content-Type`. When you do not override it, Pangolin sends `Content-Type: application/json` (or `application/x-ndjson` when using the NDJSON payload format).
<Frame>
<img src="/images/streaming-http-headers.png" alt="Headers tab for adding static HTTP headers" centered />
</Frame>
## Default payload (template off)
When custom body template is disabled, each log event is serialized as:
```json
{
"event": "request",
"timestamp": "2025-06-15T12:34:56.789Z",
"data": {
"timestamp": 1718454896,
"action": true,
"method": "GET",
"path": "/api/health"
}
}
```
| Field | Meaning |
| --- | --- |
| `event` | Log type: `request`, `access`, `action`, or `connection` |
| `timestamp` | Event time as ISO-8601 UTC |
| `data` | The **complete stored log row** for that record, not a curated subset |
The field set inside `data` depends on the log type. The same destination can stream multiple types; batches may contain heterogeneous `data` shapes. See [Log type reference](#log-type-reference) below and the dedicated log docs for full field lists.
<Warning>
Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them on the receiver if you need structured fields.
</Warning>
## Custom body template
On the **Body** tab, enable **Custom body template** and provide a JSON template string. Pangolin performs simple placeholder substitution, **not** a full templating language like Handlebars.
<Frame>
<img src="/images/streaming-http-body.png" alt="Body tab with custom body template editor" centered />
</Frame>
### Template variables
Only these three placeholders are supported:
| Variable | Source | How to use in the template |
| --- | --- | --- |
| `{{event}}` | Log type (`request`, `access`, `action`, `connection`) | Inside JSON **string quotes** |
| `{{timestamp}}` | Event time (ISO-8601 UTC) | Inside JSON **string quotes** |
| `{{data}}` | Full log row as JSON | **Never wrap in quotes**; inlined as raw JSON |
**Canonical example** (equivalent to the default payload):
```json
{
"event": "{{event}}",
"timestamp": "{{timestamp}}",
"data": {{data}}
}
```
**Remapping property names** for a downstream schema:
```json
{
"type": "{{event}}",
"ts": "{{timestamp}}",
"payload": {{data}}
}
```
You may use the same token multiple times and nest placeholders at any depth in your JSON structure. Nested objects and arrays **inside** the substituted `{{data}}` value are preserved from the log row.
### Rules and constraints
- **Simple substitution only:** No conditionals, loops, filters, or expressions.
- **No field paths:** Placeholders like `{{data.orgId}}`, `{{orgId}}`, or `{{ip}}` do **not** work. To use a single field, read it from the full `data` object on the receiver or transform after ingest.
- **Quote `{{data}}` correctly:** `"field": {{data}}` is valid; `"field": "{{data}}"` stringifies the object incorrectly and produces invalid or useless JSON.
- **One template per destination:** The same template applies to every log type enabled on that destination. You cannot define different templates per log type on one HTTP destination.
- **String escaping:** `{{event}}` and `{{timestamp}}` are JSON-escaped for safe use inside quoted strings.
- **Invalid JSON:** Pangolin does not validate templates at save time. If the rendered body is not valid JSON, delivery may still occur but your receiver may reject it. Validate templates with a JSON linter before saving.
- **Not available on other destination types:** Body templates apply to HTTP streaming only, not S3 or Datadog destinations.
## Payload format
Payload format is separate from the body template. The template defines the shape of **one event**; payload format controls **how many events** are sent per HTTP request.
| Format | HTTP body | Content-Type |
| --- | --- | --- |
| **JSON array** (default) | One POST per batch: `[{…}, {…}, …]` | `application/json` |
| **NDJSON** | One JSON object per line, no outer array | `application/x-ndjson` |
| **One event per request** | Separate POST for each event | `application/json` |
The template is applied once per event, then results are batched into an array, joined as NDJSON lines, or sent individually, depending on the format you select.
Choose **NDJSON** for aggregators that expect newline-delimited ingest (Splunk HEC, Elastic/OpenSearch bulk-style HTTP inputs, Loki). Choose **one event per request** when the endpoint cannot accept batches.
## Log type reference
The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
## Integration examples
### Generic webhook (default shape, JSON array)
Leave custom body template disabled. Select **JSON array** payload format. Point the destination at your webhook URL with bearer or custom-header auth.
Each batch POST body looks like:
```json
[
{
"event": "action",
"timestamp": "2025-06-15T12:34:56.789Z",
"data": { "action": "updateUser", "actor": "admin@example.com" }
}
]
```
### Log aggregator (NDJSON, minimal template)
Enable a custom template and select **NDJSON**:
```json
{
"type": "{{event}}",
"ts": "{{timestamp}}",
"payload": {{data}}
}
```
Each line in the POST body is one rendered event. Set any vendor-required headers on the **Headers** tab.
### Vendor schema remapping
If a tool expects your log row under a specific key, wrap `{{data}}` without quotes:
```json
{
"source": "pangolin",
"sourcetype": "_json",
"time": "{{timestamp}}",
"event": {{data}}
}
```
Adjust property names to match the vendor; field extraction beyond the three template variables happens on the receiver.
## Limitations and troubleshooting
- **Field selection:** Cannot pick individual columns in the template. Use full `{{data}}` or transform after delivery.
- **Mixed log types:** Enabling multiple log types on one destination produces heterogeneous `data` in the same batch. Enable one type per destination if your pipeline expects a uniform schema.
- **Historical logs:** New destinations do not backfill. Only events recorded after the destination is created are streamed.
- **Delivery errors:** Check the destinations **last error** in the dashboard. Common causes: wrong URL, auth failure, TLS issues, or receiver rejecting malformed JSON.
- **Quoting `{{data}}`:** `"payload": "{{data}}"` treats the entire row as a string, which is almost always wrong. Use `"payload": {{data}}`.
- **Splunk field extraction:** Pangolin does not emit Splunk-style indexed fields in the template. Parse `data` or use a receiver-side pipeline.