pangolin_mirror/docs-v2
1
0
Fork 0
mirror of https://github.com/fosrl/docs-v2.git synced 2026-10-05 20:29:08 +02:00
Code Issues Packages Projects Releases Wiki Activity

port mintlify to fumadocs

Browse Source
This commit is contained in:
miloschwartz
2026-09-25 15:31:51 -04:00
parent dc54fb1017
commit 63199a588c
373 changed files with 13533 additions and 4577 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/docs/manage/analytics/streaming/http.mdx
+199
View File
@@ -0,0 +1,199 @@
---
title: "HTTP webhook"
description: "Forward audit logs to any HTTP endpoint with optional custom body templates"
---
HTTP destinations POST your organization’s audit logs to a URL you control. Use them for generic webhooks, Splunk HEC, Elastic or OpenSearch ingest, Grafana Loki push endpoints, or any receiver that accepts JSON over HTTP.
<Note>
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
</Note>
## Overview
An HTTP destination sends **POST** requests to your endpoint. Configure:
1. **Settings:** Name, URL, and authentication.
2. **Headers:** Optional static headers on every request.
3. **Body:** Default JSON shape or a custom body template, plus payload format (how batches are packaged).
4. **Logs:** Which log types are forwarded.
Enable **Custom body template** when your receiver expects a different JSON layout than Pangolin’s default. Leave it off to send the standard `{ event, timestamp, data }` object per log record.
## Configure the connection
On the **Settings** tab, set a display name, the endpoint URL, and authentication:
| Auth type | Behavior |
| --- | --- |
| None | No `Authorization` header |
| Bearer token | `Authorization: Bearer <token>` |
| Basic auth | `Authorization: Basic <base64(user:password)>` |
| Custom header | A single header name and value (for example an API key header) |
<Frame>
<img src="/images/streaming-http-settings.png" alt="HTTP destination settings with URL and authentication options" />
</Frame>
All delivery uses **POST**. Requests time out after 30 seconds.
## Authentication and headers
On the **Headers** tab, add optional static headers sent with every request, for example a vendor-specific API key or a non-default `Content-Type`. When you do not override it, Pangolin sends `Content-Type: application/json` (or `application/x-ndjson` when using the NDJSON payload format).
<Frame>
<img src="/images/streaming-http-headers.png" alt="Headers tab for adding static HTTP headers" />
</Frame>
## Default payload (template off)
When custom body template is disabled, each log event is serialized as:
```json
{
"event": "request",
"timestamp": "2025-06-15T12:34:56.789Z",
"data": {
"timestamp": 1718454896,
"action": true,
"method": "GET",
"path": "/api/health"
}
}
```
| Field | Meaning |
| --- | --- |
| `event` | Log type: `request`, `access`, `action`, or `connection` |
| `timestamp` | Event time as ISO-8601 UTC |
| `data` | The **complete stored log row** for that record, not a curated subset |
The field set inside `data` depends on the log type. The same destination can stream multiple types; batches may contain heterogeneous `data` shapes. See [Log type reference](#log-type-reference) below and the dedicated log docs for full field lists.
<Warning>
Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them on the receiver if you need structured fields.
</Warning>
## Custom body template
On the **Body** tab, enable **Custom body template** and provide a JSON template string. Pangolin performs simple placeholder substitution, **not** a full templating language like Handlebars.
<Frame>
<img src="/images/streaming-http-body.png" alt="Body tab with custom body template editor" />
</Frame>
### Template variables
Only these three placeholders are supported:
| Variable | Source | How to use in the template |
| --- | --- | --- |
| `{{event}}` | Log type (`request`, `access`, `action`, `connection`) | Inside JSON **string quotes** |
| `{{timestamp}}` | Event time (ISO-8601 UTC) | Inside JSON **string quotes** |
| `{{data}}` | Full log row as JSON | **Never wrap in quotes**; inlined as raw JSON |
**Canonical example** (equivalent to the default payload):
```json
{
"event": "{{event}}",
"timestamp": "{{timestamp}}",
"data": {{data}}
}
```
**Remapping property names** for a downstream schema:
```json
{
"type": "{{event}}",
"ts": "{{timestamp}}",
"payload": {{data}}
}
```
You may use the same token multiple times and nest placeholders at any depth in your JSON structure. Nested objects and arrays **inside** the substituted `{{data}}` value are preserved from the log row.
### Rules and constraints
- **Simple substitution only:** No conditionals, loops, filters, or expressions.
- **No field paths:** Placeholders like `{{data.orgId}}`, `{{orgId}}`, or `{{ip}}` do **not** work. To use a single field, read it from the full `data` object on the receiver or transform after ingest.
- **Quote `{{data}}` correctly:** `"field": {{data}}` is valid; `"field": "{{data}}"` stringifies the object incorrectly and produces invalid or useless JSON.
- **One template per destination:** The same template applies to every log type enabled on that destination. You cannot define different templates per log type on one HTTP destination.
- **String escaping:** `{{event}}` and `{{timestamp}}` are JSON-escaped for safe use inside quoted strings.
- **Invalid JSON:** Pangolin does not validate templates at save time. If the rendered body is not valid JSON, delivery may still occur but your receiver may reject it. Validate templates with a JSON linter before saving.
- **Not available on other destination types:** Body templates apply to HTTP streaming only, not S3 or Datadog destinations.
## Payload format
Payload format is separate from the body template. The template defines the shape of **one event**; payload format controls **how many events** are sent per HTTP request.
| Format | HTTP body | Content-Type |
| --- | --- | --- |
| **JSON array** (default) | One POST per batch: `[{…}, {…}, …]` | `application/json` |
| **NDJSON** | One JSON object per line, no outer array | `application/x-ndjson` |
| **One event per request** | Separate POST for each event | `application/json` |
The template is applied once per event, then results are batched into an array, joined as NDJSON lines, or sent individually, depending on the format you select.
Choose **NDJSON** for aggregators that expect newline-delimited ingest (Splunk HEC, Elastic/OpenSearch bulk-style HTTP inputs, Loki). Choose **one event per request** when the endpoint cannot accept batches.
## Log type reference
The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
## Integration examples
### Generic webhook (default shape, JSON array)
Leave custom body template disabled. Select **JSON array** payload format. Point the destination at your webhook URL with bearer or custom-header auth.
Each batch POST body looks like:
```json
[
{
"event": "action",
"timestamp": "2025-06-15T12:34:56.789Z",
"data": { "action": "updateUser", "actor": "admin@example.com" }
}
]
```
### Log aggregator (NDJSON, minimal template)
Enable a custom template and select **NDJSON**:
```json
{
"type": "{{event}}",
"ts": "{{timestamp}}",
"payload": {{data}}
}
```
Each line in the POST body is one rendered event. Set any vendor-required headers on the **Headers** tab.
### Vendor schema remapping
If a tool expects your log row under a specific key, wrap `{{data}}` without quotes:
```json
{
"source": "pangolin",
"sourcetype": "_json",
"time": "{{timestamp}}",
"event": {{data}}
}
```
Adjust property names to match the vendor; field extraction beyond the three template variables happens on the receiver.
## Limitations and troubleshooting
- **Field selection:** Cannot pick individual columns in the template. Use full `{{data}}` or transform after delivery.
- **Mixed log types:** Enabling multiple log types on one destination produces heterogeneous `data` in the same batch. Enable one type per destination if your pipeline expects a uniform schema.
- **Historical logs:** New destinations do not backfill. Only events recorded after the destination is created are streamed.
- **Delivery errors:** Check the destination’s **last error** in the dashboard. Common causes: wrong URL, auth failure, TLS issues, or receiver rejecting malformed JSON.
- **Quoting `{{data}}`:** `"payload": "{{data}}"` treats the entire row as a string, which is almost always wrong. Use `"payload": {{data}}`.
- **Splunk field extraction:** Pangolin does not emit Splunk-style indexed fields in the template. Parse `data` or use a receiver-side pipeline.
content/docs/manage/analytics/streaming/s3.mdx
+221
View File
@@ -0,0 +1,221 @@
---
title: "Amazon S3"
description: "Archive audit logs to S3 or S3-compatible object storage"
---
S3 destinations upload batches of your organization's audit logs as objects in a bucket you control. Use them for long-term archival, data lakes (Athena, Glue, BigQuery), or S3-compatible stores such as MinIO and Cloudflare R2.
<Note>
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
</Note>
## Overview
An S3 destination writes **one object per batch** via `PutObject`. Each object contains up to 250 events of a **single log type**. There is no custom body template or field mapping; Pangolin serializes every event in a fixed shape and chooses the object key automatically.
Configure:
1. **Settings:** Name, credentials, region, bucket, optional prefix and custom endpoint.
2. **Format:** File format (JSON array, NDJSON, or CSV) and optional gzip compression.
3. **Logs:** Which log types are forwarded.
## Settings tab
| Field | Required | Description |
| --- | --- | --- |
| Name | Yes | Display label for this destination |
| AWS Access Key ID | Yes | Static access key for the S3 client |
| AWS Secret Access Key | Yes | Secret for the access key |
| AWS Region | Yes | S3 client region (UI default: `us-east-1`) |
| Bucket name | Yes | Target bucket |
| Key prefix | No | Prepended to every object key; trailing slashes are stripped |
| Custom endpoint | No | Base URL for MinIO, R2, etc.; leave blank for AWS S3 |
Pangolin uses static access keys only. There is no IAM role, instance profile, or OIDC picker in the UI.
<Frame>
<img src="/images/streaming-s3-settings.png" alt="S3 destination settings with credentials, region, and bucket" />
</Frame>
Uploads time out after 60 seconds per object.
## Format tab
**Gzip compression** (optional): When enabled, the object body is gzip-compressed before upload, `Content-Encoding: gzip` is set, and the object key gets a `.gz` suffix (for example `….json.gz`). Decompress before parsing unless your tool handles gzip automatically.
**File format:**
| Format | Description |
| --- | --- |
| **JSON array** (default) | One array per object: `[{…}, {…}, …]` |
| **NDJSON** | One JSON object per line, no outer array |
| **CSV** | RFC-4180 CSV with a header row; see [CSV format](#csv-format) |
<Frame>
<img src="/images/streaming-s3-format.png" alt="Format tab with file format and gzip options" />
</Frame>
## Logs tab
Choose which log categories are uploaded. Each enabled type is written to its own key prefix (`request/`, `action/`, etc.). Only log types enabled for your organization can be streamed.
<Frame>
<img src="/images/streaming-s3-logs.png" alt="Logs tab for selecting streamed log types" />
</Frame>
## Object key layout
Every upload gets a unique key:
```
{prefix}/{logType}/{YYYY}/{MM}/{DD}/{HH-mm-ss-uuid}.{ext}[.gz]
```
| Segment | Meaning |
| --- | --- |
| `prefix` | Your optional key prefix; omitted when empty |
| `logType` | `request`, `action`, `access`, or `connection` |
| `YYYY/MM/DD` | **Upload time (UTC)**, not the event timestamp |
| `HH-mm-ss-uuid` | Upload time plus a UUID so keys never collide |
| `ext` | `json` (JSON array), `ndjson`, or `csv` |
| `.gz` | Present when gzip is enabled |
**Without prefix:**
```
request/2026/06/04/14-30-45-a1b2c3d4-e5f6-7890-abcd-ef1234567890.json
```
**With prefix `pangolin/audit` and gzip:**
```
pangolin/audit/action/2026/06/04/14-30-45-a1b2c3d4-e5f6-7890-abcd-ef1234567890.json.gz
```
Enabling multiple log types on one destination produces **separate object streams** under different `logType/` segments. A single object never mixes log types.
## Event record shape
Each event in JSON and NDJSON objects uses this fixed structure:
```json
{
"event": "request",
"timestamp": "2026-06-04T12:00:00.000Z",
"data": {
"timestamp": 1717492800,
"action": true,
"method": "GET",
"path": "/api/health"
}
}
```
| Field | Meaning |
| --- | --- |
| `event` | Log type: `request`, `access`, `action`, or `connection` |
| `timestamp` | Event time as ISO-8601 UTC (connection logs use session start) |
| `data` | The **complete stored log row** for that record, not a curated subset |
<Warning>
Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them in your pipeline if you need structured fields.
</Warning>
## File formats
### JSON array (default)
- One S3 object per batch; body is `[{…}, {…}, …]`.
- Up to 250 events per object.
- `Content-Type: application/json`.
### NDJSON
- One S3 object per batch; body is one JSON record per line with no outer array.
- Good for Athena, BigQuery load jobs, Spark, and similar line-oriented pipelines.
- `Content-Type: application/x-ndjson`.
### CSV format
- Header row: `event`, `timestamp`, then **all field names** found in `data` across that batch (union of keys, in insertion order).
- Each data row flattens `event`, `timestamp`, and spreads `data` fields into columns. There is **no** nested `data` column.
- Missing fields in a given row leave an empty cell.
- Object or array values in `data` are written as `JSON.stringify` strings inside the cell.
- `Content-Type: text/csv; charset=utf-8`.
The column set can grow as new fields appear in later batches. Order is not guaranteed to stay identical across all objects over time.
## Batching and throughput
- Objects are written **per batch** (up to ~250 events), not one object per log line.
- Pangolin polls for new logs on a regular interval and may write multiple objects during catch-up after a pause.
- **No backfill:** New destinations start from the current log cursor. Historical logs already in Pangolin are not uploaded.
- **Extended outage:** If the destination is unreachable for about 24 hours, the backlog may be discarded and streaming resumes from the present cursor (same behavior as [HTTP streaming](/manage/analytics/streaming/http)).
## Gzip
When gzip is enabled:
1. The serialized body is compressed before upload.
2. The object key includes `.gz` (for example `….ndjson.gz`).
3. S3 stores `Content-Encoding: gzip`.
Consumers must decompress before parsing unless the tool auto-detects gzip (many Athena and Spark setups do when `Content-Encoding` is set). NDJSON plus gzip is a common choice for cost-sensitive archival.
## S3-compatible storage
Set **Custom endpoint** to your vendor's S3 API URL and provide access key credentials per that vendor's documentation.
| Store | Notes |
| --- | --- |
| **AWS S3** | Leave custom endpoint blank; use a bucket in the configured region |
| **MinIO** | Set endpoint to your MinIO server URL; use MinIO access keys |
| **Cloudflare R2** | Set endpoint to your R2 S3 API URL; use R2 access keys |
Pangolin does not expose path-style vs virtual-hosted addressing, ACLs, SSE-KMS, storage class, or multipart tuning. Configure those in the vendor console or bucket policy.
## IAM and bucket policy
Grant the access key permission to write under your prefix. A minimal AWS example:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": "arn:aws:s3:::your-bucket/pangolin/audit/*"
},
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": "arn:aws:s3:::your-bucket",
"Condition": {
"StringLike": { "s3:prefix": ["pangolin/audit/*"] }
}
}
]
}
```
Adjust bucket name and prefix to match your configuration. `ListBucket` is optional but useful when debugging missing objects.
Block public access, encryption at rest, lifecycle rules, and object tags are configured in AWS or your vendor console, not in Pangolin.
## Log type reference
The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
## Limitations and troubleshooting
- **No custom JSON shape:** Fixed event record only. Use an HTTP destination if you need body templates or field remapping.
- **No per-event objects:** Always batched (up to ~250 events per object).
- **No mixed log types in one object:** Each upload contains a single log type.
- **Upload-time partitioning:** Key date folders use upload time (UTC), not the event's `timestamp`.
- **CSV columns:** Automatic from batch contents; not user-selectable; column set may change over time.
- **Static credentials only:** Rotate keys by updating the destination; credentials are stored encrypted server-side.
- **Historical logs:** New destinations do not backfill.
- **Delivery errors:** Check the destination's **last error** in the dashboard. Common causes: `AccessDenied`, wrong bucket or region, bad endpoint URL, TLS issues, or expired credentials.
- **Missing objects:** Confirm prefix, lifecycle rules, and that the log type is enabled on the **Logs** tab.
- **Athena/Glue parse errors:** Verify format (JSON array vs NDJSON), gzip handling, and that the crawler/table schema matches flattened CSV columns if using CSV.