mirror of
https://github.com/fosrl/docs-v2.git
synced 2026-10-01 18:29:09 +02:00
port mintlify to fumadocs
This commit is contained in:
miloschwartz
@@ -0,0 +1,51 @@
|
||||
---
|
||||
title: "Authentication Logs"
|
||||
description: "Authentication logs are a record of each authenticated access attempt to a resource"
|
||||
---
|
||||
Authentication logs provide detailed information about each access attempt made to your Pangolin resources. These logs help you monitor and analyze user activity each time they attempt to authenticate.
|
||||
|
||||
<Note>
|
||||
Authentication logs are only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## What are Authentication Logs?
|
||||
|
||||
Authentication logs capture authentication events when users or API keys attempt to access a resource. They record whether the authentication was successful or failed, along with contextual information about the attempt. These logs are useful for:
|
||||
|
||||
- Monitoring authentication patterns and login attempts
|
||||
- Tracking which users are accessing which resources
|
||||
- Identifying failed authentication attempts for security analysis
|
||||
- Understanding geographic distribution of access attempts
|
||||
- Analyzing user agent and device information
|
||||
|
||||
<Frame>
|
||||
<img src="/images/access_logs.png" alt="Authentication logs table in the Pangolin dashboard"/>
|
||||
</Frame>
|
||||
|
||||
<Tip>Make sure to enable authentication logs in the org settings</Tip>
|
||||
|
||||
## Authentication Log Fields
|
||||
|
||||
Each authentication log entry contains the following fields:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp` | number | Unix timestamp (in seconds) when the access attempt occurred |
|
||||
| `action` | boolean | Whether the access was allowed (`true`) or denied (`false`) |
|
||||
| `type` | string | The type of authentication event (e.g., "login", "password", "pincode") |
|
||||
| `actorType` | string | The type of actor making the access attempt ("user" or "apiKey") |
|
||||
| `actor` | string | The display name of the actor (username or API key name) |
|
||||
| `actorId` | string | The unique identifier for the actor (user ID or API key ID) |
|
||||
| `resourceId` | number | The ID of the resource being accessed (if applicable) |
|
||||
| `ip` | string | The IP address of the client making the access attempt |
|
||||
| `location` | string | The geographic location (country code) based on IP address |
|
||||
| `userAgent` | string | The user agent string of the client browser or application |
|
||||
| `metadata` | string | Additional contextual information in JSON format |
|
||||
|
||||
## Log Retention
|
||||
|
||||
Authentication log retention is controlled by the organization setting. By default, authentication logs are retained for 0 days (disabled).
|
||||
|
||||
## Exporting
|
||||
|
||||
Logs can be exported into CSV format for external analysis and archival. Logs can be exported from the table view in the Pangolin dashboard or via the Pangolin API. When exporting, you can specify date ranges and filters to narrow down the logs you need.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: "Admin Action Logs"
|
||||
description: "Admin Action logs are a record of each event taken by users in the organization"
|
||||
---
|
||||
Admin Action logs provide an audit trail of administrative actions and configuration changes made within your Pangolin organization. These logs help you track who made what changes and when.
|
||||
|
||||
<Note>
|
||||
Admin Action logs are only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## What are Admin Action Logs?
|
||||
|
||||
Admin Action logs capture administrative events and configuration changes performed by users or API keys in the Pangolin dashboard. They record management operations such as creating resources, modifying settings, managing users, and other organizational changes. These logs are useful for:
|
||||
|
||||
- Maintaining an audit trail of configuration changes
|
||||
- Tracking administrative actions for compliance
|
||||
- Identifying who made specific changes to your infrastructure
|
||||
- Troubleshooting configuration issues by reviewing recent changes
|
||||
- Meeting security and compliance requirements
|
||||
|
||||
<Frame>
|
||||
<img src="/images/action_logs.png" alt="Admin action logs table in the Pangolin dashboard"/>
|
||||
</Frame>
|
||||
|
||||
<Tip>Make sure to enable access logs in the org settings</Tip>
|
||||
|
||||
## Admin Action Log Fields
|
||||
|
||||
Each action log entry contains the following fields:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp` | number | Unix timestamp (in seconds) when the action was performed |
|
||||
| `action` | string | The specific action that was performed (e.g., "createResource", "updateUser", "deleteTarget") |
|
||||
| `actorType` | string | The type of actor performing the action ("user" or "apiKey") |
|
||||
| `actor` | string | The display name of the actor (username or API key name) |
|
||||
| `actorId` | string | The unique identifier for the actor (user ID or API key ID) |
|
||||
| `metadata` | string | Additional contextual information about the action in JSON format (often contains request parameters) |
|
||||
|
||||
## Log Retention
|
||||
|
||||
Admin Action log retention is controlled by the organization settings. By default, admin action logs are retained for 0 days (disabled).
|
||||
|
||||
## Exporting
|
||||
|
||||
Logs can be exported into CSV format for external analysis and archival. Logs can be exported from the table view in the Pangolin dashboard or via the Pangolin API. When exporting, you can specify date ranges and filters to narrow down the logs you need.
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: "Network Logs"
|
||||
description: "Network logs are a record of TCP and UDP sessions between clients and private resources on sites"
|
||||
---
|
||||
|
||||
Network logs record each TCP and UDP session that traverses the tunnel between Pangolin clients and resources on your sites. They apply to private resources reached through the Pangolin client (and related tunnel traffic), not to public resources served only through the reverse proxy. You can see which clients and users opened sessions to which private resources, the source and destination addresses and protocols (TCP and UDP), the start and end times of the sessions, and more.
|
||||
|
||||
<Note>
|
||||
Network logs are only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## What are Network Logs?
|
||||
|
||||
Network logs capture tunnel sessions from clients to private resources. They are useful for:
|
||||
|
||||
- Observing which clients and users opened sessions to which private resources
|
||||
- Reviewing source and destination addresses and protocols (TCP and UDP)
|
||||
- Measuring traffic volume with transmitted and received byte counts
|
||||
- Auditing session start and end times for troubleshooting and compliance
|
||||
|
||||
Network logs are synchronized to the cloud every 30–60 seconds. A brief delay before entries appear in the table is expected.
|
||||
|
||||
<Tip>Make sure to enable network logging in the org settings</Tip>
|
||||
|
||||
## Network Log Fields
|
||||
|
||||
Each network log entry contains the following fields:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp` | number | Unix timestamp (in seconds) when the session started |
|
||||
| `protocol` | string | Transport protocol for the session (`tcp` or `udp`) |
|
||||
| `siteResourceId` | number \| null | The ID of the [private resource](/manage/resources/understanding-resources) for the session (if applicable) |
|
||||
| `clientId` | number \| null | The Pangolin [client ID](/manage/clients/understanding-clients) for the session |
|
||||
| `clientEndpoint` | string \| null | The client-side endpoint for the session (e.g. `123.123.123.123:12345`) |
|
||||
| `userId` | string \| null | The user ID when the session is tied to an authenticated user |
|
||||
| `sourceAddr` | string | Source address for the session (typically the client-side endpoint) |
|
||||
| `destAddr` | string | Destination address for the session (typically the resource-side endpoint) |
|
||||
| `duration` | number \| null | How long the session lasted (in seconds), when the session has ended |
|
||||
| `bytesTx` | number \| null | Bytes transmitted in the session |
|
||||
| `bytesRx` | number \| null | Bytes received in the session |
|
||||
|
||||
## Log Retention
|
||||
|
||||
Network log retention is controlled by the organization setting. By default, network logs are retained for 0 days (disabled).
|
||||
|
||||
<Note>
|
||||
Network logs can generate significant data volume depending on session churn and traffic. Consider your storage capacity when configuring retention periods.
|
||||
</Note>
|
||||
|
||||
## Exporting
|
||||
|
||||
Logs can be exported into CSV format for external analysis and archival. Logs can be exported from the table view in the Pangolin dashboard or via the Pangolin API. When exporting, you can specify date ranges and filters to narrow down the logs you need.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: "HTTPS Request Logs"
|
||||
description: "Request logs are a record of each HTTP request to a resource"
|
||||
---
|
||||
HTTPS Request logs provide detailed information about every HTTP request made to your Pangolin resources. These logs capture both successful and denied requests along with comprehensive request metadata.
|
||||
|
||||
## What are HTTPS Request Logs?
|
||||
|
||||
HTTPS Request logs capture every HTTPS request that passes through a reverse proxy, including the request details, the decision made (allow or deny), and the reason for that decision. These logs are useful for:
|
||||
|
||||
- Monitoring traffic patterns and request volumes
|
||||
- Debugging access issues and rule configurations
|
||||
- Analyzing API usage and endpoint popularity
|
||||
- Understanding geographic distribution of requests
|
||||
- Identifying potential security threats or unusual traffic patterns
|
||||
- Troubleshooting connectivity and routing issues
|
||||
|
||||
<Frame>
|
||||
<img src="/images/request_logs.png" alt="HTTPS request logs table in the Pangolin dashboard"/>
|
||||
</Frame>
|
||||
|
||||
## HTTPS Request Log Fields
|
||||
|
||||
Each HTTPS request log entry contains the following fields:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp` | number | Unix timestamp (in seconds) when the request was made |
|
||||
| `action` | boolean | Whether the request was allowed (`true`) or denied (`false`) |
|
||||
| `reason` | number | Numeric code indicating the reason for the decision |
|
||||
| `actorType` | string | The type of actor making the request ("user", "apiKey", or null for anonymous) |
|
||||
| `actor` | string | The display name of the authenticated actor (username or API key name) |
|
||||
| `actorId` | string | The unique identifier for the authenticated actor |
|
||||
| `resourceId` | number | The ID of the resource that received the request |
|
||||
| `ip` | string | The IP address of the client making the request |
|
||||
| `location` | string | The geographic location (country code) based on IP address |
|
||||
| `userAgent` | string | The user agent string of the client browser or application |
|
||||
| `metadata` | string | Additional contextual information in JSON format |
|
||||
| `headers` | string | HTTP request headers in JSON format |
|
||||
| `query` | string | URL query parameters in JSON format |
|
||||
| `originalRequestURL` | string | The full original URL of the request |
|
||||
| `scheme` | string | The protocol scheme (http or https) |
|
||||
| `host` | string | The hostname from the request |
|
||||
| `path` | string | The URL path of the request |
|
||||
| `method` | string | The HTTP method (GET, POST, PUT, DELETE, etc.) |
|
||||
| `tls` | boolean | Whether the connection used TLS/SSL |
|
||||
|
||||
## Log Retention
|
||||
|
||||
HTTPS Request log retention is controlled by the organization setting. By default, HTTPS request logs are retained for 7 days.
|
||||
|
||||
<Note>
|
||||
HTTPS Request logs can generate significant data volume depending on your traffic. Consider your storage capacity when configuring retention periods.
|
||||
</Note>
|
||||
|
||||
## Exporting
|
||||
|
||||
Logs can be exported into CSV format for external analysis and archival. Logs can be exported from the table view in the Pangolin dashboard or via the Pangolin API. When exporting, you can specify date ranges and filters to narrow down the logs you need.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: "Event Streaming"
|
||||
description: "Stream Pangolin log events to external collectors and SIEM tools"
|
||||
---
|
||||
|
||||
Log streaming forwards your organization's audit logs to external data collectors such as Datadog, Splunk, Microsoft Sentinel, Elastic, or any HTTP endpoint you operate. You add a **destination** (how events are delivered), choose which **log types** to include, and Pangolin pushes new events as they are recorded.
|
||||
|
||||
<Note>
|
||||
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## In the dashboard
|
||||
|
||||
Open **Organization > Logs & Analytics > Streaming** to add destinations and monitor delivery status. Each destination has its own connection settings, optional body customization (where supported), and log-type selection.
|
||||
|
||||
## Log types
|
||||
|
||||
You choose which categories each destination receives. Only log types enabled for your organization can be streamed.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-log-types.png" alt="Log type selection for a streaming destination" />
|
||||
</Frame>
|
||||
|
||||
## Destination types
|
||||
|
||||
Each destination type has its own configuration and payload behavior. Select **Add destination** and pick a delivery method.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-add-destination.png" alt="Add destination dialog in the Pangolin dashboard" />
|
||||
</Frame>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="HTTP webhook" icon="globe" href="/manage/analytics/streaming/http">
|
||||
POST JSON or NDJSON to any URL. Supports custom body templates, authentication, and payload formats for SIEMs and generic webhooks.
|
||||
</Card>
|
||||
<Card title="Amazon S3" icon="bucket" href="/manage/analytics/streaming/s3">
|
||||
Upload batched audit logs to S3 or S3-compatible storage. JSON array, NDJSON, or CSV with optional gzip.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
### Other destinations
|
||||
|
||||
Amazon S3 and HTTP webhooks are documented above. For Datadog, Microsoft Sentinel, or other vendor-specific setups, contact [sales@pangolin.net](mailto:sales@pangolin.net).
|
||||
|
||||
- **No backfill:** New destinations start from the current log cursor. Historical logs already in Pangolin are not replayed.
|
||||
- **Per-log-type cursors:** Each enabled log type on a destination is tracked independently.
|
||||
- **Errors in the UI:** When delivery fails, the destination's last error is shown in the dashboard so you can fix configuration or endpoint issues.
|
||||
@@ -0,0 +1,199 @@
|
||||
---
|
||||
title: "HTTP webhook"
|
||||
description: "Forward audit logs to any HTTP endpoint with optional custom body templates"
|
||||
---
|
||||
|
||||
HTTP destinations POST your organization’s audit logs to a URL you control. Use them for generic webhooks, Splunk HEC, Elastic or OpenSearch ingest, Grafana Loki push endpoints, or any receiver that accepts JSON over HTTP.
|
||||
|
||||
<Note>
|
||||
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## Overview
|
||||
|
||||
An HTTP destination sends **POST** requests to your endpoint. Configure:
|
||||
|
||||
1. **Settings:** Name, URL, and authentication.
|
||||
2. **Headers:** Optional static headers on every request.
|
||||
3. **Body:** Default JSON shape or a custom body template, plus payload format (how batches are packaged).
|
||||
4. **Logs:** Which log types are forwarded.
|
||||
|
||||
Enable **Custom body template** when your receiver expects a different JSON layout than Pangolin’s default. Leave it off to send the standard `{ event, timestamp, data }` object per log record.
|
||||
|
||||
## Configure the connection
|
||||
|
||||
On the **Settings** tab, set a display name, the endpoint URL, and authentication:
|
||||
|
||||
| Auth type | Behavior |
|
||||
| --- | --- |
|
||||
| None | No `Authorization` header |
|
||||
| Bearer token | `Authorization: Bearer <token>` |
|
||||
| Basic auth | `Authorization: Basic <base64(user:password)>` |
|
||||
| Custom header | A single header name and value (for example an API key header) |
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-http-settings.png" alt="HTTP destination settings with URL and authentication options" />
|
||||
</Frame>
|
||||
|
||||
All delivery uses **POST**. Requests time out after 30 seconds.
|
||||
|
||||
## Authentication and headers
|
||||
|
||||
On the **Headers** tab, add optional static headers sent with every request, for example a vendor-specific API key or a non-default `Content-Type`. When you do not override it, Pangolin sends `Content-Type: application/json` (or `application/x-ndjson` when using the NDJSON payload format).
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-http-headers.png" alt="Headers tab for adding static HTTP headers" />
|
||||
</Frame>
|
||||
|
||||
## Default payload (template off)
|
||||
|
||||
When custom body template is disabled, each log event is serialized as:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "request",
|
||||
"timestamp": "2025-06-15T12:34:56.789Z",
|
||||
"data": {
|
||||
"timestamp": 1718454896,
|
||||
"action": true,
|
||||
"method": "GET",
|
||||
"path": "/api/health"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `event` | Log type: `request`, `access`, `action`, or `connection` |
|
||||
| `timestamp` | Event time as ISO-8601 UTC |
|
||||
| `data` | The **complete stored log row** for that record, not a curated subset |
|
||||
|
||||
The field set inside `data` depends on the log type. The same destination can stream multiple types; batches may contain heterogeneous `data` shapes. See [Log type reference](#log-type-reference) below and the dedicated log docs for full field lists.
|
||||
|
||||
<Warning>
|
||||
Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them on the receiver if you need structured fields.
|
||||
</Warning>
|
||||
|
||||
## Custom body template
|
||||
|
||||
On the **Body** tab, enable **Custom body template** and provide a JSON template string. Pangolin performs simple placeholder substitution, **not** a full templating language like Handlebars.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-http-body.png" alt="Body tab with custom body template editor" />
|
||||
</Frame>
|
||||
|
||||
### Template variables
|
||||
|
||||
Only these three placeholders are supported:
|
||||
|
||||
| Variable | Source | How to use in the template |
|
||||
| --- | --- | --- |
|
||||
| `{{event}}` | Log type (`request`, `access`, `action`, `connection`) | Inside JSON **string quotes** |
|
||||
| `{{timestamp}}` | Event time (ISO-8601 UTC) | Inside JSON **string quotes** |
|
||||
| `{{data}}` | Full log row as JSON | **Never wrap in quotes**; inlined as raw JSON |
|
||||
|
||||
**Canonical example** (equivalent to the default payload):
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "{{event}}",
|
||||
"timestamp": "{{timestamp}}",
|
||||
"data": {{data}}
|
||||
}
|
||||
```
|
||||
|
||||
**Remapping property names** for a downstream schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "{{event}}",
|
||||
"ts": "{{timestamp}}",
|
||||
"payload": {{data}}
|
||||
}
|
||||
```
|
||||
|
||||
You may use the same token multiple times and nest placeholders at any depth in your JSON structure. Nested objects and arrays **inside** the substituted `{{data}}` value are preserved from the log row.
|
||||
|
||||
### Rules and constraints
|
||||
|
||||
- **Simple substitution only:** No conditionals, loops, filters, or expressions.
|
||||
- **No field paths:** Placeholders like `{{data.orgId}}`, `{{orgId}}`, or `{{ip}}` do **not** work. To use a single field, read it from the full `data` object on the receiver or transform after ingest.
|
||||
- **Quote `{{data}}` correctly:** `"field": {{data}}` is valid; `"field": "{{data}}"` stringifies the object incorrectly and produces invalid or useless JSON.
|
||||
- **One template per destination:** The same template applies to every log type enabled on that destination. You cannot define different templates per log type on one HTTP destination.
|
||||
- **String escaping:** `{{event}}` and `{{timestamp}}` are JSON-escaped for safe use inside quoted strings.
|
||||
- **Invalid JSON:** Pangolin does not validate templates at save time. If the rendered body is not valid JSON, delivery may still occur but your receiver may reject it. Validate templates with a JSON linter before saving.
|
||||
- **Not available on other destination types:** Body templates apply to HTTP streaming only, not S3 or Datadog destinations.
|
||||
|
||||
## Payload format
|
||||
|
||||
Payload format is separate from the body template. The template defines the shape of **one event**; payload format controls **how many events** are sent per HTTP request.
|
||||
|
||||
| Format | HTTP body | Content-Type |
|
||||
| --- | --- | --- |
|
||||
| **JSON array** (default) | One POST per batch: `[{…}, {…}, …]` | `application/json` |
|
||||
| **NDJSON** | One JSON object per line, no outer array | `application/x-ndjson` |
|
||||
| **One event per request** | Separate POST for each event | `application/json` |
|
||||
|
||||
The template is applied once per event, then results are batched into an array, joined as NDJSON lines, or sent individually, depending on the format you select.
|
||||
|
||||
Choose **NDJSON** for aggregators that expect newline-delimited ingest (Splunk HEC, Elastic/OpenSearch bulk-style HTTP inputs, Loki). Choose **one event per request** when the endpoint cannot accept batches.
|
||||
|
||||
## Log type reference
|
||||
|
||||
The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
|
||||
|
||||
## Integration examples
|
||||
|
||||
### Generic webhook (default shape, JSON array)
|
||||
|
||||
Leave custom body template disabled. Select **JSON array** payload format. Point the destination at your webhook URL with bearer or custom-header auth.
|
||||
|
||||
Each batch POST body looks like:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"event": "action",
|
||||
"timestamp": "2025-06-15T12:34:56.789Z",
|
||||
"data": { "action": "updateUser", "actor": "admin@example.com" }
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Log aggregator (NDJSON, minimal template)
|
||||
|
||||
Enable a custom template and select **NDJSON**:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "{{event}}",
|
||||
"ts": "{{timestamp}}",
|
||||
"payload": {{data}}
|
||||
}
|
||||
```
|
||||
|
||||
Each line in the POST body is one rendered event. Set any vendor-required headers on the **Headers** tab.
|
||||
|
||||
### Vendor schema remapping
|
||||
|
||||
If a tool expects your log row under a specific key, wrap `{{data}}` without quotes:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "pangolin",
|
||||
"sourcetype": "_json",
|
||||
"time": "{{timestamp}}",
|
||||
"event": {{data}}
|
||||
}
|
||||
```
|
||||
|
||||
Adjust property names to match the vendor; field extraction beyond the three template variables happens on the receiver.
|
||||
|
||||
## Limitations and troubleshooting
|
||||
|
||||
- **Field selection:** Cannot pick individual columns in the template. Use full `{{data}}` or transform after delivery.
|
||||
- **Mixed log types:** Enabling multiple log types on one destination produces heterogeneous `data` in the same batch. Enable one type per destination if your pipeline expects a uniform schema.
|
||||
- **Historical logs:** New destinations do not backfill. Only events recorded after the destination is created are streamed.
|
||||
- **Delivery errors:** Check the destination’s **last error** in the dashboard. Common causes: wrong URL, auth failure, TLS issues, or receiver rejecting malformed JSON.
|
||||
- **Quoting `{{data}}`:** `"payload": "{{data}}"` treats the entire row as a string, which is almost always wrong. Use `"payload": {{data}}`.
|
||||
- **Splunk field extraction:** Pangolin does not emit Splunk-style indexed fields in the template. Parse `data` or use a receiver-side pipeline.
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: "Amazon S3"
|
||||
description: "Archive audit logs to S3 or S3-compatible object storage"
|
||||
---
|
||||
|
||||
S3 destinations upload batches of your organization's audit logs as objects in a bucket you control. Use them for long-term archival, data lakes (Athena, Glue, BigQuery), or S3-compatible stores such as MinIO and Cloudflare R2.
|
||||
|
||||
<Note>
|
||||
Event streaming is only available in [Pangolin Cloud](https://app.pangolin.net/auth/signup) or self-hosted [Enterprise Edition](/self-host/enterprise-edition).
|
||||
</Note>
|
||||
|
||||
## Overview
|
||||
|
||||
An S3 destination writes **one object per batch** via `PutObject`. Each object contains up to 250 events of a **single log type**. There is no custom body template or field mapping; Pangolin serializes every event in a fixed shape and chooses the object key automatically.
|
||||
|
||||
Configure:
|
||||
|
||||
1. **Settings:** Name, credentials, region, bucket, optional prefix and custom endpoint.
|
||||
2. **Format:** File format (JSON array, NDJSON, or CSV) and optional gzip compression.
|
||||
3. **Logs:** Which log types are forwarded.
|
||||
|
||||
## Settings tab
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| Name | Yes | Display label for this destination |
|
||||
| AWS Access Key ID | Yes | Static access key for the S3 client |
|
||||
| AWS Secret Access Key | Yes | Secret for the access key |
|
||||
| AWS Region | Yes | S3 client region (UI default: `us-east-1`) |
|
||||
| Bucket name | Yes | Target bucket |
|
||||
| Key prefix | No | Prepended to every object key; trailing slashes are stripped |
|
||||
| Custom endpoint | No | Base URL for MinIO, R2, etc.; leave blank for AWS S3 |
|
||||
|
||||
Pangolin uses static access keys only. There is no IAM role, instance profile, or OIDC picker in the UI.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-s3-settings.png" alt="S3 destination settings with credentials, region, and bucket" />
|
||||
</Frame>
|
||||
|
||||
Uploads time out after 60 seconds per object.
|
||||
|
||||
## Format tab
|
||||
|
||||
**Gzip compression** (optional): When enabled, the object body is gzip-compressed before upload, `Content-Encoding: gzip` is set, and the object key gets a `.gz` suffix (for example `….json.gz`). Decompress before parsing unless your tool handles gzip automatically.
|
||||
|
||||
**File format:**
|
||||
|
||||
| Format | Description |
|
||||
| --- | --- |
|
||||
| **JSON array** (default) | One array per object: `[{…}, {…}, …]` |
|
||||
| **NDJSON** | One JSON object per line, no outer array |
|
||||
| **CSV** | RFC-4180 CSV with a header row; see [CSV format](#csv-format) |
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-s3-format.png" alt="Format tab with file format and gzip options" />
|
||||
</Frame>
|
||||
|
||||
## Logs tab
|
||||
|
||||
Choose which log categories are uploaded. Each enabled type is written to its own key prefix (`request/`, `action/`, etc.). Only log types enabled for your organization can be streamed.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/streaming-s3-logs.png" alt="Logs tab for selecting streamed log types" />
|
||||
</Frame>
|
||||
|
||||
## Object key layout
|
||||
|
||||
Every upload gets a unique key:
|
||||
|
||||
```
|
||||
{prefix}/{logType}/{YYYY}/{MM}/{DD}/{HH-mm-ss-uuid}.{ext}[.gz]
|
||||
```
|
||||
|
||||
| Segment | Meaning |
|
||||
| --- | --- |
|
||||
| `prefix` | Your optional key prefix; omitted when empty |
|
||||
| `logType` | `request`, `action`, `access`, or `connection` |
|
||||
| `YYYY/MM/DD` | **Upload time (UTC)**, not the event timestamp |
|
||||
| `HH-mm-ss-uuid` | Upload time plus a UUID so keys never collide |
|
||||
| `ext` | `json` (JSON array), `ndjson`, or `csv` |
|
||||
| `.gz` | Present when gzip is enabled |
|
||||
|
||||
**Without prefix:**
|
||||
|
||||
```
|
||||
request/2026/06/04/14-30-45-a1b2c3d4-e5f6-7890-abcd-ef1234567890.json
|
||||
```
|
||||
|
||||
**With prefix `pangolin/audit` and gzip:**
|
||||
|
||||
```
|
||||
pangolin/audit/action/2026/06/04/14-30-45-a1b2c3d4-e5f6-7890-abcd-ef1234567890.json.gz
|
||||
```
|
||||
|
||||
Enabling multiple log types on one destination produces **separate object streams** under different `logType/` segments. A single object never mixes log types.
|
||||
|
||||
## Event record shape
|
||||
|
||||
Each event in JSON and NDJSON objects uses this fixed structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "request",
|
||||
"timestamp": "2026-06-04T12:00:00.000Z",
|
||||
"data": {
|
||||
"timestamp": 1717492800,
|
||||
"action": true,
|
||||
"method": "GET",
|
||||
"path": "/api/health"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `event` | Log type: `request`, `access`, `action`, or `connection` |
|
||||
| `timestamp` | Event time as ISO-8601 UTC (connection logs use session start) |
|
||||
| `data` | The **complete stored log row** for that record, not a curated subset |
|
||||
|
||||
<Warning>
|
||||
Some columns are stored as JSON strings in the database (`headers`, `query`, and `metadata` on request logs, for example). In `data`, they appear as **string values**, not nested JSON objects. Parse them in your pipeline if you need structured fields.
|
||||
</Warning>
|
||||
|
||||
## File formats
|
||||
|
||||
### JSON array (default)
|
||||
|
||||
- One S3 object per batch; body is `[{…}, {…}, …]`.
|
||||
- Up to 250 events per object.
|
||||
- `Content-Type: application/json`.
|
||||
|
||||
### NDJSON
|
||||
|
||||
- One S3 object per batch; body is one JSON record per line with no outer array.
|
||||
- Good for Athena, BigQuery load jobs, Spark, and similar line-oriented pipelines.
|
||||
- `Content-Type: application/x-ndjson`.
|
||||
|
||||
### CSV format
|
||||
|
||||
- Header row: `event`, `timestamp`, then **all field names** found in `data` across that batch (union of keys, in insertion order).
|
||||
- Each data row flattens `event`, `timestamp`, and spreads `data` fields into columns. There is **no** nested `data` column.
|
||||
- Missing fields in a given row leave an empty cell.
|
||||
- Object or array values in `data` are written as `JSON.stringify` strings inside the cell.
|
||||
- `Content-Type: text/csv; charset=utf-8`.
|
||||
|
||||
The column set can grow as new fields appear in later batches. Order is not guaranteed to stay identical across all objects over time.
|
||||
|
||||
## Batching and throughput
|
||||
|
||||
- Objects are written **per batch** (up to ~250 events), not one object per log line.
|
||||
- Pangolin polls for new logs on a regular interval and may write multiple objects during catch-up after a pause.
|
||||
- **No backfill:** New destinations start from the current log cursor. Historical logs already in Pangolin are not uploaded.
|
||||
- **Extended outage:** If the destination is unreachable for about 24 hours, the backlog may be discarded and streaming resumes from the present cursor (same behavior as [HTTP streaming](/manage/analytics/streaming/http)).
|
||||
|
||||
## Gzip
|
||||
|
||||
When gzip is enabled:
|
||||
|
||||
1. The serialized body is compressed before upload.
|
||||
2. The object key includes `.gz` (for example `….ndjson.gz`).
|
||||
3. S3 stores `Content-Encoding: gzip`.
|
||||
|
||||
Consumers must decompress before parsing unless the tool auto-detects gzip (many Athena and Spark setups do when `Content-Encoding` is set). NDJSON plus gzip is a common choice for cost-sensitive archival.
|
||||
|
||||
## S3-compatible storage
|
||||
|
||||
Set **Custom endpoint** to your vendor's S3 API URL and provide access key credentials per that vendor's documentation.
|
||||
|
||||
| Store | Notes |
|
||||
| --- | --- |
|
||||
| **AWS S3** | Leave custom endpoint blank; use a bucket in the configured region |
|
||||
| **MinIO** | Set endpoint to your MinIO server URL; use MinIO access keys |
|
||||
| **Cloudflare R2** | Set endpoint to your R2 S3 API URL; use R2 access keys |
|
||||
|
||||
Pangolin does not expose path-style vs virtual-hosted addressing, ACLs, SSE-KMS, storage class, or multipart tuning. Configure those in the vendor console or bucket policy.
|
||||
|
||||
## IAM and bucket policy
|
||||
|
||||
Grant the access key permission to write under your prefix. A minimal AWS example:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:PutObject"],
|
||||
"Resource": "arn:aws:s3:::your-bucket/pangolin/audit/*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:ListBucket"],
|
||||
"Resource": "arn:aws:s3:::your-bucket",
|
||||
"Condition": {
|
||||
"StringLike": { "s3:prefix": ["pangolin/audit/*"] }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Adjust bucket name and prefix to match your configuration. `ListBucket` is optional but useful when debugging missing objects.
|
||||
|
||||
Block public access, encryption at rest, lifecycle rules, and object tags are configured in AWS or your vendor console, not in Pangolin.
|
||||
|
||||
## Log type reference
|
||||
|
||||
The `data` object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under **Logs & Analytics** for the complete `data` shape.
|
||||
|
||||
## Limitations and troubleshooting
|
||||
|
||||
- **No custom JSON shape:** Fixed event record only. Use an HTTP destination if you need body templates or field remapping.
|
||||
- **No per-event objects:** Always batched (up to ~250 events per object).
|
||||
- **No mixed log types in one object:** Each upload contains a single log type.
|
||||
- **Upload-time partitioning:** Key date folders use upload time (UTC), not the event's `timestamp`.
|
||||
- **CSV columns:** Automatic from batch contents; not user-selectable; column set may change over time.
|
||||
- **Static credentials only:** Rotate keys by updating the destination; credentials are stored encrypted server-side.
|
||||
- **Historical logs:** New destinations do not backfill.
|
||||
- **Delivery errors:** Check the destination's **last error** in the dashboard. Common causes: `AccessDenied`, wrong bucket or region, bad endpoint URL, TLS issues, or expired credentials.
|
||||
- **Missing objects:** Confirm prefix, lifecycle rules, and that the log type is enabled on the **Logs** tab.
|
||||
- **Athena/Glue parse errors:** Verify format (JSON array vs NDJSON), gzip handling, and that the crawler/table schema matches flattened CSV columns if using CSV.
|
||||
Reference in New Issue
Block a user