Restructuring Phase 3 (#492)

This commit is contained in:
Brandon Hopkins
2025-11-24 18:25:44 +01:00
committed by GitHub
parent c40c132caa
commit 0080ae97df
477 changed files with 800 additions and 564 deletions
@@ -0,0 +1,69 @@
# Stream Network Activity to Amazon Data Firehose
[Amazon Data Firehose](https://docs.aws.amazon.com/firehose/) is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift,
an other AWS services. You can use Amazon Data Firehose as a bridge between NetBird and other third-party providers that support Data Firehose
to ingest, transform and analyze your network activity events.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
## Prerequisites
Before you start creating and configuring an Amazon Data Firehose event streaming integration, ensure that you have the following:
- An AWS account with the permissions to create and manage Data Firehose delivery streams.
- Permissions to create and manage IAM users, roles and policies.
If you don't have the required permissions, ask your AWS administrator to grant them to you.
## Create a Data Firehose Stream
- Navigate to the [Data Firehose Dashboard](https://console.aws.amazon.com/firehose/home)
- Click `Create Firehose stream`
- As source select `Direct PUT` and the desired destination
- Give it a descriptive name like `netbird-activity-events` and configure the stream to your needs
## Create an IAM User
- Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
- Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
- Create a custom policy with the following permissions (replace the resource with the ARN of your delivery stream):
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"firehose:PutRecord",
"firehose:PutRecordBatch"
],
"Resource": "arn:aws:firehose:region:accountID:deliverystream/netbird-event-streaming"
}
]
}
```
- Attach the policy to the IAM user
- Select the user and navigate to the `Security credentials` tab
- Click `Create access key`
- Select `Third-party service` and click `Next`
- Give it a description
- Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
## Create an Integration in NetBird
- Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
<p>
<img src="/docs-static/img/manage/activity/event-streaming/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
</p>
- Enable and configure the `Amazon Data Firehose` integration
- First select the region your Firehose stream is created in
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-firehose/firehose-region-select.png" alt="firehose-region-select" className="imagewrapper" />
</p>
- Then enter the Firehose stream name you created in [Step 1](#step-1-create-a-data-firehose-stream) and click `Next`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-firehose/firehose-stream-name.png" alt="firehose-stream-name" className="imagewrapper" />
</p>
- Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user-2) and click `Connect`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-firehose/firehose-iam-credentials.png" alt="firehose-iam-credentials" className="imagewrapper" />
</p>
@@ -0,0 +1,119 @@
# Stream Network Activity to Amazon S3
[Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) is a scalable, high-speed, web-based cloud storage service.
You can use Amazon S3 to store and analyze large amounts of data, including network activity events from NetBird.
NetBird integrates with Amazon S3 and sends activity events to an S3 bucket in real-time once they occur. For every event,
NetBird creates a new object in the S3 bucket, which you can then analyze, filter, and query using Amazon tools.
Storing one event per object is not the most efficient way to save data in S3, therefore NetBird provides an
[alternative integration](/manage/activity/event-streaming/amazon-firehose) that uses Amazon Data Firehose for a more efficient
data ingestion.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
## Prerequisites
Before you start creating and configuring an Amazon S3 event streaming integration, ensure that you have the following:
- An AWS account with the permissions to create and manage S3 buckets.
- Permissions to create and manage IAM users, roles and policies.
If you don't have the required permissions, ask your AWS administrator to grant them to you.
## Create an S3 Bucket
- Navigate to the [S3 dashboard](https://console.aws.amazon.com/s3/home)
- Select the correct region in the top menu
- Click `Create bucket`
- Give it a descriptive name like `netbird-activity-events`
- (Optional) Change bucket configurations to your needs
- Click `Create bucket`
## Create an IAM User
- Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
- Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
- Create a custom policy with the following permissions:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::netbird-activity-events/*"
}
]
}
```
- Attach the policy to the IAM user
- Select the user and navigate to the `Security credentials` tab
- Click `Create access key`
- Select `Third-party service` and click `Next`
- Give it a description
- Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
## Create an Integration in NetBird
- Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
<p>
<img src="/docs-static/img/manage/activity/event-streaming/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
</p>
- Enable and configure the `Amazon S3` integration
- First select the region your S3 bucket is created in
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-s3/s3-region-select.png" alt="s3-region-select" className="imagewrapper" />
</p>
- Then enter the S3 bucket name you created in [Step 1](#step-1-create-an-s3-bucket) and click `Next`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-s3/s3-bucket-name.png" alt="s3-bucket-name" className="imagewrapper"/>
</p>
- Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user) and click `Connect`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-s3/s3-iam-credentials.png" alt="s3-iam-credentials" className="imagewrapper" />
</p>
## Verify the Integration
After configuring the integration in NetBird, you can verify that the integration is working correctly by checking
the S3 bucket for newly created objects. If the integration is successful, you should see two `.json` files in your bucket
containing the following events:
- `integration test`
- `integration created`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/amazon-s3/verify-amazon-s3-integration.png" alt="verify-amazon-s3-integration" className="imagewrapper-big" />
</p>
```json
{
"timestamp": "2024-04-11T19:01:05.381074+02:00",
"message": "integration test",
"id": 0,
"initiator_id": "netbird",
"target_id": "netbird",
"meta": null,
"reference": ""
}
```
```json
{
"timestamp": "2024-04-11T19:01:08.769292+02:00",
"message": "integration created",
"id": 0,
"initiator_id": "netbird",
"target_id": "netbird",
"meta": null,
"reference": ""
}
```
The integration test event is sent to validate whether the provided credentials are correct and NetBird can stream events.
The integration created event is sent when the integration is successfully created.
The integration is now set up and ready to stream network activity events to Amazon S3.
@@ -0,0 +1,59 @@
# Stream Network Activity to Datadog Cloud SIEM
Datadog is a monitoring and analytics platform for cloud-scale applications. Datadog Cloud SIEM provides real-time threat
detection and security monitoring for cloud environments. By using the NetBird-Datadog integration, you can stream [network activity](/manage/activity/traffic-events-logging) to Datadog Cloud SIEM for real-time monitoring
and threat detection across your private network.
NetBird integrates with Datadog using the [Datadog Log Collection HTTP API](https://docs.datadoghq.com/api/latest/logs/#send-logs)
and sends activity events to Datadog in real-time once they occur. The events appear in the [Datadog Log Explorer](https://docs.datadoghq.com/logs/explorer/), where you can
search, filter, and analyze them right away.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
## Prerequisites
Before you start creating and configuring a Datadog event streaming integration, ensure that you have the following:
- A Datadog account with permissions to create and manage API keys. If you don't have the required permissions,
ask your Datadog administrator to grant them to you.
## Create a Datdog API Key
- Navigate to the [API Keys](https://app.datadoghq.eu/organization-settings/api-keys) page
- Click `+ New Key` at the top
- Give it a descriptive name like `NetBird Event Streaming`
- Click `Create Key`
- Copy the key. You will need this key when configuring an integration in NetBird.
## Create an Integration in NetBird
- Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird Dashboard
<p>
<img src="/docs-static/img/manage/activity/event-streaming/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
</p>
- Enable and configure the Datadog integration
- First select the region of your Datadog account (for more details see [Datadog Documentation](https://docs.datadoghq.com/getting_started/site/))
<p>
<img src="/docs-static/img/manage/activity/event-streaming/datadog/datadog-region-select.png" alt="datadog-region-select" className="imagewrapper" />
</p>
- Then enter the API key you created in [Step 1](#step-1-create-a-datdog-api-key) and click `Connect`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/datadog/datadog-api-key.png" alt="datadog-api-key" className="imagewrapper" />
</p>
## Verify the Integration
After configuring the Datadog integration in NetBird, you can verify that the integration is working correctly by checking
the Datadog Log Explorer for incoming events. If the integration is successful, you should see two events from the netbird service
in the Log Explorer:
- `integration test`
- `integration created`
<p>
<img src="/docs-static/img/manage/activity/event-streaming/datadog/verify-datadog-integration.png" alt="verify-datadog-integration" className="imagewrapper"/>
</p>
The `integration test` event is sent to validate whether the provided credentials are correct and NetBird can stream events.
The `integration created` event is sent when the integration is successfully created.
The integration is now set up and ready to stream network activity events to Datadog.
@@ -0,0 +1,102 @@
# Stream Network Activity to a Generic HTTP Endpoint
The Generic HTTP integration allows you to stream your NetBird network activity events to any custom HTTP/S endpoint. This provides a flexible way to connect NetBird with a wide range of third-party systems, including custom SIEMs, log management tools like Grafana Loki, or any service that can receive POST requests.
For every event, NetBird will send a POST request to your configured endpoint. You have full control over the request's body format and headers, allowing for seamless integration with various APIs.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
## Prerequisites
Before you start, ensure you have an HTTP/S endpoint that is publicly accessible and ready to receive POST requests from NetBird. You will also need any necessary authentication details, such as an API key or credentials for Basic Authentication.
## Create an Integration in NetBird
1. Navigate to the **Integrations** tab in the NetBird dashboard and select **Event Streaming**.
2. Find the **Generic HTTP** option and click **Connect**.
3. A configuration panel will open with several tabs: **General**, **Headers**, **Body Template**, and **Danger Zone**.
### General Configuration
This tab is for the essential endpoint details.
<p>
<img src="/docs-static/img/manage/activity/event-streaming/generic-http/general-config.png" alt="Generic HTTP General Configuration" className="imagewrapper-big"/>
</p>
- **Endpoint URL**: Enter the full HTTPS or HTTP URL where NetBird should send the events. This field is mandatory.
- **Authentication**: Select the authentication method required by your endpoint.
- **No Authentication**: Sends requests without an `Authorization` header.
- **Bearer Token**: Adds an `Authorization: Bearer <your-token>` header to each request. You will need to provide your API key or token.
- **Basic Auth**: Adds an `Authorization: Basic <credentials>` header. You must provide the credentials in the format `username:password`.
- **Custom Authentication**: Allows you to specify a custom HTTP header for authentication. You will need to provide both the header name (e.g., `X-API-Key`) and its value.
<Note>
After saving the integration, the configured authentication details will be converted into an HTTP header and will appear in the **Headers** tab.
</Note>
### HTTP Headers (Optional)
You can add custom HTTP headers to every outgoing request in the **Headers** tab. This is useful for passing static tokens, setting a custom `Content-Type`, or other API requirements. By default, the `Content-Type` is `application/json`.
<p>
<img src="/docs-static/img/manage/activity/event-streaming/generic-http/headers-config.png" alt="Generic HTTP Headers Configuration" className="imagewrapper-big"/>
</p>
### Custom Body Template (Optional)
The **Body Template** tab gives you powerful control over the structure of the JSON payload sent to your endpoint.
<p>
<img src="/docs-static/img/manage/activity/event-streaming/generic-http/body-template-config.png" alt="Generic HTTP Body Template Configuration" className="imagewrapper-big"/>
</p>
If this option is disabled, NetBird sends a default JSON object for each event. When enabled, you can define your own payload structure using Go's `text/template` templating engine.
This is especially useful for integrating with services that expect a specific format, like Grafana Loki.
#### Available Template Variables
You can use the following variables from the `StreamEvent` object in your template:
| Variable | Type | Description | Example Usage |
|---------------|----------------------|--------------------------------------------------------------------------------------|-----------------------------------|
| `.ID` | `any` | The unique ID of the event. | `{"event_id": "{{.ID}}"}` |
| `.Timestamp` | `time.Time` | The timestamp of when the event occurred. Can be formatted. | `{{.Timestamp.UnixNano}}` / `{{.Timestamp.Format "2006-01-02T15:04:05.999Z07:00"}}` |
| `.Message` | `string` | A human-readable message describing the event. | `{"text": "{{.Message}}"}` |
| `.InitiatorID`| `string` | The ID of the object that initiated the event (e.g., a user or peer ID). | `{"user": "{{.InitiatorID}}"}` |
| `.TargetID` | `string` | The ID of the object that was affected by the event (e.g., a peer or group ID). | `{"resource": "{{.TargetID}}"}` |
| `.Meta` | `map[string]any` | A map containing additional metadata about the event. | `{"source": "{{.Meta.source}}"}` |
| `.Reference` | `string` | A URL to the event in the NetBird activity log for easy cross-referencing. | `{"ref": "{{.Reference}}"}` |
### Danger Zone
This tab allows you to delete the integration. This action is irreversible and will immediately stop events from being sent to your endpoint.
<p>
<img src="/docs-static/img/manage/activity/event-streaming/generic-http/danger-zone.png" alt="Generic HTTP Danger Zone" className="imagewrapper-big"/>
</p>
After configuring all settings, click **Save Changes**.
## Verify the Integration
Once you save the integration, NetBird sends a test event to your endpoint to confirm that the connection and authentication are successful.
Check the logs of your receiving service for a new POST request. You should find a JSON payload similar to this (if custom template wasn't set up):
```json
{
"ID": "test-event-1685635200",
"Timestamp": "2025-06-01T16:00:00Z",
"Message": "integration test event for generic http endpoint",
"InitiatorID": "netbird-test-system",
"TargetID": "target-id",
"Meta": {
"purpose": "configuration-test",
"source": "netbird-test"
},
"Reference": "netbird:test:generichttp"
}
@@ -0,0 +1,25 @@
# Stream Network Activity to Third-Party SIEM Platforms
<p>
<img src="/docs-static/img/manage/activity/event-streaming/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
</p>
Security Information and Event Management (SIEM) systems play a critical role in network security by monitoring,
detecting, and responding to security threats in real-time. By aggregating and analyzing activity across the network,
SIEMs help identify anomalous patterns and potential breaches, providing a centralized view of security events.
NetBird provides an event streaming feature that allows you to stream network [activity events](/manage/activity/traffic-events-logging)
to third-party SIEM systems, such as [Datadog](https://www.datadoghq.com/dg/security/siem-solution/), [Amazon S3](https://aws.amazon.com/s3/), [Amazon Data Firehose](https://aws.amazon.com/firehose/), and others through a generic HTTP integration.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
This documentation provides step-by-step guides and best practices for integrating NetBird activity event streaming with
supported third-party platforms. To get started, select one of the following integrations:
- [Datadog](/manage/activity/event-streaming/datadog)
- [Amazon S3](/manage/activity/event-streaming/amazon-s3)
- [Amazon Data Firehose](/manage/activity/event-streaming/amazon-firehose)
- [SentinelOne Data Lake](/manage/activity/event-streaming/sentinelone-data-lake)
- [Generic HTTP](/manage/activity/event-streaming/generic-http)
@@ -0,0 +1,77 @@
# Stream Network Activity to SentinelOne Singularity Data Lake
[SentinelOne Singularity Data Lake](https://www.sentinelone.com/platform/data-lake/) provides a centralized repository for storing and analyzing security data across your organization. By streaming NetBird network activity events to SentinelOne Data Lake, you can correlate network connectivity events with endpoint security data for comprehensive threat detection and investigation.
This integration leverages NetBird's Generic HTTP streaming capability with SentinelOne Singularity Data Lake-specific configurations to ensure seamless data ingestion into your Data Lake environment.
<Note>
This feature is only available in the cloud version of NetBird.
</Note>
## Prerequisites
Before you start creating and configuring a SentinelOne Singularity Data Lake event streaming integration, ensure that you have the following:
- A SentinelOne account with Singularity Data Lake access and appropriate permissions
- Administrative access to create API keys in your SentinelOne console
## Create a SentinelOne API Token
- Log in to your SentinelOne Management Console and navigate to **Visibility** from the left menu
- Click on your user email dropdown in the top right toolbar and select **API Keys**
- Under the **Log Access Keys** section, click **Add Key** dropdown menu and select **Add Write Key**
- A new API key will be created with access to write logs to Data Lake
- Click **Copy to Clipboard** and store this token securely. You will need this token when configuring the integration in NetBird.
<p>
<img src="/docs-static/img/manage/activity/event-streaming/sentinelone-data-lake/api-key-generation.png" alt="API Key Generation" className="imagewrapper-big"/>
</p>
## Get the HEC Ingestion URL
Navigate to `https://<your-tenant>.sentinelone.net/docs/en/http-event-collector--hec-.html#http-event-collector--hec-` (replace `<your-tenant>` with your actual SentinelOne tenant)
to find and copy the HTTP Event Collector base URL from the documentation page. The URL format will be `https://ingest.<region>.sentinelone.net` where `<region>` is your deployment region.
## Create an Integration in NetBird
- Navigate to the **Integrations** tab in the NetBird dashboard and select **Event Streaming**
- Find the **Generic HTTP** option and click **Connect Generic HTTP**
- Configure the endpoint URL by entering the full HEC endpoint URL: `https://ingest.<region>.sentinelone.net/services/collector/event`
- Select **Bearer Token** for authentication and provide the API token you created earlier
<p>
<img src="/docs-static/img/manage/activity/event-streaming/sentinelone-data-lake/general-settings.png" alt="General Settings Configuration" className="imagewrapper-big"/>
</p>
- Enable the custom body template and use the following template optimized for SentinelOne Singularity Data Lake:
```json
{
"id": "{{.ID}}",
"time": "{{.Timestamp.UnixNano}}",
"event": "{{.Message}}",
"initiator_id": "{{.InitiatorID}}",
"target_id": "{{.TargetID}}",
"host": "netbird",
"source": "netbird",
"sourcetype": "json",
"fields": "{{.Meta}}"
}
```
<p>
<img src="/docs-static/img/manage/activity/event-streaming/sentinelone-data-lake/custom-template.png" alt="Custom Template Configuration" className="imagewrapper-big"/>
</p>
## Verify the Integration
After configuring the SentinelOne Singularity Data Lake integration in NetBird, you can verify that the integration is working correctly by checking the SentinelOne Singularity Data Lake for incoming events. If the integration is successful, you should see test events from NetBird in your Data Lake.
- Navigate to **Data Lake** > **Search** > **New Search** in your SentinelOne console
- Select to search **All Data** and filter for recent events with source **netbird**
- Verify the test events appear with the expected structure
<p>
<img src="/docs-static/img/manage/activity/event-streaming/sentinelone-data-lake/verify-integration.png" alt="Verify Integration" className="imagewrapper-big"/>
</p>
The integration is now set up and ready to stream network activity events to SentinelOne Singularity Data Lake.