Rework activity streaming

2026-10-09 15:19:04 +02:00 · 2024-08-13 17:55:44 +02:00
parent ccdbc37118
commit 407ccf0025
10 changed files with 275 additions and 158 deletions
@@ -80,7 +80,7 @@ export const docsNavigation = [
             links: [
                 { title: 'Add users to your network', href: '/how-to/add-users-to-your-network' },
                 {
-                     title: 'Provision Users & Groups from Identity Provider',
+                     title: 'Provision Users & Groups',
                     href: '/how-to/idp-sync',
                     isOpen: false,
                     links: [
@@ -94,8 +94,17 @@ export const docsNavigation = [
         {
             title: 'Activity',
             links: [
-                 { title: 'Activity events', href: '/how-to/monitor-system-and-network-activity' },
+                 { title: 'Activity Events', href: '/how-to/monitor-system-and-network-activity' },
-              
+                 {
                     title: 'Stream Activity Events',
                     href: '/how-to/activity-event-streaming',
                     isOpen: false,
                     links: [
                         { title: 'Datadog', href: '/how-to/stream-activity-to-datadog' },
                         { title: 'Amazon S3', href: '/how-to/stream-activity-to-amazon-s3' },
                         { title: 'Amazon Firehose', href: '/how-to/stream-activity-to-amazon-firehose'},
                     ]
                 },
             ],
         },
         {
@@ -1,4 +1,8 @@
-# Stream activity events to third-party SIEM systems
+# Stream Network Activity to Third-Party SIEM Platforms
 <p>
    <img src="/docs-static/img/how-to-guides/activity-event-streaming/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 Security Information and Event Management (SIEM) systems play a critical role in network security by monitoring,
 detecting, and responding to security threats in real-time. By aggregating and analyzing activity across the network,
@@ -7,156 +11,13 @@ SIEMs help identify anomalous patterns and potential breaches, providing a centr
 NetBird provides an event streaming feature that allows you to stream network [activity events](/how-to/monitor-system-and-network-activity)
 to third-party SIEM systems, such as [Datadog](https://www.datadoghq.com/dg/security/siem-solution/), [Amazon S3](https://aws.amazon.com/s3/), [Amazon Data Firehose](https://aws.amazon.com/firehose/), and others.
 This document provides step-by-step instructions and best practices for setting up NetBird activity event streaming
 integrations to different third-party platforms.
 <Note>
-     This feature is only available in the cloud version of NetBird.
+    This feature is only available in the cloud version of NetBird.
 </Note>
-## Datadog
+This documentation provides step-by-step guides and best practices for integrating NetBird activity event streaming with
 supported third-party platforms. To get started, select one of the following integrations:
-Before you start creating and configuring a Datadog event streaming integration, ensure that you have the following:
+- [Datadog](/how-to/stream-activity-to-datadog)
- A Datadog account with the permissions to create and manage API keys. If you don't have the required permissions, ask your Datadog administrator to grant them to you.
+- [Amazon S3](/how-to/stream-activity-to-amazon-s3)
-
+- [Amazon Data Firehose](/how-to/stream-activity-to-amazon-firehose)
 ### Step 1: Create a Datdog API key
 - Navigate to the [API Keys](https://app.datadoghq.eu/organization-settings/api-keys) page
 - Click `+ New Key` at the top
 - Give it a descriptive name like `NetBird Event Streaming`
 - Click `Create Key`
 - Copy the key. You will need this key when configuring an integration in NetBird.
 ### Step 2: Create an event streaming integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the Datadog integration
  - First select the region of your Datadog account (for more details see [Datadog Documentation](https://docs.datadoghq.com/getting_started/site/))
 <p>
  <img src="/docs-static/img/how-to-guides/datadog-region-select.png" alt="datadog-region-select" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
  - Then enter the API key you created in [Step 1](#step-1-create-a-datdog-api-key) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/datadog-api-key.png" alt="datadog-api-key" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 ## Amazon S3
 Before you start creating and configuring an Amazon S3 event streaming integration, ensure that you have the following:
 - An AWS account with the permissions to create and manage S3 buckets.
 - Permissions to create and manage IAM users, roles and policies.
 If you don't have the required permissions, ask your AWS administrator to grant them to you.
 ### Step 1: Create an S3 bucket
 - Navigate to the [S3 dashboard](https://console.aws.amazon.com/s3/home)
 - Select the correct region in the top menu
 - Click `Create bucket`
 - Give it a descriptive name like `netbird-activity-events`
 - (Optional) Change bucket configurations to your needs
 - Click `Create bucket`
 ### Step 2: Create an IAM user
 - Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
 - Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
 - Create a custom policy with the following permissions:
 ```json
 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::netbird-activity-events/*"
        }
    ]
 }
 ```
 - Attach the policy to the IAM user
 - Select the user and navigate to the `Security credentials` tab
 - Click `Create access key`
 - Select `Third-party service` and click `Next`
 - Give it a description
 - Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
 ### Step 3: Create an event streaming integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the `Amazon S3` integration
 - First select the region your S3 bucket is created in
 <p>
    <img src="/docs-static/img/how-to-guides/s3-region-select.png" alt="s3-region-select" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 - Then enter the S3 bucket name you created in [Step 1](#step-1-create-an-s3-bucket) and click `Next`
 <p>
    <img src="/docs-static/img/how-to-guides/s3-bucket-name.png" alt="s3-bucket-name" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 - Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/s3-iam-credentials.png" alt="s3-iam-credentials" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 ## Amazon Data Firehose
 Before you start creating and configuring an Amazon Data Firehose event streaming integration, ensure that you have the following:
 - An AWS account with the permissions to create and manage data firehose delivery streams.
 - Permissions to create and manage IAM users, roles and policies.
 If you don't have the required permissions, ask your AWS administrator to grant them to you.
 ### Step 1: Create a Data Firehose stream
 - Navigate to the [Data Firehose Dashboard](https://console.aws.amazon.com/firehose/home)
 - Click `Create Firehose stream`
 - As source select `Direct PUT` and the desired destination
 - Give it a descriptive name like `netbird-activity-events` and configure the stream to your needs
 ### Step 2: Create an IAM user
 - Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
 - Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
 - Create a custom policy with the following permissions (replace the resource with the ARN of your delivery stream):
 ```json
 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:region:accountID:deliverystream/netbird-event-streaming"
        }
    ]
 }
 ```
 - Attach the policy to the IAM user
 - Select the user and navigate to the `Security credentials` tab
 - Click `Create access key`
 - Select `Third-party service` and click `Next`
 - Give it a description
 - Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
 ### Step 3: Create an event streaming integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the `Amazon Data Firehose` integration
 - First select the region your Firehose stream is created in
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-region-select.png" alt="firehose-region-select" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 - Then enter the Firehose stream name you created in [Step 1](#step-1-create-a-data-firehose-stream) and click `Next`
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-stream-name.png" alt="firehose-stream-name" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
 - Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user-2) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-iam-credentials.png" alt="firehose-iam-credentials" className="imagewrapper" width="400" style={{ display: 'block', marginLeft: 'auto', marginRight: 'auto' }} />
 </p>
@@ -1,5 +1,10 @@
 # Provision Users and Groups From Your Identity Provider
 <p>
    <img src="/docs-static/img/how-to-guides/supported-identity-providers.png" alt="supported-identity-providers"
         className="imagewrapper-big"/>
 </p>
 Managing user access to a private network in a business environment is a critical yet often cumbersome task.
 As companies grow and evolve, the manual process of granting network access for new employees and revoking it for
 departing ones becomes increasingly time-consuming and error-prone. This challenge strains IT resources, poses significant
@@ -23,11 +28,6 @@ eliminating the need for manual grouping.
 ## Supported Identity Providers
 <p>
    <img src="/docs-static/img/how-to-guides/supported-identity-providers.png" alt="supported-identity-providers"
         className="imagewrapper-big"/>
 </p>
 NetBird provides native support for syncing with the most popular identify providers.
 For detailed setup and configuration steps, select an IdP from the section below:
@@ -0,0 +1,69 @@
 # Stream Network Activity to Amazon Data Firehose
 [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/) is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift,
 an other AWS services. You can use Amazon Data Firehose as a bridge between NetBird and other third-party providers that support Data Firehose
 to ingest, transform and analyze your network activity events.
 <Note>
    This feature is only available in the cloud version of NetBird.
 </Note>
 ## Prerequisites
 Before you start creating and configuring an Amazon Data Firehose event streaming integration, ensure that you have the following:
 - An AWS account with the permissions to create and manage Data Firehose delivery streams.
 - Permissions to create and manage IAM users, roles and policies.
 If you don't have the required permissions, ask your AWS administrator to grant them to you.
 ## Create a Data Firehose Stream
 - Navigate to the [Data Firehose Dashboard](https://console.aws.amazon.com/firehose/home)
 - Click `Create Firehose stream`
 - As source select `Direct PUT` and the desired destination
 - Give it a descriptive name like `netbird-activity-events` and configure the stream to your needs
 ## Create an IAM User
 - Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
 - Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
 - Create a custom policy with the following permissions (replace the resource with the ARN of your delivery stream):
 ```json
 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:region:accountID:deliverystream/netbird-event-streaming"
        }
    ]
 }
 ```
 - Attach the policy to the IAM user
 - Select the user and navigate to the `Security credentials` tab
 - Click `Create access key`
 - Select `Third-party service` and click `Next`
 - Give it a description
 - Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
 ## Create an Integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the `Amazon Data Firehose` integration
 - First select the region your Firehose stream is created in
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-region-select.png" alt="firehose-region-select" className="imagewrapper" />
 </p>
 - Then enter the Firehose stream name you created in [Step 1](#step-1-create-a-data-firehose-stream) and click `Next`
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-stream-name.png" alt="firehose-stream-name" className="imagewrapper" />
 </p>
 - Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user-2) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/firehose-iam-credentials.png" alt="firehose-iam-credentials" className="imagewrapper" />
 </p>
@@ -0,0 +1,119 @@
 # Stream Network Activity to Amazon S3
 [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) is a scalable, high-speed, web-based cloud storage service.
 You can use Amazon S3 to store and analyze large amounts of data, including network activity events from NetBird.
 NetBird integrates with Amazon S3 and sends activity events to an S3 bucket in real-time once they occur. For every event,
 NetBird creates a new object in the S3 bucket, which you can then analyze, filter, and query using Amazon tools.
 Storing one event per object is not the most efficient way to save data in S3, therefore NetBird provides an
 [alternative integration](/how-to/stream-activity-to-amazon-firehose) that uses Amazon Data Firehose for a more efficient
 data ingestion.
 <Note>
    This feature is only available in the cloud version of NetBird.
 </Note>
 ## Prerequisites
 Before you start creating and configuring an Amazon S3 event streaming integration, ensure that you have the following:
 - An AWS account with the permissions to create and manage S3 buckets.
 - Permissions to create and manage IAM users, roles and policies.
 If you don't have the required permissions, ask your AWS administrator to grant them to you.
 ## Create an S3 Bucket
 - Navigate to the [S3 dashboard](https://console.aws.amazon.com/s3/home)
 - Select the correct region in the top menu
 - Click `Create bucket`
 - Give it a descriptive name like `netbird-activity-events`
 - (Optional) Change bucket configurations to your needs
 - Click `Create bucket`
 ## Create an IAM User
 - Navigate to the [IAM Dashboard](https://console.aws.amazon.com/iam/home)
 - Create an IAM User (for details see the [Amazon Docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html))
 - Create a custom policy with the following permissions:
 ```json
 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::netbird-activity-events/*"
        }
    ]
 }
 ```
 - Attach the policy to the IAM user
 - Select the user and navigate to the `Security credentials` tab
 - Click `Create access key`
 - Select `Third-party service` and click `Next`
 - Give it a description
 - Store `Access key` and `Secret access key` in a secure place. You will need these when configuring an integration in NetBird.
 ## Create an Integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the `Amazon S3` integration
 - First select the region your S3 bucket is created in
 <p>
    <img src="/docs-static/img/how-to-guides/s3-region-select.png" alt="s3-region-select" className="imagewrapper" />
 </p>
 - Then enter the S3 bucket name you created in [Step 1](#step-1-create-an-s3-bucket) and click `Next`
 <p>
    <img src="/docs-static/img/how-to-guides/s3-bucket-name.png" alt="s3-bucket-name" className="imagewrapper"/>
 </p>
 - Enter the `Access key` and `Secret access key` you created in [Step 2](#step-2-create-an-iam-user) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/s3-iam-credentials.png" alt="s3-iam-credentials" className="imagewrapper" />
 </p>
 ## Verify the Integration
 After configuring the integration in NetBird, you can verify that the integration is working correctly by checking
 the S3 bucket for newly created objects. If the integration is successful, you should see two `.json` files in your bucket
 containing the following events:
 - `integration test`
 - `integration created`
 <p>
    <img src="/docs-static/img/how-to-guides/activity-event-streaming/verify-amazon-s3-integration.png" alt="verify-amazon-s3-integration" className="imagewrapper-big" />
 </p>
 ```json
 {
    "timestamp": "2024-04-11T19:01:05.381074+02:00",
    "message": "integration test",
    "id": 0,
    "initiator_id": "netbird",
    "target_id": "netbird",
    "meta": null,
    "reference": ""
 }
 ```
 ```json
 {
    "timestamp": "2024-04-11T19:01:08.769292+02:00",
    "message": "integration created",
    "id": 0,
    "initiator_id": "netbird",
    "target_id": "netbird",
    "meta": null,
    "reference": ""
 }
 ```
 The integration test event is sent to validate whether the provided credentials are correct and NetBird can stream events.
 The integration created event is sent when the integration is successfully created.
 The integration is now set up and ready to stream network activity events to Amazon S3.
@@ -0,0 +1,59 @@
 # Stream Network Activity to Datadog Cloud SIEM
 Datadog is a monitoring and analytics platform for cloud-scale applications. Datadog Cloud SIEM provides real-time threat
 detection and security monitoring for cloud environments. By using the NetBird-Datadog integration, you can stream [network activity](/how-to/monitor-system-and-network-activity) to Datadog Cloud SIEM for real-time monitoring
 and threat detection across your private network.
 NetBird integrates with Datadog using the [Datadog Log Collection HTTP API](https://docs.datadoghq.com/api/latest/logs/#send-logs)
 and sends activity events to Datadog in real-time once they occur. The events appear in the [Datadog Log Explorer](https://docs.datadoghq.com/logs/explorer/), where you can
 search, filter, and analyze them right away.
 <Note>
    This feature is only available in the cloud version of NetBird.
 </Note>
 ## Prerequisites
 Before you start creating and configuring a Datadog event streaming integration, ensure that you have the following:
 - A Datadog account with permissions to create and manage API keys. If you don't have the required permissions,
 ask your Datadog administrator to grant them to you.
 ## Create a Datdog API Key
 - Navigate to the [API Keys](https://app.datadoghq.eu/organization-settings/api-keys) page
 - Click `+ New Key` at the top
 - Give it a descriptive name like `NetBird Event Streaming`
 - Click `Create Key`
 - Copy the key. You will need this key when configuring an integration in NetBird.
 ## Create an Integration in NetBird
 - Navigate to the [Integrations &raquo; Event Streaming](https://preview.netbird.io/integrations) tab in the NetBird Dashboard
 <p>
    <img src="/docs-static/img/how-to-guides/event-streaming-integration.png" alt="event-streaming-integration" className="imagewrapper-big"/>
 </p>
 - Enable and configure the Datadog integration
  - First select the region of your Datadog account (for more details see [Datadog Documentation](https://docs.datadoghq.com/getting_started/site/))
 <p>
  <img src="/docs-static/img/how-to-guides/datadog-region-select.png" alt="datadog-region-select" className="imagewrapper" />
 </p>
  - Then enter the API key you created in [Step 1](#step-1-create-a-datdog-api-key) and click `Connect`
 <p>
    <img src="/docs-static/img/how-to-guides/datadog-api-key.png" alt="datadog-api-key" className="imagewrapper" />
 </p>
 ## Verify the Integration
 After configuring the Datadog integration in NetBird, you can verify that the integration is working correctly by checking
 the Datadog Log Explorer for incoming events. If the integration is successful, you should see two events from the netbird service
 in the Log Explorer:
 - `integration test`
 - `integration created`
 <p>
    <img src="/docs-static/img/how-to-guides/activity-event-streaming/verify-datadog-integration.png" alt="verify-datadog-integration" className="imagewrapper"/>
 </p>
 The `integration test` event is sent to validate whether the provided credentials are correct and NetBird can stream events.
 The `integration created` event is sent when the integration is successfully created.
 The integration is now set up and ready to stream network activity events to Datadog.