# Access Private Kubernetes Clusters with NetBird Kubernetes Operator

Accessing private Kubernetes clusters can be challenging, especially when connecting from remote locations or
having multiple clusters to manage. NetBird Kubernetes operator simplifies this process by enabling secure access
to your Kubernetes clusters using custom resource configurations and annotations to expose your cluster and services in your NetBird network.

The NetBird Kubernetes operator automatically creates [Networks and Resources](/manage/networks) in your NetBird account, allowing you to
seamlessly access your Kubernetes services and control plane from your NetBird network.

## Deployment

### Prerequisites

- (Recommended) helm version 3+
- kubectl version v1.11.3+.
- Access to a Kubernetes v1.11.3+ cluster.
- (Recommended) Cert Manager.

#### Using Helm
1. Add helm repository.
```shell
helm repo add netbirdio https://netbirdio.github.io/helms
```
2. (Recommended) Install [cert-manager](https://cert-manager.io/docs/installation/#default-static-install) for k8s API to communicate with the NetBird operator.
```shell
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.yaml
```
3. Add NetBird API token. You can create a PAT by following the steps [here](/how-to/access-netbird-public-api#creating-a-service-user).
```shell
kubectl create namespace netbird
kubectl -n netbird create secret generic netbird-mgmt-api-key --from-literal=NB_API_KEY=$(cat ~/nb-pat.secret)
```
<Note>
  Replace `~/nb-pat.secret` with your NetBird API key.
</Note>

4. (Recommended) Create a [`values.yaml`](https://github.com/netbirdio/kubernetes-operator/blob/main/examples/ingress/values.yaml) file, check `helm show values netbirdio/kubernetes-operator` for more info.
```yaml
# by default the managementURL points to the NetBird cloud service: https://api.netbird.io:443
# managementURL: "https://netbird.example.io:443"
ingress:
  enabled: true
  router:
    enabled: true

netbirdAPI:
  keyFromSecret: 
    name: "netbird-mgmt-api-key"
    key: "NB_API_KEY"
```
5. Install using helm install:
```shell
helm install --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator
```
6. Check installation
```shell
kubectl -n netbird get pods
```
Output should be similar to:
```
NAME                                                    READY   STATUS    RESTARTS   AGE
netbird-operator-kubernetes-operator-67769f77db-tmnfn   1/1     Running   0          42m
```
```shell
kubectl -n netbird get services
```
Output should be similar to:
```shell
NAME                                                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
netbird-operator-kubernetes-operator-metrics           ClusterIP   192.168.194.165   <none>        8080/TCP   47m
netbird-operator-kubernetes-operator-webhook-service   ClusterIP   192.168.194.222   <none>        443/TCP    47m
```
### Updating or Modifying the Operator Configuration
The configuration or version update of the operator can be done with `helm upgrade`:
```shell
helm upgrade --create-namespace -f values.yaml -n netbird netbird-operator netbirdio/kubernetes-operator
 ```

## Expose Kubernetes Control Plane to your NetBird Network
To access your Kubernetes control plane from a NetBird network, you can expose your Kubernetes control plane as a
[NetBird resource](/manage/networks#resources) by enabling the following option in the operator values:

```yaml
ingres:
  enabled: true
  kubernetesAPI:
    enabled: true
```
The operator will create a NetBird network resource similar to the example below:

<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/kubernetes-api.png" alt="API" className="imagewrapper-big"/>
</p>

## Expose Kubernetes Services to NetBird Network

Kubernetes services is a common way to route traffic to your application pods. With the NetBird operator `ingress` you can expose services to your
NetBird network as resources by using annotations in your services. The operator will create networks, resources,
and add routing peers to your NetBird configuration.

By default, the ingress configuration is disabled. You can enable it with the following values using the helm chart:
```yaml
ingress:
  enabled: true
  router:
    enabled: true
```

You can expose services using the annotations `netbird.io/expose: "true"` and `netbird.io/groups: "resource-group"`; see the example below:
```yaml
apiVersion: v1
kind: Service
metadata:
  name: app
  annotations:
    netbird.io/expose: "true"
    netbird.io/groups: "app-access"
spec:
  selector:
    app: app
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
  type: ClusterIP
```

This will create a Network and a resource similar to the example below:
<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/resources-1.png" alt="resources" className="imagewrapper-big"/>
</p>

<Note>
 Ingress DNS Resolution requires enabled `DNS Wildcard Routing` and at least one DNS Nameserver configured for clients.
    Learn more about Networks settings [here](/manage/networks#enable-dns-wildcard-routing).
</Note>

Other annotations can be used to further configure the resources created by the operator:

|Annotation|Description|Default|Valid Values|
|:---:|:---:|:---:|:---:|
|`netbird.io/expose`| Expose service using NetBird Network Resource ||(`null`, `true`)|
|`netbird.io/groups`| Comma-separated list of group names to assign to Network Resource. If non-existing, the operator will create them for you. |`{ClusterName}-{Namespace}-{Service}`|Any comma-separated list of strings.|
|`netbird.io/resource-name`| Network Resource name |`{Namespace}-{Service}`|Any valid network resource name, make sure they're unique!|
|`netbird.io/policy`| Name(s) of NBPolicy to propagate service ports as destination. ||Comma-separated list of names of any NBPolicy resource|
|`netbird.io/policy-ports`| Narrow down exposed ports in a policy. Leave empty for all ports. ||Comma-separated integer list, integers must be between 0-65535|
|`netbird.io/policy-protocol`| Narrow down protocol for use in a policy. Leave empty for all protocols. ||(`tcp`,`udp`)|

## Control Access to Your Kubernetes Resources with Access Control Policies
By default, resources created by the operator will not have any access control policies assigned to them.
To allow the operator to manage your access control policies, configure policy bases in your `values.yaml` file.
In this file, you can define source groups, name suffixes, and other settings related to access control policies.
Afterward, you can tag the policies in your service annotations using the annotation `netbird.io/policy: "policy-base"`.
See the examples `values.yaml` below:

```yaml
ingress:
  enabled: true
  router:
    enabled: true
  policies:
    app-users:
      name: App users # Required, name of policy in NetBird console
      description: Policy for app users access # Optional
      sourceGroups: # Required, name of groups to assign as source in Policy.
        - app-users
      protocols: # Optional, restricts protocols allowed to resources, defaults to ['tcp', 'udp'].
        - tcp
      bidirectional: false
    k8s-admins:
      name: App admins
      sourceGroups:
        - app-admins
```
After adding the policy base and [applying the configuration](#updating-or-modifying-the-operator-configuration),
you can use the `app-users` and `k8s-admins` bases for your services and Kubernetes API configurations.

### Linking Policy Bases to the Kubernetes API Service
To link a policy base to the Kubernetes API, we need to update the operator configuration by adding the policy and groups to the `kubernetesAPI` key as follows:
```yaml
  kubernetesAPI:
    enabled: false
    groups:
      - k8s-access
    policies:
      - k8s-admins
```
After updating and [applying the configuration](#updating-or-modifying-the-operator-configuration), you should see a policy similar to the one below:
<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/kubernetes-api-policy.png" alt="resources policy" className="imagewrapper-big"/>
</p>

### Linking Policy Bases to Kubernetes Services
You can link policy bases with the annotation `netbird.io/policy:` where you can simply add one or more bases to the service,
see the example below where we link the base `"app-users"` to our app service:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: app
  annotations:
    netbird.io/expose: "true"
    netbird.io/groups: "app-access"
    netbird.io/policy: "app-users"
spec:
  selector:
    app: app
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
  type: ClusterIP
```

The operator will create a policy in your management account similar to the one below:
<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/resources-policy.png" alt="resources policy" className="imagewrapper-big"/>
</p>

You can reference multiple policy bases using a comma separated list of policy bases: `netbird.io/policy: "app-users,app-admins"`

### Policy auto-creation

1. Ensure `ingress.allowAutomaticPolicyCreation` is set to true in the Helm chart and apply.
2. Annotate a service with `netbird.io/policy` with the name of the policy as a kubernetes object, for example `netbird.io/policy: default`. This will create an NBPolicy with the name `default-<Service Namespace>-<Service Name>`.
3. Annotate the same service with `netbird.io/policy-source-groups` with a comma-separated list of group names to allow as a source, for example `netbird.io/policy-source-groups: dev`.
4. (Optional) Annotate the service with `netbird.io/policy-name` for a human-friendly name, for example `netbird.io/policy-name: "default:Default policy for kubernetes cluster"`.
Example:
```yaml
apiVersion: v1
kind: Service
metadata:
  name: app
  annotations:
    netbird.io/expose: "true"
    netbird.io/groups: "app-access"
    netbird.io/policy: "app-users"
    netbird.io/policy-source-groups: "dev"
    netbird.io/policy-name: "dev:Developers to app"
spec:
  selector:
    app: app
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
  type: ClusterIP
```

<Note>
 If a policy already exists with the name specified in `netbird.io/policy`, the other settings will be ignored in favor of the existing policy.
</Note>

## Accessing Remote Services Using Sidecars
To access services running in different locations from your Kubernetes clusters, you can deploy NetBird sidecars—additional
containers that run alongside your Kubernetes service containers within the same pod.

A NetBird sidecar joins your network as a regular peer and becomes a subject to access control, routing,
and DNS configurations as any other peer in your NetBird network. This allows your Kubernetes application traffic to be securely routed
through the NetBird network, enabling egress-like access to remote services from your Kubernetes services across various locations or cloud providers.

To enable sidecar functionality in your deployments, you first need to generate a setup key, either via the UI (see image below)
or by following [this guide](/manage/peers/register-machines-using-setup-keys) for more details.

<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/side-cars-setup-key.png" alt="Setup Keys" className="imagewrapper"/>
</p>

Next, you'll create a secret in Kubernetes and add a new resource called `NBSetupKey`. The `NBSetupKey` name can then be
referenced in your deployments or daemon sets to specify which setup key should be used when injecting a sidecar into
your application pods. Below is an example of a secret and an `NBSetupKey` resource:

```yaml
apiVersion: v1
stringData:
  setupkey: EEEEEEEE-EEEE-EEEE-EEEE-EEEEEEEEEEEE
kind: Secret
metadata:
  name: app-setup-key
```
NBSetupKey:
```yaml
apiVersion: netbird.io/v1
kind: NBSetupKey
metadata:
  name: app-setup-key
spec:
  # Optional, overrides management URL for this setupkey only
  # defaults to https://api.netbird.io
  # managementURL: https://netbird.example.com
  secretKeyRef:
    name: app-setup-key # Required
    key: setupkey # Required
```

After adding the resource, you can reference the `NBSetupKey` in your deployments or daemon-sets as shown below:

```yaml
kind: Deployment
...
spec:
...
  template:
    metadata:
      annotations:
        netbird.io/setup-key: app-setup-key # Must match the name of an NBSetupKey object in the same namespace
...
    spec:
      containers:
...
```

### Using Extra Labels to Access Multiple Pods Using the Same Name
Starting with `v0.27.0`, NetBird supports extra DNS labels, allowing you to define extended DNS names for peers. This enables grouping peers under a shared DNS name and distributing traffic using DNS round-robin load balancing.

To use this feature, create a setup key with the “Allow Extra DNS Labels” option enabled. See the example below for reference:

<p>
    <img src="/docs-static/img/how-to-guides/kubernetes/side-cars-setup-key-with-extra-labels.png" alt="Setup keys with extra labels" className="imagewrapper"/>
</p>

And add the annotation `netbird.io/extra-dns-labels` to your pod; see the example below:

```yaml
kind: Deployment
...
spec:
...
  template:
    metadata:
      annotations:
        netbird.io/setup-key: app-setup-key # Must match the name of an NBSetupKey object in the same namespace
        netbird.io/extra-dns-labels: "app"
...
    spec:
      containers:
...
```
With this setup, other peers in your NetBird network can reach these pods using the domain `app.<NETBIRD_DOMAIN>`
(e.g., for NetBird cloud, app.netbird.cloud). The access will be made using a DNS round-robin fashion for multiple pods.

## Get started
<p float="center" >
    <Button name="button" className="button-5" onClick={() => window.open("https://netbird.io/pricing")}>Use NetBird</Button>
</p>

- Make sure to [star us on GitHub](https://github.com/netbirdio/netbird)
- Follow us [on X](https://x.com/netbird)
- Join our [Slack Channel](/slack-url)
- NetBird [latest release](https://github.com/netbirdio/netbird/releases) on GitHub
