mirror of
https://github.com/netbirdio/docs.git
synced 2026-08-24 16:51:26 +02:00
docs: add "Route to a Kubernetes service with HA" how-to (#810)
* docs: add Highly Available Routing Peers use-case page (Kubernetes operator) Add a standalone use-case page under a new Use Cases group in the Kubernetes nav, covering how to run the operator's routing peers in HA: NetworkRouter workloadOverride.replicas (default 3), the auto-created PodDisruptionBudget (maxUnavailable: 1), equal-metric automatic failover, and spreading replicas across failure domains via workloadOverride.podTemplate. Models least-privilege (named destination group + access policy) rather than the All group. * docs: add topology diagrams to HA routing peers page Two SVG topology diagrams: replicas on a single node (single point of failure) and replicas spread one-per-node via topologySpreadConstraints. Embedded in Step 1 and the failure-domains section. * docs: correct HA scheduling framing; drop single-node diagram kube-scheduler spreads a Deployment's replicas across nodes by default (best-effort, via built-in PodTopologySpread defaults). The earlier text/ diagram wrongly implied replicas co-locate by default. Reframe: multi-node spread is the default; topologySpreadConstraints turns it into a guarantee (or spans zones). Remove the single-node diagram (non-HA case, out of scope). * docs: add Friendly DNS names appendix to HA routing peers page Document exposing a service under a cleaner name via a CNAME in a custom zone pointing at the operator's <service>.<namespace>.<zone> record (verified end-to-end). Placed as an appendix for now; can move to a shared location later. * docs: use ScheduleAnyway in spread example; note DoNotSchedule rollout deadlock Multi-node verification: default scheduling already spreads replicas one-per-node; the operator merges workloadOverride.podTemplate.topologySpreadConstraints into the Deployment. DoNotSchedule with replicas == schedulable nodes deadlocks rolling updates (surge pod can't place). Switch the example to ScheduleAnyway (verified clean rollout) and document DoNotSchedule + the node-count/maxSurge caveat for a hard guarantee. * docs: clarify custom-zone records are per-name (no whole-domain shadowing) Verified on the lab: a NetBird custom zone serves only the records you add; other names under the domain fall through to upstream DNS. Reusing a real internal domain for friendly names is safe except for exact-name collisions. * docs: expand into full 'Route to a Kubernetes service' how-to Restructure the HA use-case page into an end-to-end guide covering the whole journey: create the custom DNS zone, groups, and access policy (dashboard) -> deploy HA routing peers (NetworkRouter, replicas:3) -> expose a Service (NetworkResource) -> verify + failover. Generic, human-readable example names (k8s.company.internal, kubernetes-clients/-services, network 'kubernetes', nginx). Keeps the failure-domains diagram + ScheduleAnyway/DoNotSchedule note and the friendly-DNS appendix. Adds <img> slots for 5 dashboard/terminal screenshots (to be supplied). Renames the page + nav entry to route-to-a-kubernetes-service; old slug removed. * docs: add dashboard/terminal screenshots to the K8s how-to Four screenshots (DNS zone, access policy, the kubernetes network with HA + 3 routing peers, kubectl pods-across-nodes). Drop the groups screenshot and renumber the <img> refs to match. * docs: swap in cleaner pods-across-nodes screenshot for Step 5 * docs: make node-spread central to the HA guide Node-spread is the point of an HA guide, not a tail-end section. Move the topology diagram up to 'What you'll achieve', fold the node-spread story into Step 3 (deploy HA routing peers) - leading with the verified fact that the scheduler spreads replicas across nodes by default (HA out of the box), with topologySpreadConstraints as optional hardening - and drop the orphaned 'Spread across failure domains' section. * docs: clarify the custom zone is created empty (operator fills the record) Step 1 showed the auto-created A record without saying you don't enter it. Note that you create only the zone (no hostname/IP/TTL by hand) and the operator adds <service>.<namespace>.<zone> -> ClusterIP (5-min TTL) in Step 4. * docs: replace Excalidraw topology with a custom dark-mode SVG Hand-authored dark-background topology diagram (NetBird overlay -> routing peers one-per-node -> Service) that matches the dark docs theme, replacing the light Excalidraw-derived SVG. Removes the orphaned ha-routing-peers-spread-nodes.svg. * docs: add CNAME dialog screenshot to the friendly-DNS appendix Show the Add DNS Record dialog (CNAME 'app' -> nginx.default.k8s.company.internal) and align the example hostname to 'app' to match. * docs: drop maxSurge:0 workaround (not configurable via the operator) The operator's workloadOverride only exposes annotations, labels, podTemplate, and replicas — there is no hook for the Deployment's strategy.rollingUpdate.maxSurge. Keep the achievable workaround (more schedulable nodes than replicas). * docs: drop manual topology spread guidance (operator handles it by default)
This commit is contained in:
Jack Carter
Binary file not shown.
|
After Width: | Height: | Size: 135 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 186 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 173 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
@@ -0,0 +1,60 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 480" width="880" height="480" font-family="ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||
<defs>
|
||||
<marker id="ah" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0,0 L10,5 L0,10 z" fill="context-stroke"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- card background -->
|
||||
<rect x="2" y="2" width="876" height="476" rx="16" fill="#0d1117" stroke="#21262d" stroke-width="1.5"/>
|
||||
|
||||
<text x="40" y="46" font-size="21" font-weight="600" fill="#e6edf3">Highly available routing peers</text>
|
||||
|
||||
<!-- NetBird overlay zone -->
|
||||
<rect x="40" y="80" width="230" height="356" rx="14" fill="#0f1b2d" stroke="#1f4f8f" stroke-width="1.5"/>
|
||||
<text x="62" y="110" font-size="15" font-weight="600" fill="#58a6ff">NetBird overlay</text>
|
||||
|
||||
<!-- client -->
|
||||
<rect x="66" y="236" width="180" height="78" rx="12" fill="#15294a" stroke="#4d8bf0" stroke-width="1.5"/>
|
||||
<text x="156" y="270" text-anchor="middle" font-size="14" fill="#dbeafe">Your laptop</text>
|
||||
<text x="156" y="290" text-anchor="middle" font-size="12" fill="#93b6e6">NetBird client</text>
|
||||
|
||||
<!-- Kubernetes cluster zone -->
|
||||
<rect x="330" y="80" width="510" height="356" rx="14" fill="#0d1f15" stroke="#1f6f3a" stroke-width="1.5"/>
|
||||
<text x="352" y="110" font-size="15" font-weight="600" fill="#3fb950">Kubernetes cluster</text>
|
||||
|
||||
<!-- node 1 -->
|
||||
<rect x="360" y="132" width="330" height="84" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
|
||||
<text x="376" y="154" font-size="12" fill="#8b949e">node-1</text>
|
||||
<rect x="470" y="152" width="200" height="44" rx="8" fill="#122a1c" stroke="#2ea043" stroke-width="1.5"/>
|
||||
<text x="570" y="179" text-anchor="middle" font-size="13" fill="#56d364">routing-peer pod</text>
|
||||
|
||||
<!-- node 2 -->
|
||||
<rect x="360" y="240" width="330" height="84" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
|
||||
<text x="376" y="262" font-size="12" fill="#8b949e">node-2</text>
|
||||
<rect x="470" y="260" width="200" height="44" rx="8" fill="#122a1c" stroke="#2ea043" stroke-width="1.5"/>
|
||||
<text x="570" y="287" text-anchor="middle" font-size="13" fill="#56d364">routing-peer pod</text>
|
||||
|
||||
<!-- node 3 -->
|
||||
<rect x="360" y="348" width="330" height="84" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
|
||||
<text x="376" y="370" font-size="12" fill="#8b949e">node-3</text>
|
||||
<rect x="470" y="368" width="200" height="44" rx="8" fill="#122a1c" stroke="#2ea043" stroke-width="1.5"/>
|
||||
<text x="570" y="395" text-anchor="middle" font-size="13" fill="#56d364">routing-peer pod</text>
|
||||
|
||||
<!-- service -->
|
||||
<rect x="735" y="235" width="92" height="80" rx="12" fill="#11272b" stroke="#2bb3c0" stroke-width="1.5"/>
|
||||
<text x="781" y="269" text-anchor="middle" font-size="14" fill="#9fe7ef">Service</text>
|
||||
<text x="781" y="289" text-anchor="middle" font-size="13" fill="#5fcdd8">nginx</text>
|
||||
|
||||
<!-- client -> routing peers (overlay) -->
|
||||
<polyline points="246,274 470,174" fill="none" stroke="#4d8bf0" stroke-width="2" marker-end="url(#ah)"/>
|
||||
<polyline points="246,274 470,282" fill="none" stroke="#4d8bf0" stroke-width="2" marker-end="url(#ah)"/>
|
||||
<polyline points="246,274 470,390" fill="none" stroke="#4d8bf0" stroke-width="2" marker-end="url(#ah)"/>
|
||||
|
||||
<!-- routing peers -> service -->
|
||||
<polyline points="670,174 735,266" fill="none" stroke="#3fb950" stroke-width="2" marker-end="url(#ah)"/>
|
||||
<polyline points="670,282 735,274" fill="none" stroke="#3fb950" stroke-width="2" marker-end="url(#ah)"/>
|
||||
<polyline points="670,390 735,283" fill="none" stroke="#3fb950" stroke-width="2" marker-end="url(#ah)"/>
|
||||
|
||||
<text x="40" y="463" font-size="13" fill="#8b949e">Clients reach the Service through any routing peer — lose a peer or a node and traffic fails over automatically.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -552,6 +552,16 @@ export const docsNavigation = [
|
||||
title: 'Gateway API',
|
||||
href: '/manage/integrations/kubernetes/gateway-api',
|
||||
},
|
||||
{
|
||||
title: 'Use Cases',
|
||||
isOpen: false,
|
||||
links: [
|
||||
{
|
||||
title: 'Route to a Kubernetes Service',
|
||||
href: '/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
import { Note } from '@/components/mdx'
|
||||
|
||||
# Route to a Kubernetes service with high availability
|
||||
|
||||
This guide walks the whole journey: create the NetBird-side pieces the operator doesn't make (a custom DNS zone, groups, an access policy), deploy a redundant pool of routing peers, expose an in-cluster Service as a NetBird resource, and reach it by name from a NetBird client. Because the routing peers run as a high-availability pool, access keeps working when a routing-peer pod or a node fails.
|
||||
|
||||
## What you'll achieve
|
||||
|
||||
A NetBird client (for example, your laptop) reaches a private Kubernetes `ClusterIP` Service by a stable DNS name, with traffic flowing through a pool of routing-peer pods spread across your nodes. Lose a pod or a node and clients fail over automatically to a healthy peer.
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/topology.svg" alt="A NetBird client reaching a Kubernetes service through three routing-peer pods, one per node" className="imagewrapper-big"/>
|
||||
</p>
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Kubernetes cluster (multiple nodes recommended, so routing peers can spread across them).
|
||||
- The NetBird operator installed — see [Getting Started](/manage/integrations/kubernetes).
|
||||
- A NetBird account and a [Personal Access Token](/manage/public-api#creating-a-service-user).
|
||||
- A NetBird client (the device that will reach the service) enrolled in your account.
|
||||
|
||||
In this guide the example objects are named `k8s.company.internal` (DNS zone), `kubernetes-clients` / `kubernetes-services` (groups), `kubernetes` (the network), and `nginx` (the Service). Substitute your own.
|
||||
|
||||
## Step 1: Create a custom DNS zone
|
||||
|
||||
The operator publishes each exposed Service as a DNS record inside a custom zone, so clients reach it by name instead of by its (ephemeral) ClusterIP. The zone must exist **before** you deploy the routing peers.
|
||||
|
||||
In the dashboard, go to **DNS > Zones > Add Zone**:
|
||||
|
||||
- **Name**: `k8s.company.internal`
|
||||
- **Distribution Groups**: `kubernetes-clients` — only peers in these groups can resolve the zone's records.
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/01-dns-zone.png" alt="The k8s.company.internal custom DNS zone, distributed to the kubernetes-clients group, with the operator-created nginx record" className="imagewrapper"/>
|
||||
</p>
|
||||
|
||||
<Note>
|
||||
Create only the zone — **leave its records empty**. You don't enter a hostname, IP, or TTL here. When you expose a Service in [Step 4](#step-4-expose-your-service), the operator automatically adds the `A` record (named `<service>.<namespace>.<zone>`, e.g. `nginx.default.k8s.company.internal`, pointing at the Service's ClusterIP, with a 5-minute TTL) — that's the record shown above.
|
||||
</Note>
|
||||
|
||||
See [Custom Zones](/manage/dns/custom-zones) for details.
|
||||
|
||||
## Step 2: Create groups and an access policy
|
||||
|
||||
NetBird is deny-by-default: nothing is reachable until a policy allows it, and the operator does **not** create groups or policies for you. Set up two groups and one policy under **Access Control**.
|
||||
|
||||
Create the groups via **Access Control > Groups**:
|
||||
|
||||
- `kubernetes-clients` — the peers that should reach your services (put your client device in it).
|
||||
- `kubernetes-services` — the destination group the exposed Services will be placed in.
|
||||
|
||||
Then create a policy via **Access Control > Policies > Add policy**:
|
||||
|
||||
- **Name**: `kubernetes-access`
|
||||
- **Source**: `kubernetes-clients`
|
||||
- **Destination**: `kubernetes-services`
|
||||
- **Protocol/Ports**: `TCP` `80` (match your Service's port)
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/02-access-policy.png" alt="The kubernetes-access policy allowing kubernetes-clients to reach kubernetes-services over TCP 80" className="imagewrapper"/>
|
||||
</p>
|
||||
|
||||
See [Manage network access](/manage/access-control/manage-network-access).
|
||||
|
||||
<Note>
|
||||
Create the group **and** the policy before the `NetworkResource` in Step 4. Until both exist, traffic is denied even though DNS resolves.
|
||||
</Note>
|
||||
|
||||
## Step 3: Deploy the routing peers (HA)
|
||||
|
||||
A `NetworkRouter` creates a NetBird network and deploys routing-peer pods. Set `spec.workloadOverride.replicas` to run a redundant pool:
|
||||
|
||||
```yaml
|
||||
apiVersion: netbird.io/v1alpha1
|
||||
kind: NetworkRouter
|
||||
metadata:
|
||||
name: kubernetes
|
||||
namespace: netbird
|
||||
spec:
|
||||
dnsZoneRef:
|
||||
name: k8s.company.internal
|
||||
workloadOverride:
|
||||
replicas: 3
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl apply -f networkrouter.yaml
|
||||
```
|
||||
|
||||
The operator registers all replicas in a single routing-peer group at one metric, so each client connects through its lowest-latency peer and fails over automatically if that peer becomes unreachable (the equal-metric behavior in [How Routing Peers Work — High availability](/manage/networks/how-routing-peers-work#high-availability)). When `replicas > 1`, it also creates a **PodDisruptionBudget** with `maxUnavailable: 1`, so node drains and rolling updates never take down more than one routing peer at a time.
|
||||
|
||||
<Note>
|
||||
The operator already defaults to **3** replicas — set the field explicitly to be intentional, or raise it for more redundancy. See the [Routing Peer](/manage/integrations/kubernetes/routing-peer) page for the full `NetworkRouter` reference.
|
||||
</Note>
|
||||
|
||||
On a multi-node cluster, the operator spreads these replicas across nodes by default, so you already have node-level high availability: a node failure takes out only one routing peer and clients fail over to the rest.
|
||||
|
||||
## Step 4: Expose your Service
|
||||
|
||||
A `NetworkResource` maps a Kubernetes `ClusterIP` Service to a NetBird resource and creates a DNS record for it in the router's zone. Place it in the `kubernetes-services` group from Step 2:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nginx
|
||||
namespace: default
|
||||
labels: { app: nginx }
|
||||
spec:
|
||||
replicas: 1
|
||||
selector: { matchLabels: { app: nginx } }
|
||||
template:
|
||||
metadata: { labels: { app: nginx } }
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:stable
|
||||
ports: [{ containerPort: 80 }]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: nginx
|
||||
namespace: default
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector: { app: nginx }
|
||||
ports:
|
||||
- { name: http, port: 80, targetPort: 80, protocol: TCP }
|
||||
---
|
||||
apiVersion: netbird.io/v1alpha1
|
||||
kind: NetworkResource
|
||||
metadata:
|
||||
name: nginx
|
||||
namespace: default
|
||||
spec:
|
||||
networkRouterRef:
|
||||
name: kubernetes
|
||||
namespace: netbird
|
||||
serviceRef:
|
||||
name: nginx
|
||||
groups:
|
||||
- name: kubernetes-services
|
||||
```
|
||||
|
||||
The Service must be type `ClusterIP`. The operator creates the record `nginx.default.k8s.company.internal` (`<service>.<namespace>.<zone>`) pointing at the Service's ClusterIP. The `kubernetes` network now shows its routing peers and the resource:
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/03-network.png" alt="The kubernetes network with its routing peers and the nginx resource" className="imagewrapper"/>
|
||||
</p>
|
||||
|
||||
## Step 5: Verify and test failover
|
||||
|
||||
Confirm the routing-peer pods are running, spread across nodes, and protected by a PodDisruptionBudget:
|
||||
|
||||
```shell
|
||||
kubectl -n netbird get pods -l app.kubernetes.io/name=networkrouter -o wide
|
||||
kubectl -n netbird get pdb
|
||||
```
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/04-pods-across-nodes.png" alt="kubectl get pods -o wide showing the routing peers on different nodes" className="imagewrapper"/>
|
||||
</p>
|
||||
|
||||
From a NetBird client in `kubernetes-clients`, resolve and reach the service:
|
||||
|
||||
```shell
|
||||
curl http://nginx.default.k8s.company.internal/
|
||||
```
|
||||
|
||||
Then delete one routing-peer pod (or drain its node) while curling in a loop — the connection keeps working as another peer takes over, and the Deployment reschedules the missing pod:
|
||||
|
||||
```shell
|
||||
kubectl -n netbird delete pod <routing-peer-pod>
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Routing Peer](/manage/integrations/kubernetes/routing-peer) — the `NetworkRouter` / `NetworkResource` reference.
|
||||
- [How Routing Peers Work](/manage/networks/how-routing-peers-work) — failover, metrics, and access control.
|
||||
|
||||
## Appendix: Friendly DNS names
|
||||
|
||||
Each `NetworkResource` is published at `<service>.<namespace>.<zone>` — here, `nginx.default.k8s.company.internal`. The operator always uses this form and it can't be customized on the `NetworkResource`.
|
||||
|
||||
To expose a service under a cleaner name, add a **CNAME** in a [custom DNS zone](/manage/dns/custom-zones) pointing at the operator's record:
|
||||
|
||||
```text
|
||||
app.k8s.company.internal CNAME nginx.default.k8s.company.internal
|
||||
```
|
||||
|
||||
In the dashboard, that's **DNS > Zones >** the zone's **Add** button: a `CNAME` record with hostname `app` and the operator's record as the target.
|
||||
|
||||
<p>
|
||||
<img src="/docs-static/img/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service/friendly-dns-cname.png" alt="Adding a CNAME record with hostname app targeting nginx.default.k8s.company.internal in the k8s.company.internal zone" className="imagewrapper"/>
|
||||
</p>
|
||||
|
||||
Because it targets the operator-managed record, the alias keeps resolving if the service's ClusterIP changes. A static `A` record straight to the ClusterIP also works, but it goes stale when the ClusterIP changes — prefer the CNAME.
|
||||
|
||||
<Note>
|
||||
The friendly name is only a DNS alias — traffic still routes through the `NetworkResource`, so keep it in place. NetBird serves only the specific records you add to a zone; other names under the same domain keep resolving through your existing DNS, so reusing a real internal domain is safe — just avoid a name that already exists in your corporate DNS. These manual records are not managed by the operator, so you maintain them yourself.
|
||||
</Note>
|
||||
Reference in New Issue
Block a user