mirror of
https://github.com/netbirdio/docs.git
synced 2026-09-23 15:29:06 +02:00
Add vLLM Provider doc (#826)
This commit is contained in:
Binary file not shown.
|
After Width: | Height: | Size: 219 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 165 KiB |
+1275
-1273
File diff suppressed because it is too large
Load Diff
@@ -17,6 +17,8 @@ Replace `<your-endpoint>` in the snippets below with the endpoint shown on the
|
|||||||
- [Codex](/agent-network/integrations/codex) — point the Codex CLI at the endpoint.
|
- [Codex](/agent-network/integrations/codex) — point the Codex CLI at the endpoint.
|
||||||
- [LiteLLM](/agent-network/integrations/litellm) — use a LiteLLM gateway with identity-based
|
- [LiteLLM](/agent-network/integrations/litellm) — use a LiteLLM gateway with identity-based
|
||||||
attribution and budgets.
|
attribution and budgets.
|
||||||
|
- [vLLM](/agent-network/integrations/vllm) — connect a self-hosted, OpenAI-compatible vLLM
|
||||||
|
server.
|
||||||
- [Google Vertex AI](/agent-network/integrations/vertex-ai) — connect Gemini and Claude on
|
- [Google Vertex AI](/agent-network/integrations/vertex-ai) — connect Gemini and Claude on
|
||||||
Vertex AI with a Google Cloud service account.
|
Vertex AI with a Google Cloud service account.
|
||||||
- [AWS Bedrock](/agent-network/integrations/bedrock) — connect Claude, Llama, and Nova on
|
- [AWS Bedrock](/agent-network/integrations/bedrock) — connect Claude, Llama, and Nova on
|
||||||
|
|||||||
@@ -0,0 +1,71 @@
|
|||||||
|
import { Note } from '@/components/mdx'
|
||||||
|
|
||||||
|
export const description =
|
||||||
|
'Serve a self-hosted vLLM model behind NetBird Agent Network: keyless, tunnel-only access to your OpenAI-compatible vLLM endpoint with policy-based authorization.'
|
||||||
|
|
||||||
|
# vLLM
|
||||||
|
|
||||||
|
[vLLM](https://docs.vllm.ai) is a high-throughput inference server for self-hosted models
|
||||||
|
that exposes an **OpenAI-compatible API**. You run it yourself — typically on a GPU host
|
||||||
|
inside the same network as the NetBird proxy — and connect it as a provider so agents reach
|
||||||
|
it keyless over the tunnel, authorized by your [policies](/agent-network/policies).
|
||||||
|
|
||||||
|
This gives you keyless access to your private vLLM from anywhere: the endpoint stays
|
||||||
|
off the public internet, and any authorized user or agent reaches it over the NetBird
|
||||||
|
tunnel without shared API keys.
|
||||||
|
|
||||||
|
vLLM behaves like a [custom / self-hosted provider](/agent-network/providers#custom-and-self-hosted-providers);
|
||||||
|
it simply has its own named entry in the provider picker.
|
||||||
|
|
||||||
|
## Connect vLLM as a Provider
|
||||||
|
|
||||||
|
Because vLLM is self-hosted, the upstream URL points at your own instance. Host it in the
|
||||||
|
same network as the proxy so the proxy can reach it directly.
|
||||||
|
|
||||||
|
1. Go to **Agent Network → Providers** and click **Connect Provider**.
|
||||||
|
2. Select **vLLM** and set the **Upstream URL** to your vLLM server's OpenAI-compatible base
|
||||||
|
URL, for example `https://vllm.internal:8000`.
|
||||||
|
3. If your vLLM server was started with an API key (`--api-key`), paste it as the **API key**;
|
||||||
|
NetBird stores it server-side and sends it as a bearer token. Leave it empty if the server
|
||||||
|
requires none.
|
||||||
|
4. _(Optional)_ Enable **Skip TLS Verification** if your vLLM endpoint serves a self-signed
|
||||||
|
certificate — see [Skip TLS Verification](/agent-network/providers#skip-tls-verification).
|
||||||
|
Prefer mounting trusted certificates on your proxy instances for anything beyond testing.
|
||||||
|
5. _(Optional)_ List the **models** your server exposes with per-model pricing so cost shows
|
||||||
|
up in [Usage & Logs](/agent-network/usage-and-logs). Leaving the list empty accepts any
|
||||||
|
model name the server serves.
|
||||||
|
6. Save the provider.
|
||||||
|
|
||||||
|
<p>
|
||||||
|
<img src="/docs-static/img/agent-network/integrations/agent-network-connect-vllm.png" alt="connect a self-hosted vLLM provider in NetBird Agent Network" className="imagewrapper" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## Create a Policy
|
||||||
|
|
||||||
|
By default nothing is allowed — a policy must connect a source group to the vLLM provider
|
||||||
|
before anyone can route through it.
|
||||||
|
|
||||||
|
1. Go to **Agent Network → Policies** and add a policy.
|
||||||
|
2. Set the **Source** to the users or agents who should be able to use vLLM (for example your
|
||||||
|
`Engineering` group from your IdP).
|
||||||
|
3. Set the **Provider** to the vLLM provider you just connected.
|
||||||
|
4. Optionally attach per-user or per-group [token and budget limits](/agent-network/policies/limits)
|
||||||
|
and [guardrails](/agent-network/policies/guardrails) such as a model allowlist.
|
||||||
|
|
||||||
|
<p>
|
||||||
|
<img src="/docs-static/img/agent-network/integrations/agent-network-create-policy-vllm.png" alt="create a NetBird Agent Network policy authorizing vLLM" className="imagewrapper" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
See [Policies](/agent-network/policies) for details.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
Agents point at the NetBird endpoint with no key and call your vLLM models by name. NetBird
|
||||||
|
enforces your policies, limits, and guardrails, then forwards the request to your vLLM server.
|
||||||
|
Every call is recorded in [Usage & Logs](/agent-network/usage-and-logs) with the caller's
|
||||||
|
identity, auth group, model, tokens, cost, and status.
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
vLLM speaks the OpenAI API shape, so point OpenAI-compatible clients at
|
||||||
|
`https://<your-endpoint>` and select a model your server hosts.
|
||||||
|
</Note>
|
||||||
Reference in New Issue
Block a user