mirror of
https://github.com/fosrl/docs-v2.git
synced 2026-08-31 03:01:30 +02:00
add model routing and catalog
This commit is contained in:
miloschwartz
@@ -194,6 +194,7 @@
|
||||
"pages": [
|
||||
"manage/ai/providers/overview",
|
||||
"manage/ai/providers/configuration",
|
||||
"manage/ai/providers/model-routing",
|
||||
"manage/ai/providers/openai",
|
||||
"manage/ai/providers/anthropic",
|
||||
"manage/ai/providers/google-gemini",
|
||||
@@ -225,7 +226,8 @@
|
||||
"manage/ai/configure-ai-clients/claude-desktop",
|
||||
"manage/ai/configure-ai-clients/openclaw"
|
||||
]
|
||||
}
|
||||
},
|
||||
"manage/ai/model-catalog"
|
||||
]
|
||||
},
|
||||
"manage/endpoints-and-pops",
|
||||
|
||||
77
manage/ai/model-catalog.mdx
Normal file
77
manage/ai/model-catalog.mdx
Normal file
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: "Model Catalog"
|
||||
description: "Known models, pricing for budgets, and how to load a custom catalog"
|
||||
---
|
||||
|
||||
The model catalog is Pangolin's list of known model ids, which provider type owns them, and token pricing. It is used for:
|
||||
|
||||
- The **Known Models** picker on a provider's Models tab
|
||||
- Expanding wildcard allows in `GET /v1/models`
|
||||
- [Provider selection](/manage/ai/providers/model-routing#provider-selection) when more than one attached provider allows the same id
|
||||
- **Budgets and usage cost**, which look up `in` / `out` / `cache` / `reasoning` rates for a model key
|
||||
|
||||
See [Model Routing](/manage/ai/providers/model-routing) for allow lists, inherit vs select, and how those lists combine with the catalog at request time.
|
||||
|
||||
## Catalog Providers
|
||||
|
||||
Default source: `https://api.fossorial.io/api/v1/models`. Typed AI provider types map as follows:
|
||||
|
||||
| Provider type | Catalog |
|
||||
|---|---|
|
||||
| OpenAI | `openai` |
|
||||
| Anthropic | `anthropic` |
|
||||
| Google Gemini | `gemini` |
|
||||
| Vertex AI | `vertex` |
|
||||
| Amazon Bedrock | `bedrock` |
|
||||
| Microsoft Foundry | `azure` |
|
||||
| OpenRouter, Vercel AI Gateway, Custom | (not mapped) |
|
||||
|
||||
Unmapped types skip Known Models and wildcard expansion against the catalog. Enter exact keys if you need them listed in discovery.
|
||||
|
||||
## Pricing and Budgets
|
||||
|
||||
Each catalog entry can include token rates: `in`, `out`, `cache`, and `reasoning`. Budget enforcement and usage analytics use those rates when the requested model matches a catalog id. Unknown keys have no catalog price, so add custom models to the catalog if you need cost tracking for them.
|
||||
|
||||
## Loading a Custom Catalog
|
||||
|
||||
Self-hosted Pangolin loads and refreshes the catalog from [`config.yml`](/self-host/advanced/config-file#ai-model-catalog):
|
||||
|
||||
```yaml
|
||||
ai:
|
||||
model_catalog:
|
||||
upstream_url: "https://api.fossorial.io/api/v1/models"
|
||||
file: "config/ai-models.json"
|
||||
merge_file: "config/ai-models-extra.json"
|
||||
refresh_interval_min_hours: 6
|
||||
refresh_interval_max_hours: 12
|
||||
```
|
||||
|
||||
| Key | Purpose |
|
||||
|---|---|
|
||||
| `upstream_url` | HTTP GET for the catalog JSON. Defaults to the Fossorial API. Point this at your own endpoint to serve a custom catalog. |
|
||||
| `file` | Local JSON file that **replaces** the HTTP catalog. |
|
||||
| `merge_file` | Local JSON merged on top of the base catalog. Base entries win on duplicates; the merge file only adds missing models. |
|
||||
| `refresh_interval_min_hours` / `max_hours` | Background refresh window. The interval is jittered between min and max so many instances do not hit the upstream at the same moment. Defaults are 6 and 12. |
|
||||
|
||||
If both `file` and `upstream_url` are set, `file` is used as the base. `merge_file` still applies.
|
||||
|
||||
Catalog files and API responses are JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"data": [
|
||||
{
|
||||
"model": "gpt-4o",
|
||||
"provider": "openai",
|
||||
"pricing": {
|
||||
"in": 2.5,
|
||||
"out": 10,
|
||||
"cache": 1.25,
|
||||
"reasoning": null
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`provider` must be `openai`, `anthropic`, `gemini`, `vertex`, `azure`, or `bedrock`. Prefixes `bedrock*`, `vertex*`, and `azure*` are accepted and normalized to those catalogs. Unknown providers are skipped.
|
||||
@@ -12,6 +12,7 @@ An AI Gateway resource is a normal Pangolin resource that proxies requests to on
|
||||
- **[Providers](/manage/ai/providers/overview)** are configured once per organization - the upstream URL, API key, and capabilities for OpenAI, Anthropic, etc.
|
||||
- **Resources** (type `AI Gateway`) attach one or more of those providers and get a normal Pangolin domain.
|
||||
- **Keys** are what clients authenticate to the resource with. Public resources check them; private resources don't, since only devices on the Pangolin network can reach them at all.
|
||||
- The **[Model Catalog](/manage/ai/model-catalog)** is the known-model list and token pricing used by [model routing](/manage/ai/providers/model-routing) and budgets.
|
||||
|
||||
A resource only understands the API format(s) its attached providers support. An Anthropic provider makes the resource speak the Anthropic Messages API; an OpenAI provider makes it speak Chat Completions/Responses; a Gemini provider makes it speak Gemini's `generateContent` API. Attach whichever providers match the clients you plan to connect. See [AI Providers](/manage/ai/providers/overview) for per-type setup and [Provider Configuration](/manage/ai/providers/configuration) for capabilities, auth, and model lists.
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ A capability is an API format the gateway will accept and proxy. The incoming re
|
||||
|
||||
Typed providers start with recommended capabilities. Custom providers need at least one selected. You can change capabilities later on the provider's **General** tab. Select every format the upstream actually speaks. Custom providers can mix any combination.
|
||||
|
||||
If two attached providers share a capability for the same model, the gateway picks one using allow lists and catalog ownership. Two providers of the same type both allowing `*` can leave the request ambiguous.
|
||||
If two attached providers share a capability for the same model, the gateway picks one at request time. See [Model Routing](/manage/ai/providers/model-routing#provider-selection).
|
||||
|
||||
## Auth Type
|
||||
|
||||
@@ -58,11 +58,7 @@ See [Ollama](/manage/ai/providers/custom/ollama), [vLLM](/manage/ai/providers/cu
|
||||
|
||||
## Models
|
||||
|
||||
Allow and block lists restrict which model keys this provider may serve. A request must match an allow entry and must not match a block entry. An empty allow list denies all traffic. Patterns support `*` and `?` (`gpt-*`, `claude-3-5-sonnet-?`).
|
||||
|
||||
When you attach a provider to a resource you can **inherit** these lists or **select** a subset for that resource only.
|
||||
|
||||
OpenRouter, Vercel AI Gateway, and Custom are treated as aggregators: a wildcard allow such as `*` matches at request time but does not expand in `GET /v1/models`. Add exact model keys to make those models show up in discovery.
|
||||
Allow and block lists, inherit vs select, and provider selection are covered in [Model Routing](/manage/ai/providers/model-routing). Known Models, ownership scoring, and token pricing for budgets come from the [Model Catalog](/manage/ai/model-catalog).
|
||||
|
||||
## Headers and TLS
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ See [AI Providers](/manage/ai/providers/overview) if you haven't created a provi
|
||||
3. Select every **API capability** the upstream speaks. You can add any combination.
|
||||
4. Choose **Routing Mode**. For **Upstream URL**, paste the base URL. For **Site Targets**, add targets after save.
|
||||
5. Pick **Auth Type** and paste a key if that type needs one.
|
||||
6. Set **Allow** and **Block** lists. Add exact model keys if you want them listed in `GET /v1/models`. See [Models](/manage/ai/providers/configuration#models).
|
||||
6. Set **Allow** and **Block** lists. Add exact model keys if you want them listed in `GET /v1/models`. See [Model Routing](/manage/ai/providers/model-routing).
|
||||
7. Save, then attach the provider to an [AI Gateway resource](/manage/ai/overview).
|
||||
|
||||
## Reach a Self-Hosted API
|
||||
|
||||
@@ -18,7 +18,7 @@ See [Capabilities](/manage/ai/providers/configuration#capabilities) for the full
|
||||
3. Select **OpenAI Chat Completions** (and any other formats Bifrost should accept).
|
||||
4. Set **Routing Mode** to **Site Targets** if Bifrost runs on a site's network. Use **Upstream URL** (`http://<host>:8080`) only when the Pangolin node can reach Bifrost directly.
|
||||
5. Set **Auth Type** to **No Auth** on a private LAN. Use **Bearer** if Bifrost requires a virtual key or other API key.
|
||||
6. Allow the model ids Bifrost should serve, including provider-prefixed keys such as `openai/gpt-4o-mini` if that is how Bifrost names them. See [Models](/manage/ai/providers/configuration#models).
|
||||
6. Allow the model ids Bifrost should serve, including provider-prefixed keys such as `openai/gpt-4o-mini` if that is how Bifrost names them. See [Model Routing](/manage/ai/providers/model-routing).
|
||||
7. Save. On the **Network** tab, add a target: the Bifrost host, port `8080`, method HTTP.
|
||||
8. Attach the provider to an [AI Gateway resource](/manage/ai/overview).
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ See [Capabilities](/manage/ai/providers/configuration#capabilities) for the full
|
||||
3. Select **OpenAI Chat Completions**.
|
||||
4. Set **Routing Mode** to **Site Targets** if Ollama runs on a site's network. Use **Upstream URL** (`http://<host>:11434`) only when the Pangolin node can reach Ollama directly.
|
||||
5. Set **Auth Type** to **No Auth** on a private LAN. Use **Bearer** if you set an Ollama API key.
|
||||
6. Allow the exact model ids you plan to call, for example `llama3.2`. See [Models](/manage/ai/providers/configuration#models).
|
||||
6. Allow the exact model ids you plan to call, for example `llama3.2`. See [Model Routing](/manage/ai/providers/model-routing).
|
||||
7. Save. On the **Network** tab, add a target: the Ollama host, port `11434`, method HTTP.
|
||||
8. Attach the provider to an [AI Gateway resource](/manage/ai/overview).
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ See [Capabilities](/manage/ai/providers/configuration#capabilities) for the full
|
||||
3. Select **OpenAI Chat Completions**.
|
||||
4. Set **Routing Mode** to **Site Targets** if vLLM runs on a site's network. Use **Upstream URL** (`http://<host>:8000`) only when the Pangolin node can reach vLLM directly.
|
||||
5. Set **Auth Type** to **No Auth** if the server has no key. Use **Bearer** and paste the key if you started vLLM with `--api-key`.
|
||||
6. Allow the exact served model id (the `--model` name or `--served-model-name`). See [Models](/manage/ai/providers/configuration#models).
|
||||
6. Allow the exact served model id (the `--model` name or `--served-model-name`). See [Model Routing](/manage/ai/providers/model-routing).
|
||||
7. Save. On the **Network** tab, add a target: the vLLM host, port `8000`, method HTTP.
|
||||
8. Attach the provider to an [AI Gateway resource](/manage/ai/overview).
|
||||
|
||||
|
||||
74
manage/ai/providers/model-routing.mdx
Normal file
74
manage/ai/providers/model-routing.mdx
Normal file
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: "Model Routing"
|
||||
description: "Allow and block lists, inherit vs select, and how the gateway picks a provider"
|
||||
---
|
||||
|
||||
Model routing is how Pangolin decides which attached provider handles a request, and which model keys that provider may serve. Configure lists on the provider, optionally narrow them on the resource, then the gateway picks a provider per request.
|
||||
|
||||
See [AI Providers](/manage/ai/providers/overview) for what a provider is, and [Provider Configuration](/manage/ai/providers/configuration) for capabilities, auth, and routing mode. The [Model Catalog](/manage/ai/model-catalog) supplies Known Models, wildcard discovery, ownership scoring, and pricing for budgets.
|
||||
|
||||
## Choosing Models on a Provider
|
||||
|
||||
On the provider's **Models** tab, every request must match an **allow** entry and must not match a **block** entry. An empty allow list denies all traffic. A key cannot sit on both lists.
|
||||
|
||||
You add keys three ways:
|
||||
|
||||
| How | What it is | Example |
|
||||
|---|---|---|
|
||||
| **[Catalog (Known Models)](/manage/ai/model-catalog)** | Pick from the catalog for that provider type | `gpt-4o`, `claude-sonnet-4-6` |
|
||||
| **Custom key** | Type an exact model id the upstream accepts | `llama3.2`, `openai/gpt-4o-mini` |
|
||||
| **Pattern** | Glob with `*` and `?` | `gpt-*`, `claude-3-5-sonnet-?`, `*` |
|
||||
|
||||
`*` matches every model key. Typed providers (OpenAI, Anthropic, Google Gemini, Vertex AI, Amazon Bedrock, Microsoft Foundry) show Known Models from the catalog. [OpenRouter](/manage/ai/providers/open-router), [Vercel AI Gateway](/manage/ai/providers/vercel-ai-gateway), and [Custom](/manage/ai/providers/custom) skip that list; enter custom keys or patterns.
|
||||
|
||||
### Allow vs Block
|
||||
|
||||
Allow is the positive set. Block subtracts even when allow matched. Allow `*` and block `gpt-4o-mini` to permit every key except that one. Allow `gpt-*` and leave block empty to permit only keys that start with `gpt-`.
|
||||
|
||||
### Model Discovery
|
||||
|
||||
`GET /v1/models` is answered from these lists plus the catalog. The gateway does not proxy that listing call upstream.
|
||||
|
||||
- Exact allow keys are listed as themselves.
|
||||
- Patterns expand against the catalog for typed providers (`claude-*` becomes every matching catalog id).
|
||||
- On OpenRouter, Vercel AI Gateway, and Custom, add exact allow keys to make those models show up in discovery. A wildcard still matches at request time.
|
||||
|
||||
A block pattern hides a model from discovery the same way it would reject the request.
|
||||
|
||||
## Inherit vs Select
|
||||
|
||||
On an AI Gateway resource, attach providers from the **AI Gateway** settings tab. Each attachment has an access mode:
|
||||
|
||||
| Mode | What the resource uses |
|
||||
|---|---|
|
||||
| **Inherit** | The provider's allow and block lists as configured on the provider |
|
||||
| **Select** | A subset of that provider's allow-list models, chosen on this resource |
|
||||
|
||||
The same provider can be inherited on one resource and selected on another. You can disable an attachment without removing the provider.
|
||||
|
||||
Select only offers models already on the provider's allow list. Use inherit when every resource should see the same keys. Use select when one gateway should expose a narrower set (for example a public resource with `gpt-4o` only, while the provider still allows `gpt-*`).
|
||||
|
||||
## Provider Selection
|
||||
|
||||
When a resource has more than one attached provider, the gateway picks one per request. Overlapping allows are allowed at save time. Collisions are resolved, or rejected, when the request arrives.
|
||||
|
||||
Steps, in order. Later steps run only while more than one candidate remains:
|
||||
|
||||
1. **Capability.** The path must match a [capability](/manage/ai/providers/configuration#capabilities) the provider advertises.
|
||||
2. **Allow and block.** The effective lists (inherit or select) must pass.
|
||||
3. **Specificity.** The most specific matching allow wins: exact keys beat patterns, fewer wildcards win, longer literals win. `gpt-4o` beats `gpt-*` beats `*`.
|
||||
4. **Catalog ownership.** A typed provider whose catalog contains the id beats an aggregator that also allows it.
|
||||
5. **Class.** Native typed providers beat aggregators (OpenRouter, Vercel AI Gateway), which beat Custom.
|
||||
6. **Ambiguous.** If more than one distinct provider remains, the gateway returns `403` with `Model "<id>" is ambiguous across multiple AI providers on this resource`.
|
||||
|
||||
Typical ambiguous cases: two OpenAI providers both allowing `*`, or two aggregators both allowing `*` with no native owner on the resource.
|
||||
|
||||
`GET /v1/models` skips steps 3-6. There is no requested model to disambiguate, so the gateway returns the union of what every attached provider advertising Anthropic Models would accept.
|
||||
|
||||
### Examples
|
||||
|
||||
**OpenAI + Anthropic, both `*`.** `POST /v1/chat/completions` with `gpt-4o` goes to OpenAI. Anthropic never reaches scoring; capability alone decides. `POST /v1/messages` with a Claude model goes to Anthropic.
|
||||
|
||||
**OpenAI + OpenRouter, both `*`.** `gpt-4o` on Chat Completions matches both. Specificity is a tie (`*` vs `*`). Catalog ownership sends it to OpenAI.
|
||||
|
||||
**Two OpenAI providers, both `*`.** Capability, lists, specificity, catalog, and class all leave both candidates. Result is the ambiguous error. Narrow at least one allow list, disable one attachment, or split them across resources.
|
||||
@@ -27,7 +27,7 @@ OpenRouter's default capability is Chat Completions only. Clients that require R
|
||||
|
||||
## Models
|
||||
|
||||
OpenRouter is an aggregator. See [Models](/manage/ai/providers/configuration#models) for how allow lists and `GET /v1/models` discovery work. Add exact keys such as `openai/gpt-4o` or `anthropic/claude-sonnet-4`.
|
||||
OpenRouter is an aggregator. See [Model Routing](/manage/ai/providers/model-routing) for how allow lists and `GET /v1/models` discovery work. Add exact keys such as `openai/gpt-4o` or `anthropic/claude-sonnet-4`.
|
||||
|
||||
## Clients
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ description: "Connect upstream model APIs once per organization, then attach the
|
||||
|
||||
A provider is an organization-level connection to an upstream model API. It stores the provider type, credentials, base URL, API capabilities, and optional model allow and block lists. Create providers once, then attach them to any AI Gateway resource in the org. Clients call the resource's domain, not the provider directly.
|
||||
|
||||
See [AI Gateway Overview](/manage/ai/overview) for creating a resource and issuing keys. See [Provider Configuration](/manage/ai/providers/configuration) for capabilities, auth, routing, and model lists.
|
||||
See [AI Gateway Overview](/manage/ai/overview) for creating a resource and issuing keys. See [Provider Configuration](/manage/ai/providers/configuration) for capabilities, auth, and routing. See [Model Routing](/manage/ai/providers/model-routing) for allow lists, inherit vs select, and how the gateway picks a provider. See [Model Catalog](/manage/ai/model-catalog) for known models and pricing used by routing and budgets.
|
||||
|
||||
## What a Provider Is
|
||||
|
||||
@@ -30,7 +30,7 @@ The AI Gateway → Providers → Create form, showing the provider type dropdown
|
||||
|
||||
## Attach to a Resource
|
||||
|
||||
On an AI Gateway resource, attach one or more providers from the **AI Gateway** settings tab. For each attachment you can inherit the provider's model lists or select a subset. Details are in [Provider Configuration](/manage/ai/providers/configuration#models).
|
||||
On an AI Gateway resource, attach one or more providers from the **AI Gateway** settings tab. For each attachment you can inherit the provider's model lists or select a subset. Details are in [Model Routing](/manage/ai/providers/model-routing#inherit-vs-select).
|
||||
|
||||
The resource only speaks the API formats its attached providers advertise. Attach whichever providers match the [clients](/manage/ai/overview#4-connect-a-client) you plan to connect.
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ See [AI Providers](/manage/ai/providers/overview) if you haven't created a provi
|
||||
|
||||
## Models
|
||||
|
||||
Vercel AI Gateway is an aggregator. See [Models](/manage/ai/providers/configuration#models) for how allow lists and `GET /v1/models` discovery work. Add exact model keys to the allow list.
|
||||
Vercel AI Gateway is an aggregator. See [Model Routing](/manage/ai/providers/model-routing) for how allow lists and `GET /v1/models` discovery work. Add exact model keys to the allow list.
|
||||
|
||||
## Clients
|
||||
|
||||
|
||||
@@ -918,6 +918,60 @@ This section contains the complete reference for all configuration options in `c
|
||||
</Expandable>
|
||||
</ResponseField>
|
||||
|
||||
### AI Model Catalog
|
||||
|
||||
The catalog feeds Known Models pickers, wildcard discovery, provider selection, and usage pricing. See [Model Catalog](/manage/ai/model-catalog) for the JSON format, catalog providers, and how budgets use pricing.
|
||||
|
||||
<ResponseField name="ai" type="object">
|
||||
AI Gateway catalog settings. Omit this block to use the defaults.
|
||||
|
||||
<Expandable title="AI">
|
||||
<ResponseField name="model_catalog" type="object">
|
||||
Where Pangolin loads the model catalog from, and how often it refreshes.
|
||||
|
||||
<Expandable title="Model catalog">
|
||||
<ResponseField name="upstream_url" type="string" default="https://api.fossorial.io/api/v1/models">
|
||||
HTTP endpoint that returns catalog JSON. Point this at your own API to serve a custom catalog.
|
||||
|
||||
```yaml
|
||||
ai:
|
||||
model_catalog:
|
||||
upstream_url: "https://api.fossorial.io/api/v1/models"
|
||||
```
|
||||
</ResponseField>
|
||||
|
||||
<ResponseField name="file" type="string">
|
||||
Path to a local catalog JSON file. When set, this file is the base catalog instead of `upstream_url`.
|
||||
|
||||
```yaml
|
||||
ai:
|
||||
model_catalog:
|
||||
file: "config/ai-models.json"
|
||||
```
|
||||
</ResponseField>
|
||||
|
||||
<ResponseField name="merge_file" type="string">
|
||||
Path to a local JSON file whose entries are merged into the base catalog. Base entries win on duplicates; the merge file only adds models that are not already present.
|
||||
|
||||
```yaml
|
||||
ai:
|
||||
model_catalog:
|
||||
merge_file: "config/ai-models-extra.json"
|
||||
```
|
||||
</ResponseField>
|
||||
|
||||
<ResponseField name="refresh_interval_min_hours" type="number" default="6">
|
||||
Lower bound, in hours, for the jittered background refresh interval.
|
||||
</ResponseField>
|
||||
|
||||
<ResponseField name="refresh_interval_max_hours" type="number" default="12">
|
||||
Upper bound, in hours, for the jittered background refresh interval. Pangolin waits a random duration between min and max so many instances do not hit the upstream at the same moment.
|
||||
</ResponseField>
|
||||
</Expandable>
|
||||
</ResponseField>
|
||||
</Expandable>
|
||||
</ResponseField>
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Some configuration values can be set using environment variables for enhanced security:
|
||||
|
||||
Reference in New Issue
Block a user