sendnrw/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-09-02 13:01:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

[proxy] Route the Bedrock count-tokens action

Browse Source 
Both the request parser and the router enumerated Bedrock actions without
count-tokens, so the path carried no model and the request denied as
not-routable. Nothing breaks outright, because the client falls back to
counting context through the inference endpoint, but that fallback is
billable and the dedicated endpoint exists to avoid exactly that.

The action carries a model in the path and returns no usage, so it routes
like any other Bedrock action and meters to zero.
This commit is contained in:
mlsmaycon
2026-08-11 02:56:00 +00:00
parent 789d416215
commit 03e02c86ce
4 changed files with 80 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
proxy/internal/middleware/builtin/llm_request_parser/middleware.go
View File

@@ -350,7 +350,9 @@ func trimBedrockNamespace(reqPath string) string {
//
// /model/{modelId}/{action}
//
// action ∈ {invoke, invoke-with-response-stream, converse, converse-stream}.
// action ∈ {invoke, invoke-with-response-stream, converse, converse-stream,
// count-tokens}. Token counting carries a model and no usage, so it routes
// like any other action and meters to zero.
// The modelId may be URL-encoded and may carry a cross-region inference-profile
// prefix and a version suffix; normalizeBedrockModel strips both so the model
// matches catalog pricing.
@@ -374,7 +376,7 @@ func parseBedrockPath(reqPath string) (bedrockRequest, bool) {
return bedrockRequest{}, false
}
switch action {
case "invoke", "converse":
case "invoke", "converse", "count-tokens":
return bedrockRequest{model: model}, true
case "invoke-with-response-stream", "converse-stream":
return bedrockRequest{model: model, stream: true}, true