[e2e] Spike the vendors' own model-discovery endpoints

Management could populate the provider-config model picker from live
vendor data instead of the hand-curated catalog, which today carries
comments tracking which models a vendor retired on which date and which
cannot know what a particular account may actually invoke.

Whether that is a small feature or a large one turns on one unknown:
ListInferenceProfiles is a control-plane operation on
bedrock.<region>.amazonaws.com, while a provider record's upstream must
be bedrock-runtime.<region> for InvokeModel. Nothing documents whether a
Bedrock API key authorises the control plane, and if it does not, live
discovery for Bedrock needs SigV4 signing in management.

Probe each vendor directly — no proxy, no tunnel, no containers — under
the credential an operator would actually supply, and print a verdict
table. Bedrock gets three probes so the control-plane answer is read
against the runtime host we already know 404s, rather than as a lone
data point; Vertex gets three because which path serves the publisher
listing is itself part of the question. Vertex reuses the same JWT mint
and cloud-platform scope llm_router uses in production, so a result here
transfers.

No status code is asserted: the status is the finding. A probe fails
only when its credential is present and the request could not be made at
all, which is a broken probe rather than an answer.

Add a test_pattern dispatch input so this can run on its own in about a
minute instead of behind the sixteen-minute container suite. It reaches
the shell through an env var, since a dispatch input interpolated into a
run script is a script-injection seam.
This commit is contained in:
mlsmaycon
2026-08-19 06:31:30 +00:00
parent ecfbd686b8
commit 47cb96c010
2 changed files with 383 additions and 1 deletions
+12 -1
View File
@@ -12,6 +12,13 @@ on:
AWS issues it. Leave empty for the Sonnet 4.6 default.
required: false
default: ""
test_pattern:
description: >-
Package pattern to run. Defaults to the whole suite; narrow it to one
package (e.g. ./e2e/providerdiscovery/...) when a run only needs that
package's answer and not the sixteen minutes the container suite costs.
required: false
default: "./e2e/..."
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -77,4 +84,8 @@ jobs:
GOOGLE_VERTEX_PROJECT: ${{ secrets.E2E_GOOGLE_VERTEX_PROJECT }}
GOOGLE_VERTEX_REGION: ${{ secrets.E2E_GOOGLE_VERTEX_REGION }}
GOOGLE_VERTEX_MODEL: ${{ secrets.E2E_GOOGLE_VERTEX_MODEL }}
run: go test -tags e2e -timeout 40m -v ./e2e/...
# Read through an env var rather than interpolated into the run
# script: a dispatch input reaching a shell command directly is a
# script-injection seam, however trusted the dispatcher.
TEST_PATTERN: ${{ inputs.test_pattern || './e2e/...' }}
run: go test -tags e2e -timeout 40m -v "$TEST_PATTERN"