mirror of
https://github.com/netbirdio/netbird.git
synced 2026-07-16 19:49:56 +00:00
* [management,proxy] Agent network: per-account LLM gateway (policy, metering, multi-provider) (#6555) * [agent-network] Shared proto, OpenAPI schema, and generated types * [agent-network] Management: store, manager, synthesizer, policy engine, provider catalog, HTTP/gRPC API Adds the account-scoped agent-network module: provider/policy/budget CRUD and store, the reverse-proxy service synthesizer, policy selection + limit enforcement, the provider catalog (incl. Vertex AI and AWS Bedrock entries), and the management HTTP + proxy gRPC surfaces. * [management] Fix agent-network proxy-peer fan-out on affected-peer recompute The affected-peers resolver loaded only persisted reverse-proxy services, but agent-network services are synthesized on demand and never persisted. As a result the embedded proxy peer was never folded into the affected set when a client's group changed, so the proxy received no network-map update for a newly authorised client and rejected its handshake until a full resync (restart). loadProxyServices now merges the synthesized agent-network services (injected via a registration hook to avoid an import cycle), so proxy peers learn newly authorised clients immediately. * [proxy] Reverse-proxy middleware framework, chain, and request plumbing The per-target middleware chain (slots, dispatcher, mutation gate, metadata merger), body capture, access-log terminal sink, and the proxy wiring that builds + runs chains for synthesized agent-network services. * [proxy] LLM parsers, pricing, and builtin middlewares (OpenAI, Anthropic, Vertex AI, AWS Bedrock) Request/response parsers and SSE/event-stream metering, the embedded pricing table, and the builtin middleware set: request parser, router, policy limit-check/record, cost meter, guardrail, identity inject, response parser. Includes the path-routed providers — Google Vertex AI (keyfile:: service-account OAuth minting) and AWS Bedrock (bearer auth, invoke/converse/streaming, optional /bedrock prefix) — plus the Models allowlist and unmeterable-publisher deny. * [proxy] IPv6 in-place apply and TCP accept-loop hardening on netstack listeners * [agent-network] End-to-end test suite, module docs, and deployment preset * [agent-network] Fix codespell typos and exclude false positives - labelgen word pool: vermillion -> vermilion, racoon -> raccoon. - codespell ignore list: add flate (Go compress/flate package), recordin (a test-local identifier), and unparseable (a valid alternative spelling used consistently across identifiers + a metadata-value constant). * [management] Set LastSeen on injected proxy peer in realstack test (MySQL strict-mode) The injected embedded proxy peer had a PeerStatus with a zero LastSeen, which serializes to '0000-00-00' and is rejected by MySQL in strict mode (SQLite tolerates it). Set LastSeen to a valid time so SaveAccount succeeds on both engines. * [agent-network] Remove e2e shell-script suite from this branch The end-to-end shell scripts under scripts/e2e/ are maintained in a separate testing suite and are not part of this change set. * [agent-network] Polish module docs: remove internal review scaffolding, fix links, verify diagrams Strip PR-review framing, commit references, absolute paths, and stale internal references from the agent-network module docs; fix broken relative links; verify all diagrams against the current architecture. Remove the internal AI-reviewer prompt file. * [management] Refine session expiration handling to support 3-state encoding for SSO deadlines * [agent-network] Relocate agentnetwork package to internals/modules Move management/server/agentnetwork (and its catalog/, labelgen/, types/ subpackages) to management/internals/modules/agentnetwork, alongside the reverse-proxy module, and rewrite all importers. Pure relocation: package names, the synthesizer + affectedpeers registration hook, and store access (shared store.Store) are unchanged, so no import cycle is introduced (affectedpeers still depends only on the agentnetwork/types leaf). * [agent-network] Co-locate HTTP handlers in the module (RegisterEndpoints) Move the agent-network HTTP handlers from server/http/handlers/agentnetwork into the module at internals/modules/agentnetwork/handlers (package handlers) and rename the entrypoint AddEndpoints -> RegisterEndpoints, matching the reverse-proxy module convention. Wiring in http/handler.go updated accordingly. * Update getting started to point to rc when agent network enabled * Add a reference to a commercial license * Fix docs localhost link * Fix docs localhost link * Add private services domain note * [management] Add agent-network telemetry metrics (#6561) Surface agent-network adoption and usage in the self-hosted metrics worker: distinct accounts, providers, policies, budget rules, accounts with log collection enabled, and aggregated input/output tokens plus cost. Tokens and cost are summed from agent_network_request_usage (the always-written per-request ledger) so the figures are accurate regardless of the log-collection toggle and carry no double-counting. All values come from a handful of indexed aggregate queries run only on the worker's periodic tick. Adds store.AgentNetworkMetrics with GetAgentNetworkMetrics on the Store interface, the SqlStore implementation, and a zero-valued FileStore stub. * Update NetBird server and proxy image versions to 0.74.0-rc.2 * [management,proxy] Reduce agent-network cognitive complexity (#6566) Address the SonarCloud quality-gate findings in new agent-network code by extracting focused helpers. No behavior change. - synthesizer.go: split buildIdentityInjectConfigJSON into per-shape rule builders; extract mergeGuardrail from mergeGuardrails to cut nesting depth. - llm_identity_inject: extract injectionEmitsAnything validation predicate from New. - llm_response_parser/streaming.go: extract applyOpenAIStreamUsage and applyAnthropicStreamUsage (via a named anthropicStreamUsage type) and simplify the OpenAI scanner loop. - reverseproxy.go: decompose ServeHTTP into serveRouteError, buildTargetContext, serveDirect, serveWithChain, captureRequestForChain, serveDeny, newResponseWriter, observeResponse, and forwardUpstream, preserving the defer ordering so response observation still reads the captured writer before it is released. * [management] Move agent-network access-log ingest into the agentnetwork module (#6568) The agent-network access-log ingest path (metaKey wire contract, flatten, usage derivation, and the dual-write of the usage ledger + settings-gated full row) lived in the reverseproxy accesslogs manager, even though the agentnetwork module already owns the rest of that domain — types, read (ListAccessLogs / GetUsageOverview), the budget-counter writes, and retention cleanup. Move it next to the rest: a stateless agentnetwork.IngestAccessLog(ctx, store, entry) that the reverseproxy SaveAccessLog delegates to when the entry is agent-network. Removes the agentNetworkTypes import from the reverseproxy manager. No behavior change; the write/read table separation is unchanged. Adds real-store coverage for the disable->enable log-collection toggle (usage ledger always written, full row gated) plus the metadata parse and group-dedup helpers, which previously had no dedicated tests. * Add session view support in the access log * [management,proxy] Container-based agent-network e2e harness (#6577) * [e2e] Add container-based agent-network e2e harness (Pillar 1) Introduce a self-contained, OIDC-free e2e harness that stands up NetBird in containers, so suites no longer depend on the hand-maintained Tilt stack or a real IdP. - harness brings up the combined server (management + signal + relay + STUN + embedded IdP) in a single container built from combined/Dockerfile.multistage, and mints an admin PAT through the unauthenticated /api/setup bootstrap (NB_SETUP_PAT_ENABLED). API access goes through the existing shared/management/client/rest typed client. - the image is built via the docker CLI (BuildKit) so the Dockerfile's cache mounts are honored; testcontainers then runs the tagged image. - everything is behind the `e2e` build tag so normal builds and unit tests never pull in testcontainers. Adds BuildKit cache mounts to combined/Dockerfile.multistage so source changes recompile incrementally rather than from scratch. Pillar 1 proven by TestCombinedBootstrap: server builds, boots, mints a PAT, and the PAT authenticates a real management API call. * [e2e] Add management-side agent-network scenarios (Pillar 2) Port the API-driven agent-network scenarios from the bash suites to Go, sharing one combined server per package run (TestMain) with each test owning its resource cleanup. Drives the /api/agent-network/* endpoints through the shared REST client's NewRequest primitive with the generated api types. Scenarios: - provider lifecycle (create/get/list/delete + 404 after delete) - provider validation (missing api_key, unknown catalog id → 4xx) - settings collection-toggle round-trip with cluster/subdomain immutability - policy window floor (reject <60s enabled limit, accept at 60s) - consumption read endpoint returns an array All deterministic and dependency-free (dummy provider keys; no upstream calls), so they run headless in CI. * [e2e] Add live chat-through-proxy scenario (Pillar 3) Stand up the full agent-network data path in containers and drive a real chat-completion through the gateway: - harness: a shared docker network (combined server reachable by alias), a proxy container built from the published reverse-proxy image (NB_PROXY_PRIVATE, NB_PROXY_ALLOW_INSECURE, NB_RELAY_TRANSPORT=ws to match the combined server's WS-multiplexed relay) with a generated self-signed wildcard cert, and a netbird client container that joins via a setup key. - the combined image, proxy image, and client image default to the published rc.2 releases (overridable via NB_E2E_*_IMAGE; a bare local tag is built from source instead). Geolocation download is disabled so the server starts without external fetches. - one shared domain is used for the management exposed address, the proxy domain, and the agent-network cluster; the proxy token is minted via the server CLI (global) to match the manual install. TestChatCompletionThroughProxy provisions provider+policy+group+setup key, runs proxy+client, drives an OpenAI chat-completion through the tunnel, and asserts a 200 plus the ingested access-log row. Requires OPENAI_TOKEN (skips otherwise). The provider must be created with enabled=true explicitly — the create default is false despite the API doc. * [e2e] Run the live chat scenario across a provider matrix Replace the single-provider chat test with a data-driven matrix that runs the same scenario through every provider whose credentials are present in the environment (keys/URLs sourced from ~/.llm-keys locally, Actions secrets in CI): - OpenAI (chat), Anthropic (messages), Vercel, OpenRouter, Cloudflare (OpenAI-compatible gateways), and Bedrock (path-routed, bearer, via the messages shape) — covering both wire shapes and the gateway routing. - all providers are created enabled with a unique model string so the proxy's connect-time snapshot carries them all and model->provider routing is unambiguous (provider toggles after connect don't reconcile to a connected proxy). - the client supports both wire shapes (/v1/chat/completions and /v1/messages); Cloudflare gets the openai provider segment appended to its gateway URL. Each provider must return 200 through the tunnel and produce an ingested access-log row. Vertex is intentionally excluded from the uniform matrix: it needs a bespoke rawPredict request shape rather than the shared chat/messages path, so it warrants a dedicated scenario. * [ci] Add manual workflow for the agent-network e2e suite The e2e suite (build tag `e2e`) stands up the combined server + proxy + client in Docker and drives live chat-completions, so it is slow and needs provider credentials. Gate it out of normal CI (it already is, via the build tag) and run it on demand via workflow_dispatch. Provider scenarios skip when their secret is unset, so it degrades gracefully. * [e2e] Add Vertex to the provider matrix; run e2e on ubuntu-latest Vertex (Anthropic-on-Vertex) doesn't share the chat/messages wire shapes: the model travels in a rawPredict path and the proxy mints the service account's OAuth token. Add a Vertex client method that posts /v1/projects/<project>/locations/<region>/publishers/anthropic/models/<model>:rawPredict with the Vertex anthropic_version body, and wire it into the matrix as a path-routed provider (created without a models array). It is keyed off GOOGLE_VERTEX_SA_BASE64 + GOOGLE_VERTEX_PROJECT (region defaults to "global", model to a pinned claude snapshot, both overridable). Also bump the e2e workflow runner to ubuntu-latest and add the Vertex secrets. * Add docker/docker and docker/go-connections as direct dependencies in go.mod * [ci] Trigger agent-network e2e workflow on push to main and pull requests * [e2e] Fix proxy cert permission denied on Linux CI runners The proxy bind-mounts a temp dir of self-signed certs. MkdirTemp creates it 0700 and the key was 0600, which Docker Desktop on macOS ignores but a non-root proxy container on Linux runners cannot traverse/read, so the cert watcher failed with "open /certs/tls.crt: permission denied" and the container exited. Widen the cert dir to 0755 and write the throwaway key 0644 so the proxy uid can read the bind-mounted material. * [e2e] Build images from source by default instead of pulling rc.2 The agent-network code under test lives in this branch, so the e2e should exercise it rather than a frozen published release. Flip the harness default: combined/proxy/client are now built from their in-repo Dockerfiles (combined/Dockerfile.multistage, proxy/Dockerfile.multistage, e2e/harness/Dockerfile.client) under local tags. Pulling a published image stays available by setting NB_E2E_*_IMAGE to a registry reference. Builds now go through buildx --load so the Dockerfile cache mounts are honored and the result is loaded for testcontainers. The CI workflow adds a container-driver builder and a local layer cache (NB_E2E_BUILDX_CACHE) persisted via actions/cache, which caches the base/apt/dep-download layers across runs. The Go compile still re-runs each time, as BuildKit mount caches cannot be exported to the GitHub cache. * [e2e] Cover real providers in lifecycle + assert real consumption metering - TestProviderLifecycle now runs per available real provider (create → get → list → delete → 404) instead of a single dummy provider, exercising each catalog's create and field round-trip. Create is offline, so it stays fast and burns no provider quota; falls back to a synthetic OpenAI provider when no keys are set. - TestProvidersMatrix attaches a token limit (high caps, 60s window) to its policy, which switches on usage metering, and asserts consumption rows are recorded with positive token counts after the live traffic. Consumption is account-scoped (keyed by source group / user and window, not per provider), so the assertion is aggregate. - TestProviderValidation gains invalid-upstream and blank-name cases. Create validation is uniform across catalogs (no per-provider required-field rules), so per-provider rejection cases would be redundant. * [e2e] Assert session id propagates per provider Each matrix request now sends a unique session id as the universal x-session-id header and asserts it round-trips into that provider's access-log row. This guards the session-grouping contract end to end for every provider (header extraction runs in llm_request_parser ahead of the parser-specific body extraction, so it is provider-agnostic). * [e2e] Drop accidentally committed sync-phases dashboard netbird-sync-phases.json was swept into the Pillar 1 commit by a broad git add; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Remove it from the branch so the PR diff is scoped to the e2e changes. * [e2e] Revert accidentally committed sync-phase ingest spec The netbird_sync_phase measurement spec in metrics ingest was swept into the Pillar 1 commit; it belongs to the unrelated sync-phases metrics work, not this e2e harness. Its emission side never landed here, so the spec was orphaned anyway. Restore ingest/main.go to its origin/main state. * Fix golint issues * Fix sonar * Add access log session test * Fix access log tests --------- Co-authored-by: braginini <bangvalo@gmail.com> Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>
1968 lines
58 KiB
Go
1968 lines
58 KiB
Go
package tcp
|
|
|
|
import (
|
|
"context"
|
|
"crypto/ecdsa"
|
|
"crypto/elliptic"
|
|
"crypto/rand"
|
|
"crypto/tls"
|
|
"crypto/x509"
|
|
"math/big"
|
|
"net"
|
|
"testing"
|
|
"time"
|
|
|
|
log "github.com/sirupsen/logrus"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/netbirdio/netbird/proxy/internal/restrict"
|
|
"github.com/netbirdio/netbird/proxy/internal/types"
|
|
)
|
|
|
|
func TestRouter_HTTPRouting(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
router := NewRouter(logger, nil, addr)
|
|
router.AddRoute("example.com", Route{Type: RouteHTTP})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// Dial in a goroutine. The TLS handshake will block since nothing
|
|
// completes it on the HTTP side, but we only care about routing.
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
// Send a TLS ClientHello manually.
|
|
tlsConn := tls.Client(conn, &tls.Config{
|
|
ServerName: "example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
_ = tlsConn.Handshake()
|
|
tlsConn.Close()
|
|
}()
|
|
|
|
// Verify the connection was routed to the HTTP channel.
|
|
select {
|
|
case conn := <-router.httpCh:
|
|
assert.NotNil(t, conn)
|
|
conn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("no connection received on HTTP channel")
|
|
}
|
|
}
|
|
|
|
func TestRouter_TCPRouting(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
// Set up a TLS backend that the relay will connect to.
|
|
backendCert := generateSelfSignedCert(t)
|
|
backendLn, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
|
|
Certificates: []tls.Certificate{backendCert},
|
|
})
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
backendAddr := backendLn.Addr().String()
|
|
|
|
// Accept one connection on the backend, echo data back.
|
|
backendReady := make(chan struct{})
|
|
go func() {
|
|
close(backendReady)
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := conn.Read(buf)
|
|
_, _ = conn.Write(buf[:n])
|
|
}()
|
|
<-backendReady
|
|
|
|
dialResolve := func(accountID types.AccountID) (types.DialContextFunc, error) {
|
|
return func(ctx context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
router.AddRoute("tcp.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "test-service",
|
|
Target: backendAddr,
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// Connect as a TLS client; the proxy should passthrough to the backend.
|
|
clientConn, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "tcp.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
defer clientConn.Close()
|
|
|
|
testData := []byte("hello through TCP passthrough")
|
|
_, err = clientConn.Write(testData)
|
|
require.NoError(t, err)
|
|
|
|
buf := make([]byte, 1024)
|
|
n, err := clientConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "should receive echoed data through TCP passthrough")
|
|
}
|
|
|
|
func TestRouter_UnknownSNIGoesToHTTP(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
router := NewRouter(logger, nil, addr)
|
|
// No routes registered.
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
tlsConn := tls.Client(conn, &tls.Config{
|
|
ServerName: "unknown.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
_ = tlsConn.Handshake()
|
|
tlsConn.Close()
|
|
}()
|
|
|
|
select {
|
|
case conn := <-router.httpCh:
|
|
assert.NotNil(t, conn)
|
|
conn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("unknown SNI should be routed to HTTP")
|
|
}
|
|
}
|
|
|
|
// TestRouter_NonTLSConnectionDropped verifies that a non-TLS connection
|
|
// on the shared port is closed by the router (SNI peek fails to find a
|
|
// valid ClientHello, so there is no route match).
|
|
func TestRouter_NonTLSConnectionDropped(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
// Register a TLS passthrough route. Non-TLS should NOT match.
|
|
dialResolve := func(accountID types.AccountID) (types.DialContextFunc, error) {
|
|
return func(ctx context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
router.AddRoute("tcp.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "test-service",
|
|
Target: "127.0.0.1:9999",
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// Send plain HTTP (non-TLS) data.
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
defer conn.Close()
|
|
|
|
_, _ = conn.Write([]byte("GET / HTTP/1.1\r\nHost: tcp.example.com\r\n\r\n"))
|
|
|
|
// Non-TLS traffic on a port with RouteTCP goes to the HTTP channel
|
|
// because there's no valid SNI to match. Verify it reaches HTTP.
|
|
select {
|
|
case httpConn := <-router.httpCh:
|
|
assert.NotNil(t, httpConn, "non-TLS connection should fall through to HTTP")
|
|
httpConn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("non-TLS connection was not routed to HTTP")
|
|
}
|
|
}
|
|
|
|
// TestRouter_TLSAndHTTPCoexist verifies that a shared port with both HTTP
|
|
// and TLS passthrough routes correctly demuxes based on the SNI hostname.
|
|
func TestRouter_TLSAndHTTPCoexist(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
backendCert := generateSelfSignedCert(t)
|
|
backendLn, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
|
|
Certificates: []tls.Certificate{backendCert},
|
|
})
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
// Backend echoes data.
|
|
go func() {
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := conn.Read(buf)
|
|
_, _ = conn.Write(buf[:n])
|
|
}()
|
|
|
|
dialResolve := func(accountID types.AccountID) (types.DialContextFunc, error) {
|
|
return func(ctx context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
// HTTP route.
|
|
router.AddRoute("app.example.com", Route{Type: RouteHTTP})
|
|
// TLS passthrough route.
|
|
router.AddRoute("tcp.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "test-service",
|
|
Target: backendLn.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// 1. TLS connection with SNI "tcp.example.com" → TLS passthrough.
|
|
tlsConn, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "tcp.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
testData := []byte("passthrough data")
|
|
_, err = tlsConn.Write(testData)
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "TLS passthrough should relay data")
|
|
tlsConn.Close()
|
|
|
|
// 2. TLS connection with SNI "app.example.com" → HTTP handler.
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
c := tls.Client(conn, &tls.Config{
|
|
ServerName: "app.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
_ = c.Handshake()
|
|
c.Close()
|
|
}()
|
|
|
|
select {
|
|
case httpConn := <-router.httpCh:
|
|
assert.NotNil(t, httpConn, "HTTP SNI should go to HTTP handler")
|
|
httpConn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("HTTP-route connection was not delivered to HTTP handler")
|
|
}
|
|
}
|
|
|
|
func TestRouter_AddRemoveRoute(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, nil, addr)
|
|
|
|
router.AddRoute("a.example.com", Route{Type: RouteHTTP, ServiceID: "svc-a"})
|
|
router.AddRoute("b.example.com", Route{Type: RouteTCP, ServiceID: "svc-b", Target: "10.0.0.1:5432"})
|
|
|
|
route, ok := router.lookupRoute("a.example.com")
|
|
assert.True(t, ok)
|
|
assert.Equal(t, RouteHTTP, route.Type)
|
|
|
|
route, ok = router.lookupRoute("b.example.com")
|
|
assert.True(t, ok)
|
|
assert.Equal(t, RouteTCP, route.Type)
|
|
|
|
router.RemoveRoute("a.example.com", "svc-a")
|
|
_, ok = router.lookupRoute("a.example.com")
|
|
assert.False(t, ok)
|
|
}
|
|
|
|
func TestChanListener_AcceptAndClose(t *testing.T) {
|
|
ch := make(chan net.Conn, 1)
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
ln := newChanListener(ch, addr)
|
|
|
|
assert.Equal(t, addr, ln.Addr())
|
|
|
|
// Send a connection.
|
|
clientConn, serverConn := net.Pipe()
|
|
defer clientConn.Close()
|
|
defer serverConn.Close()
|
|
|
|
ch <- serverConn
|
|
|
|
conn, err := ln.Accept()
|
|
require.NoError(t, err)
|
|
assert.Equal(t, serverConn, conn)
|
|
|
|
// Close should cause Accept to return error.
|
|
require.NoError(t, ln.Close())
|
|
// Double close should be safe.
|
|
require.NoError(t, ln.Close())
|
|
|
|
_, err = ln.Accept()
|
|
assert.ErrorIs(t, err, net.ErrClosed)
|
|
}
|
|
|
|
func TestRouter_HTTPPrecedenceGuard(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, nil, addr)
|
|
|
|
host := SNIHost("app.example.com")
|
|
|
|
t.Run("http takes precedence over tcp at lookup", func(t *testing.T) {
|
|
router.AddRoute(host, Route{Type: RouteHTTP, ServiceID: "svc-http"})
|
|
router.AddRoute(host, Route{Type: RouteTCP, ServiceID: "svc-tcp", Target: "10.0.0.1:443"})
|
|
|
|
route, ok := router.lookupRoute(host)
|
|
require.True(t, ok)
|
|
assert.Equal(t, RouteHTTP, route.Type, "HTTP route must take precedence over TCP")
|
|
assert.Equal(t, types.ServiceID("svc-http"), route.ServiceID)
|
|
|
|
router.RemoveRoute(host, "svc-http")
|
|
router.RemoveRoute(host, "svc-tcp")
|
|
})
|
|
|
|
t.Run("tcp becomes active when http is removed", func(t *testing.T) {
|
|
router.AddRoute(host, Route{Type: RouteHTTP, ServiceID: "svc-http"})
|
|
router.AddRoute(host, Route{Type: RouteTCP, ServiceID: "svc-tcp", Target: "10.0.0.1:443"})
|
|
|
|
router.RemoveRoute(host, "svc-http")
|
|
|
|
route, ok := router.lookupRoute(host)
|
|
require.True(t, ok)
|
|
assert.Equal(t, RouteTCP, route.Type, "TCP should take over after HTTP removal")
|
|
assert.Equal(t, types.ServiceID("svc-tcp"), route.ServiceID)
|
|
|
|
router.RemoveRoute(host, "svc-tcp")
|
|
})
|
|
|
|
t.Run("order of add does not matter", func(t *testing.T) {
|
|
router.AddRoute(host, Route{Type: RouteTCP, ServiceID: "svc-tcp", Target: "10.0.0.1:443"})
|
|
router.AddRoute(host, Route{Type: RouteHTTP, ServiceID: "svc-http"})
|
|
|
|
route, ok := router.lookupRoute(host)
|
|
require.True(t, ok)
|
|
assert.Equal(t, RouteHTTP, route.Type, "HTTP takes precedence regardless of add order")
|
|
|
|
router.RemoveRoute(host, "svc-http")
|
|
router.RemoveRoute(host, "svc-tcp")
|
|
})
|
|
|
|
t.Run("same service id updates in place", func(t *testing.T) {
|
|
router.AddRoute(host, Route{Type: RouteTCP, ServiceID: "svc-1", Target: "10.0.0.1:443"})
|
|
router.AddRoute(host, Route{Type: RouteTCP, ServiceID: "svc-1", Target: "10.0.0.2:443"})
|
|
|
|
route, ok := router.lookupRoute(host)
|
|
require.True(t, ok)
|
|
assert.Equal(t, "10.0.0.2:443", route.Target, "route should be updated in place")
|
|
|
|
router.RemoveRoute(host, "svc-1")
|
|
_, ok = router.lookupRoute(host)
|
|
assert.False(t, ok)
|
|
})
|
|
|
|
t.Run("double remove is safe", func(t *testing.T) {
|
|
router.AddRoute(host, Route{Type: RouteHTTP, ServiceID: "svc-1"})
|
|
router.RemoveRoute(host, "svc-1")
|
|
router.RemoveRoute(host, "svc-1")
|
|
|
|
_, ok := router.lookupRoute(host)
|
|
assert.False(t, ok, "route should be gone after removal")
|
|
})
|
|
|
|
t.Run("remove does not affect other hosts", func(t *testing.T) {
|
|
router.AddRoute("a.example.com", Route{Type: RouteHTTP, ServiceID: "svc-a"})
|
|
router.AddRoute("b.example.com", Route{Type: RouteTCP, ServiceID: "svc-b", Target: "10.0.0.2:22"})
|
|
|
|
router.RemoveRoute("a.example.com", "svc-a")
|
|
|
|
_, ok := router.lookupRoute(SNIHost("a.example.com"))
|
|
assert.False(t, ok)
|
|
|
|
route, ok := router.lookupRoute(SNIHost("b.example.com"))
|
|
require.True(t, ok)
|
|
assert.Equal(t, RouteTCP, route.Type, "removing one host must not affect another")
|
|
|
|
router.RemoveRoute("b.example.com", "svc-b")
|
|
})
|
|
}
|
|
|
|
func TestRouter_SetRemoveFallback(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, nil)
|
|
|
|
assert.True(t, router.IsEmpty(), "new port router should be empty")
|
|
|
|
router.SetFallback(Route{Type: RouteTCP, ServiceID: "svc-fb", Target: "10.0.0.1:5432"})
|
|
assert.False(t, router.IsEmpty(), "router with fallback should not be empty")
|
|
|
|
router.AddRoute("a.example.com", Route{Type: RouteTCP, ServiceID: "svc-a", Target: "10.0.0.2:443"})
|
|
assert.False(t, router.IsEmpty())
|
|
|
|
router.RemoveFallback("svc-fb")
|
|
assert.False(t, router.IsEmpty(), "router with SNI route should not be empty")
|
|
|
|
router.RemoveRoute("a.example.com", "svc-a")
|
|
assert.True(t, router.IsEmpty(), "router with no routes and no fallback should be empty")
|
|
}
|
|
|
|
func TestPortRouter_FallbackRelaysData(t *testing.T) {
|
|
// Backend echo server.
|
|
backendLn, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
go func() {
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := conn.Read(buf)
|
|
_, _ = conn.Write(buf[:n])
|
|
}()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "test-service",
|
|
Target: backendLn.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Plain TCP (non-TLS) connection should be relayed via fallback.
|
|
// Use exactly 5 bytes. PeekClientHello reads 5 bytes as the TLS
|
|
// header, so a single 5-byte write lands as one chunk at the backend.
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
defer conn.Close()
|
|
|
|
testData := []byte("hello")
|
|
_, err = conn.Write(testData)
|
|
require.NoError(t, err)
|
|
|
|
buf := make([]byte, 1024)
|
|
n, err := conn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "should receive echoed data through fallback relay")
|
|
}
|
|
|
|
func TestPortRouter_FallbackOnUnknownSNI(t *testing.T) {
|
|
// Backend TLS echo server.
|
|
backendCert := generateSelfSignedCert(t)
|
|
backendLn, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
|
|
Certificates: []tls.Certificate{backendCert},
|
|
})
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
go func() {
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := conn.Read(buf)
|
|
_, _ = conn.Write(buf[:n])
|
|
}()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
// Only a fallback, no SNI route for "unknown.example.com".
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "test-service",
|
|
Target: backendLn.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// TLS with unknown SNI → fallback relay to TLS backend.
|
|
tlsConn, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "tcp.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
defer tlsConn.Close()
|
|
|
|
testData := []byte("hello through fallback TLS")
|
|
_, err = tlsConn.Write(testData)
|
|
require.NoError(t, err)
|
|
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "unknown SNI should relay through fallback")
|
|
}
|
|
|
|
func TestPortRouter_SNIWinsOverFallback(t *testing.T) {
|
|
// Two backend echo servers: one for SNI match, one for fallback.
|
|
sniBacked := startEchoTLS(t)
|
|
fbBacked := startEchoTLS(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.AddRoute("tcp.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "sni-service",
|
|
Target: sniBacked.Addr().String(),
|
|
})
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "fb-service",
|
|
Target: fbBacked.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// TLS with matching SNI should go to SNI backend, not fallback.
|
|
tlsConn, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "tcp.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
defer tlsConn.Close()
|
|
|
|
testData := []byte("SNI route data")
|
|
_, err = tlsConn.Write(testData)
|
|
require.NoError(t, err)
|
|
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "SNI match should use SNI route, not fallback")
|
|
}
|
|
|
|
func TestPortRouter_NoFallbackNoHTTP_Closes(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, nil)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
defer conn.Close()
|
|
|
|
_, _ = conn.Write([]byte("hello"))
|
|
|
|
// Connection should be closed by the router (no fallback, no HTTP).
|
|
buf := make([]byte, 1)
|
|
_ = conn.SetReadDeadline(time.Now().Add(3 * time.Second))
|
|
_, err = conn.Read(buf)
|
|
assert.Error(t, err, "connection should be closed when no fallback and no HTTP channel")
|
|
}
|
|
|
|
func TestRouter_FallbackAndHTTPCoexist(t *testing.T) {
|
|
// Fallback backend echo server (plain TCP).
|
|
fbBackend, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer fbBackend.Close()
|
|
|
|
go func() {
|
|
conn, err := fbBackend.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := conn.Read(buf)
|
|
_, _ = conn.Write(buf[:n])
|
|
}()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
|
|
// HTTP route for known SNI.
|
|
router.AddRoute("app.example.com", Route{Type: RouteHTTP})
|
|
// Fallback for non-TLS / unknown SNI.
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "test-account",
|
|
ServiceID: "fb-service",
|
|
Target: fbBackend.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// 1. TLS with known HTTP SNI → should go to HTTP channel.
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
c := tls.Client(conn, &tls.Config{
|
|
ServerName: "app.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
_ = c.Handshake()
|
|
c.Close()
|
|
}()
|
|
|
|
select {
|
|
case httpConn := <-router.httpCh:
|
|
assert.NotNil(t, httpConn, "known HTTP SNI should go to HTTP channel")
|
|
httpConn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("HTTP-route connection was not delivered to HTTP handler")
|
|
}
|
|
|
|
// 2. Plain TCP (non-TLS) → should go to fallback, not HTTP.
|
|
// Use exactly 5 bytes to match PeekClientHello header size.
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
defer conn.Close()
|
|
|
|
testData := []byte("plain")
|
|
_, err = conn.Write(testData)
|
|
require.NoError(t, err)
|
|
|
|
buf := make([]byte, 1024)
|
|
n, err := conn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, testData, buf[:n], "non-TLS should be relayed via fallback, not HTTP")
|
|
}
|
|
|
|
// startEchoTLS starts a TLS echo server and returns the listener.
|
|
func startEchoTLS(t *testing.T) net.Listener {
|
|
t.Helper()
|
|
|
|
cert := generateSelfSignedCert(t)
|
|
ln, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
|
|
Certificates: []tls.Certificate{cert},
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { ln.Close() })
|
|
|
|
go func() {
|
|
conn, err := ln.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
buf := make([]byte, 1024)
|
|
for {
|
|
n, err := conn.Read(buf)
|
|
if err != nil {
|
|
return
|
|
}
|
|
if _, err := conn.Write(buf[:n]); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}()
|
|
|
|
return ln
|
|
}
|
|
|
|
func generateSelfSignedCert(t *testing.T) tls.Certificate {
|
|
t.Helper()
|
|
|
|
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
|
require.NoError(t, err)
|
|
|
|
template := &x509.Certificate{
|
|
SerialNumber: big.NewInt(1),
|
|
DNSNames: []string{"tcp.example.com"},
|
|
NotBefore: time.Now(),
|
|
NotAfter: time.Now().Add(time.Hour),
|
|
}
|
|
|
|
certDER, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key)
|
|
require.NoError(t, err)
|
|
|
|
return tls.Certificate{
|
|
Certificate: [][]byte{certDER},
|
|
PrivateKey: key,
|
|
}
|
|
}
|
|
|
|
func TestRouter_DrainWaitsForRelays(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
backendLn, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
// Accept connections: echo first message, then hold open until told to close.
|
|
closeBackend := make(chan struct{})
|
|
go func() {
|
|
for {
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
go func(c net.Conn) {
|
|
defer c.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := c.Read(buf)
|
|
_, _ = c.Write(buf[:n])
|
|
<-closeBackend
|
|
}(conn)
|
|
}
|
|
}()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
Target: backendLn.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
serveDone := make(chan struct{})
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
close(serveDone)
|
|
}()
|
|
|
|
// Open a relay connection (non-TLS, hits fallback).
|
|
conn, err := net.Dial("tcp", ln.Addr().String())
|
|
require.NoError(t, err)
|
|
_, _ = conn.Write([]byte("hello"))
|
|
|
|
// Wait for the echo to confirm the relay is fully established.
|
|
buf := make([]byte, 16)
|
|
_ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
|
n, err := conn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "hello", string(buf[:n]))
|
|
_ = conn.SetReadDeadline(time.Time{})
|
|
|
|
// Drain with a short timeout should fail because the relay is still active.
|
|
assert.False(t, router.Drain(50*time.Millisecond), "drain should timeout with active relay")
|
|
|
|
// Close backend connections so relays finish.
|
|
close(closeBackend)
|
|
_ = conn.Close()
|
|
|
|
// Drain should now complete quickly.
|
|
assert.True(t, router.Drain(2*time.Second), "drain should succeed after relays end")
|
|
|
|
cancel()
|
|
<-serveDone
|
|
}
|
|
|
|
func TestRouter_DrainEmptyReturnsImmediately(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, nil)
|
|
|
|
start := time.Now()
|
|
ok := router.Drain(5 * time.Second)
|
|
elapsed := time.Since(start)
|
|
|
|
assert.True(t, ok)
|
|
assert.Less(t, elapsed, 100*time.Millisecond, "drain with no relays should return immediately")
|
|
}
|
|
|
|
// TestRemoveRoute_KillsActiveRelays verifies that removing a route
|
|
// immediately kills active relay connections for that service.
|
|
func TestRemoveRoute_KillsActiveRelays(t *testing.T) {
|
|
backendLn, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer backendLn.Close()
|
|
|
|
// Backend echoes first message, then holds connection open.
|
|
go func() {
|
|
for {
|
|
conn, err := backendLn.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
go func(c net.Conn) {
|
|
defer c.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := c.Read(buf)
|
|
_, _ = c.Write(buf[:n])
|
|
// Hold the connection open.
|
|
for {
|
|
if _, err := c.Read(buf); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}(conn)
|
|
}
|
|
}()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-1",
|
|
Target: backendLn.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Establish a relay connection.
|
|
conn, err := net.Dial("tcp", ln.Addr().String())
|
|
require.NoError(t, err)
|
|
defer conn.Close()
|
|
_, err = conn.Write([]byte("hello"))
|
|
require.NoError(t, err)
|
|
|
|
// Wait for echo to confirm relay is established.
|
|
buf := make([]byte, 16)
|
|
_ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
|
n, err := conn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "hello", string(buf[:n]))
|
|
_ = conn.SetReadDeadline(time.Time{})
|
|
|
|
// Remove the fallback: should kill the active relay.
|
|
router.RemoveFallback("svc-1")
|
|
|
|
// The client connection should see an error (server closed).
|
|
_ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
|
_, err = conn.Read(buf)
|
|
assert.Error(t, err, "connection should be killed after service removal")
|
|
}
|
|
|
|
// TestRemoveRoute_KillsSNIRelays verifies that removing an SNI route
|
|
// kills its active relays without affecting other services.
|
|
func TestRemoveRoute_KillsSNIRelays(t *testing.T) {
|
|
backend := startEchoTLS(t)
|
|
defer backend.Close()
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
router.AddRoute("tls.example.com", Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-tls",
|
|
Target: backend.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Establish a TLS relay.
|
|
tlsConn, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "tls.example.com", InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err)
|
|
defer tlsConn.Close()
|
|
|
|
_, err = tlsConn.Write([]byte("ping"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "ping", string(buf[:n]))
|
|
|
|
// Remove the route: active relay should die.
|
|
router.RemoveRoute("tls.example.com", "svc-tls")
|
|
|
|
_ = tlsConn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
|
_, err = tlsConn.Read(buf)
|
|
assert.Error(t, err, "TLS relay should be killed after route removal")
|
|
}
|
|
|
|
// TestPortRouter_SNIAndTCPFallbackCoexist verifies that a single port can
|
|
// serve both SNI-routed TLS passthrough and plain TCP fallback simultaneously.
|
|
func TestPortRouter_SNIAndTCPFallbackCoexist(t *testing.T) {
|
|
sniBackend := startEchoTLS(t)
|
|
fbBackend := startEchoPlain(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
|
|
// SNI route for a specific domain.
|
|
router.AddRoute("tcp.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "acct-1",
|
|
ServiceID: "svc-sni",
|
|
Target: sniBackend.Addr().String(),
|
|
})
|
|
// TCP fallback for everything else.
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "acct-2",
|
|
ServiceID: "svc-fb",
|
|
Target: fbBackend.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// 1. TLS with matching SNI → goes to SNI backend.
|
|
tlsConn, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "tcp.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
_, err = tlsConn.Write([]byte("sni-data"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "sni-data", string(buf[:n]), "SNI match → SNI backend")
|
|
tlsConn.Close()
|
|
|
|
// 2. Plain TCP (no TLS) → goes to fallback.
|
|
tcpConn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
|
|
_, err = tcpConn.Write([]byte("plain"))
|
|
require.NoError(t, err)
|
|
n, err = tcpConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "plain", string(buf[:n]), "plain TCP → fallback backend")
|
|
tcpConn.Close()
|
|
|
|
// 3. TLS with unknown SNI → also goes to fallback.
|
|
unknownBackend := startEchoTLS(t)
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
AccountID: "acct-2",
|
|
ServiceID: "svc-fb",
|
|
Target: unknownBackend.Addr().String(),
|
|
})
|
|
|
|
unknownTLS, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "unknown.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
_, err = unknownTLS.Write([]byte("unknown-sni"))
|
|
require.NoError(t, err)
|
|
n, err = unknownTLS.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "unknown-sni", string(buf[:n]), "unknown SNI → fallback backend")
|
|
unknownTLS.Close()
|
|
}
|
|
|
|
// TestPortRouter_UpdateRouteSwapsSNI verifies that updating a route
|
|
// (remove + add with different target) correctly routes to the new backend.
|
|
func TestPortRouter_UpdateRouteSwapsSNI(t *testing.T) {
|
|
backend1 := startEchoTLS(t)
|
|
backend2 := startEchoTLS(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Initial route → backend1.
|
|
router.AddRoute("db.example.com", Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-db",
|
|
Target: backend1.Addr().String(),
|
|
})
|
|
|
|
conn1, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "db.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
_, err = conn1.Write([]byte("v1"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := conn1.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "v1", string(buf[:n]))
|
|
conn1.Close()
|
|
|
|
// Update: remove old route, add new → backend2.
|
|
router.RemoveRoute("db.example.com", "svc-db")
|
|
router.AddRoute("db.example.com", Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-db",
|
|
Target: backend2.Addr().String(),
|
|
})
|
|
|
|
conn2, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "db.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
_, err = conn2.Write([]byte("v2"))
|
|
require.NoError(t, err)
|
|
n, err = conn2.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "v2", string(buf[:n]))
|
|
conn2.Close()
|
|
}
|
|
|
|
// TestPortRouter_RemoveSNIFallsThrough verifies that after removing an
|
|
// SNI route, connections for that domain fall through to the fallback.
|
|
func TestPortRouter_RemoveSNIFallsThrough(t *testing.T) {
|
|
sniBackend := startEchoTLS(t)
|
|
fbBackend := startEchoTLS(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.AddRoute("db.example.com", Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-db",
|
|
Target: sniBackend.Addr().String(),
|
|
})
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
Target: fbBackend.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Before removal: SNI matches → sniBackend.
|
|
conn1, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "db.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
_, err = conn1.Write([]byte("before"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := conn1.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "before", string(buf[:n]))
|
|
conn1.Close()
|
|
|
|
// Remove SNI route. Should fall through to fallback.
|
|
router.RemoveRoute("db.example.com", "svc-db")
|
|
|
|
conn2, err := tls.Dial("tcp", ln.Addr().String(), &tls.Config{
|
|
ServerName: "db.example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
require.NoError(t, err)
|
|
_, err = conn2.Write([]byte("after"))
|
|
require.NoError(t, err)
|
|
n, err = conn2.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "after", string(buf[:n]), "after removal, should reach fallback")
|
|
conn2.Close()
|
|
}
|
|
|
|
// TestPortRouter_RemoveFallbackCloses verifies that after removing the
|
|
// fallback, non-matching connections are closed.
|
|
func TestPortRouter_RemoveFallbackCloses(t *testing.T) {
|
|
fbBackend := startEchoPlain(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return func(_ context.Context, network, address string) (net.Conn, error) {
|
|
return net.Dial(network, address)
|
|
}, nil
|
|
}
|
|
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, dialResolve)
|
|
router.SetFallback(Route{
|
|
Type: RouteTCP,
|
|
ServiceID: "svc-fb",
|
|
Target: fbBackend.Addr().String(),
|
|
})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// With fallback: plain TCP works.
|
|
conn1, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
_, err = conn1.Write([]byte("hello"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := conn1.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "hello", string(buf[:n]))
|
|
conn1.Close()
|
|
|
|
// Remove fallback.
|
|
router.RemoveFallback("svc-fb")
|
|
|
|
// Without fallback on a port router (no HTTP channel): connection should be closed.
|
|
conn2, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
defer conn2.Close()
|
|
_, _ = conn2.Write([]byte("bye"))
|
|
_ = conn2.SetReadDeadline(time.Now().Add(3 * time.Second))
|
|
_, err = conn2.Read(buf)
|
|
assert.Error(t, err, "without fallback, connection should be closed")
|
|
}
|
|
|
|
// TestPortRouter_HTTPToTLSTransition verifies that switching a service from
|
|
// HTTP-only to TLS-only via remove+add doesn't orphan the old HTTP route.
|
|
func TestPortRouter_HTTPToTLSTransition(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
tlsBackend := startEchoTLS(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Phase 1: HTTP-only. SNI connections go to HTTP channel.
|
|
router.AddRoute("app.example.com", Route{Type: RouteHTTP, AccountID: "acct-1", ServiceID: "svc-1"})
|
|
|
|
httpConn := router.HTTPListener()
|
|
connDone := make(chan struct{})
|
|
go func() {
|
|
defer close(connDone)
|
|
c, err := httpConn.Accept()
|
|
if err == nil {
|
|
c.Close()
|
|
}
|
|
}()
|
|
tlsConn, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "app.example.com", InsecureSkipVerify: true},
|
|
)
|
|
if err == nil {
|
|
tlsConn.Close()
|
|
}
|
|
select {
|
|
case <-connDone:
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("HTTP listener did not receive connection for HTTP-only route")
|
|
}
|
|
|
|
// Phase 2: Simulate update to TLS-only (removeMapping + addMapping).
|
|
router.RemoveRoute("app.example.com", "svc-1")
|
|
router.AddRoute("app.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "acct-1",
|
|
ServiceID: "svc-1",
|
|
Target: tlsBackend.Addr().String(),
|
|
})
|
|
|
|
tlsConn2, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "app.example.com", InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err, "TLS connection should succeed after HTTP→TLS transition")
|
|
defer tlsConn2.Close()
|
|
|
|
_, err = tlsConn2.Write([]byte("hello-tls"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn2.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "hello-tls", string(buf[:n]), "data should relay to TLS backend")
|
|
}
|
|
|
|
// TestPortRouter_TLSToHTTPTransition verifies that switching a service from
|
|
// TLS-only to HTTP-only via remove+add doesn't orphan the old TLS route.
|
|
func TestPortRouter_TLSToHTTPTransition(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
tlsBackend := startEchoTLS(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
router := NewRouter(logger, dialResolve, addr)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Phase 1: TLS-only. Route relays to backend.
|
|
router.AddRoute("app.example.com", Route{
|
|
Type: RouteTCP,
|
|
AccountID: "acct-1",
|
|
ServiceID: "svc-1",
|
|
Target: tlsBackend.Addr().String(),
|
|
})
|
|
|
|
tlsConn, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "app.example.com", InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err, "TLS relay should work before transition")
|
|
_, err = tlsConn.Write([]byte("tls-data"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "tls-data", string(buf[:n]))
|
|
tlsConn.Close()
|
|
|
|
// Phase 2: Simulate update to HTTP-only (removeMapping + addMapping).
|
|
router.RemoveRoute("app.example.com", "svc-1")
|
|
router.AddRoute("app.example.com", Route{Type: RouteHTTP, AccountID: "acct-1", ServiceID: "svc-1"})
|
|
|
|
// TLS connection should now go to the HTTP listener, NOT to the old TLS backend.
|
|
httpConn := router.HTTPListener()
|
|
connDone := make(chan struct{})
|
|
go func() {
|
|
defer close(connDone)
|
|
c, err := httpConn.Accept()
|
|
if err == nil {
|
|
c.Close()
|
|
}
|
|
}()
|
|
tlsConn2, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "app.example.com", InsecureSkipVerify: true},
|
|
)
|
|
if err == nil {
|
|
tlsConn2.Close()
|
|
}
|
|
select {
|
|
case <-connDone:
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("HTTP listener should receive connection after TLS→HTTP transition")
|
|
}
|
|
}
|
|
|
|
// TestPortRouter_MultiDomainSamePort verifies that two TLS services sharing
|
|
// the same port router are independently routable and removable.
|
|
func TestPortRouter_MultiDomainSamePort(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
backend1 := startEchoTLSMulti(t)
|
|
backend2 := startEchoTLSMulti(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
router := NewPortRouter(logger, dialResolve)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
router.AddRoute("svc1.example.com", Route{Type: RouteTCP, AccountID: "acct-1", ServiceID: "svc-1", Target: backend1.Addr().String()})
|
|
router.AddRoute("svc2.example.com", Route{Type: RouteTCP, AccountID: "acct-1", ServiceID: "svc-2", Target: backend2.Addr().String()})
|
|
assert.False(t, router.IsEmpty())
|
|
|
|
// Both domains route independently.
|
|
for _, tc := range []struct {
|
|
sni string
|
|
data string
|
|
}{
|
|
{"svc1.example.com", "hello-svc1"},
|
|
{"svc2.example.com", "hello-svc2"},
|
|
} {
|
|
conn, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: tc.sni, InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err, "dial %s", tc.sni)
|
|
_, err = conn.Write([]byte(tc.data))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := conn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, tc.data, string(buf[:n]))
|
|
conn.Close()
|
|
}
|
|
|
|
// Remove svc1. Router should NOT be empty (svc2 still present).
|
|
router.RemoveRoute("svc1.example.com", "svc-1")
|
|
assert.False(t, router.IsEmpty(), "router should not be empty with one route remaining")
|
|
|
|
// svc2 still works.
|
|
conn2, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "svc2.example.com", InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err)
|
|
_, err = conn2.Write([]byte("still-alive"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := conn2.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "still-alive", string(buf[:n]))
|
|
conn2.Close()
|
|
|
|
// Remove svc2. Router is now empty.
|
|
router.RemoveRoute("svc2.example.com", "svc-2")
|
|
assert.True(t, router.IsEmpty(), "router should be empty after removing all routes")
|
|
}
|
|
|
|
// TestPortRouter_SNIAndFallbackLifecycle verifies the full lifecycle of SNI
|
|
// routes and TCP fallback coexisting on the same port router, including the
|
|
// ordering of add/remove operations.
|
|
func TestPortRouter_SNIAndFallbackLifecycle(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
sniBackend := startEchoTLS(t)
|
|
fallbackBackend := startEchoPlain(t)
|
|
|
|
dialResolve := func(_ types.AccountID) (types.DialContextFunc, error) {
|
|
return (&net.Dialer{}).DialContext, nil
|
|
}
|
|
|
|
router := NewPortRouter(logger, dialResolve)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Step 1: Add fallback first (port mapping), then SNI route (TLS service).
|
|
router.SetFallback(Route{Type: RouteTCP, AccountID: "acct-1", ServiceID: "pm-1", Target: fallbackBackend.Addr().String()})
|
|
router.AddRoute("tls.example.com", Route{Type: RouteTCP, AccountID: "acct-1", ServiceID: "svc-1", Target: sniBackend.Addr().String()})
|
|
assert.False(t, router.IsEmpty())
|
|
|
|
// SNI traffic goes to TLS backend.
|
|
tlsConn, err := tls.DialWithDialer(
|
|
&net.Dialer{Timeout: 2 * time.Second},
|
|
"tcp", ln.Addr().String(),
|
|
&tls.Config{ServerName: "tls.example.com", InsecureSkipVerify: true},
|
|
)
|
|
require.NoError(t, err)
|
|
_, err = tlsConn.Write([]byte("sni-traffic"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 1024)
|
|
n, err := tlsConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "sni-traffic", string(buf[:n]))
|
|
tlsConn.Close()
|
|
|
|
// Plain TCP goes to fallback.
|
|
plainConn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
_, err = plainConn.Write([]byte("plain"))
|
|
require.NoError(t, err)
|
|
n, err = plainConn.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "plain", string(buf[:n]))
|
|
plainConn.Close()
|
|
|
|
// Step 2: Remove SNI route. Fallback still works, router not empty.
|
|
router.RemoveRoute("tls.example.com", "svc-1")
|
|
assert.False(t, router.IsEmpty(), "fallback still present")
|
|
|
|
plainConn2, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
require.NoError(t, err)
|
|
// Must send >= 5 bytes so the SNI peek completes immediately
|
|
// without waiting for the 5-second peek timeout.
|
|
_, err = plainConn2.Write([]byte("after"))
|
|
require.NoError(t, err)
|
|
n, err = plainConn2.Read(buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "after", string(buf[:n]))
|
|
plainConn2.Close()
|
|
|
|
// Step 3: Remove fallback. Router is now empty.
|
|
router.RemoveFallback("pm-1")
|
|
assert.True(t, router.IsEmpty())
|
|
}
|
|
|
|
// TestPortRouter_IsEmptyTransitions verifies IsEmpty reflects correct state
|
|
// through all add/remove operations.
|
|
func TestPortRouter_IsEmptyTransitions(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
router := NewPortRouter(logger, nil)
|
|
|
|
assert.True(t, router.IsEmpty(), "new router")
|
|
|
|
router.AddRoute("a.com", Route{Type: RouteTCP, ServiceID: "svc-a"})
|
|
assert.False(t, router.IsEmpty(), "after adding route")
|
|
|
|
router.SetFallback(Route{Type: RouteTCP, ServiceID: "svc-fb1"})
|
|
assert.False(t, router.IsEmpty(), "route + fallback")
|
|
|
|
router.RemoveRoute("a.com", "svc-a")
|
|
assert.False(t, router.IsEmpty(), "fallback only")
|
|
|
|
router.RemoveFallback("svc-fb1")
|
|
assert.True(t, router.IsEmpty(), "all removed")
|
|
|
|
// Reverse order: fallback first, then route.
|
|
router.SetFallback(Route{Type: RouteTCP, ServiceID: "svc-fb2"})
|
|
assert.False(t, router.IsEmpty())
|
|
|
|
router.AddRoute("b.com", Route{Type: RouteTCP, ServiceID: "svc-b"})
|
|
assert.False(t, router.IsEmpty())
|
|
|
|
router.RemoveFallback("svc-fb2")
|
|
assert.False(t, router.IsEmpty(), "route still present")
|
|
|
|
router.RemoveRoute("b.com", "svc-b")
|
|
assert.True(t, router.IsEmpty(), "fully empty again")
|
|
}
|
|
|
|
// startEchoTLSMulti starts a TLS echo server that accepts multiple connections.
|
|
func startEchoTLSMulti(t *testing.T) net.Listener {
|
|
t.Helper()
|
|
|
|
cert := generateSelfSignedCert(t)
|
|
ln, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
|
|
Certificates: []tls.Certificate{cert},
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { ln.Close() })
|
|
|
|
go func() {
|
|
for {
|
|
conn, err := ln.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
go func(c net.Conn) {
|
|
defer c.Close()
|
|
buf := make([]byte, 1024)
|
|
n, _ := c.Read(buf)
|
|
_, _ = c.Write(buf[:n])
|
|
}(conn)
|
|
}
|
|
}()
|
|
|
|
return ln
|
|
}
|
|
|
|
// startEchoPlain starts a plain TCP echo server that reads until newline
|
|
// or connection close, then echoes the received data.
|
|
func startEchoPlain(t *testing.T) net.Listener {
|
|
t.Helper()
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { ln.Close() })
|
|
|
|
go func() {
|
|
for {
|
|
conn, err := ln.Accept()
|
|
if err != nil {
|
|
return
|
|
}
|
|
go func(c net.Conn) {
|
|
defer c.Close()
|
|
// Set a read deadline so we don't block forever waiting for more data.
|
|
_ = c.SetReadDeadline(time.Now().Add(2 * time.Second))
|
|
buf := make([]byte, 1024)
|
|
n, _ := c.Read(buf)
|
|
_, _ = c.Write(buf[:n])
|
|
}(conn)
|
|
}
|
|
}()
|
|
|
|
return ln
|
|
}
|
|
|
|
// fakeAddr implements net.Addr with a custom string representation.
|
|
type fakeAddr string
|
|
|
|
func (f fakeAddr) Network() string { return "tcp" }
|
|
func (f fakeAddr) String() string { return string(f) }
|
|
|
|
// fakeConn is a minimal net.Conn with a controllable RemoteAddr.
|
|
type fakeConn struct {
|
|
net.Conn
|
|
remote net.Addr
|
|
}
|
|
|
|
func (f *fakeConn) RemoteAddr() net.Addr { return f.remote }
|
|
|
|
func TestCheckRestrictions_UnparseableAddress(t *testing.T) {
|
|
router := NewPortRouter(log.StandardLogger(), nil)
|
|
filter := restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}})
|
|
route := Route{Filter: filter}
|
|
|
|
conn := &fakeConn{remote: fakeAddr("not-an-ip")}
|
|
assert.NotEqual(t, restrict.Allow, router.checkRestrictions(conn, route), "unparsable address must be denied")
|
|
}
|
|
|
|
func TestCheckRestrictions_NilRemoteAddr(t *testing.T) {
|
|
router := NewPortRouter(log.StandardLogger(), nil)
|
|
filter := restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}})
|
|
route := Route{Filter: filter}
|
|
|
|
conn := &fakeConn{remote: nil}
|
|
assert.NotEqual(t, restrict.Allow, router.checkRestrictions(conn, route), "nil remote address must be denied")
|
|
}
|
|
|
|
func TestCheckRestrictions_AllowedAndDenied(t *testing.T) {
|
|
router := NewPortRouter(log.StandardLogger(), nil)
|
|
filter := restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}})
|
|
route := Route{Filter: filter}
|
|
|
|
allowed := &fakeConn{remote: &net.TCPAddr{IP: net.IPv4(10, 1, 2, 3), Port: 1234}}
|
|
assert.Equal(t, restrict.Allow, router.checkRestrictions(allowed, route), "10.1.2.3 in allowlist")
|
|
|
|
denied := &fakeConn{remote: &net.TCPAddr{IP: net.IPv4(192, 168, 1, 1), Port: 1234}}
|
|
assert.NotEqual(t, restrict.Allow, router.checkRestrictions(denied, route), "192.168.1.1 not in allowlist")
|
|
}
|
|
|
|
func TestCheckRestrictions_NilFilter(t *testing.T) {
|
|
router := NewPortRouter(log.StandardLogger(), nil)
|
|
route := Route{Filter: nil}
|
|
|
|
conn := &fakeConn{remote: fakeAddr("not-an-ip")}
|
|
assert.Equal(t, restrict.Allow, router.checkRestrictions(conn, route), "nil filter should allow everything")
|
|
}
|
|
|
|
func TestCheckRestrictions_IPv4MappedIPv6(t *testing.T) {
|
|
router := NewPortRouter(log.StandardLogger(), nil)
|
|
filter := restrict.ParseFilter(restrict.FilterConfig{AllowedCIDRs: []string{"10.0.0.0/8"}})
|
|
route := Route{Filter: filter}
|
|
|
|
// net.IPv4() returns a 16-byte v4-in-v6 representation internally.
|
|
// The restriction check must Unmap it to match the v4 CIDR.
|
|
conn := &fakeConn{remote: &net.TCPAddr{IP: net.IPv4(10, 1, 2, 3), Port: 5678}}
|
|
assert.Equal(t, restrict.Allow, router.checkRestrictions(conn, route), "v4-in-v6 TCPAddr must match v4 CIDR")
|
|
|
|
// Explicitly v4-mapped-v6 address string.
|
|
conn6 := &fakeConn{remote: fakeAddr("[::ffff:10.1.2.3]:5678")}
|
|
assert.Equal(t, restrict.Allow, router.checkRestrictions(conn6, route), "::ffff:10.1.2.3 must match v4 CIDR")
|
|
|
|
connOutside := &fakeConn{remote: fakeAddr("[::ffff:192.168.1.1]:5678")}
|
|
assert.NotEqual(t, restrict.Allow, router.checkRestrictions(connOutside, route), "::ffff:192.168.1.1 not in v4 CIDR")
|
|
}
|
|
|
|
// TestRouter_PlainHTTP_RoutesToPlainChannel verifies that a plain (non-TLS)
|
|
// connection lands on the plain HTTP channel when the router was built
|
|
// with WithPlainHTTP, leaving the TLS channel untouched.
|
|
func TestRouter_PlainHTTP_RoutesToPlainChannel(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
router := NewRouter(logger, nil, addr, WithPlainHTTP(addr))
|
|
router.AddRoute("example.com", Route{Type: RouteHTTP})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err, "test listener bind must succeed")
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() {
|
|
_ = router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// Plain HTTP request (no TLS handshake byte).
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
_, _ = conn.Write([]byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"))
|
|
}()
|
|
|
|
plainListener := router.HTTPListenerPlain()
|
|
require.NotNil(t, plainListener, "plain listener must be exposed when WithPlainHTTP is set")
|
|
|
|
acceptDone := make(chan net.Conn, 1)
|
|
go func() {
|
|
conn, err := plainListener.Accept()
|
|
if err == nil {
|
|
acceptDone <- conn
|
|
}
|
|
}()
|
|
|
|
tlsListener, ok := router.HTTPListener().(*chanListener)
|
|
require.True(t, ok, "router.HTTPListener() must be the test's chanListener; the test relies on observing its channel directly")
|
|
|
|
select {
|
|
case conn := <-acceptDone:
|
|
require.NotNil(t, conn)
|
|
_ = conn.Close()
|
|
case <-tlsListener.ch:
|
|
t.Fatal("plain HTTP request leaked into TLS channel")
|
|
case <-time.After(3 * time.Second):
|
|
t.Fatal("plain HTTP connection never reached plain channel")
|
|
}
|
|
}
|
|
|
|
// TestRouter_TLS_StaysOnTLSChannel_WhenPlainEnabled verifies that the
|
|
// presence of a plain channel does not divert TLS traffic — TLS still
|
|
// goes to the TLS channel as before.
|
|
func TestRouter_TLS_StaysOnTLSChannel_WhenPlainEnabled(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
|
|
router := NewRouter(logger, nil, addr, WithPlainHTTP(addr))
|
|
router.AddRoute("example.com", Route{Type: RouteHTTP})
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err, "test listener bind must succeed")
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
go func() { _ = router.Serve(ctx, ln) }()
|
|
|
|
// Send a TLS ClientHello.
|
|
go func() {
|
|
conn, err := net.DialTimeout("tcp", ln.Addr().String(), 2*time.Second)
|
|
if err != nil {
|
|
return
|
|
}
|
|
tlsConn := tls.Client(conn, &tls.Config{
|
|
ServerName: "example.com",
|
|
InsecureSkipVerify: true, //nolint:gosec
|
|
})
|
|
_ = tlsConn.Handshake()
|
|
_ = tlsConn.Close()
|
|
}()
|
|
|
|
select {
|
|
case conn := <-router.httpCh:
|
|
require.NotNil(t, conn, "TLS conn should land on the TLS channel")
|
|
_ = conn.Close()
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatal("TLS conn never reached the TLS channel")
|
|
}
|
|
}
|
|
|
|
// scriptedAcceptListener is a net.Listener whose Accept() returns
|
|
// pre-scripted errors. Used by the accept-loop exit tests to simulate
|
|
// the failure mode that triggers the tight-loop bug: a netstack
|
|
// listener whose endpoint has been destroyed and now returns the gVisor
|
|
// "endpoint is in invalid state" error from every Accept call.
|
|
type scriptedAcceptListener struct {
|
|
errs chan error
|
|
closed chan struct{}
|
|
}
|
|
|
|
func newScriptedAcceptListener(errs ...error) *scriptedAcceptListener {
|
|
s := &scriptedAcceptListener{
|
|
errs: make(chan error, len(errs)+1),
|
|
closed: make(chan struct{}),
|
|
}
|
|
for _, e := range errs {
|
|
s.errs <- e
|
|
}
|
|
return s
|
|
}
|
|
|
|
func (s *scriptedAcceptListener) Accept() (net.Conn, error) {
|
|
select {
|
|
case <-s.closed:
|
|
return nil, net.ErrClosed
|
|
case err := <-s.errs:
|
|
return nil, err
|
|
}
|
|
}
|
|
|
|
func (s *scriptedAcceptListener) Close() error {
|
|
select {
|
|
case <-s.closed:
|
|
default:
|
|
close(s.closed)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (s *scriptedAcceptListener) Addr() net.Addr {
|
|
return &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0}
|
|
}
|
|
|
|
// TestRouter_Serve_ExitsOnGVisorInvalidEndpoint is the regression guard
|
|
// for the tight-loop bug: when the underlying netstack endpoint is
|
|
// destroyed, Accept returns "endpoint is in invalid state" forever. The
|
|
// loop must recognise that signal and return, otherwise it pegs a CPU
|
|
// core and floods logs.
|
|
func TestRouter_Serve_ExitsOnGVisorInvalidEndpoint(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, nil, addr)
|
|
|
|
gvisorErr := &net.OpError{
|
|
Op: "accept",
|
|
Net: "tcp",
|
|
Addr: addr,
|
|
Err: errSentinel("endpoint is in invalid state"),
|
|
}
|
|
ln := newScriptedAcceptListener(gvisorErr)
|
|
defer ln.Close()
|
|
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
done <- router.Serve(context.Background(), ln)
|
|
}()
|
|
|
|
select {
|
|
case err := <-done:
|
|
assert.NoError(t, err, "Serve must return cleanly on a recognised closed-listener error")
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("Serve did not exit on gVisor 'endpoint is in invalid state' — accept loop is spinning")
|
|
}
|
|
}
|
|
|
|
// TestRouter_Serve_BacksOffOnTransientError verifies the defence-in-
|
|
// depth path: when Accept returns an unknown transient error, the loop
|
|
// MUST not spin. It backs off, then exits cleanly once ctx is cancelled.
|
|
// "Bounded call count" stands in for "no CPU spin" — without backoff
|
|
// the goroutine would issue thousands of Accept calls in this window.
|
|
func TestRouter_Serve_BacksOffOnTransientError(t *testing.T) {
|
|
logger := log.StandardLogger()
|
|
addr := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 443}
|
|
router := NewRouter(logger, nil, addr)
|
|
|
|
const transientErrCount = 5
|
|
errs := make([]error, transientErrCount)
|
|
for i := range errs {
|
|
errs[i] = errSentinel("transient: too many open files")
|
|
}
|
|
ln := newScriptedAcceptListener(errs...)
|
|
defer ln.Close()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
done := make(chan error, 1)
|
|
start := time.Now()
|
|
go func() {
|
|
done <- router.Serve(ctx, ln)
|
|
}()
|
|
|
|
// Cancel after enough time for the backoff to climb (5ms + 10ms +
|
|
// 20ms + 40ms = 75ms minimum), but short enough that a spinning
|
|
// loop would have made thousands of calls by now.
|
|
time.AfterFunc(150*time.Millisecond, cancel)
|
|
|
|
select {
|
|
case err := <-done:
|
|
assert.NoError(t, err, "Serve must return cleanly on ctx cancellation")
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("Serve did not exit on ctx cancellation — backoff or exit path broken")
|
|
}
|
|
|
|
// Without backoff the loop would burn through all 5 scripted errors
|
|
// in microseconds and then block on the channel. With backoff the
|
|
// total wall time should be at least 5ms (the first backoff).
|
|
elapsed := time.Since(start)
|
|
assert.GreaterOrEqual(t, elapsed, minAcceptDelay,
|
|
"loop ran without backing off — would burn CPU in production")
|
|
}
|
|
|
|
// errSentinel mirrors gVisor's tcpip error message exactly. We can't
|
|
// import the gVisor package without dragging in the whole netstack, so
|
|
// the test uses the canonical string the production error formatter
|
|
// emits — same shape IsClosedListenerErr matches in production.
|
|
type errSentinel string
|
|
|
|
func (e errSentinel) Error() string { return string(e) }
|
|
|