ADR-029: Relay ingress topology — a dedicated host, internal in dev, public front door deferred

Context

Deploying the Yard Relay (SPEC-080) — the standalone WSS byte-tunnel broker that a customer's Mac (aucert agent) dials into and an in-cluster rover-worker connects through — forces a decision on how the relay is exposed at the ingress. This is separable from the relay's internals; it is about the edge.

The relay's traffic profile is unlike the rest of the platform and drives the decision:

• Long-lived, binary, streaming. The agent's control channel stays open for hours; data pipes stream raw adb bytes with idle gaps. This is the opposite of the JSON request/response the API host (dev-api.aucert.ai) serves.
• Its own auth. The relay authenticates agents at the WSS upgrade with an RS256 JWT of audience yard-relay, minted by the Yard control plane — a different credential family from the user-JWT Kong verifies for /v1 API calls (kong-plugins.yaml already records this distinction).
• Stateful. If the relay ever scales past one replica, an agent's control channel and its pipes must land on the same pod (session affinity).

Two constraints fix the environment side:

• The dev cluster has no public front door. The only ingress LB is kong-dev-kong-proxy with azure-load-balancer-internal: true at private IP 10.0.0.62; there is no public LoadBalancer. Developers reach dev-api.aucert.ai over the VNet path (VPN), and that is accepted for the team's own dev/testing.
• The agent already models a distinct relay origin. RelayClientConfig.DEFAULT_RELAY_ORIGIN is wss://relay.aucert.ai — a host separate from api.aucert.ai, overridable via AUCERT_RELAY_URL. The client design already assumes the relay is its own host.

Decision

1. Expose the relay on a dedicated host relay.<env>.aucert.ai (dev: dev-relay.aucert.ai) via its own Kong ingress — WebSocket upgrade on, proxy_read/send_timeout in the hours, response buffering off, and none of the API host's auth/rate-limit/body plugins. Not a path on the API host.
2. In dev, place that host on the existing internal Kong LB. It is reachable from a developer's Mac over the VPN. Dev gets no public front door.
3. Defer the production public front door — a public ingress for the agent-facing surfaces (the relay WSS + /agent/v1/* enrollment/refresh) — to a later, explicit decision made when external customers onboard. That is where the internet-exposure concerns (DDoS surface, WAF, abuse controls, a public TLS edge) belong; dev neither needs nor should take them on now.

Alternatives considered

Option	Pros	Cons
Dedicated host, internal in dev (chosen)	Clean Kong plugin/timeout/affinity isolation for a long-lived binary WSS hub; separate security blast radius from the API; dev/prod parity with the agent's modeled relayOrigin; team can test over VPN today; the prod front-door decision is cleanly isolated and deferrable	A new host + DNS record + cert coverage to manage; a public front door is still owed later (explicitly parked)
Path on dev-api.aucert.ai/relay/v1/*	No new DNS/TLS; one host to reason about	Inherits the API host's plugins (rate-limit/body/auth/buffering — all wrong for a binary WSS pipe) unless every one is overridden per-route; entangles relay timeouts with the API on one ingress; awkward stateful-affinity upstream; shared security blast radius; diverges from prod (wss://relay.aucert.ai is a distinct host) — "works on dev, breaks on prod"
Public front door for dev now	External-customer-shaped testing (real NAT, public TLS edge) immediately	Exposes the dev cluster to the internet — a real security surface (DDoS, abuse, WAF) — for no dev benefit; premature before the prod exposure model is designed

Consequences

What becomes easier

• The relay ingress is tuned for long-lived WSS (timeouts, buffering, no hostile plugins) without touching the API host's latency-sensitive config.
• Dev/prod parity: the same topology and the same AUCERT_RELAY_URL shape on both — prod cutover is a DNS + a public-front-door decision, not a re-architecture.
• The team can exercise the full device loop over VPN now (SPEC-080 validation).
• The prod public-exposure decision is isolated behind this ADR and can be taken deliberately, with a security review, when it is actually needed.

What becomes harder

• One more host (DNS + cert coverage) to provision and monitor.
• A production public front door is now a known, explicit piece of owed work — this ADR parks it rather than solving it.

Risks

• Dev-internal masks external edge cases. Testing only over VPN won't surface real-world NAT traversal, public-TLS-edge behaviour, or latency until the prod front door lands — a gap to re-validate then, not now.
• The deferred front door is a dependency for GA, not optional: no external customer can use the product until it exists. Tracking it as owed work (not "done") is the honest posture.
• If the relay later needs HA, session affinity must be designed before scaling past one replica (out of scope here; noted in SPEC-080).

Amendment (2026-08-02) — dev edge is a Cloudflare Tunnel, not internal-LB-over-VPN

Decision point 2 above assumed developers reach dev-relay.aucert.ai over "the VNet path (VPN)". Building it revealed that path does not exist: the dev product hosts (api.dev, console.dev, relay.dev) resolved nowhere — there is no VPN into the VNet, and the internal Kong LB (10.0.0.62) is unreachable from a developer's Mac. Dev "access" was in fact a local Caddy + /etc/hosts + kubectl port-forward improvisation. So the edge had to be built for real. Two options:

1. Public Azure LB → Kong (flip the Kong service to a public IP).
2. Cloudflare Tunnel (cloudflared dials Cloudflare's edge outbound; no public IP).

Chosen: Cloudflare Tunnel (aucert-dev-edge), on a security-architecture basis:

• Attack surface. The tunnel keeps zero inbound exposure — no public IP, no open ports; the origin (relay, Kong, backend) has no internet-reachable address. A public LB is standing 24/7 exposure that's scannable and must be locked to Cloudflare IP ranges (an allow-list that drifts and fails open). For a relay that brokers customer device bytes, eliminating any direct path to the origin is the right control.
• Non-bypassable edge controls. With the tunnel, Cloudflare WAF/DDoS/rate-limit sit in front and cannot be bypassed (there is no origin IP to hit around them). The public-LB model's classic failure — origin-IP leak → WAF bypass — is structurally impossible here.
• Posture consistency. Preserves the established "no public IPs on Azure" control; the public-LB option would regress it.

Revisit only if compliance bars routing through Cloudflare, or if the relay's long-lived binary WSS proves throughput-bound at scale — and even then the answer is a hardened public LB (mTLS origin auth + NSG allow-list + Cloudflare origin-pull), not a naked one.

Consequences of the amendment:

• Cloudflare Access is bypassed for api.dev + relay.dev: a headless agent cannot pass interactive OAuth, so app-layer auth is the sole perimeter — Kong jwt-validation for the API, the relay's agent-JWT for the relay. Those must stay tight and monitored.
• TLS gap: Cloudflare Universal SSL covers only one label (aucert.ai + *.aucert.ai), so the two-level *.dev.aucert.ai hosts get no edge cert (handshake_failure). Serving them requires Advanced Certificate Manager / Total TLS for a *.dev.aucert.ai cert (or renaming to one-level hosts). Owed work before this edge is usable end-to-end.
• The tunnel credential lives in Key Vault (yard-edge-tunnel-credentials), read via CSI + the cloudflared-edge UAMI (edge-tunnel-identity.tf) — no hand-managed K8s Secret.
• Manifests: k8s/platform/edge-tunnel/ (tunnel + SA + SecretProviderClass). The tunnel routes dev-relay straight to the relay service and dev-api through Kong. Because nothing reaches the relay via Kong anymore, decision point 1's dedicated relay Kong ingress is retired (the KongIngress relay-proxy + its Ingress and the Service's konghq.com/override annotation were removed): the relay wanted none of Kong's plugins, so with the tunnel going direct there is nothing left for that ingress to do. The relay's long-WSS-timeout requirement now lives at the tunnel, not Kong.
• The "production public front door" (decision point 3) is now partially realized in dev via this tunnel; the GA exposure/abuse-control review is still owed.