ADR-022: Rover scan-execution worker runs as a standalone process, not embedded in the monolith

Context

The coupling problem

RoverWorkerRegistrar.start() is currently invoked from Application.module() in //backend/app (Application.kt ≈ line 105) — the same JVM process as the Ktor HTTP API and all 14 platform domain modules (organization, identity, subscription, metering, payment, catalog, onboarding, entitlement, hangar, validationgraph, generation, execution, analysis, pipeline). This was a deliberate POC shortcut to prove the Temporal scaffolding before committing to a separate process.

The coupling creates three concrete problems:

1. Boot-time overhead. The worker pod must initialise every domain module, Koin graph entry, and Flyway migration set that the HTTP API needs, even though scan-execution activities require only the rover module, the validation-graph module, the database pool, and the Temporal client. A cold start that should take ~3 s takes ~15 s today.

2. Independent scaling is impossible. Scan workloads are bursty and asynchronous; API request load is steady and latency-sensitive. With a shared process the only scaling axis is "add a replica of everything" — overpaying for API capacity when scan throughput is the bottleneck, and vice versa.

3. Netty version conflict. Ktor (HTTP) and the Temporal gRPC client both manage a Netty EventLoopGroup. Ktor 2.x ships io.netty:netty-*:4.1.x; the Temporal Java SDK 1.25+ requires io.netty:netty-*:4.2.x. Running both in the same classpath caused the backend to stop serving HTTP responses on main (tracked, root-cause confirmed). Pinning one version breaks the other; the real fix is to stop sharing the process.

The standalone-worker precedent

The internal spec-agent (atlas) already follows the correct pattern: internal/backend's SpecAgentWorker.kt is a standalone main() function with no Ktor dependency. It runs as its own container (aucertacr41e0x5.azurecr.io/spec-agent:latest) under its own K8s Deployment (infra/k8s/internal-platform/spec-agent-worker/deployment.yaml), connected to spec-agent-queue with no inbound Service — exec-based liveness/readiness probes (kill -0 1), 120 s termination grace to drain in-flight activities, and Azure Workload Identity for secret access. RoverTemporalClient.kt's KDoc states explicitly that Rover should mirror this pattern.

SPEC-038 alignment

SPEC-038 §D-F2 specifies that scan jobs run on the existing Temporal deployment (ADR-011) as durable workflows: acquire-device → acquire-account-lease → checkpointed-crawl-segments → release → trigger-enrichment. The spec makes no assumption that the worker is co-located with the HTTP API.

Related open question

Whether v1 Rover orchestration keeps Temporal (gRPC-based durable workflows, already scaffolded) or is simplified to a REST-triggered + DB-state runner (lower operational cost for the dev cluster, no Temporal pods) is still under discussion (see open questions below). That choice determines whether the standalone process is a Temporal worker or a lightweight runner — but it does not change this decision: scan execution is a standalone process regardless of which orchestration mechanism is chosen.

Decision

Rover scan execution runs as a standalone JVM process, separate from //backend/app.

Binary and image

• A new Bazel kt_jvm_binary target //backend/rover/workflow:worker provides the main() entry point (mirroring SpecAgentWorker.kt's structure).
• A dedicated oci_image rule produces aucert-rover-worker (distinct from aucert-backend).
• The //backend/app kt_jvm_binary and oci_image (aucert-backend) continue unchanged, serving only the Ktor HTTP API. RoverWorkerRegistrar.start() is removed from Application.module().

Koin graph (minimal)

The worker process loads a minimal Koin graph:

• roverModule — scan workflows, activities, registrar
• validationGraphModule — static analysis used by scan activities
• Database connection pool (Hikari → aucert-pg)
• RoverTemporalClient (Temporal WorkflowClient)

Explicitly excluded: Ktor, every non-rover domain module, the authentication stack, the routing layer, the observability HTTP endpoint, metering, payment, identity, subscription. There is no io.netty 4.1 dependency in the worker image.

Kubernetes deployment

The Rover worker pod mirrors spec-agent-worker/deployment.yaml:

• Lives under a new manifest at infra/k8s/charts/aucert-rover-worker/ (or infra/k8s/platform/rover-worker/).
• No inbound Service resource — Temporal workers connect outbound only.
• Exec-based liveness/readiness probes (kill -0 1), consistent with the spec-agent pattern.
• Termination grace period ≥ 120 s to allow in-flight Temporal activities to complete.
• Azure Workload Identity for Key Vault secret access (same mechanism as spec-agent-worker).
• Replica count and resource requests/limits sized independently from aucert-backend.

API-side residual

//backend/app retains only what is needed to trigger scans:

• RoverTemporalClient (read-only: starts workflows, queries run status).
• RoverRestAdapter (HTTP surface: POST /rover/scans, GET /rover/scans/{id}, etc.).
• No RoverWorkerRegistrar — that lives exclusively in //backend/rover/workflow:worker.

Alternatives considered

Option	Pros	Cons
Standalone process (this ADR)	Solves Netty conflict, enables independent scaling, lean image, blast-radius isolation	Adds one K8s Deployment and one OCI image to maintain
Keep worker in //backend/app (status quo)	Zero infra change	Netty conflict is structural not fixable by pinning; scaling couples API and scan load; boot time grows with every new domain module
Separate Gradle subproject, still same binary	Cleaner module graph without deployment change	Doesn't solve Netty conflict or scaling; just moves the coupling one layer down
Extract to a fully independent microservice (now)	Maximum isolation	Premature — ADR-003 monolith-first policy; Temporal already provides durable-workflow isolation without inter-service HTTP

Consequences

What becomes easier

• Independent scaling. The Rover worker HPA/KEDA can respond to Temporal rover-scan-queue backlog depth; the API deployment scales on HTTP RPS. They no longer move in lockstep.
• Lean image. The worker image carries no Ktor, no domain modules other than rover and validationgraph, and no io.netty 4.1 — smaller image, faster cold start, smaller attack surface.
• Blast-radius isolation. A crash in a scan activity (OOM, device driver panic) kills the worker pod, not the HTTP API. API availability is decoupled from scan execution health.
• Netty conflict resolved for the worker. The worker image uses only Temporal's io.netty 4.2; Ktor's io.netty 4.1 is absent. The API image (aucert-backend) has the inverse: Temporal's gRPC client is absent, only RoverTemporalClient (a lightweight stub) for workflow dispatch.
• Precedent is already production. The spec-agent worker uses the identical pattern; ops runbooks, monitoring dashboards, and Workload Identity wiring are already proven.

What becomes harder

• Two images to build and push per release. CI pipelines and deployment scripts must handle aucert-backend and aucert-rover-worker independently. Versioning discipline (same git SHA → same image tag) must be enforced.
• Migration from the inline POC. RoverWorkerRegistrar.start() must be removed from Application.module(), a new //backend/rover/workflow:worker binary wired up, and the K8s manifests created. Until the migration is complete and deployed, the Netty conflict and the scaling constraint persist.
• Local development. Running the full system locally now requires two JVM processes (or one combined docker compose profile). docker-compose.dev.yaml should add a rover-worker service.

Risks

• Temporal dependency still in //backend/app. RoverTemporalClient keeps a gRPC Temporal stub in the API process for workflow dispatch. This is a thin client (no worker, no activity registration) and uses only WorkflowClient.newWorkflowStub() — but the io.netty version it pulls in must be verified against Ktor's netty transport. If the conflict persists on the API side, the fix is to wrap workflow dispatch in a thin REST call to the worker (pushing the gRPC dependency entirely out of //backend/app), or to switch the Temporal client to HTTP transport. This is tracked as an open question.

Open questions

1. Temporal vs. REST-triggered runner for v1. Whether scan orchestration in v1 uses Temporal (gRPC durable workflows, currently scaffolded under RoverWorkflow / RoverWorkerRegistrar) or a simpler REST-triggered + DB-state runner (no Temporal pods on the dev cluster) is still under discussion. This ADR records the "standalone process" boundary — the orchestration mechanism choice is a dependency of this ADR: whichever mechanism is chosen, the worker runs as its own process outside //backend/app. The Netty conflict analysis above assumes Temporal is retained; if the decision is a DB-state runner, the Netty issue becomes moot but the standalone deployment shape and the scaling arguments still apply.

2. RoverTemporalClient in //backend/app — Netty on the API side. If Temporal is retained, confirm whether the lightweight WorkflowClient (dispatch-only, no worker) in the API process pulls in io.netty 4.2 and whether that conflicts with Ktor's io.netty 4.1. If it does, the options are: (a) Temporal HTTP transport (beta), (b) a thin HTTP dispatch endpoint on the worker (the API calls the worker's probe port), or (c) accept the conflict with a forced resolution until the Ktor or Temporal SDK upgrades converge.

3. K8s manifest location. Should the Rover worker manifest live under infra/k8s/internal-platform/ (alongside spec-agent-worker, since Rover is a platform worker) or under infra/k8s/charts/aucert-rover-worker/ (as a Helm chart for parity with aucert-backend)? The spec-agent pattern (plain YAML under internal-platform/) is simpler for a v1; the Helm chart pattern is more consistent with the backend deployment. Resolve before the first staging deploy.

References

• ADR-003 — monolith-first backend (this ADR is consistent; standalone worker ≠ service extraction)
• ADR-011 — Temporal as the agent/workflow harness
• SPEC-038 — Rover app exploration agent (D-F2: Temporal scan orchestration)
• internal/backend/src/main/kotlin/dev/aucert/internal/agents/spec/temporal/SpecAgentWorker.kt — precedent: standalone Temporal worker main()
• infra/k8s/internal-platform/spec-agent-worker/deployment.yaml — precedent: K8s Deployment for a standalone Temporal worker
• backend/rover/src/main/kotlin/ai/aucert/rover/api/RoverTemporalClient.kt — KDoc: "mirror SpecAgentWorker"
• backend/app/src/main/kotlin/ai/aucert/app/Application.kt — current inline RoverWorkerRegistrar.start() (to be removed)