ADR-023: Durable Workflow Engine — Temporal, wrapped by Orbit
Context
The product has crossed the threshold where multiple concerns need durable execution — work that survives a process restart, suspends for minutes-to-days, and resumes deterministically:
- Agentic human-in-the-loop (SPEC-049 / SPEC-050). An agent returns
NeedsInput(a credential, an OTP, a confirmation) and must park until a human supplies it, then resume its reasoning where it left off. - Domain orchestration. Job scheduling (reservation auto-release, failed-webhook retry, subscription period-end), cross-module sagas (Payment → Entitlement → Subscription), and long flows (a Rover scan campaign).
Today the only durable flow — Rover's scan (ADR-022) — hand-wires Temporal directly in the domain, with no shared runtime, no CallPrincipal propagation across the durable boundary, and no reusable pattern.
We are introducing Orbit (SPEC-050) as a durable-workflow backbone and need to fix the engine underneath it.
Constraints / forcing functions. (1) Greenfield decision — the existing internal Temporal deployment and the internal "atlas" agents are not decision inputs (SPEC-049); we choose on merits. (2) Isolation matters — domains must keep clean, non-durable cores and delegate durability, so the engine must sit behind a wrapper (Orbit) and never leak types into domain code (ADR-020). (3) On-prem / self-managed packaging is a real future requirement — the backbone must ship as a self-contained deployable. (4) Reuse existing infra — we run PostgreSQL (product flexible server) already; a new stateful datastore per engine is a cost. (5) Maturity on the JVM — the backend is Kotlin.
Decision
Use Temporal as the durable-workflow engine, wrapped by Orbit.
- Orbit owns the Temporal client, worker runtime, a
ContextPropagatorthat carriesCallPrincipal(incl.tenantId) across the durable boundary, an OTel interceptor, and per-worker DI. Domain code depends on Orbit's ports, never on Temporal types (same reversibility discipline as Tower and the storage engines). - Temporal runs on the existing product PostgreSQL (
temporal+temporal_visibilitydatabases, already provisioned) — no new datastore. - The whole backbone ships as a single deployable (Temporal server + Orbit workers, replicated for HA) — a docker image with its database configured, which makes on-prem a packaging concern, not an architecture one.
The engine choice is deliberately reversible: because domains touch only Orbit's ports, a future swap is an Orbit-internal change.
Alternatives considered
| Option | Pros | Cons |
|---|---|---|
| Temporal (chosen), wrapped by Orbit | Mature JVM SDK; strong workflow-as-code + durable timers/signals (ideal for HITL park/resume); runs on our existing Postgres; battle-tested isolation of workflow vs activity workers; single-image packaging for on-prem | Operationally heavier (a stateful service + 2 DBs); workflow determinism constraints; another system to run |
| DBOS (Postgres-native durable execution) | Lightweight; durability in Postgres; minimal infra | Durability is ingrained in application code — every domain service must understand workflows, defeating the "clean core, delegate to a central backbone" isolation goal; younger ecosystem; JVM story weaker |
| Restate | Modern, low-latency durable execution; nice programming model | Younger; smaller JVM/Kotlin maturity; another new datastore/runtime to operate; less proven at HITL-day-scale durability |
| Cadence | Temporal's predecessor, similar model | Smaller ecosystem/momentum than Temporal; no compelling advantage given Temporal exists |
| Inngest / hosted workflow SaaS | No infra to run | SaaS dependency conflicts with on-prem/self-managed requirement; data-egress + tenancy concerns for a QA product handling customer app data |
| Postgres-native (durable-jobs table + transactional outbox), no engine | Simplest; no new infra; fine for single-step async jobs | Does not model multi-step orchestration, durable HITL suspend/resume, or sagas without reinventing a workflow engine; the agentic HITL loop is exactly what it can't express cleanly |
The decisive axis was isolation + maturity: Temporal keeps durability out of domain cores (behind Orbit) while being the most mature JVM option, and the two objections that usually count against it — new datastore and on-prem weight — are neutralized by running on existing Postgres and single-image packaging.
Consequences
What becomes easier
- Durable HITL (agent park/resume) is a first-class engine primitive (durable timers + signals), not a hand-rolled state machine.
- Any domain gets durable orchestration by adding a
workflow/module against Orbit's ports (SPEC-050 §E). - Tenant identity + trace survive the durable boundary via the
ContextPropagator. - The engine choice is reversible — domains never see Temporal types.
- No new datastore; on-prem is a packaging exercise (single image + Postgres).
What becomes harder
- An operational surface to run and monitor (Temporal server, 2 databases, worker fleet).
- Workflow code must respect determinism constraints (no wall-clock/random in workflow bodies; side effects in activities).
- Every activity must be idempotent; side-effecting device activities need explicit state checks (SPEC-050 §F).
Risks
- Operational maturity on-prem — self-managed customers inherit running Temporal and its two databases: schema upgrades, backup/restore, and monitoring of a stateful service. This is an explicit product + ops commitment, not free. Mitigated by single-image packaging and reuse of the product Postgres; a managed control-plane option (e.g. Temporal Cloud) should be offered for customers who will not operate it. The Orbit wrapper's reversibility caps the downside if the operational cost proves too high.
ContextPropagatoris load-bearing for isolation — a dropped or spoofed principal across the durable boundary is a tenant-isolation defect (SPEC-040). Must be tested as rigorously as the gRPC principal interceptor.- Lock-in creep — if Temporal types leak past Orbit, reversibility erodes. Enforced by a grep-clean check that no domain core module imports Temporal.
References
- Specs: SPEC-050 (Orbit — the wrapper this ADR sits under), SPEC-049 (agent framework — Mission/HITL that needs durability), SPEC-048 (Input Resolver — HITL resolution)
- ADRs: ADR-022 (rover-scan-worker standalone process — the existing hand-wired Temporal use Orbit generalizes), ADR-020 (generated contract types at domain boundaries — the reversibility discipline), ADR-011 (agent-harness strategy — internal, not load-bearing here)