ADR-027: Run CI on self-hosted GitHub Actions runners on AKS, funded by Azure credits

Context

Aucert's GitHub organisation is on the Free plan, which includes 2,000 Actions minutes per month for private repositories. Measured consumption on 2026-07-25 was 234 workflow runs producing roughly 1.08 billable minutes per run, or about 7,600 minutes per month — the allowance is exhausted in roughly eight days. Current overage is about $35/month and rising.

The dollar figure understates the problem, because the measured demand is post-austerity. Three separate retreats have already been made to stay inside the allowance, all recorded in the workflow files themselves:

Retreat	Evidence
Backend Kotlin/Bazel suite removed from PRs, moved to a scheduled run	.github/workflows/nightly-health.yml — "Runs the EXPENSIVE backend suite on a SCHEDULE against main, instead of on every PR push, to cut CI spend." (Was daily; raised to 12-hourly on 2026-08-09 once the job moved to the self-hosted fleet and its marginal cost went to zero. The paired frontend job was deleted at the same time as a duplicate of ci.yml / frontend — the retreat recorded here still stands for the backend.)
tilt-ci full boot job restricted to nightly and manual	.github/workflows/tilt-ci.yml:6-9 — "a cold bazel backend build is far too costly to run on every PR (it's why the per-PR backend job was removed)"
Three docs workflows fully paused	.github/workflows/docs-{preview,public-deploy,internal-deploy}.yml:5-6 — "Paused 2026-05-08: auto-deploy disabled to conserve GitHub Actions minutes."

The forcing function is therefore a capacity ceiling, not a cash bill. Aucert today merges backend changes with no per-PR compile or test gate.

A second, compounding cause sits upstream of the runners. The repository has no Bazel cache of any kind: .bazelrc is twelve lines of Java toolchain and test strategy flags with no --remote_cache, --disk_cache, or --remote_executor, and deploy-platform.yml:87-90 invokes bazel-contrib/setup-bazel with no cache configuration. Every CI Bazel invocation builds from cold. This is why the backend gate was unaffordable in the first place.

Meanwhile Azure Founders Hub credits are available and currently fund AKS, two PostgreSQL Flexible Servers, Redis, and AI Foundry. The aucert-aks cluster, Container Registry, Key Vault, Workload Identity, and CSI secret-mounting patterns are all already provisioned and in production use (ADR-005, ADR-008).

Decision

Run Aucert CI on self-hosted GitHub Actions runners hosted on the existing aucert-aks cluster, using the official actions/actions-runner-controller (ARC) runner scale sets deployment mode, on a dedicated, tainted, scale-to-zero node pool — and pair the migration with a Bazel remote cache so that restored CI is fast as well as unmetered.

Three constraints are part of the decision, not implementation detail:

1. Ephemeral runners only. One pod per job, destroyed on completion. No persistent or reusable runners under any circumstance.
2. Deploy workflows stay on GitHub-hosted runners. deploy-platform.yml, deploy-astra.yml, and deploy-spec-agent.yml deploy into aucert-aks. Hosting them inside the cluster they mutate creates a circular dependency in which a bad deploy removes the ability to deploy the fix.
3. The migration is partial by design. Sub-minute lint and docs jobs remain on GitHub-hosted runners; ARC pod cold-start exceeds their runtime, and they fit inside the free allowance once the heavy jobs leave.

The full design, phasing, and node-pool sizing are specified in SPEC-079.

Alternatives considered

Option	Pros	Cons
ARC runner scale sets on AKS (chosen)	Officially supported by GitHub; ephemeral pods by default; scales to zero; reuses every existing pattern (Workload Identity, Key Vault CSI, ACR, helm_release); runs-on: is a one-line change per workflow; funded by credits, not cash	New node pool to operate; ARC controller is another component to upgrade; requires an org-level GitHub App
Upgrade the GitHub org plan (Team, $4/user/month → 3,000 minutes)	Zero engineering work; no new infrastructure	Raises the ceiling by 50% against demand that is 3-4x the current allowance once suppressed jobs return; spends cash while Azure credits sit unused; does nothing about cold Bazel builds
Azure VMs running the runner agent directly	Simplest possible setup; no Kubernetes involved	No autoscaling or scale-to-zero — pays 24/7 for burst workload; runners are long-lived, which is the dominant self-hosted security risk; separate patching and image lifecycle outside the existing AKS/Terraform surface
Legacy ARC (summerwind/actions-runner-controller)	Larger body of community documentation; more mature webhook autoscaler	Community-maintained, superseded by the official runner scale sets; requires webhook plumbing for scaling; ephemeral mode is opt-in rather than default
Bazel remote cache alone, keep GitHub-hosted runners	Much smaller change; directly attacks cold builds	Caching reduces minutes but does not remove the 2,000-minute ceiling; restored per-PR backend CI would still breach it
Do nothing	No work	Ceiling tightens as the team and merge rate grow; backend changes continue to merge with no per-PR gate

Consequences

What becomes easier

• Per-PR backend compile and test gating becomes affordable again, closing the gap where Kotlin changes merge without CI verification.
• The three paused docs workflows and the tilt-ci boot job can be restored.
• CI concurrency becomes a capacity decision (node pool max) rather than a billing decision.
• Runners sit inside the VNet, so ACR pulls, Key Vault reads, and any future in-cluster cache are local rather than egress traffic.
• Marginal CI cost moves from cash to Founders Hub credits.

What becomes harder

• Aucert now operates CI capacity. Node pool sizing, ARC controller upgrades, and runner image maintenance become ongoing responsibilities.
• CI availability becomes coupled to AKS availability for the migrated jobs. The deploy-workflow carve-out limits but does not eliminate this.
• Debugging shifts from a documented GitHub-hosted image to an image Aucert builds and pins; tool version drift becomes Aucert's problem.
• With min_count = 0, the first job of a burst waits for node provisioning (roughly 2-4 minutes) instead of starting immediately.

Risks

• Self-hosted runner compromise. Any job that executes untrusted code on a self-hosted runner can reach the cluster network. Mitigated by ephemeral pods, a private repository (no fork-PR execution), a dedicated namespace, and keeping the agent-driven workflows (claude.yml, claude-code-review.yml) on GitHub-hosted runners.
• Noisy-neighbour eviction. The system pool is a static 3 × Standard_D2s_v6 (6 vCPU total, no autoscaling) carrying backend, Kong, Temporal, ClickStack and Plane. A Bazel build scheduled there would evict platform pods. Mitigated by a dedicated pool with a NoSchedule taint; runners must never tolerate the system pool.
• Credit exhaustion. Founders Hub credits are finite and already committed to other services. CI burn is estimated at roughly $70/month at min_count = 0, but this must be tracked against the remaining balance and expiry date.
• Org-admin dependency. ARC authenticates via a GitHub App installed at the organisation level, which requires an org owner.
• ACR capacity. The registry is Basic SKU (10 GiB, acr.tf:10). Runner images plus increased push frequency may require a Standard upgrade.