Secrets management

Aucert keeps every persistent infrastructure secret in Azure Key Vault (aucertdev-kv-41e0x5) as the source of truth. Pods receive credentials by way of Kubernetes Secrets that are mirrored from Key Vault by idempotent shell scripts. Application-layer agent platform tokens (GitHub PATs, Slack bots, etc.) live in the Astra Token Vault — a Postgres-backed AES-256-GCM store distinct from Key Vault. No secret is ever committed to git, code, or Terraform values.

This page documents the current state as of 2026-04-29, the known gaps, and the migration to Azure Workload Identity that closes them.

Layers

Layer	Backing store	What lives here	How it lands on a pod
Cloud	Azure Key Vault aucertdev-kv-41e0x5 (RBAC mode, soft-delete 7d, no purge protection in dev)	DB passwords, encryption master keys, Foundry/Redis/Storage keys, Cloudflare token	Mirrored into K8s Secrets by tools/scripts/setup-*.sh; scripts are idempotent and re-runnable
Cluster	Kubernetes Secret objects in internal-platform, temporal namespaces	Per-service env-shaped credentials (JDBC URLs split per database, encryption key, Cloudflare credentials)	Pod consumes via envFrom: secretRef or explicit secretKeyRef
Application	astra_db.agent_tokens table (AES-256-GCM, master key from astra-secrets)	Agent platform tokens — GitHub PAT, Slack bot, Google OAuth, AWS Bedrock keys	Astra backend serves them through the Token Vault API to other agents
CI/CD	GitHub Actions Secrets	Service Principal credentials, deploy tokens	${{ secrets.* }} in workflow files only

Current pattern — shell-script-bridged K8s Secrets

The dominant pattern today is: Terraform generates a password (random_password) → writes it to Key Vault (azurerm_key_vault_secret) → an idempotent shell script reads from KV at deploy time and creates a K8s Secret via kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -. The pod consumes the K8s Secret as env vars.

Trade-offs:

• Source of truth is Key Vault. Scripts are stateless mirrors.
• Tracked in IaC? Partially. KV writes are Terraform; the K8s Secret values are not — they exist only as the in-cluster object. There is no kubernetes_secret Terraform resource and no GitOps reconciler.
• Re-creating the cluster requires re-running each setup-*.sh script in the right order. Bootstrap is manual.
• At-rest encryption. etcd is not configured for KMS-backed encryption in this cluster, so K8s Secrets are base64-encoded blobs. RBAC restricts read access; treat them as secret-by-policy, not secret-by-cryptography.

This pattern is being deprecated in favor of Azure Workload Identity for new services and Postgres credentials. See Migration to Workload Identity below.

Service inventory

Workload	Manifest	Cred path	IaC tracked	Status
astra-backend	infra/k8s/internal-platform/astra/backend.yaml	astra-db-credentials + astra-secrets (K8s Secret) + llm-config (ConfigMap)	Partial (KV via TF; K8s Secret via script)	Working
dispatcher	infra/k8s/internal-platform/dispatcher/deployment.yaml	dispatcher-secrets + dispatcher-config ConfigMap	Partial	Working
db-migrate (product)	infra/k8s/jobs/db-migrate/job.yaml	db-ddl-credentials-product (K8s Secret created by CI)	Partial	Working
db-migrate (internal)	infra/k8s/jobs/db-migrate/job.yaml	db-ddl-credentials-internal (K8s Secret created by CI)	Partial	Working
cloudflared	infra/k8s/internal-platform/cloudflared/tunnel.yaml	cloudflared-tunnel-credentials (K8s Secret) + tunnel ID in ConfigMap	Partial	Working
temporal-server	Helm chart	temporal-postgres-credentials	Partial	Working
spec-agent-worker	infra/k8s/internal-platform/spec-agent-worker/deployment.yaml	KV SDK direct via KeyVaultSecretProvider (DefaultAzureCredentialBuilder)	KV via TF; no UAMI/FIC for the pod	Broken — CredentialUnavailableException on every activity. Tracked in .context/drift/drift-2026-04-29-spec-agent-keyvault-auth.md
astra-frontend, astra-proxy, plane-proxy	various	None — only public ConfigMap values (URLs, model names)	Yes	Working

Verify deployed shape: kubectl get deploy,job -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)"'

Key Vault — secret inventory

Secret name	Written by	Consumers
internal-pg-admin-user	infra/terraform/foundation/database.tf	setup-astra-secrets.sh, setup-temporal-secrets.sh, KeyVaultSecretProvider (spec-agent)
internal-pg-admin-password	infra/terraform/foundation/database.tf (Terraform random_password)	Same
internal-pg-connection-string	Same	Convenience only — services build per-DB URLs
pg-admin-password	infra/terraform/environments/dev/database.tf	Product backend (future)
pg-connection-string	Same	Product backend (future)
agent-tokens-encryption-key	Astra deploy bootstrap	astra-secrets K8s Secret (master key for AES-256-GCM token encryption)
foundry-api-key, foundry-endpoint	infra/terraform/foundation/foundry.tf	Backend pods (LLM adapter, primary region)
foundry-east-api-key, foundry-east-endpoint	infra/terraform/foundation/foundry-east.tf	Backend pods (GPT-5.4-Pro routing, eastus2)
redis-primary-key, redis-connection-string	infra/terraform/environments/dev/redis.tf	Product backend
storage-key	infra/terraform/foundation/storage.tf	Backup tooling
cloudflare-tunnel-token	Manual / Terraform	cloudflared-tunnel-credentials K8s Secret

Verify: az keyvault secret list --vault-name aucertdev-kv-41e0x5 --query "[].name" -o tsv | sort

Kubernetes Secrets — actual deployed mirrors

K8s Secret	Namespace	Source	Consumers
astra-db-credentials	internal-platform	setup-astra-secrets.sh reads internal-pg-admin-user/password from KV; emits INTERNAL_SHARED_DB_URL/USER/PASSWORD, ASTRA_DB_URL/USER/PASSWORD	astra-backend
astra-secrets	internal-platform	setup-astra-secrets.sh; emits ENCRYPTION_MASTER_KEY (from KV agent-tokens-encryption-key), CF_AUDIENCE, DISPATCHER_SERVICE_TOKEN	astra-backend
dispatcher-secrets	internal-platform	deploy-dispatcher.sh	dispatcher
temporal-postgres-credentials	temporal	setup-temporal-secrets.sh reads internal-pg-admin-password from KV	temporal-server
cloudflared-tunnel-credentials	internal-platform	Manual / Cloudflare Tunnel bootstrap	cloudflared
db-ddl-credentials-product, db-ddl-credentials-internal	internal-platform (job-scoped)	Created by .github/workflows/deploy-astra.yml migration step	db-migrate jobs

Verify: kubectl get secret -A --field-selector type=Opaque -o name

Astra Token Vault — application-layer secrets

Distinct from Key Vault. The Token Vault is a Postgres-backed store inside astra_db.agent_tokens, encrypted with AES-256-GCM using a master key sourced from the astra-secrets K8s Secret (which mirrors KV agent-tokens-encryption-key). It holds agent platform tokens, not infrastructure credentials.

Token	Use	Notes
bedrock-access-key, bedrock-secret-key	AWS Bedrock — coding agents	Static IAM keys; tracked in .context/drift/drift-2026-04-25-aws-static-credentials.md
github-app-private-key, github-app-id	Spec agent → GitHub MR creation
slack-bot-token	Spec agent → Slack updates
google-sa-key	Atlas → Workspace integration	Superseded by ADR-014 OAuth-as-real-user
plane-api-key	Spec agent → Plane (issue tracking)

The master key must never rotate — every existing encrypted token would become permanently undecryptable. See PostgreSQL configuration → Rotating credentials for the infrastructure-credential rotation runbook.

Local development

az login        # Interactive — Azure permissions are user-scoped
az keyvault secret show --vault-name aucertdev-kv-41e0x5 --name <secret>  # ad-hoc reads

Developers never have shared secrets on disk. The Astra backend in local mode runs against a Docker-Compose Postgres with a static dev password (aucert_dev) — see PostgreSQL configuration → Local development.

Migration to Workload Identity

The cluster has workload_identity_enabled = true and OIDC issuer enabled at the AKS level (infra/terraform/foundation/aks.tf). What's missing per workload is the User Assigned Managed Identity, the federated credential, the service account annotation, and the pod label.

Verify cluster-level: az aks show -g aucert-foundation-rg -n aucert-aks --query oidcIssuerProfile

Plan, rolling — one workload at a time:

1. spec-agent-worker (unblocks the live bug in drift-2026-04-29-spec-agent-keyvault-auth.md). New UAMI + federated credential + KV Secrets User role + service account + deployment patch. App code already uses DefaultAzureCredentialBuilder, no app changes required.
2. astra-backend — same wiring; replace setup-astra-secrets.sh with a thin DB-credential fetch via the SDK on startup, or keep the script for DB credentials and use WI only for KV-only paths (Foundry key).
3. dispatcher, temporal, cloudflared — convert in the same pattern; retire the bridge scripts.
4. Once all five mirror scripts are retired, delete tools/scripts/setup-astra-secrets.sh, setup-temporal-secrets.sh, deploy-dispatcher.sh. KV writes stay in Terraform.

Until step 1 ships, every Temporal workflow on spec-agent-queue fails at activity start with retry exhaustion.

Hard rules

1. Never store secrets in code, Terraform values, or version control. Terraform may generate a password (random_password) and write it to Key Vault — that's fine. The plaintext value never lives in *.tfvars or commit history.
2. Never commit .env files. .gitignore blocks them.
3. Never log secrets — even at DEBUG. KeyVaultSecretProvider returns String; never include the value in error messages or stack traces.
4. Always use Key Vault references, not literal values, in any new K8s Secret or Terraform resource.
5. Always rotate compromised credentials immediately — see Rotating credentials.
6. Never rotate the AES master key. Rotating it bricks every encrypted token in agent_tokens.

Troubleshooting

Pod logs SecretNotFoundException with a chained CredentialUnavailableException

The visible "Secret '...' not found" is a wrapper. Look at the chained cause — if it says "WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured", the pod is missing one or more of:

• serviceAccountName: set to a SA annotated with azure.workload.identity/client-id: <UAMI client_id>
• Pod template label azure.workload.identity/use: "true"
• An azurerm_federated_identity_credential whose subject matches system:serviceaccount:<ns>:<sa-name>
• Role assignment Key Vault Secrets User on the UAMI

This is exactly the spec-agent-worker bug captured in .context/drift/drift-2026-04-29-spec-agent-keyvault-auth.md.

403 Forbidden when reading from Key Vault

az role assignment list \
  --scope $(az keyvault show -n aucertdev-kv-41e0x5 --query id -o tsv) \
  --query "[].{principalName:principalName,role:roleDefinitionName}" -o table

The principal must have Key Vault Secrets User (or higher). The KV is RBAC-mode (no access policies).

K8s Secret value drifted from KV after a Terraform rotation

The mirror scripts run on demand, not on every Terraform apply. After terraform taint on a random_password and re-apply, you must re-run the relevant setup-*.sh script and kubectl rollout restart every consuming Deployment. Full steps in PostgreSQL configuration → Rotating credentials.

What's next

• PostgreSQL configuration — Database credential lifecycle and rotation
• Cloudflare setup — Tunnel credentials and Cloudflare Access
• Azure topology — Where Key Vault and AKS sit in the wider VNet
• .context/drift/drift-2026-04-29-spec-agent-keyvault-auth.md — Active drift item: spec-agent-worker WI wiring
• .context/drift/drift-2026-04-25-aws-static-credentials.md — Bedrock static IAM keys (separate hardening item)