How to provision a tenant's BYOK

BYOK ("bring your own key") runs a tenant's inference on their Azure OpenAI resource, billed to their account, instead of Aucert's. The runtime is built; provisioning is deliberately manual — there is no self-serve flow, no API, no console screen. Work through this page once per tenant.

What you are actually setting up

Two credentials, in two custodies. Keeping them straight is the whole job:

Credential	Who holds it	Who presents it	Where it lives
Provider key — the customer's Azure OpenAI key	LiteLLM	LiteLLM → Azure	Key Vault, referenced from the engine's model row
Virtual key — a LiteLLM key scoped to this tenant's groups	Tower	Tower → LiteLLM	Key Vault, referenced from tower.tenant_engine_binding

Tower never holds the provider key. If Tower is compromised, what leaks is a revocable virtual key, not the customer's Azure credential. Preserve that property; nothing below should tempt you to put the provider key anywhere Tower can read it.

Before you start

• The engine must be on litellm ≥ 1.96. Earlier versions send a Key Vault reference to Azure verbatim as the API key. You get a provider 401 and nothing explains why. Check with cat infra/docker/litellm/upstream.pin.
• Collect from the customer, not just a key:
  • the Azure OpenAI endpoint (api_base) and api-version
  • a deployment name per tier they want served — vision, judgment, reasoning, synthesis, reporting. Their deployment names are theirs; do not assume they match ours (gpt-5.4, Kimi-K2.6).
  • their per-token prices, input and output, for each of those deployments. Skip this and their spend records as zero — see step 3.
• Embeddings are NOT covered by BYOK. Do not register a …-embedding group; nothing would ever request it. Rover's embedder calls LiteLLM directly rather than through Tower, using the process-wide ROVER_EMBEDDING_MODEL (aucert-embedding) and LITELLM_SERVICE_KEY — both fixed at pod startup, neither aware of a tenant. So a BYOK tenant's embedding calls still run on Aucert's account and are billed to us. That is a known gap, not a provisioning mistake; say so if a customer asks why their Azure bill shows no embedding usage.
• The vault is fixed. LiteLLM builds one SecretClient from AZURE_KEY_VAULT_URI, so the provider key must live in the vault the engine already points at (dev: aucertdev-kv-41e0x5). Tower reads its virtual key from the same vault. Per-tenant vault isolation is not available; isolation is by secret name, and litellm's identity holds Key Vault Secrets User at vault scope, so it can read any secret there by design.

Names you will need

Everything derives from the tenant id, deterministically — CredentialProfiles.forTenant computes exactly this, so hand-computed names and what Tower sends cannot drift:

tenant_id            tnt_9z2mw6qr8j1mw3x1
credential profile   tenant-9z2mw6qr8j1mw3x1      (tnt_ → tenant-)
model groups         tenant-9z2mw6qr8j1mw3x1-vision, …-judgment, …
KV: provider key     byok-tenant-9z2mw6qr8j1mw3x1-provider
KV: virtual key      byok-tenant-9z2mw6qr8j1mw3x1-virtual-key

The tnt_ → tenant- rewrite is not cosmetic: Azure Key Vault secret names permit only alphanumerics and dashes, so a raw tnt_… id cannot name a secret.

1. Create the binding as PENDING

Do this first, and leave it PENDING. Tower routes a tenant to its own account only on ACTIVE, so nothing changes for the tenant until the final step — which is what lets you provision at leisure and cut over in one move.

INSERT INTO tower.tenant_engine_binding
  (binding_id, tenant_id, engine, credential_profile, status, provider,
   created_at, created_by, updated_at, updated_by)
VALUES
  (gen_random_uuid(), 'tnt_9z2mw6qr8j1mw3x1', 'litellm', 'TENANT_BYOK', 'PENDING', 'azure',
   now(), '<your actor uuid>', now(), '<your actor uuid>');

2. Store the customer's provider key

Use --file, not --value — --value puts the customer's key in your shell history.

az keyvault secret set --vault-name aucertdev-kv-41e0x5 --name byok-tenant-9z2mw6qr8j1mw3x1-provider --file ./key.txt

Then delete the local file.

3. Register one model per tier

MASTER_KEY=$(az keyvault secret show --vault-name aucertdev-kv-41e0x5 --name litellm-master-key --query value -o tsv)

kubectl -n litellm port-forward svc/litellm 4000:4000

Repeat per tier, substituting the tenant's deployment name:

curl -sS -X POST http://localhost:4000/model/new -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' -d '{"model_name":"tenant-9z2mw6qr8j1mw3x1-vision","litellm_params":{"model":"azure/THEIR-DEPLOYMENT","api_base":"https://theirs.openai.azure.com/","api_version":"2024-12-01-preview","api_key":"os.environ/byok-tenant-9z2mw6qr8j1mw3x1-provider","input_cost_per_token":0.000004,"output_cost_per_token":0.000016}}'

api_key is the reference, never the key. LiteLLM stores it encrypted, and resolves it from Key Vault when it adds the deployment to its router.

The two cost fields are not optional. LiteLLM derives response_cost from its own price table, keyed by model name — and it has no entry for a customer's private deployment, so without these it reports nothing and every one of that tenant's ledger rows records engine_cost_micros = 0. The call still succeeds, which is what makes this easy to miss: you find out when someone asks what a scan cost and the answer is zero. Use the customer's own per-token prices; this figure is what you would show them as their spend, so it should match what their provider actually bills.

4. Mint the tenant's virtual key

Scope it to exactly this tenant's groups. That scoping is what makes the key an isolation boundary rather than a formality.

curl -sS -X POST http://localhost:4000/key/generate -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' -d '{"models":["tenant-9z2mw6qr8j1mw3x1-*"],"key_alias":"byok-tenant-9z2mw6qr8j1mw3x1"}'

The prefix pattern is deliberate, and it is still a hard boundary: it matches this tenant's groups and nothing else — not aucert-*, not another tenant's. Enumerating instead would mean every later tier needs the key updated as well as the model registered, and a forgotten update is a 403 that surfaces mid-scan rather than at provisioning time. Verified working on this engine (1.90+): a key scoped ["<prefix>-*"] resolves concrete group names at request time.

Store the returned key, then record its handle on the binding:

az keyvault secret set --vault-name aucertdev-kv-41e0x5 --name byok-tenant-9z2mw6qr8j1mw3x1-virtual-key --file ./vk.txt

UPDATE tower.tenant_engine_binding
   SET engine_credential_ref = 'byok-tenant-9z2mw6qr8j1mw3x1-virtual-key',
       secret_ref            = 'byok-tenant-9z2mw6qr8j1mw3x1-provider',
       updated_at = now(), updated_by = '<your actor uuid>'
 WHERE tenant_id = 'tnt_9z2mw6qr8j1mw3x1' AND engine = 'litellm';

engine_credential_ref is the one Tower reads. secret_ref is recorded for offboarding only — Tower never reads it, and it never enters the data-plane model.

5. Validate before cutting over

A wrong deployment name is an Azure 404 on first use. Find it here, not in a customer's scan.

curl -sS -X POST http://localhost:4000/v1/chat/completions -H "Authorization: Bearer <the virtual key>" -H 'Content-Type: application/json' -d '{"model":"tenant-9z2mw6qr8j1mw3x1-vision","messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

Run it once per registered group, with the tenant's virtual key rather than the master key — that also proves the allowlist is right.

• 403 key not allowed to access model → the group is missing from the key's models list.
• Provider 401 → the reference did not resolve. Wrong secret name, or the engine is pre-1.96.
• Azure 404 → wrong deployment name or api_base.

6. Cut over

UPDATE tower.tenant_engine_binding SET status = 'ACTIVE', updated_at = now(), updated_by = '<uuid>'
 WHERE tenant_id = 'tnt_9z2mw6qr8j1mw3x1' AND engine = 'litellm';

Takes effect within the resolver's refresh interval (60s). Confirm on the next scan:

SELECT credential_profile, count(*) FROM tower.llm_usage_ledger
 WHERE tenant_id = 'tnt_9z2mw6qr8j1mw3x1' AND created_at > now() - interval '1 hour'
 GROUP BY 1;

It should read tenant-9z2mw6qr8j1mw3x1. If it still says aucert, the binding is not ACTIVE, the kind is not TENANT_BYOK, or the pod has not refreshed yet.

To roll back, set status back to PENDING. Traffic returns to the platform account within 60s.

Adding a tier later

One step: register the model as in step 3, with its prices. The virtual key needs no change, because step 4 scoped it to the tenant's whole prefix — that is what the pattern buys.

If you inherited a key minted with an enumerated list instead, either add the new group to it or, better, switch it to the pattern once:

curl -sS -X POST http://localhost:4000/key/update -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' -d '{"key":"<the virtual key>","models":["tenant-9z2mw6qr8j1mw3x1-*"]}'

/key/update replaces the models array rather than merging into it. Capture the existing value with /key/info before you overwrite it.

Revocation and offboarding

Revocation is not instant. Tower caches the resolved credential for 60s, so calls keep succeeding for up to a minute after you revoke. That is expected — do not conclude the revoke failed.

To offboard, in this order:

1. status = 'SUSPENDED' on the binding — traffic returns to the platform account within 60s.
2. POST /key/delete for the virtual key.
3. POST /model/delete for each registered model.
4. Delete both Key Vault secrets.

Reversing steps 1 and 2 gives a window where Tower still presents a deleted key and every call fails 401 — which reads like an outage rather than a deprovision.

What happens when provisioning is incomplete

Tower fails the call rather than falling back to the platform account. A binding that is TENANT_BYOK + ACTIVE but has no engine_credential_ref, or whose vault secret is missing, raises ByokCredentialUnavailable.

This is the opposite of how the rest of Tower degrades, and deliberately so: an unresolvable tier is a degraded answer, but an unresolvable credential is a billing error. Falling back would run the customer's scan on Aucert's Foundry account, bill us for it, and look entirely successful.