Production readiness — reproducible-deploy audit

CLAUDE.md's hard rule is the bar this page measures against: the entire deployed state of any environment must be rebuildable from git plus a clean deploy with zero manual steps. This page enumerates every remaining gap between "a clean from-scratch helm deploy" and "a working environment" — the operator-set secrets, the one-time steps, the deploy-ordering constraints — so the team has one authoritative pre-production checklist.

This is a map, not a set of fixes. Every gap below is mapped to an owning Jira ticket; the fixes land under those tickets, not here. The one gap this audit found that was not already tracked was filed as SSO-2139.

How to read the "fail-loud vs silent" column

The single most important property of every operator-set value is what happens when it is unset on a fresh deploy. There are three shapes, and only the first is safe by construction:

• Fail-loud (required helper). The chart refuses to render — helm upgrade aborts with a message naming the value. An environment cannot ship without it. This is the SSO-1129 / SSO-1134 pattern and the gold standard.
• Auto-generate + lookup-preserve. The chart mints a random value at first install and re-uses the already-installed Secret on every subsequent upgrade via a Helm lookup. Safe and self-healing for a single, long-lived environment; not reproducible for disaster recovery or a fresh-namespace rebuild, where lookup finds nothing and mints a new value — leaving data encrypted under the old value undecryptable.
• Silent placeholder. The chart renders a git-known default (typically one that matches an immutable Flyway seed). The deploy is green and the pod starts, but the credential is world-readable in the repo. This is the class SSO-2101 / SSO-2137 exist to close.

The required fail-loud helpers (the safe set)

These five values already fail the render when unset — the correct posture. They are listed so a production overlay author knows exactly what a fresh deploy will demand up front:

Value	Template	Ticket origin
hub.publicBaseUrl	hub-deployment.yaml	SSO-1135
identity.publicBaseUrl	identity-deployment.yaml	SSO-1134
apple.publicBaseUrl	apple-deployment.yaml	SSO-832 / SSO-1138
productApi.federatedIdentityIssuer (wired into both hub and product-api)	hub-deployment.yaml, product-api-deployment.yaml	SSO-2019
per-service / global.image.tag	_helpers.tpl	SSO-799 / SSO-1459

Consequence for production: values-production.yaml supplies none of these today (see the gap table). A helm upgrade -f values-production.yaml therefore aborts at render on the first required helper. That is the fail-loud property working as designed — but it means the production overlay is incomplete, which is the SSO-1485 gap.

Gap table

Each row is a gap between a clean from-scratch deploy and a working environment. "Fail mode" is what happens on a fresh deploy when the value/step is missing.

Gap	What breaks if unset/missing	Fail mode	Owning ticket	Pre-cutover action
No production overlay / no production deploy pipeline. values-production.yaml is a near-empty placeholder; the only live deploy workflow is deploy-staging.yml targeting namespace thoryn.	A production deploy has no overlay supplying the required helpers, the backup bucket, or the imperatively-created secrets (thoryn-db-credentials, thoryn-smtp-credentials, thoryn-saml-credentials, ghcr-pull-secret) that deploy-staging.yml creates via kubectl create.	Fail-loud at render (missing required helpers) — but there is no pipeline to run at all.	SSO-1485 (P1)	Populate values-production.yaml with the full required-helper set + backup config; stand up a deploy-production.yml that mints the same repo-secret-backed Secrets the staging workflow does.
Category B: chart-generated per-service AES keys are outside the backup/DR set. thoryn-identity-tenant-smtp-credential-encryption-key (SMTP passwords), thoryn-identity-phone-encryption-key, thoryn-identity-oath-seed-encryption-key (TOTP seeds), thoryn-identity-hooks-secret-encryption-key, and thoryn-hub-pairwise-salt (pairwise sub stability).	A Postgres restore into a fresh namespace restores the ciphertext while lookup mints new keys — MFA phone numbers, OATH seeds, hook secrets, tenant BYO-SMTP passwords silently undecryptable; every pairwise sub moves (breaks R4).	Silent (lookup auto-generate). Green restore, broken data.	SSO-2129	Fold these Secrets into the backup set (or move the key class into OpenBao Transit, which is backed up). Until then: back them up out-of-band and restore before the first helm upgrade of a rebuilt env. See Backup and recovery.
Category C: infra client secrets default to a git-known placeholder that matches a {noop} Flyway seed. product-api-internal (V58), credential-issuer-admin (V62), synthetic monitor-client (V45).	Nothing breaks functionally — but the credential is world-readable in the repo. The issuer-admin and product-api-internal templates literally say "PRODUCTION MUST set a real secret".	Silent placeholder.	SSO-2101 (design), SSO-2137 (impl); SSO-741 (bcrypt sweep) done	Set a real random secret via productApi.credentials.* / synthetic.credentials.hubClientSecret AND land a fix-forward migration bcrypt-hashing the matching seed row.
Synthetic-monitor login password is operator-set out-of-band. synthetic.credentials.identityPassword is not wired from a repo secret in deploy-staging.yml; V45 seeds a matching hash.	On a from-scratch deploy the Secret falls to its fail-loud placeholder, so synthetic Flow A / Flow L good-creds legs get 401 — the SSO-1789 auth-bypass guard (Flow L) stops guarding, and the monitor is red every minute.	Fail-loud placeholder → runtime auth failure (not a deploy block).	SSO-2030	Thread a SYNTHETIC_MONITOR_IDENTITY_PASSWORD repo secret as --set-string synthetic.credentials.identityPassword — exactly the treatment SSO-2025 gave the Slack webhook.
Hub wildcard TLS cert depends on a manual one-time DNS step + an operator secret. The DNS-01 ClusterIssuer needs ACME_DNS_CREDENTIALS (repo secret, synced to the cert-manager namespace) AND a hand-created Hetzner _acme-challenge.hub.<env>... CNAME to <uuid>.auth.acme-dns.io.	If ACME_DNS_CREDENTIALS is unset the sync step silently skips; if the CNAME is absent the challenge is stuck; either way the wildcard cert never issues and every HTTPS host is down.	Silent skip (secret) + manual out-of-band DNS (CNAME).	SSO-2119 (High)	Provision ACME_DNS_CREDENTIALS and the CNAME before the cutover; record the CNAME as a documented one-time prerequisite (it cannot be chart-reproduced — it lives in the DNS zone).
Backup destination secrets are operator-created, not chart-owned. backup.enabled: true by default with s3Bucket: "" in base values.yaml; the CronJobs reference thoryn-backup-secret (encryption key + region) and thoryn-backup-aws-credentials (S3 creds ini blob) by name, both non-optional.	With an empty bucket or a missing Secret the backup pod fails its fail-loud guard / won't start — nightly Postgres + OpenBao backups silently never run.	Silent at deploy (pod fails at first scheduled run).	SSO-1485 + operator runbook backup-cronjob-security.adoc	Set backup.s3Bucket in the production overlay and mint both Secrets before the first backup window; verify the first run in S3.
OpenBao unseal key is an operator-readable Kubernetes Secret. thoryn-openbao-keys is generated by the openbao-init Job's operator init and read by the per-pod unseal sidecar.	Loss of the OpenBao node and this Secret is permanent Transit-key loss — the ciphertext/audit-signing keys become unrecoverable even with a data-dir backup.	Auto-generate at init; self-healing across restarts; not in the backup set by default.	Documented later phase in ADR 2026-06-07-vault-to-openbao.md (KMS/HSM auto-unseal); backup handling in SSO-2129 neighbourhood	Include thoryn-openbao-keys in the cluster Secret/etcd backup; plan the KMS/HSM auto-unseal seal for production rather than promoting the operator-readable-key model.
SSA field-ownership auto-reclaim omits namespaced Role/RoleBinding/PersistentVolumeClaim. RECLAIM_KINDS in deploy-staging.yml lists 9 kinds but not these three (OpenBao k8s-auth RBAC + OpenBao PVC).	An emergency kubectl edit/patch on the OpenBao RBAC (exactly what the 2026-06-07 cascade did) scars SSA ownership; the next helm upgrade wedges on a conflict the auto-reclaim does not cover.	Silent until the next deploy, then a hard helm upgrade failure.	SSO-2139 (newly filed)	Add role rolebinding persistentvolumeclaim to RECLAIM_KINDS.
Redis default user is still on nopass ~* &* +@all. SSO-1120 shipped BFF-only ACLs; hub / wallet-broker / identity / product-api still connect unrestricted as default.	Not a deploy blocker, but a production posture gap: a compromised consumer has full keyspace access.	Silent (works, but unrestricted).	Epic SSO-1481 (posture punch-list)	Per-service ACLs for the remaining consumers before prod, swept together (tightening default is a multi-PR change per CLAUDE.md).

Migration / deploy-ordering constraints a prod cutover must respect

These are not "unset value" gaps — they are ordering hazards where two correct changes break if applied in the wrong sequence. A production cutover (especially the cross-repo hub then thoryn-web sequence) must honour them.

Constraint	Why	Reference
Hub scope-grant migration must deploy before the console/BFF/CLI requests that scope. The V82…V101 family (roles, idp, security-policy, migration, fga, hooks, orgs, users.import, events.read, geo-policy) each UPDATE a customer-plane client's scopes. If thoryn-web requests a scope the hub has not yet granted, sign-in loops on invalid_scope.	oathy (hub) and thoryn-web (BFF/console/CLI) are separate repos with separate deploys — the hub migration must land first.	CLAUDE.md; the V82…V101 migrations under servers/authorization-hub/.../db/migration/
productApi.federatedIdentityIssuer must be byte-for-byte identical between the hub Path C headless-resolve issuer and the product-api V55 federated_identity seed.	A mismatch changes seeded users' sub at subject resolution and locks them out. Both are wired from the same required helper — do not diverge them.	SSO-2019 (the required helper wires both hub-deployment.yaml and product-api-deployment.yaml)
A new product-api migration placeholder must be mirrored in the api-gateway route-alignment tests. api-gateway tests boot product-api's Flyway migrations; an unmirrored placeholder reddens module-tests main-wide.	Cross-module test coupling, not a runtime deploy order — but it blocks the merge that would ship the migration.	SSO-2021
Category C: a real infra client secret and its bcrypt-hash migration must co-deploy. Overriding product-api-internal / credential-issuer-admin / monitor-client secret via --set only authenticates if a fix-forward migration also updates the seeded row's hash.	The immutable {noop} seed is the only value that authenticates until a forward migration rewrites it.	SSO-741 / SSO-2101 / SSO-2137
OpenBao init before Vault-dependent services are gated ready. The openbao-init post-upgrade hook provisions Transit keys + k8s-auth; services with a vaultTransit readiness indicator gate on it, which is why the staging helm upgrade omits --wait.	A --wait deploy would deadlock on pods that cannot be ready until the hook they follow has run.	deploy-staging.yml (the --wait-omission comment); backup-and-recovery OpenBao section

kubectl-scar coverage (SSA field-ownership)

The Reclaim SSA field-ownership step in deploy-staging.yml self-heals emergency-kubectl scars for a fixed RECLAIM_KINDS list. Verified coverage against every kind the chart renders:

• Covered (namespaced, in RECLAIM_KINDS): Deployment, StatefulSet, ConfigMap, Service, ServiceAccount, Secret, CronJob, Job, Ingress, NetworkPolicy.
• Not covered, but documented as manual-recipe cases in CLAUDE.md: cluster-scoped ClusterRoleBinding and ClusterIssuer; absent-CRD PrometheusRule and Certificate.
• Not covered and not previously flagged — the gap SSO-2139 tracks: namespaced core Role, RoleBinding (OpenBao k8s-auth RBAC), and PersistentVolumeClaim (OpenBao file storage). These are exactly the resources the 2026-06-07 OpenBao cascade patched by hand.

Operator pre-flight checklist

Run this before the first production cutover. Items are ordered so a from-scratch deploy converges.

1. Populate the production overlay (values-production.yaml, SSO-1485): set every required helper (hub.publicBaseUrl, identity.publicBaseUrl, apple.publicBaseUrl, productApi.federatedIdentityIssuer), backup.s3Bucket, and any per-env host values.
2. Mint the imperatively-created Secrets the staging workflow creates via kubectl create: thoryn-db-credentials, thoryn-smtp-credentials, thoryn-saml-credentials, ghcr-pull-secret — from a production deploy pipeline, not by hand (SSO-1485).
3. Set the Category C infra client secrets to real values and land the matching bcrypt fix-forward migrations (SSO-2101 / SSO-2137): productApi.credentials.hubInternalClientSecret, productApi.credentials.credentialIssuerAdminClientSecret, synthetic.credentials.hubClientSecret.
4. Wire the synthetic-monitor login password as a repo-secret --set-string (synthetic.credentials.identityPassword, SSO-2030) so Flow A / Flow L authenticate.
5. Wire the alert + at-rest secrets: SYNTHETIC_MONITOR_SLACK_WEBHOOK (SSO-2025, done for staging) and — only if pinning a known key for DR — IDENTITY_TENANT_SMTP_CREDENTIAL_ENCRYPTION_KEY (SSO-2090). Leave the SMTP key unset for a first install so the chart generates it.
6. Provision TLS: set ACME_DNS_CREDENTIALS and create the Hetzner _acme-challenge CNAME before the cutover (SSO-2119). The CNAME is a documented one-time DNS prerequisite.
7. Back up the un-backed-up secret material: include thoryn-openbao-keys and the five Category B AES-key Secrets in the cluster Secret/etcd backup, and rehearse restoring them before the first helm upgrade of a rebuilt environment (SSO-2129).
8. Mint the backup-destination Secrets (thoryn-backup-secret, thoryn-backup-aws-credentials) and verify the first nightly Postgres + OpenBao backups land in S3 (SSO-1485, runbook backup-cronjob-security.adoc).
9. Close the posture gaps tracked under SSO-1481 that this audit does not re-derive: observability/paging, per-service Redis ACLs, HA/PDB/anti-affinity.
10. Add the three namespaced kinds to RECLAIM_KINDS (SSO-2139) so an emergency OpenBao-RBAC patch does not silently wedge the next deploy.

Dual-delivery note

For the managed SaaS, Thoryn SRE owns every item above. For self-managed customers on the same artefacts, items 1–2, 6, and 8 (the overlay, the ingress DNS/TLS, and the backup destination) are the customer's shared-responsibility surface; the chart's fail-loud helpers are what make a mis-provisioned self-managed install abort at render rather than ship a broken environment.

Verification boundary

What this audit verified directly (read the chart, workflows, runbooks, and ADRs on origin/main at the time of review):

• The five required fail-loud helpers, and that values-production.yaml supplies none of them.
• The Category B / Category C secret templates and their default-resolution chains.
• That SYNTHETIC_MONITOR_IDENTITY_PASSWORD is not wired in deploy-staging.yml (only the Slack webhook and the SMTP key are threaded there).
• That RECLAIM_KINDS omits Role, RoleBinding, and PersistentVolumeClaim, and which templates render them.
• The ACME_DNS_CREDENTIALS silent-skip and the manual Hetzner CNAME prerequisite text.
• The status of every referenced ticket in Jira.

What this audit inferred rather than executed (a read-only audit does not deploy): it did not run a from-scratch production helm install, drive a DR restore, or exercise a cutover. The ordering constraints and fail modes are read from the chart/workflow/runbook text and the CLAUDE.md pitfalls that recorded each incident, not observed live. The owning tickets carry the validation work; per the CLAUDE.md "validate against the real system" rule, none of these gaps should be reported closed until the exact failing operation has been observed to succeed on the real target.