Production go-live cutover runbook

This is the ordered, gated sequence for taking the Thoryn Identity Platform to production — the phases, the go/no-go gates between them, and the rollback path out of each. It is the orchestration layer over the individual production-readiness artefacts; it references them and does not restate them:

• Production readiness — reproducible-deploy audit — the authoritative gap map (every operator-set secret, one-time step, and deploy-ordering constraint between a clean helm deploy and a working environment).
• Test mode: trustTestIssuers flip-readiness assessment — the evidence-based go/no-go for the sandbox/test-mode flip (Phase 3 below).
• Deploy, Upgrade, and Backup and recovery — the per-topic runbooks each phase leans on.
• The two in-flight design ADRs — production monitoring + alerting (SSO-1482) and the production deploy pipeline + overlay (SSO-1485) — plus the sandbox-mode ADR (adrs/2026-07-21-per-tenant-test-sandbox-mode.md). See the verification boundary for their merge status.

Everything here sits under epic SSO-1481 (production readiness — staging-to-prod hardening).

This runbook is a map and a gate discipline, not a green light. As of the lastReviewed date the honest verdict is NO-GO: most Phase 0 pre-flight gates are not green — the monitoring stack is uninstalled, there is no production pipeline, there is no HA, the security guards do not gate merge, and the test-mode flip sweep is incomplete. Two gates are already closed (TLS issuance SSO-2119, at-rest-key backup coverage SSO-2129). The point of the page is to make what is left legible at a glance and to keep the cutover honest.

The one bar every phase is measured against

Per CLAUDE.md, a phase is "done" only when a clean, from-scratch helm deploy reproduces the working state, with zero manual steps — not when the live cluster has been patched into shape. kubectl edit / patch / apply against live resources, console clicks, and hand-rolled secrets are not how changes are made; they leave no reproducible trail. And a gate is only truly green when the exact operation it guards has been observed to succeed on the real target (the "validate against the real system" rule) — "it renders", "CI is green", and "the PR merged" are not validation.

Current blocking gates (at a glance)

Status is the Jira state as of lastReviewed. Ready means the gate's work is Done; every other row is work still to land before it can be checked.

#	Gate	Tracking ticket	Status	Phase
G1	High availability — Postgres + OpenBao are single-instance; no PDB / anti-affinity / HPA; PVCs on local-path	SSO-1486 (P1)	Not ready — Backlog (no ADR/PR yet)	0
G2	Monitoring + alerting live so outages page (stack is uninstalled today)	SSO-1482 (P0)	Not ready — Backlog (design ADR only)	0
G3	Security CI guards gate merge (SSRF / tenant-isolation / audit-auth / secret guards run but do not block)	SSO-2162 (High)	Not ready — Backlog	0
G4	Secrets custody — unseal key + at-rest AES keys in the backup/DR set, restore-ordering documented	SSO-2129	Ready — Done	0
G5	Staging TLS / cert-manager DNS-01 issuance fixed + validated	SSO-2119 (High)	Ready — Done	0 / 4
G6	Production overlay + production deploy pipeline (human-gated)	SSO-1485 (P1)	Not ready — Backlog (scaffold designed)	1
G7	Category C infra client secrets are real + prod-vs-dev seed gating	SSO-2101 / SSO-2137	Not ready — Backlog (design)	1
G8	Synthetic-monitor password reproducible from a scratch deploy	SSO-2030	Not ready — Backlog	1
G9	SSA-reclaim covers namespaced Role / RoleBinding / PVC	SSO-2139	Not ready — Backlog	1
G10	Deploy-workflow imperative mutations back-filled to declarative config	SSO-2166 / SSO-2167 / SSO-2168 / SSO-2169	Not ready — Backlog	1
G11	trustTestIssuers flip — gated property + cross-mode E2E + PO go/no-go	SSO-2146 (blocked by SSO-2148/2157/…; SSO-2149 resolved)	Not ready — Backlog (NO-GO)	3
G12	Post-cutover synthetic + smoke green	SSO-2025	In Progress	5

How to read the phases

Each phase has entry gates (what must be true to start), steps, and exit / verification gates (what must be true to proceed). A gate is a checkbox with its tracking ticket; an unchecked box against a Backlog ticket is filed-but-unbuilt work. The phases are ordered — Phase n+1 must not start until Phase n's exit gate is green — with one deliberate exception: Phase 3 (the flip) is independent of the cutover and happens after the platform is live and stable.

Phase 0 — Pre-flight readiness gates (blocking)

Entry gate: none — this is where a cutover begins.

Every box below must be green before any production cutover starts. This is the phase that says "not yet"; production has no "we'll patch it live."

• G1 — High availability in place. Postgres and OpenBao are no longer single-instance; PodDisruptionBudget, podAntiAffinity / topologySpreadConstraints, and HPA are set; PVCs are off single-node local-path storage. Tracked by SSO-1486 (Backlog) and sequenced as slice S6 of the deploy-pipeline design ADR. There is no ADR and no PR for this yet — it is the least-built of the Phase 0 gates.
• G2 — Monitoring + alerting live so outages page. The monitoring stack (kube-prometheus-stack) was uninstalled 2026-04-26 after it OOM'd the k3s control-plane node; metrics are produced at /actuator/prometheus but nothing scrapes them, no rules evaluate, and no human is paged. The design (SSO-1482, a design ADR only) sequences the install-as-code (SSO-2170), scrape wiring (SSO-2171), the PrometheusRule paging pack (SSO-2172), Alertmanager routing (SSO-2173), Grafana-as-code (SSO-2174), and the synthetic-monitor alert integration (SSO-2175). Until it lands, the only floor is the secret-less synthetic-monitor watchdog (SSO-2025). This is epic SSO-1481's P0 gap.
• G3 — Security CI guards gate merge. The SSRF, tenant-isolation, audit-auth, and secret guards run on every PR but are not in main's branch-protection required_status_checks, so a red guard does not block a merge. Promote them (SSO-2162, High) before prod.
• G4 — Secrets custody + backup coverage. OpenBao holds signing keys via Transit with auto-unseal. The chart-generated at-rest AES keys and the thoryn-openbao-keys unseal Secret are folded into the backup/DR set (SSO-2129, Done), and the restore ordering is documented in Backup and recovery. Residual: the unseal key is still an operator-readable Kubernetes Secret; KMS/HSM auto-unseal for production is a later phase of ADR adrs/2026-06-07-vault-to-openbao.md.
• G5 — Staging TLS / cert-manager validated. The DNS-01 wildcard-cert issuance path that sat broken for 30 days is fixed and validated (SSO-2119, Done). Note the alerting on cert expiry still depends on G2 (the monitoring pack's TLSCertExpiry alert cross-checks the served cert, because cert-manager's own Ready condition lied during the incident).
• No unresolved main-red. Required checks are green; known pre-existing reds on the non-required module-tests slice (tracked under SSO-1945) are triaged and understood, not carried in as unknowns.

Exit gate: every box above checked. If any is red — today, G1/G2/G3 and the flip sweep are — the verdict is NO-GO and the cutover does not proceed.

Phase 1 — Environment provisioning (from-scratch reproducibility)

Entry gate: Phase 0 green. Production cluster capacity available; the production domain/apex confirmed with the platform owner (an open question in the deploy-pipeline ADR — the hub.thoryn.org family is proposed, not final).

The bar is the reproducibility rule: provision a cluster, then deploy the whole platform from git with zero manual steps.

Steps

1. Provision the production cluster. A separate production cluster (not a shared namespace on staging), namespace thoryn. The single self-hosted Hetzner runner is a deploy SPOF the audit flagged; production runner topology is a pipeline-slice decision.
2. Populate values-production.yaml (SSO-1485). Supply the four required URL helpers so the render stops aborting: hub.publicBaseUrl, identity.publicBaseUrl, productApi.federatedIdentityIssuer (byte-identical to the V55 federated_identity seed and to identity.publicBaseUrl, per SSO-2019), and apple.publicBaseUrl (only when Apple is enabled). Add prod hostnames/TLS, replica counts, backup S3 bucket + retention, and monitoring.prometheusRules.enabled: true (CRD-auto-detected, overridden to false on a cluster without the monitoring CRDs). The image tag stays out of the file — the pipeline pins an immutable commit SHA; latest is forbidden (SSO-799).
3. Stand up the production pipeline (SSO-1485). A dedicated deploy-production.yml, not a branch of deploy-staging.yml: triggered by a v* release tag (and manual workflow_dispatch for DR replay), not by every merge to main. It runs inside a protected production GitHub Environment with required reviewers — that human approval is the go-live gate. A non-activated scaffold (the overlay + a skeleton workflow) is the concrete artefact of the design ADR.
4. Mint the operator secrets the pipeline injects — never by hand, always from the pipeline's production secret set (distinct from staging). Follow the audit's operator pre-flight checklist: the imperatively-created thoryn-db-credentials / thoryn-smtp-credentials / thoryn-saml-credentials / ghcr-pull-secret; the Category C infra client secrets plus their bcrypt fix-forward migrations (G7 — SSO-2101 / SSO-2137); the synthetic-monitor login password (G8 — SSO-2030); the Slack/pager webhook; and the two backup-destination Secrets.
5. Deploy from git. helm upgrade --install thoryn ./deploy/helm/thoryn -f values-production.yaml with --set global.image.tag= the immutable SHA, --timeout 15m --cleanup-on-fail.

Known manual-step scars still to back-fill — reproducibility debt that must be closed before prod, not carried into it. Each is an imperative deploy-workflow mutation that a fresh deploy silently reverts:

Scar	Ticket	Priority
CoreDNS upstream resolver is sed-patched imperatively every deploy → reverts on k3s restart; make declarative	SSO-2166	High
SSA-reclaim scans only -n thoryn → cert-manager / kube-system field-ownership scars are uncleanable	SSO-2167	High
Hub domain mapping seeded via kubectl exec psql every deploy → move to Flyway / product-API (no-direct-DB-config rule)	SSO-2168	Medium
Deploy-workflow imperative-mutation cleanup batch (redundant deletes, cert force-reissue surgery, PrometheusRule strip, stale dev-token scripts)	SSO-2169	Low
SSA-reclaim add namespaced Role / RoleBinding / PersistentVolumeClaim (OpenBao k8s-auth RBAC + PVC)	SSO-2139	Low

One more finding to resolve in this phase: the thoryn.springBootStagingEnv Helm macro hardcodes SPRING_PROFILES_ACTIVE=staging, so the hub and identity-service render with the staging profile even under the production overlay (api-gateway and product-api set the profile directly and render production correctly). The deploy-pipeline ADR recommends parameterising the macro (its option A / slice S3); until then the staging profile carries in-cluster wiring that production also uses, so nothing breaks, but the hub's application-production.yml is dead.

Exit / verification gate:

• helm template -f values-production.yaml renders cleanly (the deploy-pipeline ADR validated 72 resources, exit 0).
• A from-scratch deploy converges with zero manual steps, and no kubectl scars remain (the SSA-reclaim step is clean). Validate by reasoning precisely about a from-scratch deploy, or by running one — not by inspecting a hand-patched cluster.

Phase 2 — Data & migrations

Entry gate: Phase 1 pipeline provisioned; a fresh production Postgres reachable.

Steps

1. Flyway applies forward-only on boot. Every service runs its versioned migrations at startup (the hub uses R2DBC at runtime but Flyway needs a JDBC DataSource; both are wired side by side). A fresh prod DB reaches the intended state by replaying the full migration history — there is no snapshot-import shortcut.
2. Seed genuine platform reference data via migration (the default tenant, the platform reference clients/tenants that the extraction ADRs deliberately kept in oathy). This is platform reference-data seeding, not demo/use-case configuration — no demo or monitor Flyway seeds reach a production DB (the product-boundary rule). Static seed client secrets are {bcrypt}, never {noop} (SSO-741), with real prod values threaded per G7.
3. Honour the deploy-ordering constraints the audit enumerated:
   • The hub client-scope-grant migrations (the V82…V101 family) must be applied before the console / BFF / CLI request those scopes, or sign-in loops on invalid_scope. Because the hub (oathy) and thoryn-web deploy from separate repos, the hub migration must land first.
   • productApi.federatedIdentityIssuer must byte-match the V55 seed issuer, or seeded users' sub moves at subject resolution and they are locked out. Both are wired from the same required helper — do not let them diverge.
   • A Category C real infra secret and its bcrypt-hash migration co-deploy — the immutable seed value is the only credential that authenticates until a forward migration rewrites it. The production pipeline can assert these against the rendered values and the target DB before rolling pods.

Exit gate: flyway_schema_history clean, no pending or failed migrations; the pipeline's pre-flight ordering assertions (issuer byte-match, scope-grant presence) pass.

Recovery discipline (carried into Phase 6): forward-patch only. DROP DATABASE / TRUNCATE / DROP SCHEMA, and the k8s-level equivalent (PVC-delete-and-re-bootstrap), are forbidden in every environment and CI-guarded. A bad migration is corrected with a new V<n+1> migration, never a destructive reset. See Backup and recovery.

Phase 3 — The trustTestIssuers flip (independent, PO-gated)

Entry gate: this is not part of the initial cutover switch. It is a separately-gated security widening that happens after the platform is live and stable, only once the sweep is complete and the product owner records a go. The authoritative current verdict, from the flip-readiness assessment, is NO-GO.

Where it sits and why it is safe to defer. Per-tenant test/sandbox mode (livemode, ADR adrs/2026-07-21-per-tenant-test-sandbox-mode.md) is inert until trustTestIssuers flips to true at the three customer-plane resource servers. Today it is false everywhere: a mode=test token is rejected fail-safe by issuer and signature, with no JWKS fetch. The production overlay keeps it false (a hard invariant of the deploy-pipeline ADR). So production ships with test mode off and nothing depends on the flip for go-live.

The flip is a code change at three sites, gated on a completed sweep. The blockers, from the flip-readiness assessment (do not restate its matrix — read it):

• [HARD] SSO-2147 — identity-service livemode data plane + user-facade mode-forwarding. identity-service has zero livemode today, so a test credential would read and mutate live user data through the product-api user facade. This alone is disqualifying.
• [HARD] SSO-2148 — auth-runtime email senders (magic-link, account-unlock, step-up, email-verification) suppressed in test mode.
• [MED] SSO-2149 — SIEM + DSAR mode decision — RESOLVED: SSO-2151 partitions SIEM by livemode (merged #2545) and SSO-2152 rejects test-mode tokens at the DSAR surface (merged #2542). No longer a flip blocker.
• [MED/LOW] SSO-2097 (SCIM facade + /internal org-routing), SSO-2136 (login mode through JIT), SSO-2106 / SSO-2114 (audit tail), SSO-2075 (default test-key rotation).
• [GATE] SSO-2146 — introduce the gated trustTestIssuers property (default false) at the three resource servers, add the cross-mode rejection E2E tests, and take the PO go/no-go.

Required verification gate: a live mode=test round-trip on staging — provision a mode=test client, mint a token, and confirm (a) the prod-shaped resource servers accept it only with the flip on, (b) it reads only livemode=false rows, and (c) it is rejected fail-safe with the flip off. A green code review is not sufficient for a security-relevant widening.

Exit gate: blockers closed, PO GO recorded, flip PR merged with cross-mode E2E, staging round-trip proven.

Phase 4 — Cutover & DNS / TLS

Entry gate: Phases 0–2 green (Phase 3 is independent). Production apex confirmed.

Steps

1. Issue TLS. cert-manager issues the wildcard cert via the DNS-01 path fixed in SSO-2119. Prerequisites the chart cannot reproduce: the ACME_DNS_CREDENTIALS Secret synced into the cert-manager namespace, and the one-time Hetzner _acme-challenge CNAME (it lives in the DNS zone — a documented one-time prerequisite, not chart state).
2. Verify the cert is served, not just "Ready". The SSO-2119 lesson: cert-manager's own Ready condition stayed stale-true for 30 days while the cert was expired. Check the actual notAfter and what a TLS client receives on the live endpoint.
3. DNS cutover. Point the production hostnames (the hub / identity / api apex family) at the production ingress.
4. Traffic cutover. The deploy-production pipeline runs behind the required-reviewer gate (the human go), helm upgrade with the immutable image SHA, then the per-service rollout gates and the crash-loop scan.

Exit / verification gate: every Ingress host serves a valid, unexpired TLS cert; kubectl rollout status is green for every service; no CrashLoopBackOff.

Phase 5 — Post-cutover verification

Entry gate: Phase 4 cutover applied.

Steps / gates

1. Smoke + flows are the release gate. Run scripts/smoke-test.sh and scripts/smoke-flow-runner.sh against the production host. The standing Tier-3 synthetic monitor is not enabled by default in production until its threat-model review (SSO-1212), so the one-shot smoke + flow-runner is the gate; the synthetic-monitor work (SSO-2025, In Progress) is the ongoing floor.
2. Readiness deep-checks UP. Every service's Tier-2 readiness group (SSO-1211) is UP — redisAcl, vaultTransit, r2dbc, JWKS reachability, hub OIDC discovery, and per-service indicators. Readiness gates traffic; a DOWN deep-check means the pod is not serving.
3. Smoke the OAuth / OIDC flows directly. /.well-known/openid-configuration; an authorization-code sign-in; token issuance at /oauth2/token; /userinfo; a refresh; a revoke. These are the exact classes (/oauth2/token, /oauth2/authorize, /login) whose 5xx went undetected in the incident that motivates G2.
4. Confirm signals flow — once G2's monitoring lands: scrape targets up, the paging pack loaded, and a test alert routes to the configured pager.

Exit gate: smoke + flows green; readiness green; monitoring green (once installed).

Phase 6 — Rollback plan

Roll back per phase; the reproducibility bar is what makes rollback safe (every prior state is a git ref), and the database is the one place rollback is forward-only.

Phase	How to roll back	Hard rule
1 / 4 — app, config, cutover	helm rollback to the previous release, or re-deploy the prior immutable image SHA; revert the DNS cutover.	Reproducible because every prior release is a pinned SHA.
2 — database	Forward-patch only. A bad migration is corrected with a new V<n+1> migration. Restore-from-backup uses the documented restore ordering, then forward-patches.	Never DROP DATABASE / TRUNCATE / PVC-delete-and-re-bootstrap (CI-guarded).
3 — the flip	Set trustTestIssuers back to false — instant, safe revert; test tokens are 401'd fail-safe with no JWKS fetch. No data to undo: livemode is a partition discriminator, not a signed field, so live rows verify byte-for-byte.	Fail-safe by construction.
custody	Restore the pinned OpenBao unseal key + at-rest AES keys.	Never regenerate them on rollback — a lookup minting new keys orphans every ciphertext (MFA phone numbers, OATH seeds, hook secrets, tenant SMTP passwords).

Emergency manual mitigation is permitted to restore service, but it is a mitigation, not the fix: back-fill it into the chart / migration / config immediately, and file a Highest-priority removal ticket for anything security-relaxing. A manual fix that is not back-filled is an outage scheduled to recur on the next deploy.

Dual-delivery note

• Managed sovereign SaaS: Thoryn SRE owns every phase, the monitoring runtime, and the pager receiver.
• Self-managed: the customer owns Phases 1, 2, and 4 (their cluster, DNS, TLS, provisioning), runs their own Prometheus / Alertmanager runtime (Thoryn ships the ServiceMonitors, PrometheusRule pack, and Grafana dashboards as chart artefacts, gated behind monitoring.*), and wires their own pager. Phase 3 is a platform-level decision — trustTestIssuers stays false for everyone until the sweep completes.

Verification boundary

This runbook synthesizes read-only artefacts and the Jira status of each gate as of the lastReviewed date. It has not driven a production cutover.

• The readiness audit and flip-readiness assessment are on main and are cited in sources:.
• The monitoring (SSO-1482) and deploy-pipeline (SSO-1485) inputs are design ADRs on unmerged PRs at the time of writing (adrs/2026-07-24-production-monitoring-alerting.md and adrs/2026-07-23-production-deploy-pipeline.md, plus a docs/content/operate/production-deploy-pipeline.mdx how-to). When those PRs merge, add their ADR paths to this page's sources: and link the deploy-pipeline how-to directly. The HA work (SSO-1486) has no ADR and no PR yet — it is the least-built Phase 0 gate.
• Each phase's gate is only truly green when its owning ticket's exact operation has been observed to succeed on the real target. Two gates are Done and validated (SSO-2119, SSO-2129); the rest of the critical path is Backlog or In Progress. Treat any "ready" claim here as a pointer to the ticket that must prove it, not as proof itself.