Backup and recovery

This page covers what the chart backs up and how to recover. The recovery procedures for the high-risk paths — schema drift, OpenBao unseal, Helm field-ownership — are summarised here and linked to the authoritative, step-by-step runbooks. Those runbooks are the source of truth and carry the exact commands; this page tells you which one you need and states the one principle that governs all of them.

The governing rule: forward-patch only, never destroy user-created data. DROP DATABASE, TRUNCATE TABLE, DROP SCHEMA, and their Kubernetes equivalent (kubectl delete pvc on the Postgres volume followed by a re-bootstrap) are forbidden as a recovery mechanism in every environment, staging included — a CI guard (scripts/check-no-drop-database.sh) rejects the SQL forms. Recovery converges the schema with a new migration, or it escalates. It never nukes. The rule exists because a 2026-05-20 "nuclear reset" silently destroyed real test accounts and surfaced 48 hours later as a login failure; the 24-hour cost of an escalation is always cheaper than that.

What is backed up

The chart provisions offsite backups as Kubernetes CronJobs (backup.enabled: true). Each job dumps or copies its data, encrypts it (AES-256 via openssl, key from backup.encryptionKeySecret), and uploads to backup.s3Bucket.

Two honesty notes about the OpenBao backup you must internalise:

• It is best-effort online, not atomic. The file backend writes each entry atomically (temp-file + rename) but is not transactional and has no filesystem-level locking, so a tar walk of the live tree can capture files across a logical write. For append-mostly Transit key material the cross-file window is small but non-zero. The strictly-consistent answer is an offline (sealed) backup or a migration to the raft backend (native atomic snapshots + HA), which is a documented HA follow-up. See docs/modules/ROOT/pages/operator/openbao-file-backup.adoc.
• The unseal Secret is part of the backup story. OpenBao's Transit keys are useless without the unseal key in thoryn-openbao-keys. Losing the node and that Secret is permanent key loss. Ensure the Secret is included in your cluster's etcd/Secret backup, or export and store it out-of-band per the OpenBao runbook.

The per-service encryption keys

A family of AES-256-GCM keys encrypt application data at rest inside Postgres and live only as Kubernetes Secrets, generated at first install and preserved across helm upgrade by a Helm lookup: thoryn-identity-hooks-secret-encryption-key, thoryn-identity-phone-encryption-key, thoryn-identity-oath-seed-encryption-key, and thoryn-identity-tenant-smtp-credential-encryption-key (mirrored hook secrets, MFA phone numbers, OATH/TOTP seeds, and tenant BYO-SMTP passwords, respectively).

Why they must be backed up with the data. These keys are not derivable and not stored anywhere else. If you restore a Postgres dump into a fresh namespace or cluster where the chart's lookup finds no existing Secret, it mints new random keys — and every ciphertext the old keys protected is silently undecryptable, on a restore that otherwise looks successful. (The BYO-SMTP case is the most insidious: a decrypt failure falls back to the platform sender with fail-closed: false, so mail keeps flowing from the wrong relay rather than erroring.) A backup of the data without the keys that decrypt it is not a recovery.

They are now in the backup set (SSO-2129). The encryption-key-backup CronJob dumps each key Secret (kubectl get secret -o json), encrypts it with the same openssl AES-256-CBC pipeline and backup.encryptionKeySecret passphrase as the Postgres dumps, and uploads it to the same backup.s3Bucket under an encryption-keys/ prefix. It runs on backup.enabled and identity.enabled — independent of the OpenBao backend path, so the keys are captured regardless of which secrets-backend the cluster runs. A CI guard (scripts/check-encryption-key-secrets-backed-up.sh) fails the build if a new *-encryption-key-secret.yaml is ever added without being wired into this CronJob, so the class cannot silently drift out of coverage again.

This backup is raw key material — its destination protection is load-bearing. Each artefact is an AES key in the clear inside the Secret JSON. Riding the shared pipeline is safe only because every artefact is openssl AES-256-CBC-encrypted with the backup.encryptionKeySecret passphrase before it leaves the pod, and the backup.s3Bucket destination is access-controlled by the backup.awsCredentialsSecret credentials — the same protection the OpenBao unseal Secret already rides. Do not relax either: an unencrypted or world-readable destination for these keys is worse than the gap this closes. As with every other backup, an operator must have configured a real backup.s3Bucket + backup.awsCredentialsSecret (and the encryption-key passphrase in backup.encryptionKeySecret) for the job to upload anywhere — the chart wires the job; the destination is the operator's to supply.

Restore ordering — keys before the first helm upgrade. On a rebuild, restore these key Secrets (decrypt the encryption-keys/*.enc artefacts, kubectl apply the Secret JSON) before the first helm upgrade of the rebuilt environment, alongside the Postgres dump they decrypt. That way the chart's lookup finds the original key and preserves it, rather than minting a replacement that orphans the restored ciphertext. For the BYO-SMTP key specifically, the consequences and the rotation path are in Tenant BYO-SMTP activation → rotation.

Longer term (not yet implemented): the OATH-seed key already carries an "interim local key pending the SSO-313 / SSO-314 Vault Transit migration" note. Moving this key class into OpenBao Transit — which is backed up (SSO-1483) and is already the platform's custody story for signing keys — would retire the local-AES-key-in-a-Secret pattern entirely and fold these keys into the OpenBao backup rather than a dedicated CronJob. Tracked under SSO-2129 as the alternative to the backup-set approach shipped here.

Source: deploy/helm/thoryn/templates/encryption-key-backup-cronjob.yaml, .../backup-cronjob.yaml, .../openbao-file-backup-cronjob.yaml, values.yaml, scripts/check-encryption-key-secrets-backed-up.sh.

Restore drill

Backups are only real if they restore. The quarterly disaster-recovery drill restores the encrypted Postgres dumps (and, historically, a Vault snapshot) into a scratch namespace and runs the smoke test against them — a red smoke test means the backup shape is no longer compatible with the application code, which is a P1. The full procedure, cadence, and ownership are in docs/modules/ROOT/pages/runbooks/dr-restore.adoc.

Accuracy note: the DR-restore runbook's Vault steps (vault operator raft snapshot restore, and a vault.devRootToken chart value) describe the pre-OpenBao Vault path. On the current OpenBao file backend the restore is a data-directory restore, not a raft-snapshot restore, and there is no devRootToken — follow docs/modules/ROOT/pages/operator/openbao-file-backup.adoc for the OpenBao restore, and the Postgres steps of the DR runbook for the database restore.

Flyway schema drift, forward-patch only

When a Spring pod won't start on a FlywayValidateException, the recovery is always forward-patch — a new V<n+1>__<slug>.sql that converges the schema, plus a targeted flyway_schema_history repair — never a destructive reset. The authoritative decision tree is docs/modules/ROOT/pages/operator/staging-migration-recovery.adoc; it maps each error shape to a fix:

The one thing you never do is delete a success=true row or drop a table — those hold the work reflected in the live schema. This is the operational form of the CLAUDE.md "Databases: forward-patch only, never DROP/CREATE" rule.

OpenBao secrets-backend recovery

OpenBao boots sealed on file storage and is unsealed automatically — the openbao-init Job unseals it after operator init, and each pod carries an unseal init-container that re-unseals on restart from the key in thoryn-openbao-keys. So a routine pod roll needs no operator action. The two failure modes that do:

• The store is sealed and not recovering (e.g. the keys Secret is present but the init/unseal path did not run). Confirm with bao status inside the pod; unseal proceeds from the stored key. The cutover runbook (docs/modules/ROOT/pages/operator/openbao-cutover.adoc) carries the diagnosis and the unseal steps.
• The stored root token is stale/revoked, so the init Job's provisioning calls return 403. The openbao-init Job fails loud on this with an explicit message rather than crash-looping opaquely, and recovery is a forward re-init of the secrets backend — which is distinct from the forbidden destructive-Postgres class, because the secrets backend's durable material is the Transit keys, and re-init regenerates them (the runbook covers preserving/re-provisioning them). The step-by-step is in docs/modules/ROOT/pages/operator/openbao-init-stale-root-token.adoc (linked from the Job's own error output).

Do not hand-roll OpenBao recovery from memory — the init Job's messages point at the exact runbook section, and the runbooks are kept in lock-step with the Job.

Production hardening (not yet implemented): staging keeps the unseal key in a RBAC-scoped Kubernetes Secret. A production deployment should move to a KMS/HSM auto-unseal seal so no operator-readable unseal key exists. This is a documented later phase in ADR adrs/2026-06-07-vault-to-openbao.md and the cutover runbook — plan for it rather than promoting the Secret-stored-key model to production unchanged.

Helm SSA field-ownership reclaim

If an emergency kubectl set / edit / patch was run against a Helm-managed resource, the next helm upgrade fails with a Server-Side Apply conflict of the shape conflict with "kubectl-set" using apps/v1: .spec.template.... The emergency write took field-ownership away from Helm's field-manager.

• Automatic: the Deploy to staging workflow includes a Reclaim SSA field-ownership step that scans a set of SSA-conflict-prone kinds in the namespace for kubectl-* field-managers and removes those managedFields entries via JSON patch. It is idempotent (a no-op on a clean release) and never touches Helm's own manager. So in the standard workflow the deploy self-heals.
• Manual (uncovered kinds / other namespaces): the recipe is in docs/modules/ROOT/pages/runbooks/helm-field-ownership-recovery.adoc. The critical gotcha: kubectl get -o json strips managedFields unless you pass --show-managed-fields=true (kubectl 1.21+), so a scan without the flag silently finds nothing.

The durable fix for any manual change is always to back-fill it into the chart/values so the next clean deploy reproduces it — a manual fix that isn't back-filled is an outage scheduled to recur on the next deploy.

Configuration

The backup surface is a single values block (staging shown; a self-managed overlay supplies its own bucket and schedule):

backup:
  enabled: true
  s3Bucket: "thoryn-<env>-backups"
  retentionDays: 14
  encryptionKeySecret: "thoryn-backup-secret"        # AES-256 key
  awsCredentialsSecret: "thoryn-backup-aws-credentials"  # S3 upload creds (ini blob)
  schedule:
    postgres:       "0 1 * * *"   # nightly pg_dump
    vault:          "30 1 * * *"  # raft snapshot (rollback path only)
    openbao:        "0 2 * * *"   # file-backend tar (the default OpenBao path)
    encryptionKeys: "30 2 * * *"  # per-service AES key Secrets (SSO-2129)

The openbao-file-backup and vault-snapshot schedules are mutually exclusive — one renders on the file-backend default, the other only on the devMode: true rollback path, so the backend is never left without a backup job. The encryption-key-backup schedule is staggered last so it does not contend with the three data-backup windows for offsite-S3 upload bandwidth. Backup encryption, S3 credentials, and the per-CronJob security context are documented in docs/modules/ROOT/pages/operator/backup-cronjob-security.adoc and its OpenBao sibling.

Security notes

• The backup encryption key never touches local disk during a restore — the DR runbook reads it straight from OpenBao into an environment variable that is unset after decryption. A failed openssl decrypt means the backup is unrecoverable — try the previous day and escalate as a P1.
• The unseal Secret is as sensitive as the keys it unlocks. Back it up, but treat it with the same custody as the Transit key material — anyone with the unseal key + a data-dir copy can reconstruct the signing keys.
• Recovery preserves user data by contract. Staging Postgres carries real test users and production carries audit rows on a ≥ 7-year retention horizon (eIDAS); the forward-patch and never-DROP rules make the two environments symmetric. If forward-patching is impossible, escalate — never re-bootstrap.
• Back-fill every emergency change. A live kubectl fix is a mitigation, not the fix; the durable fix is a chart/migration/config change so a clean deploy reproduces the working state.