Upgrade and rollback

An upgrade is the same command as a first deploy — helm upgrade --install against a new image SHA. Because every change ships as code (chart, migrations, config, source) and the deployed state is rebuildable from git, an upgrade is a re-render of the same chart at a newer commit, not a bespoke procedure.

helm upgrade --install thoryn ./deploy/helm/thoryn \
  --namespace thoryn \
  -f deploy/helm/thoryn/values-<env>.yaml \
  --set global.image.tag="$(git rev-parse HEAD)" \
  --timeout 15m --cleanup-on-fail

The immutable image-SHA contract

Every image tag the platform deploys must be an immutable commit SHA. The mutable :latest tag is forbidden both in the publish pipeline and at deploy time. This is enforced structurally:

• global.image.tag has no default in values.yaml. The thoryn.imageTag Helm helper wraps it in a required assertion, so a render with no tag fails rather than silently resolving to :latest.
• Each app service can carry its own <service>.image.tag; when set it overrides the fleet-wide global.image.tag, so a deploy rolls only the services whose content changed. A service with no per-service tag falls back to the global tag. The staging workflow computes this per-service map from the commit that last touched each service; a manual deploy typically just sets global.image.tag and rolls everything to the same SHA.

Why this matters for upgrades: an immutable SHA is what makes an upgrade reproducible and reversible. Rolling back is re-deploying the previous SHA; there is no ambiguity about what :latest pointed at last week. See docs/modules/ROOT/pages/operator/image-tag-contract.adoc for the full contract.

Source: deploy/helm/thoryn/templates/_helpers.tpl (thoryn.imageTag), values.yaml.

Schema migrations are forward-only, applied on boot

There is no separate migration step. Each Spring service runs Flyway on boot against its database and applies any new V*__*.sql migrations before it reports ready. Two rules govern what an upgrade may carry:

• Migrations are immutable once applied. Never edit a V*__*.sql that has shipped — a CI gate (scripts/check-migration-immutability.sh) rejects in-place edits, deletions, and renames of a migration already on main. Correct a mistake by adding a new V<n+1>__<slug>.sql, never by editing the old one.
• Recovery is forward-patch only. If a migration drifts or fails on an existing database, the fix is a new migration that converges the schema — never DROP DATABASE, TRUNCATE, or a re-bootstrap. That rule holds in every environment; the procedure is on the backup-and-recovery page.

Because migrations gate readiness, an upgrade that ships a bad migration surfaces as a pod that never becomes ready — the rolling update holds the old pod in service rather than cutting traffic to a broken new one (see readiness below).

Rolling behaviour and readiness

The chart is built so a routine upgrade does not churn the whole stack:

• Infra stays up. Postgres, Redis, and OpenBao are image-digest-pinned with stable pod templates, so they do not roll on a code deploy. Only app services (hub, identity, product-api, api-gateway, …) surge-roll — a new pod becomes Ready before the old one terminates. See docs/modules/ROOT/pages/operator/deploy-rollout-model.adoc.
• Probes are tiered so a cold-start blip cannot crash-loop a healthy pod. The startup probe targets the cheap canonical liveness endpoint (/actuator/health/liveness) — "is the JVM up and the context initialised?" — while the readiness group folds in the deep checks (Redis ACL, OpenBao Transit, JWKS reachability, R2DBC pool). A transient Transit warm-up or Redis-ACL render delay flips readiness, not liveness, so the pod is given time to become ready instead of being killed. A CI guard enforces the canonical startup-probe path.
• OpenBao consumers wait for their role. Services that authenticate to OpenBao carry a wait-for-vault-role init container that blocks the JVM until the service's Kubernetes-auth role is bound — so on a fresh backend the pod waits rather than crash-looping past an unprovisioned OpenBao.

Watch an upgrade land:

for d in thoryn-hub thoryn-identity thoryn-product-api thoryn-api-gateway; do
  kubectl rollout status deployment/$d -n thoryn --timeout=10m
done

Then re-run the smoke test (scripts/smoke-test.sh) as the from-scratch verify — a green render is not a validated upgrade.

Source: CLAUDE.md (tiered readiness; startup-probe canonical path), docs/content/product/architecture/delivery-models.mdx.

Rollback

Because the previous release is a previous immutable SHA, rollback is a first-class Helm operation:

# Roll back to the immediately-previous release revision.
helm rollback thoryn -n thoryn
 
# Or re-deploy a specific known-good SHA explicitly.
helm upgrade --install thoryn ./deploy/helm/thoryn \
  -n thoryn -f deploy/helm/thoryn/values-<env>.yaml \
  --set global.image.tag=<known-good-sha> --timeout 15m --cleanup-on-fail

A helm rollback reverts the workloads and config, but it does not reverse a Flyway migration. Migrations are forward-only by design — rolling application pods back to an older image against a newer schema is only safe when the newer schema is backward-compatible (the platform's additive-migration convention aims for this). If a schema change is genuinely incompatible with the previous image, the recovery is a forward migration that restores compatibility, plus a code fix — not a schema rollback. See the forward-patch rule on the backup-and-recovery page.

If a helm upgrade gets stuck in pending-upgrade (e.g. it timed out mid-roll), scripts/deploy.sh handles it by rolling the release back before retrying; manually, helm rollback thoryn -n thoryn (or helm uninstall as a last resort on a first install) clears the stuck state.

Troubleshooting

Security notes

• Never deploy a mutable tag. :latest breaks reproducibility and makes rollback ambiguous; the required helper and the publish pipeline both forbid it.
• Never edit an applied migration to "fix" an upgrade. Add a forward migration. The immutability gate and the never-DROP rule protect user-created data in every environment.
• Prefer a re-deploy over kubectl edit. A manual live edit both violates the rebuildable-from-git rule and scars Helm's Server-Side Apply ownership, breaking the next upgrade until reclaimed.