Production deploy pipeline (design)

This page describes the design for making a production deploy reproducible — the production values overlay, the values and secrets it needs, and the pipeline that applies it. It is the operator-facing companion to the ADR adrs/2026-07-23-production-deploy-pipeline.md and story SSO-1485 under the production-readiness epic SSO-1481.

The reproducibility bar (see Deploy (self-managed)) is: a clean, from-scratch helm deploy reproduces the working state with zero manual steps. The production pipeline and overlay are what make that true for the managed production environment. The full self-managed deploy runbook lives on the Deploy page; this page is the production-pipeline design that the managed service runs.

Design stage. deploy/helm/thoryn/values-production.yaml is a reviewable scaffold, and .github/workflows/deploy-production.yml is a non-activated skeleton (manual trigger, gated on a repo variable that does not exist, targeting a protected production Environment that must be created with required reviewers). Turning either on is the first implementation slice, not the design.

The production overlay

The overlay supplies every non-secret production value and documents every operator-supplied secret. It never contains a credential — credentials are Kubernetes Secrets or --set-string injected at deploy time.

Required Helm values (fail the render if missing)

The chart aborts helm template / helm upgrade when one of these is unset — a misconfigured deploy stops at render time rather than shipping a broken or host-header-poisonable URL.

productApi.federatedIdentityIssuer and identity.publicBaseUrl must be byte-identical — they are the same issuer viewed from two services.

Required Kubernetes Secrets

Production sources these from a secret store distinct from staging's. The names match the chart's secretKeyRef wiring:

The four AES-256-GCM keys are chart-generated and preserved across helm upgrade by a lookup. They must be in the backup set (SSO-2129): a Postgres restore into a fresh namespace restores every ciphertext, but a missing key Secret makes the chart mint a NEW random key and silently orphan those columns.

The pipeline

A separate workflow from deploy-staging.yml, hardened rather than forked:

• Trigger — a git tag v* (a release is an explicit, immutable artefact) plus manual workflow_dispatch. Not workflow_run on every main build: production deploys are deliberate, not a side effect of merge.
• Approval — the deploy job runs in a protected production GitHub Environment with required reviewers. A deploy pauses for a named approver before any cluster mutation.
• Secret injection — the same --set-string-from-secret and pre-created-Secret pattern staging uses, sourced from the production Environment's own secret set.
• Image pin — --set global.image.tag=<the tag's commit SHA>; the release tag resolves to one immutable artefact set.

Pre-flight (fail fast, before helm upgrade)

• DNS preflight for the wildcard hub subdomain.
• The image tag is a real pushed SHA, not a moving ref.
• Migration ordering — the hub client-scope grants (Flyway UPDATE client.scopes) are present before the console/CLI request those scopes, or sign-in loops on invalid_scope; and productApi.federatedIdentityIssuer byte-matches the V55 seed issuer.

Deploy

helm upgrade --install thoryn ./deploy/helm/thoryn \
  -f values-production.yaml \
  --set global.image.tag=<sha> \
  --set monitoring.prometheusRules.enabled=<crd-detected> \
  --timeout 15m --cleanup-on-fail

The pipeline CRD-auto-detects monitoring.coreos.com/v1 and overrides prometheusRules.enabled to false on a cluster without kube-prometheus-stack, so a fresh-cluster first deploy still succeeds.

Post-deploy verification

Per-service kubectl rollout status plus the crash-loop scan, then scripts/smoke-test.sh and scripts/smoke-flow-runner.sh against the production host. The standing Tier-3 synthetic monitor is not enabled in production by default — that needs a separate threat-model review (SSO-1212) — so the one-shot smoke and flow-runner are the release gate until then.

Ordering and disaster recovery

• Restore ordering (SSO-2129). A DR rebuild into a fresh namespace restores the OpenBao unseal Secret and the four AES-at-rest key Secrets before the first helm upgrade, or the chart's lookup mints new keys and orphans every ciphertext. See Backup and recovery.
• Forward-patch only. Recovery never drops the database or deletes the PVC and re-bootstraps — that destroys user-created data. See CLAUDE.md "Databases: forward-patch only".
• SSA-reclaim covers RBAC + PVC (SSO-2139). The production reclaim step's RECLAIM_KINDS adds role, rolebinding, and persistentvolumeclaim so an emergency patch of the OpenBao k8s-auth RBAC or PVC self-heals instead of wedging the next deploy.

Security invariants that must hold in production

• Test / sandbox mode stays off. Production resource servers are mode-unaware: TrustedTenantIssuers.trustTestIssuers keeps its false code default, so a test-mode issuer is rejected with 401 and no JWKS fetch is attempted. There is no chart knob that flips it true; do not add one to the overlay.
• NetworkPolicy default-deny stays on; TLS/ssl-redirect stays on every public ingress.
• No demo/monitor Flyway seeds — they inject demo data.
• The audit chain signs via OpenBao Transit, never the in-JVM local heap-key stand-in.

Design finding: the Spring profile

The chart's thoryn.springBootStagingEnv macro hardcodes SPRING_PROFILES_ACTIVE: staging for every service that includes it (hub, identity, apple, linkedin, synthetic). A helm template of the production overlay renders hub and identity with the staging profile, while api-gateway and product-api — which set the profile directly and render production. So the hub's existing application-production.yml cannot activate under today's chart. Parameterising the macro's profile (recommended) is a deferred slice; until then the staging profile means "in-cluster wiring" and every environment-specific value is carried by the overlay's env overrides. The ADR records the trade-off.

What is deferred

Each slice is its own Jira story under SSO-1485 / epic SSO-1481: activate the pipeline skeleton; fail-loud seed secrets (SSO-2101); Spring-profile parameterisation; SSA-reclaim RBAC/PVC (SSO-2139); backup/DR green; HA primitives (HPA / PodDisruptionBudget / anti-affinity); and the observability install so the enabled PrometheusRules have an operator to run against.