Deploy Thoryn (self-managed)

This is the full deploy runbook for the self-managed distribution — the depth the self-managed quickstart defers to. The unit of deployment is the thoryn Helm chart (deploy/helm/thoryn/); the same chart Thoryn runs for the managed service. A deploy is a single helm upgrade --install, plus a small set of Kubernetes Secrets you provide and a small set of Helm values that fail the render loudly if you forget them.

Two reference overlays exist in the repository. values-staging.yaml is the live worked overlay, driven by the Deploy to staging workflow (.github/workflows/deploy-staging.yml) and scripts/deploy.sh. values-production.yaml is now a structured production scaffold — it supplies the required values described below and documents every operator secret — but the production pipeline that consumes it is still a non-activated skeleton (.github/workflows/deploy-production.yml); see the production deploy pipeline design and ADR adrs/2026-07-23-production-deploy-pipeline.md. A self-managed deployment supplies its own environment overlay that provides the same required values described below. Use either overlay as the reference for the shape of a complete overlay, not as a file to deploy verbatim.

Prerequisites

• A Kubernetes cluster whose CNI enforces standard networking.k8s.io/v1 NetworkPolicy (Calico, Cilium, or k3s' default Calico). The chart's default-deny + per-service policies render but have no effect on a CNI that ignores NetworkPolicy.
• kubectl configured for the target cluster, and helm (v3, Server-Side Apply).
• An ingress controller (the chart is written against ingress-nginx) and, for automatic TLS, cert-manager with a configured issuer.
• A container registry the cluster can pull from. Images are published as ghcr.io/thoryn-io/<service>:<sha>; a private registry needs an image-pull secret referenced through global.imagePullSecrets.
• Persistent storage — Postgres and OpenBao each claim a PVC (ReadWriteOnce is sufficient; OpenBao file storage is single-replica).

Step 1 — Provide the Kubernetes Secrets

Credentials are injected from Kubernetes Secrets; never put plaintext in a values file. The core Secrets the chart consumes (creation instructions in deploy/k8s/secrets-README.md):

The OpenBao unseal key + root token Secret (thoryn-openbao-keys, name set by vault.keysSecretName) is not pre-seeded — OpenBao generates the unseal key at operator init, so the init Job creates it (Step 3). Do not attempt to create it by hand.

Several per-service encryption keys are chart-generated too — the Redis ACL passwords, the hub's pairwise-sub salt, and identity-service's AES keys for mirrored hook secrets, MFA phone numbers, OATH seeds, and tenant BYO-SMTP passwords. Each is created at first install and preserved across every helm upgrade via a Helm lookup, so it never rotates out from under existing ciphertext. Do not create or edit them by hand — a kubectl-created Secret is exactly the state a clean deploy cannot reproduce. For the BYO-SMTP key, activation and the (destructive) rotation procedure are in Tenant BYO-SMTP activation; for how they are backed up and the restore-ordering requirement see Backup and recovery.

If you enable wildcard-subdomain TLS via a DNS-01 solver, the solver's API-token Secret (the staging overlay uses hetzner-dns-api-token in the cert-manager namespace) is an additional prerequisite.

Step 2 — Set the required Helm values

Four values are enforced by the chart's required helpers — the render fails rather than shipping a silently-broken or insecure deployment. This is deliberate: each guards a real incident class.

Beyond the required four, an environment overlay sets at least the public issuer, the tenant-subdomain suffix, ingress hosts, and the in-cluster service URLs. The minimum shape, mirroring values-staging.yaml:

global:
  image:
    registry: ghcr.io/thoryn-io   # your registry
  imagePullSecrets:
    - name: ghcr-pull-secret
 
hub:
  publicBaseUrl: "https://hub.<env>.example.org"
  ingress:
    host: hub.<env>.example.org
    tls: true
  env:
    # Spring AS advertises this issuer in discovery even though nginx terminates TLS.
    SPRING_SECURITY_OAUTH2_AUTHORIZATIONSERVER_ISSUER: "https://hub.<env>.example.org"
    # Per-tenant issuer suffix: https://<slug>.hub.<env>.example.org
    OAUTHY_TENANCY_PLATFORM_DOMAIN: ".hub.<env>.example.org"
 
identity:
  publicBaseUrl: "https://identity.<env>.example.org"
 
networkPolicy:
  enabled: true        # only once you've confirmed the CNI enforces NetworkPolicy

The image tag is passed at deploy time, not baked into the overlay: --set global.image.tag=<sha>. See Configuration reference for the full set of properties, and the image-tag contract on the upgrade page for why the SHA is mandatory.

Step 3 — Deploy

A deploy is one command. Against your own overlay:

helm upgrade --install thoryn ./deploy/helm/thoryn \
  --namespace thoryn --create-namespace \
  -f deploy/helm/thoryn/values-<env>.yaml \
  --set global.image.tag="$(git rev-parse HEAD)" \
  --timeout 15m --cleanup-on-fail

scripts/deploy.sh wraps this for the staging cluster (it also unlocks a stuck Helm release and runs the smoke tests). Note that the Vault-token handling in that script is the dev-mode rollback path; the default backend is OpenBao, whose bootstrap runs as a Helm hook and needs no token passed in.

What the deploy does, in order

Two Helm post-install/post-upgrade hooks run before the application pods settle, then each service migrates its own schema on boot:

1. Ensure databases (postgres-ensure-databases-job, hook-weight -5). Waits for Postgres to be ready, then CREATE DATABASE for each entry in postgres.databases that does not yet exist. This is what lets you add a database to an existing cluster — the image's first-boot init script only runs against an empty data directory.
2. OpenBao init + unseal + provision (openbao-init-job, hook-weight 0). On an uninitialised store it runs bao operator init (1 share / 1 threshold), captures the unseal key + root token, and writes them to the thoryn-openbao-keys Secret via the Kubernetes API — the only moment that material exists in plaintext. It then unseals and provisions the Transit engine, the signing keys, the KV v2 mount, the Kubernetes auth method, and the per-consumer policies/roles. On a re-run it reads the material back from the Secret and every provisioning call is an idempotent "already exists" skip. Per-pod unseal init-containers unseal each OpenBao replica on restart, so a pod roll is a non-event.
3. Application pods start and each Spring service runs Flyway on boot against its database, applying any new V*__*.sql migrations. A service does not report ready until its context (and therefore its migrations) has come up. Services that authenticate to OpenBao carry a wait-for-vault-role init container that blocks the JVM until the service's Kubernetes-auth role is bound, so a service never crash-loops past an not-yet-provisioned OpenBao.

The first-boot ordering — Postgres up → databases ensured → OpenBao initialised and unsealed → app pods migrate and become ready — is what makes a from-scratch deploy converge instead of racing.

Source: deploy/helm/thoryn/templates/postgres-ensure-databases-job.yaml, .../openbao-init-job.yaml, .../_helpers.tpl (thoryn.waitForVaultRoleInitContainer), scripts/deploy.sh.

Step 4 — DNS and TLS

Point DNS at your ingress controller's load balancer, then let the chart wire TLS:

• Per-host TLS via cert-manager: set <service>.ingress.tls: true and a cert-manager.io/cluster-issuer annotation (HTTP-01). The hub, identity-service, and api-gateway ingresses follow this shape.
• Wildcard-subdomain TLS (needed for per-tenant issuers https://<slug>.hub.<env>): set hub.ingress.wildcardHost and provision a DNS-01 solver — the staging overlay uses the Hetzner DNS-01 solver and an explicitly-managed Certificate (it deliberately omits the cluster-issuer annotation on that ingress so ingress-shim does not create a duplicate Certificate that deadlocks issuance).
• Every ingress inherits modern-TLS + HSTS + HTTP→HTTPS hardening from the thoryn.ingressTlsAnnotations helper.

Multi-tenancy makes each tenant its own OIDC issuer at https://<slug>.hub.<platform-domain>; the wildcard host and OAUTHY_TENANCY_PLATFORM_DOMAIN are what make those subdomains resolve and validate. See Multi-tenancy.

Step 5 — Verify from scratch

Do not trust "the pods are up". Verify the operations you care about actually succeed — the platform's standing rule is that for anything depending on OpenBao policy, Helm/k8s wiring, or a multi-service flow, "it rendered" and "CI is green" are not validation.

# 1. Every first-party rollout is complete.
for d in thoryn-hub thoryn-identity thoryn-product-api thoryn-api-gateway; do
  kubectl rollout status deployment/$d -n thoryn --timeout=10m
done
 
# 2. The OpenBao init hook completed (keys provisioned).
kubectl logs -n thoryn job/thoryn-openbao-init --tail=20
 
# 3. OIDC discovery answers on the public issuer.
curl -fsS https://hub.<env>.example.org/.well-known/openid-configuration | jq .issuer

Then run the hub smoke test, which asserts discovery, JWKS, health, and the token endpoint respond correctly and within latency thresholds (a slow token endpoint is a failure, not just a slow success):

BASE_URL="https://hub.<env>.example.org" bash scripts/smoke-test.sh

A from-scratch deploy is "done" only when this passes against a clean install — not when a live cluster was patched into shape.

What the chart deploys

Infra components (Postgres/Redis/OpenBao) are image-digest-pinned and keep a stable pod template, so they do not roll on a routine code deploy — only app services surge-roll. See docs/modules/ROOT/pages/operator/deploy-rollout-model.adoc.

Troubleshooting

Security notes

• The four required values are security gates, not conveniences. Three of them (*.publicBaseUrl) close Host-header injection into user-facing URLs; the fourth pins an immutable, reproducible artefact. The render failing is the intended behaviour.
• OpenBao holds every signing key. The unseal key lives in the RBAC-scoped thoryn-openbao-keys Secret (etcd-at-rest). Production hardening should move to a KMS/HSM auto-unseal seal so no operator-readable unseal key exists — this is a documented later phase, not yet implemented; see ADR adrs/2026-06-07-vault-to-openbao.md and docs/modules/ROOT/pages/operator/openbao-cutover.adoc.
• Redis is TLS-only and per-service ACL'd. Passwords are per-service Kubernetes Secrets rendered into the ACL at deploy time, never in application.yml.
• Deploy only immutable SHAs. :latest is forbidden in the publish pipeline and at deploy time; see Upgrade → the immutable image-SHA contract.