OpenBao raft HA + OTC KMS auto-unseal (production, gated)

This page documents the gated, default-off chart scaffold added under SSO-2182 for Thoryn's own managed-OTC production high-availability posture, and — just as importantly — the careful operational steps the scaffold deliberately does not perform. Per the accepted design ADRs (adrs/2026-07-25-managed-otc-production-ha.md and adrs/2026-07-24-production-high-availability.md), OpenBao is the one stateful tier that stays in-cluster even on managed OTC CCE — there is no managed-Transit equivalent — running integrated raft for its own HA and delegating only the unseal to OTC / T Cloud KMS via the first-party tcloudpublic seal plugin.

Security-critical — this is the unseal mechanism. Flipping any flag below changes how the platform's cryptographic custody layer boots. The scaffold ships inert: with vault.raft.enabled and vault.seal.tcloudpublic.enabled both false (the defaults) the rendered OpenBao config is byte-identical to the current SSO-1466 file-backend + Shamir / K8s-Secret unseal path. The live cutover is a separately staging-validated ops step, not a chart flip. Do not enable these in the staging or self-managed defaults.

What the scaffold renders (and what it does not)

Rendered by the chart today (gated, default off)

The credential values never live in a values file or the config.hcl ConfigMap. The Secret is created out-of-band (--set-string from a protected environment, or a sealed-secret) and the three fields are injected into the OpenBao pod as env vars only. A missing Secret name aborts helm template / helm upgrade at render time — there is no operator-readable unseal key on the KMS path, so a missing credential means a pod that can never auto-unseal, and failing loud at render is the correct posture (the CLAUDE.md fail-loud-secrets rule).

Deliberately NOT performed by this scaffold

These are the careful parts — each lands under the separately staging-validated cutover, not this PR:

1. The OpenBao image is not rebuilt. The tcloudpublic plugin must be bundled + registered in the image before the seal works (see below). The chart still references the stock digest-pinned openbao/openbao image.
2. The Deployment is not converted to a StatefulSet. Raft HA needs stable per-pod identity, a headless peer Service, and per-pod PVCs. The scaffold renders only the storage "raft" stanza; the workload stays a single-replica Deployment (a 3-replica Deployment on one ReadWriteOnce PVC would not schedule, which is why vault.raft.replicas does not drive the Deployment replica count here).
3. No seal-migration or raft-migration is run. Migrating a live file-backend + Shamir store to raft + KMS auto-unseal is an ordered, forward-only, rehearsed operation validated live as the real service identity — never a chart flip.
4. The unseal orchestration is unchanged. On the KMS path the per-pod openbao-unseal sidecar and the operator init-generated unseal key in openbao-init-job.yaml are replaced by KMS auto-unseal (recovery keys, not an unseal key). That rewrite is part of the validated cutover.

(i) Bundle the kms-tcloudpublic plugin into the OpenBao image

The tcloudpublic seal is an external plugin (kms-tcloudpublic, shipped in openbao/openbao-plugins); the stock openbao/openbao binary does not carry it. Two version facts from the ADR gate this:

• The tcloudpublic seal requires OpenBao >= v2.6.0 (auto-unseal became pluggable in that release). The chart is pinned to openbao/openbao:2.5.4 today, so the version bump is part of this work.
• In v2.7.0 the built-in cloud KMS seals (awskms / azurekeyvault / gcpckms / ocikms / alicloudkms / pkcs11) become external-plugin-only, while transit stays built-in — relevant to the fallback below.

Build a thin image FROM openbao/openbao:<version >= 2.6.0> that adds the plugin binary into the plugin directory and registers it, then digest-pin it the same way the base image is pinned (scripts/check-infra-image-digests.sh enforces the @sha256: suffix). Sketch:

FROM openbao/openbao:2.6.x@sha256:<pin>
# Copy the tcloudpublic seal plugin into the plugin directory and register it in
# the image's plugin catalog. The exact plugin filename, the plugin_directory,
# and the registration command are confirmed against openbao/openbao-plugins
# during the staging validation (the plugin is v0.0.1).
COPY kms-tcloudpublic /openbao/plugins/kms-tcloudpublic
# plugin_directory = "/openbao/plugins" in config.hcl, or `bao plugin register`.

Maturity caveat. The plugin is v0.0.1 (upstream marks it experimental). Treat every mechanism here — the env-var names the plugin reads for key_id / access_key / secret_key, the exact registration command, the config field names — as confirmed by the staging validation gate, not assumed. The env-var names the chart injects (BAO_SEAL_TCLOUDPUBLIC_KEY_ID / _ACCESS_KEY / _SECRET_KEY) are a documented placeholder shape; adjust them to whatever the plugin build actually reads.

(ii) The staging validation gate (the gating deliverable)

Before any production use, run this validation on OTC CCE as the real service identity (the CLAUDE.md validate-live-before-cutover rule). This is the deliverable that gates the launch — not "the chart renders" and not "CI is green".

1. Seal reachability. Deploy the plugin-bundled image with vault.seal.tcloudpublic.enabled=true and a real credentialsSecret. Confirm the OpenBao pod's egress reaches OTC KMS (the allow-egress-openbao-kms NetworkPolicy allows it when the egress baseline is on).
2. Fresh-init auto-unseal. On a fresh store, bao operator init must auto-unseal via KMS and return recovery keys (not an unseal key). Confirm the pod comes up unsealed with no per-pod unseal sidecar.
3. Restart auto-unseal. Delete the pod; confirm it restarts and auto-unseals without manual intervention — this is the mechanism that removes the SSO-1724 sealed-cascade class of outage.
4. Raft standby join. Bring up the raft replicas (StatefulSet), confirm a follower joins quorum via retry_join, and confirm a bao operator raft snapshot succeeds (re-enabling the native snapshot backup the file backend could not take — vault-snapshot-cronjob.yaml).
5. Seal migration. Rehearse the forward-only migration from the current file-backend + Shamir store to raft + KMS auto-unseal (bao operator unseal -migrate / the seal-rewrap flow), keeping the file PVC as a rollback anchor.
6. Exercise the real Transit operations as each consuming service identity (kubectl create token <sa> then bao login then a real sign / verify / encrypt round-trip) — proving the key material survived the migration, per the ADR.

Only when steps 1-6 pass against the real deployed system is the production cutover authorized.

(iii) Fallback — Transit seal-OpenBao (mature, always available)

If the v0.0.1 tcloudpublic plugin proves unready at the launch date, fall back to Transit auto-unseal via a small dedicated "seal-OpenBao" that holds the cluster's unseal key. The transit seal stays built-in through OpenBao v2.7.0, so this path needs no external plugin. It still removes the operator-readable-unseal-key custody risk and the SSO-2129 restore-ordering exposure (the AES-256-GCM at-rest key ordering constraint is unchanged). A PKCS#11 seal against an OTC Dedicated HSM (Data Encryption Workshop) is a possible future hardware-backed compliance path, but its GA on T Cloud Public is unconfirmed, so it is not the launch choice.

What this posture buys (per the ADRs)

• No SPOF / no key loss — raft replicates the Transit key material across quorum members, so a single pod/node loss neither seals the platform nor loses keys, and it re-enables the native raft-snapshot backup that the file backend could not take.
• No sealed-cascade — KMS auto-unseal removes the manual-unseal failure mode that kept the store sealed for 6h in the SSO-1724 staging outage; a restarted pod auto-unseals.
• No operator-readable unseal key — the unseal is delegated to OTC KMS, so there is no thoryn-openbao-keys-style unseal Secret to custody or to restore in the right order (the SSO-2129 restore-ordering risk is reduced; the four AES-256-GCM at-rest keys still have their ordering constraint).

Related

• adrs/2026-07-25-managed-otc-production-ha.md — the ratified managed-OTC posture and the "Resolved — OTC KMS auto-unseal compatibility" spike result.
• adrs/2026-07-24-production-high-availability.md — the self-managed in-cluster HA design (raft 3-node + KMS/HSM or transit auto-unseal) this scaffold shares.
• adrs/2026-06-07-vault-to-openbao.md — the OpenBao seal/unseal + Transit custody boundary this extends.
• docs/modules/ROOT/pages/operator/openbao-cutover.adoc — the current file-backend cutover + unseal-key recover/rotate runbook (the posture this replaces in production).