Signing & cryptographic custody

Every signature the platform produces — access tokens, ID tokens, and audit-row assertions — is computed by a secrets backend (Vault/OpenBao Transit), not in the JVM. Private key material never enters application memory. This page covers the signing algorithm and custody model, per-tenant HMAC, version-prefixed identifiers, and the verifiable audit trail those primitives back.

ES256, computed in the secrets backend

Hub-minted JWTs are signed with ES256 (ECDSA over P-256, SHA-256). The signing operation is delegated to Vault/OpenBao Transit through core/lib/signing/hashicorp-vault/HashiCorpVaultSigningStrategy.kt: the strategy builds the JWS signing input, calls the Transit sign endpoint, and assembles the compact JWS from the returned signature. It then immediately calls Transit verify on the result as a self-check before returning the token.

Keys never leave the backend. Transit performs the signing server-side and returns only the signature; the hub process never holds the private key. A JVM heap dump or memory-disclosure bug cannot leak signing authority — the contrast is with a keystore- or in-memory-keypair strategy where the private key lives in process memory from startup to shutdown.

ES256 was chosen over RS256 for token size (64-byte signatures vs. 256-byte), faster verification on every relying party and resource server, cheap rotation, and the broadest JWT library support. RS256 remains available in the codebase (VaultTransitKeyType.RSA_2048) for relying-party compatibility but is not the default. Cloud KMS was considered and rejected in favour of Transit's cross-cloud portability behind the same SigningStrategy seam.

Source: ADR 2026-04-26-jwt-signing-strategy.md; HashiCorpVaultSigningStrategy.kt.

Per-tenant keys and versioned kid

The signing key is selected per tenant from the active tenant context: tenant-{slug} for a named tenant, sas-ecdsa-jwt-key for the default tenant. Each key has its own Transit ACL, version history, and rotation schedule, so one tenant's key compromise does not touch another's.

Every JWS header stamps a kid of the form <keyname>-v<version>, taken from the key version that signed it. A scheduled VaultKeyRotationJob rotates keys in the backend; because the kid names the exact version, in-flight tokens keep verifying after a rotation. The hub's JWKS endpoint therefore surfaces every active key version, not just the latest — otherwise a token minted seconds before a rotation would fail verification at a relying party.

Source: HashiCorpVaultSigningStrategy.resolveKeyName(); VaultKeyRotationJob.kt; ADR 2026-04-26-jwt-signing-strategy.md → Consequences.

Availability trade-off (stated honestly)

Because signing is a Transit round-trip, token minting depends on the backend being available — a backend outage blocks new authorizations (surfaced as 503 at the token endpoint rather than a slow hang). Token verification is unaffected: relying parties verify offline against the cached JWKS. Signing latency is one round-trip per mint (sub-5 ms in-region; co-locate the backend with the hub out-of-region).

Per-tenant HMAC and version-prefixed identifiers

Any value derived from a tenant-scoped key is computed in Transit and stored version-prefixed as v<n>:<hex>, where v<n> is the backend key version:

• HMAC-SHA-256, not plain SHA-256, for audit content hashing. Plain hashing would make a digest a tenant-independent identifier — the same credential presented to two tenants would collide, giving a platform operator a cross-tenant linkability set. A per-tenant HMAC key neutralises that while preserving the same-tenant lookup affordance.
• The HMAC key lives in the backend (tenant-audit-hmac-<tenantId>), never in the JVM — a heap dump must not let an operator forge audit-row hashes. HMAC is a fresh Transit round-trip, not a locally-cached key.

The version prefix is what makes rotation non-destructive: rotating a tenant's key bumps the version, and historical rows still verify against the retired version (retained in the backend for the row's retention horizon) while new rows use the rotated key. Without the prefix, a rotation would either break every historical row's lookup or force a re-key migration over years of data.

Source: ADR 2026-05-09-verifiable-audit-chain.md → "Why per-tenant HMAC"; CLAUDE.md → "Cryptographic / signing pipeline rules"; VaultAuditChainCryptoClient.kt.

The verifiable audit trail

The customer-plane audit trail (servers/product-api/.../compliance/audit/) is a per-tenant, hash-chained, cryptographically-signed log — tamper-evident, not merely tamper-resistant.

Partitioned by mode (SSO-2105). With per-tenant test/sandbox mode, each tenant has two independent chains — one for live activity, one for test — keyed on (tenant_id, livemode). A test event chains only to the previous test event and a live event only to the previous live event; the two never interleave, so test activity can neither be counted by, nor fork the verifiability of, the live chain. livemode is a partition discriminator, not a signed field — it is deliberately not in the canonical bytes below, so the pre-existing live rows' HMAC + ES256 signatures stay byte-for-byte unchanged (no schema-version bump, no re-signing). Live audit export selects live rows only.

Append (AuditChainAppendService)

Appending one row is atomic per (tenant, mode):

1. A transaction-scoped Postgres advisory lock keyed on (tenant, livemode) (pg_advisory_xact_lock(hashtext(tenant_id), livemode)) serialises concurrent same-(tenant, mode) appends so chain_seq is gap-free and the prev_hash link is unambiguous. Different tenants — and a tenant's live vs. test chains — never contend, so a live append never waits on a test append.
2. The current head of that mode's chain sets the next sequence (head.chainSeq + 1) and prev_hash (head.rowHash, or a genesis sentinel for the first row).
3. The row is canonicalised (below) and its per-tenant HMAC row_hash and ES256 row_signature are computed in Transit.
4. The row is INSERTed inside the same transaction; the lock releases on commit.

Failure modes are deliberate. The HMAC row_hash is the structural chain link, so an HMAC failure aborts the append — the caller's event is not silently dropped, the exception propagates, and the caller (e.g. an outbox drainer) retries. The ES256 row_signature degrades gracefully: if signing fails (backend unreachable), the row is still written with a NULL signature plus a WARN and the productapi.audit.chain.row_unsigned_total counter. The design optimises for not losing audit data over no-unsigned-rows; verify treats a NULL-signature row as "hash-chain intact, signature unverifiable", distinct from tamper.

Canonical form (AuditChainCanonicalizer)

The bytes fed to both the HMAC and the JWS signer are a single-line, UTF-8, whitespace-free JSON object with a pinned field order and a leading schema-version field. Timestamps render as createdAtEpochMilli (a base-10 epoch-millis long — no timezone or fractional-second ambiguity); null fields render as the literal null (never omitted, so presence can't shift the byte layout); strings use RFC 8259 minimal escaping. The encoding is pinned explicitly rather than delegated to a JSON library whose key ordering could shift across versions. The prev_hash is inside the signed bytes — signing the previous link inside each row is what makes the chain tamper-evident rather than a set of independently-signed rows.

Verify (AuditChainVerifier)

Walking a tenant's chain in chain_seq order, the verifier independently for each row:

1. re-derives the canonical bytes from the persisted content columns (it trusts no stored canonical blob — a mutated row produces different bytes);
2. recomputes the per-tenant HMAC and asserts it equals the stored row_hash;
3. asserts the link — prev_hash equals the previous row's row_hash, and the sequence is contiguous;
4. verifies the ES256 signature over the same bytes against the public key for the row's signing_kid.

The first failing row is the divergence point, classified as MALFORMED_ROW, SEQUENCE_GAP, BROKEN_LINK, HASH_MISMATCH, or BAD_SIGNATURE. A present-but-invalid signature (BAD_SIGNATURE, i.e. forgery) is distinct from a NULL signature (UNSIGNED, backend was down at append time). The response surfaces only the row id and sequence — never the tampered content — per the error-sanitisation rule.

Long-lived verification without a live JWKS

An audit row may be verified years after it was written, long after its signing-key version has rotated out of the live set. Retired versions stay verifiable because Vault/OpenBao Transit retains every key version: the verifier reads the signing key's public material for the exact version named by the row's kid (v<version>:<keyname>) and checks the signature locally. This is the "historical-jwks" principle — long-lived signed artefacts must verify against retired kids — realised through backend key-version retention and a kid pinned inside the signed bytes (so it can't be swapped after signing).

Scope note. This retired-version verification is the product-api audit chain's mechanism. It is distinct from the hub's live-token JWKS endpoint (which surfaces active versions for hot-path token verification). The dossier does not claim a separate published historical-jwks HTTP endpoint in this repository.

Source: AuditChainAppendService.kt, AuditChainVerifier.kt, AuditChainCanonicalizer.kt, VaultAuditChainCryptoClient.kt (all servers/product-api/.../compliance/audit/); ADR 2026-05-09-verifiable-audit-chain.md.

Deletion detection (out of scope)

The chain is tamper-evident against mutation and reordering, and a contiguous walk detects a removed row as a SEQUENCE_GAP / BROKEN_LINK. An independent Merkle-anchor scheme that would prove no row was ever excised is a documented future option, not shipped — the ADR is honest about this gap rather than implying coverage.

The secrets backend: Vault → OpenBao

The cryptographic custody layer is the Transit secrets engine, reached through spring-vault / VaultTemplate. The platform is migrating this backend from HashiCorp Vault to OpenBao (the Linux Foundation's MPL-2.0 fork) on persistent storage with auto-unseal. Transit is API- compatible, so the core/lib/signing/ code, per-tenant key paths, and version-prefixed identifiers are unchanged — core/lib/signing/ stays the vendor seam, so the platform is not locked to either backend.

Production seal — a self-managed responsibility. Staging stores the OpenBao unseal key in a Kubernetes Secret. That model is explicitly not promotable to production: production must use a KMS/HSM auto-unseal seal (seal "awskms" / seal "pkcs11") so no operator-readable unseal key exists. A self-managed operator owns this step.

Source: ADR 2026-06-07-vault-to-openbao.md.