The orders database password started in a .env file. Then it was copied into a CI variable for migrations, a Kubernetes Secret for the service and two laptops during an incident. Two years later nobody knows every copy, so nobody dares to rotate it.
GitGuardian counted 28.65 million new hardcoded secrets in public GitHub commits in 2025, up 34% in a year, and over 64% of the secrets it confirmed as valid in 2022 still worked in January 2026 (State of Secrets Sprawl 2026).
This post builds a secrets management architecture without long-lived database passwords: workload identity, short-lived dynamic credentials, KMS envelope encryption, rotation and audit, with a diagram of the flow.
Where static secrets go wrong
Every copy of a static secret is a place it can leak from:
- Git history keeps the value after you delete the line.
- CI variables can be read by anyone who can change a job that receives them.
- Environment variables are visible to every process in the container and end up in logs and crash dumps. OWASP's secrets cheat sheet advises against them unless nothing else works.
- Kubernetes Secrets are stored unencrypted in etcd by default, and anyone who can create a pod in a namespace can read every Secret there, as the Kubernetes documentation warns.
A secrets manager fixes storage, not bootstrap: the app still needs a credential to fetch its credential. The fix is a platform-issued identity, traded for credentials that expire on their own.
The building blocks
| Component | Job | Examples |
|---|---|---|
| Secrets manager | Stores and issues secrets, enforces policy, tracks leases | HashiCorp Vault, OpenBao, AWS Secrets Manager, Google Secret Manager, Azure Key Vault |
| KMS or HSM | Holds key-encryption keys that never leave it | AWS KMS, Google Cloud KMS, Azure Key Vault Managed HSM |
| Workload identity | Proves which workload is calling, with short-lived platform tokens | Kubernetes service account tokens, EKS Pod Identity, Workload Identity Federation for GKE, Microsoft Entra Workload ID |
| Delivery | Gets the secret into the process | Vault Agent, Secrets Store CSI Driver, External Secrets Operator, SDK calls |
| Audit | Records who read what, and when | Vault audit devices, AWS CloudTrail, a SIEM |
Envelope encryption with KMS
A KMS is not built for bulk data; AWS KMS Encrypt takes at most 4,096 bytes. Envelope encryption splits the job:
GenerateDataKeyreturns a unique data key twice: in plaintext and encrypted under your KMS key, the key-encryption key (KEK).- Encrypt the data locally with the plaintext key, then erase it from memory.
- Store the encrypted data key next to the ciphertext. To read, send it to
Decryptand decrypt locally.
aws kms generate-data-key --key-id alias/orders-data --key-spec AES_256
The KEK never leaves the KMS unencrypted, so every use is an API call you can authorize and log (AWS KMS cryptography details). Automatic rotation, once enabled, runs yearly by default and changes only the key material for new encryptions; old ciphertext still decrypts.
The pattern appears twice here. The Kubernetes KMS provider wraps the data keys that encrypt Secrets in etcd; use KMS v2, since v1 is disabled by default from Kubernetes 1.29. Vault's keyring is encrypted by a root key, and with auto-unseal a cloud KMS decrypts that root key when Vault starts.
How a pod proves who it is
Every pod already has an identity, its service account, which also backs workload identity in a service mesh. With the Vault Kubernetes auth method:
- The kubelet requests a projected service account token with audience
vaultthrough the TokenRequest API. It lasts one hour by default (10 minutes minimum) and is refreshed at 80% of its TTL. - Vault Agent logs in with the token and a Vault role.
- Vault validates the token with the Kubernetes TokenReview API. A bound token becomes invalid when it expires or its pod is deleted.
- Vault checks the service account, namespace and audience against the role and returns a token with its policies.
vault write auth/kubernetes/role/payments \
bound_service_account_names=payments-api \
bound_service_account_namespaces=payments \
audience=vault policies=payments-db token_ttl=1h
The token is a bearer credential, and the Vault Kubernetes auth documentation warns against sharing it with other services. A dedicated audience makes services that check aud reject it.
The workload identity services in the table exchange a projected service account token for short-lived cloud credentials, so no cloud key is stored. GitHub Actions OIDC does the same for deploy jobs in a CI/CD pipeline. The JWT authentication flow explains how such tokens are verified.
Dynamic database credentials
A dynamic secret is created on request and destroyed when its lease ends. With the Vault database secrets engine, each read of database/creds/payments creates a new PostgreSQL user:
vault write database/roles/payments db_name=orders \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl=1h max_ttl=24h
Leases can be renewed up to max_ttl. Vault Agent renews renewable secrets after two thirds of the lease and re-fetches non-renewable leased ones at 90% of their TTL. When a lease expires or is revoked, Vault runs the revocation statements, by default a generic drop-user statement. Without a TTL, Vault's system default of 768 hours (32 days) applies.
What you gain:
- One user per pod, so a suspicious query points to one lease, not to every service sharing a password.
- Leaks expire, at most an hour after issue unless the lease is renewed.
- One-command response:
vault lease revoke -prefix database/creds/paymentsrevokes every active credential from the role.
One catch: VALID UNTIL expires the password, which matters at login. Give pooled connections a maximum lifetime shorter than the lease (HikariCP's default is 30 minutes) and reload the credentials when Vault Agent rewrites the file.
Delivery options and trade-offs
| Delivery | How it works | New values | Watch out for |
|---|---|---|---|
| Environment variables | Set when the container starts | After a restart | Visible to every process; leak into logs and dumps |
| Synced Kubernetes Secret | External Secrets Operator (hourly refresh by default) or Vault Secrets Operator copies values into a Secret | Mounted volumes update; environment variables do not | A copy in etcd: enable encryption at rest and tight RBAC |
| Agent sidecar | Vault Agent Injector renders secrets to an in-memory volume at /vault/secrets |
Yes, if the app re-reads the file | A sidecar and an init container in every pod |
| CSI volume | Secrets Store CSI Driver mounts secrets when the pod starts | Only with alpha auto-rotation, off by default | Optional sync to a Secret brings back the etcd copy |
| SDK call | The app calls the secrets manager API | Yes, in code | Auth, caching and retries in every service |
For dynamic credentials, agent-rendered files fit best: the agent handles login, renewal and re-rendering. Use synced Secrets when Helm charts expect one or when you keep secrets in a GitOps repository as references, and the SDK outside Kubernetes.
Rotation without downtime
Rotation has one rule: the new credential must work before the old one stops.
- Dynamic credentials rotate by design, as long as consumers reload in time.
- Static database passwords need a rotation job. In AWS Secrets Manager, the single-user strategy keeps open connections but can deny calls briefly, so clients need retries; the alternating-users strategy switches between two users, so a client that fetched the secret mid-rotation still holds valid credentials. Rotation can run as often as every four hours.
- API keys: deploy the new key, confirm traffic has moved, then revoke the old one.
Audit logs and break-glass access
Vault sends an entry for every request and response to all enabled audit devices and refuses requests that no device can record, so HashiCorp recommends enabling at least two. Most strings are logged as HMAC-SHA256 hashes, and /sys/audit-hash computes the hash of a known value so you can search for it. Ship it to your SIEM and alert on root token use and policy changes.
Break-glass access should be boring and tested:
- No standing root token. HashiCorp recommends root tokens only for setup or emergencies, revoked right after.
vault operator generate-rootneeds a quorum of unseal or recovery key holders. - Know your hard dependencies. With auto-unseal, recovery keys cannot decrypt the root key, so Vault cannot unseal without its KMS key. AWS KMS waits 7 to 30 days before deleting a key; alert on every scheduled deletion.
The secrets flow diagram
Diagram contents
- Kubernetes API server
- Pod: payments-api
- Projected SA token
- Vault Agent (sidecar)
- App container
- Cloud KMS key
- Vault: auth, policies, leases
- Vault audit log to SIEM
- orders DB (PostgreSQL)
- 1 TokenRequest via kubelet
- read JWT
- 2 login: JWT + role payments
- 3 TokenReview
- 4 read database/creds/payments
- 5 CREATE ROLE ... VALID UNTIL
- 6 render /vault/secrets/db
- 7 connect as the short-lived user (TLS)
- 8 lease ends: DROP ROLE
- at startup: decrypt root key
- every request and response
How to read it:
- Left: the pod frame with the projected token, the Vault Agent sidecar and the app, under the Kubernetes API server.
- Purple edges 1 to 3, identity: TokenRequest (1), login with role payments (2), TokenReview (3).
- Teal edges 4 to 7, the credential: read
database/creds/payments(4),CREATE ROLE ... VALID UNTIL(5), render/vault/secrets/db(6), connect over TLS (7). - Red dashed edge 8: the lease ends and Vault drops the user.
- Right: KMS decrypts Vault's root key at startup; the audit log receives every request and response.
- The Markdown block lists every lifetime.
Drawing secrets management diagrams clearly
- Number the steps and color them by phase: identity, delivery, revocation.
- Write lifetimes and audiences on the diagram. "aud vault, 1h" invites review; "short-lived" invites argument.
- Show where plaintext exists: memory, in-memory volumes, environment variables, etcd.
- Draw revocation and audit paths, not only the happy path.
- Never put real keys, tokens or internal hostnames in a diagram.
- Leave Vault's storage and cluster internals to an operations diagram.
The threat modeling data flow diagram adds trust boundaries, the zero trust model shows the wider picture of access, and the Kubernetes architecture diagram covers the API server and kubelet.
Build it in Archboard
- Sign in with Google or GitHub and create a diagram.
- Add a Frame from the Core pack named Pod: payments-api. Inside it, stack a Token from the Security pack and two Container nodes from the Docker pack: Vault Agent (sidecar) and App container.
- Above the frame, add a Kubernetes Cluster from the Cloud pack labeled Kubernetes API server.
- Add a Vault from the Security pack in the middle. On the right, add a Cloud resource from the Cloud pack for the KMS key, Logs from the DevOps pack for the audit log and Postgres from the Data pack.
- Draw edges 1 to 8 with block arrows (purple for identity, teal for the credential, red dashed for DROP ROLE), plus the startup edge to the KMS key and a dashed edge to the audit log.
- Add a Markdown block from the Core pack with the lifetimes, and two Core Text annotations for the Vault role and the database user.
The Kubernetes YAML to diagram tool turns existing manifests into a first draft. The example uses 10 diagram nodes, within the Free plan.
FAQ
What is the difference between a secrets manager and a KMS? A KMS holds keys that never leave it and encrypts small items such as data keys. A secrets manager stores and issues the secrets themselves, such as passwords and API keys, often protected by a KMS.
Are Kubernetes Secrets encrypted? Not by default. Enable encryption at rest for etcd, ideally with the KMS v2 provider, and restrict RBAC: anyone who can create pods in a namespace can read its Secrets.
What are dynamic secrets? Credentials created on request and deleted when their lease ends, such as the one-hour database users that Vault's database secrets engine creates per request.
Should I pass secrets as environment variables? Avoid it when you can: every process in the container can read them, they leak into logs and crash dumps, and they change only on restart. Agent-rendered files are easier to protect and rotate.
How often should secrets be rotated? Dynamic credentials expire within hours and need no schedule. Rotate static secrets on a schedule and after any suspected exposure or staff change, and test rotation in staging first.