Guide G8 · DevOps path
Secrets Management: Vault, Dynamic Secrets, and Rotation
Where secrets should live and how they reach workloads: AWS Secrets Manager vs Parameter Store, HashiCorp Vault (KV, dynamic database credentials, Kubernetes auth), injection patterns, automatic rotation, envelope encryption with KMS, and short-lived credentials everywhere.
Start here
The mental model
A static password is a house key copied to everyone who ever needed it: after a year nobody knows how many copies exist. Good secrets management moves towards HOTEL KEY CARDS: issued on demand to one person, programmed for one room, expiring automatically at checkout, and every issue is logged.
The maturity ladder: secrets in code (never) → secrets in a vault, fetched at runtime → secrets ROTATED automatically → DYNAMIC secrets created per workload with a short lease → no secrets at all, only workload identity (IAM roles, OIDC, SPIFFE). Every step up shrinks what a leak can do and for how long.
Go deeper
How it works inside
01The AWS options
SSM PARAMETER STORE: hierarchical key-value (/shoplite/prod/db/host), SecureString values encrypted with KMS, free standard tier, no built-in rotation. Great for configuration and simple secrets. SECRETS MANAGER: built for secrets, with managed ROTATION (Lambda-based, native for RDS/Aurora/Redshift/DocumentDB), cross-region replication, resource policies, and ~$0.40 per secret per month. RDS can even manage the master password in Secrets Manager for you (manage_master_user_password). Workloads read both through IAM (Pod Identity/IRSA, instance roles), so there are no credentials to fetch the credentials (AWS course, secrets and KMS).
02HashiCorp Vault
Vault is a secrets platform that runs anywhere (self-hosted, or HCP Vault). Clients AUTHENTICATE with an auth method (Kubernetes service account token, AWS IAM, OIDC for humans, AppRole for machines), receive a TOKEN with POLICIES attached, and use SECRETS ENGINES: KV v2 (versioned static secrets), DATABASE (dynamic credentials), AWS (temporary IAM credentials), PKI (issue certificates), and TRANSIT (encryption as a service). Every request is written to an audit log. OpenBao is the open-source fork maintained under the Linux Foundation.
Operationally, Vault starts SEALED: its storage is encrypted and it can't read its own data until unsealed. Production uses AUTO-UNSEAL with a cloud KMS key, runs 3–5 nodes with integrated Raft storage (Stateful Systems course, Unit 0.1), and needs backups (Raft snapshots) and DR planning like any critical database.
03Dynamic secrets
Instead of one shared database password, Vault's database engine CREATES a new database user for each requester, with a LEASE (e.g. 1 hour). When the lease expires or is revoked, Vault deletes the user. Consequences: every pod has its own credentials (audit shows exactly who ran a query), a leaked credential stops working on its own, and revoking access for one compromised workload doesn't affect others. The application (or a sidecar) must renew leases or re-read credentials; connection pools must handle credential changes (e.g. HikariCP maxLifetime shorter than the lease).
04Getting secrets into workloads
SYNC TO KUBERNETES SECRETS: External Secrets Operator (GitOps course, Mission 2.1) or the Vault Secrets Operator copy values into native Secrets. Simplest for apps, but the secret then also exists in etcd (enable KMS encryption of Secrets at rest on EKS). AGENT/SIDECAR INJECTION: Vault Agent Injector renders secrets into files in a shared in-memory volume and renews them, so values never touch etcd. CSI DRIVER: Secrets Store CSI mounts secrets as files from Vault, AWS Secrets Manager, Azure Key Vault, or GCP. DIRECT SDK: the app calls Vault or Secrets Manager itself (Spring Cloud Vault, Spring Cloud AWS), which is most flexible but couples code to the store. Prefer FILES over environment variables for sensitive values: env vars leak into crash dumps, /proc, child processes, and debug endpoints.
05Rotation and envelope encryption
ROTATION changes a secret on a schedule (and immediately after any suspected leak) without downtime. The standard pattern uses two valid credentials during the switch: create the new one, update the store, let consumers pick it up, then revoke the old one. Secrets Manager's rotation Lambdas implement this ('alternating users'). Test rotation in staging, because the first automated rotation is where apps that cache credentials forever break.
ENVELOPE ENCRYPTION is how KMS protects large data and secrets: a DATA KEY encrypts the data, and the KMS KEY (which never leaves KMS) encrypts the data key; you store the encrypted data key next to the ciphertext. Decrypting requires an IAM-authorised KMS call, which is logged in CloudTrail. It's what S3, EBS, RDS, Secrets Manager, and SOPS use under the hood.
Do it
Hands-on lab
- 1
Run Vault in dev mode and store a static secret
Dev mode is in-memory, unsealed, and uses a root token: for learning only, never production.
terminal$ docker run -d --name vault -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault:1.20export VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=rootvault kv put secret/shoplite/stripe api_key=sk_test_examplevault kv get -field=api_key secret/shoplite/stripevault kv metadata get secret/shoplite/stripe | grep current_version── expected output ──Success! Data written to: secret/data/shoplite/stripesk_test_examplecurrent_version 1 - 2
Dynamic PostgreSQL credentials
Point Vault at a Postgres it can administer, define a role with a creation statement and TTL, and request credentials. Each read returns a brand-new database user.
terminal$ docker run -d --name pg --link vault -e POSTGRES_PASSWORD=pw postgres:17vault secrets enable databasevault write database/config/shop plugin_name=postgresql-database-plugin allowed_roles=checkout \connection_url='postgresql://{{username}}:{{password}}@pg:5432/postgres' username=postgres password=pwvault write database/roles/checkout db_name=shop default_ttl=1h max_ttl=24h \creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"vault read database/creds/checkout── expected output ──Key Value--- -----lease_id database/creds/checkout/Vx3kF2...lease_duration 1hlease_renewable truepassword A1a-8sD0fQ...username v-token-checkout-8f2kQz1c-1790487120 - 3
Let pods authenticate with their service account
The Kubernetes auth method trusts service account tokens validated by the cluster; the role binds a service account in a namespace to a Vault policy.
vault-k8s-auth.shwhole filebash vault auth enable kubernetes vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_API" vault policy write checkout-db - <<'EOF' path "database/creds/checkout" { capabilities = ["read"] } EOF vault write auth/kubernetes/role/checkout \ bound_service_account_names=checkout \ bound_service_account_namespaces=shoplite-prod \ policies=checkout-db ttl=1h - 4
Turn on rotation in AWS Secrets Manager
For RDS, AWS provides the rotation function; you set the schedule. Then verify the app survives a rotation in staging.
terminal$ aws secretsmanager rotate-secret --secret-id shoplite/prod/db --rotation-lambda-arn arn:aws:lambda:ap-south-1:123456789012:function:SecretsManagerRDSPostgreSQLRotationMultiUser --rotation-rules '{"ScheduleExpression":"rate(30 days)"}'aws secretsmanager describe-secret --secret-id shoplite/prod/db --query '{Rotation:RotationEnabled,Last:LastRotatedDate}'── expected output ──{ "Rotation": true, "Last": "2026-09-28T04:12:09+05:30" }
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| Lease TTL / max TTL (Vault) | 32 days system default | How long dynamic credentials live and how long they can be renewed. | Short (1h) default TTLs with renewal; max TTL a day or less for database credentials. |
| Rotation schedule (Secrets Manager) | off | How often a secret is rotated. | 30–90 days for static secrets, plus on-demand after any exposure. |
| Kubernetes Secret encryption at rest | EKS: envelope encryption available | Encrypts Secrets in etcd with a KMS key. | Enable on every cluster that holds synced secrets. |
| Connection pool maxLifetime | 30 min (HikariCP) | How long a pooled DB connection lives. | Shorter than the credential lease so pools reconnect with fresh credentials. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
Rotation day outage
Secrets Manager rotates the prod database password for the first time. Ten minutes later, new connections from checkout start failing, and existing ones keep working until pods restart.
Drill #2
Vault sealed after a restart
All Vault pods restart during a node upgrade. Every service that fetches secrets at startup begins crash-looping.
Decide
Where to keep secrets
| Option | Strengths | Watch out for | Good for |
|---|---|---|---|
| SSM Parameter Store | Free tier, simple, KMS-encrypted SecureString | No managed rotation, throughput limits | Config and simple secrets on AWS |
| AWS Secrets Manager | Managed rotation (native for RDS), replication, IAM | Cost per secret, AWS-only | Database and API credentials on AWS |
| HashiCorp Vault / OpenBao | Dynamic secrets, PKI, transit encryption, multi-cloud, fine-grained policies | You operate a critical cluster (or pay for HCP) | Multi-cloud, dynamic credentials, large estates |
| Kubernetes Secrets alone | Native, simple | Base64 not encryption; in etcd; no rotation | Only as the delivery target of ESO/VSO, with encryption at rest |
| Sealed Secrets / SOPS | Encrypted secrets in Git | Rotation means re-encrypting; key management | Small GitOps setups |
The bigger picture
Connects to
System Design · Security
Authentication, authorization, OAuth2, JWT, mTLS, secrets, encryption — the seven walls of a real system.
DevSecOps · Secrets chapter
What happens when secrets leak, and how leaks are detected.
GitOps · Secrets in GitOps
ESO, Sealed Secrets, and SOPS in practice.
AWS · Secrets & KMS
Secrets Manager, Parameter Store, and KMS on AWS.
Kubernetes · Secrets management
How Secrets are stored and consumed in the cluster.
Prove it
Interview questions
What are dynamic secrets and why are they better than static credentials?
How do secrets get from a secret store into a Kubernetes pod?
Explain envelope encryption.