Mission 2.1 · Stage 2 — GitOps at Scale: Secrets, Rollouts, and Many Clusters
Secrets in GitOps: Sealed Secrets, SOPS, and External Secrets
Goal: ShopLite's Redis password and DB credentials reach the cluster without ever sitting in Git as plain text.
By the end of this mission
- Explain why a Kubernetes Secret in Git is not secret
- Compare the three standard approaches and choose one
- Encrypt a Secret for Git with Sealed Secrets
- Sync secrets from AWS Secrets Manager with the External Secrets Operator
Part 1
Understand it first
The problem
A Kubernetes Secret's data is base64, which is an encoding, not encryption; anyone who can read the repo can decode it. And Git history is forever: a secret committed once and 'deleted' in the next commit is still in history and in every clone (DevSecOps course, secrets chapter). So GitOps needs a way to put a REFERENCE or CIPHERTEXT in Git and produce the real Secret only inside the cluster.
Three approaches
SEALED SECRETS (Bitnami): a controller in the cluster holds a private key; you encrypt with its public key using kubeseal, commit the SealedSecret, and only that cluster can decrypt it. Simple and self-contained, but the key is per cluster (back it up!) and rotating the actual secret still means re-sealing.
SOPS (+ age or KMS): encrypts only the VALUES in YAML files, leaving keys readable so diffs stay meaningful. Argo CD needs a plugin (or Flux's built-in support) to decrypt at render time. Good when you want encrypted files in Git with cloud KMS access control.
EXTERNAL SECRETS OPERATOR (ESO): Git holds only an ExternalSecret saying 'fetch key shoplite/prod/db from AWS Secrets Manager'. The operator reads it (using IRSA/Pod Identity, no static keys) and creates the Kubernetes Secret, refreshing on an interval. Secrets are managed and rotated in the secret manager; nothing sensitive ever touches Git. This is the most common choice on cloud platforms, and what ShopLite uses in prod.
Part 2
Your project after this mission · 5 files change
- shoplite-gitops/
- apps/
- web/
- overlays/
- dev/
- redis-auth.sealed.yamlnew
- prod/
- db-secret.yamlnew
- argocd/
- apps/
- external-secrets.yamlnew
- sealed-secrets.yamlnew
- platform/
- external-secrets/
- cluster-store.yamlnew
Part 3
Build it, step by step
- 1
Install both controllers through Argo CD
Platform components are Applications too, living in the same repo. Here's ESO; sealed-secrets is identical with chart
sealed-secretsfromhttps://bitnami-labs.github.io/sealed-secretsintokube-system. They go in aplatformproject in real setups.shoplite-gitops/argocd/apps/external-secrets.yamlwhole fileyaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: external-secrets namespace: argocd annotations: argocd.argoproj.io/sync-wave: "-10" # before apps that need secrets spec: project: default source: repoURL: https://charts.external-secrets.io chart: external-secrets targetRevision: 0.20.1 helm: valuesObject: installCRDs: true destination: server: https://kubernetes.default.svc namespace: external-secrets syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [CreateNamespace=true, ServerSideApply=true] - 2
Seal the dev Redis password
Create the Secret locally with
--dry-run(it never reaches the cluster or disk as plain text), pipe it throughkubeseal, and commit the output. The ciphertext is bound to this name and namespace, so copying it elsewhere won't decrypt.terminal$ kubectl create secret generic redis-auth -n shoplite-dev \--from-literal=redis-password="$(openssl rand -hex 16)" --dry-run=client -o yaml \| kubeseal --format yaml > apps/web/overlays/dev/redis-auth.sealed.yamlgrep -A2 encryptedData apps/web/overlays/dev/redis-auth.sealed.yaml── expected output ──encryptedData:redis-password: AgBy3i4OJSWK+PiTySYZZA1rO43cGDEq... - 3
Connect ESO to AWS Secrets Manager
A
ClusterSecretStoresays where secrets live and how to authenticate. On EKS the operator's service account is bound to an IAM role via IRSA or Pod Identity (AWS course, EKS topic), allowedsecretsmanager:GetSecretValueonly onshoplite/*. No access keys anywhere.shoplite-gitops/platform/external-secrets/cluster-store.yamlwhole fileyaml apiVersion: external-secrets.io/v1 kind: ClusterSecretStore metadata: name: aws-secrets-manager spec: provider: aws: service: SecretsManager region: ap-south-1 auth: jwt: serviceAccountRef: name: external-secrets namespace: external-secrets - 4
Reference the prod DB secret
This file is safe to commit: it contains names, not values. ESO creates a normal Secret
web-dbthat the Deployment mounts as usual.refreshIntervalpicks up rotations; pods need a restart (or Reloader) to see new env values.shoplite-gitops/apps/web/overlays/prod/db-secret.yamlwhole fileyaml apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: web-db spec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: aws-secrets-manager target: name: web-db creationPolicy: Owner data: - secretKey: DB_PASSWORD remoteRef: key: shoplite/prod/db property: password - secretKey: DB_USER remoteRef: key: shoplite/prod/db property: username - 5
Verify
Add the new file to the prod overlay's
resources, push, and check the ExternalSecret's status.SecretSyncedmeans the Secret exists in the cluster.terminal$ git add . && git commit -m 'secrets: sealed (dev) + ESO (prod)' && git pushkubectl -n shoplite-prod get externalsecret web-dbkubectl -n shoplite-dev get secret redis-auth── expected output ──NAME STORE REFRESH INTERVAL STATUS READYweb-db aws-secrets-manager 1h SecretSynced TrueNAME TYPE DATA AGEredis-auth Opaque 1 2m
Checkpoint — you should now have
- ✓
git grep -i passwordin the config repo finds only references and ciphertext. - ✓The ExternalSecret is
SecretSyncedand the Sealed Secret has been unsealed into a Secret. - ✓You've backed up the sealed-secrets controller key somewhere safe (or chosen ESO for everything).
Part 4
Break it on purpose
Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.
Break #1
The operator can't read the secret
Reference shoplite/prod/db-v2, a key the IAM role isn't allowed to read (or that doesn't exist).
Break #2
Lose the sealing key
Delete the kind cluster, create a new one, reinstall sealed-secrets, and let Argo sync the old SealedSecret.
Part 5
Interview questions from this mission
How do you manage secrets in a GitOps workflow?
A developer committed a real password to the config repo. What do you do?