Command Palette

Search for a command to run...

Hectal

Mission 2.1 · Stage 2 — GitOps at Scale: Secrets, Rollouts, and Many Clusters

Secrets in GitOps: Sealed Secrets, SOPS, and External Secrets

Goal: ShopLite's Redis password and DB credentials reach the cluster without ever sitting in Git as plain text.

50 min Free locally; Secrets Manager ~$0.40/secret/month 5 steps 2 break-it drills

By the end of this mission

  • Explain why a Kubernetes Secret in Git is not secret
  • Compare the three standard approaches and choose one
  • Encrypt a Secret for Git with Sealed Secrets
  • Sync secrets from AWS Secrets Manager with the External Secrets Operator

Part 1

Understand it first

The problem

A Kubernetes Secret's data is base64, which is an encoding, not encryption; anyone who can read the repo can decode it. And Git history is forever: a secret committed once and 'deleted' in the next commit is still in history and in every clone (DevSecOps course, secrets chapter). So GitOps needs a way to put a REFERENCE or CIPHERTEXT in Git and produce the real Secret only inside the cluster.

Three approaches

SEALED SECRETS (Bitnami): a controller in the cluster holds a private key; you encrypt with its public key using kubeseal, commit the SealedSecret, and only that cluster can decrypt it. Simple and self-contained, but the key is per cluster (back it up!) and rotating the actual secret still means re-sealing.

SOPS (+ age or KMS): encrypts only the VALUES in YAML files, leaving keys readable so diffs stay meaningful. Argo CD needs a plugin (or Flux's built-in support) to decrypt at render time. Good when you want encrypted files in Git with cloud KMS access control.

EXTERNAL SECRETS OPERATOR (ESO): Git holds only an ExternalSecret saying 'fetch key shoplite/prod/db from AWS Secrets Manager'. The operator reads it (using IRSA/Pod Identity, no static keys) and creates the Kubernetes Secret, refreshing on an interval. Secrets are managed and rotated in the secret manager; nothing sensitive ever touches Git. This is the most common choice on cloud platforms, and what ShopLite uses in prod.

External Secrets flowdiagram
Rendering diagram…

Part 2

Your project after this mission · 5 files change

shoplite-gitops/
  • shoplite-gitops/
    • apps/
      • web/
        • overlays/
          • dev/
            • redis-auth.sealed.yamlnew
          • prod/
            • db-secret.yamlnew
    • argocd/
      • apps/
        • external-secrets.yamlnew
        • sealed-secrets.yamlnew
    • platform/
      • external-secrets/
        • cluster-store.yamlnew

Part 3

Build it, step by step

  1. 1

    Install both controllers through Argo CD

    Platform components are Applications too, living in the same repo. Here's ESO; sealed-secrets is identical with chart sealed-secrets from https://bitnami-labs.github.io/sealed-secrets into kube-system. They go in a platform project in real setups.

    shoplite-gitops/argocd/apps/external-secrets.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: external-secrets
      namespace: argocd
      annotations:
        argocd.argoproj.io/sync-wave: "-10"   # before apps that need secrets
    spec:
      project: default
      source:
        repoURL: https://charts.external-secrets.io
        chart: external-secrets
        targetRevision: 0.20.1
        helm:
          valuesObject:
            installCRDs: true
      destination:
        server: https://kubernetes.default.svc
        namespace: external-secrets
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true, ServerSideApply=true]
  2. 2

    Seal the dev Redis password

    Create the Secret locally with --dry-run (it never reaches the cluster or disk as plain text), pipe it through kubeseal, and commit the output. The ciphertext is bound to this name and namespace, so copying it elsewhere won't decrypt.

    terminal
    $ kubectl create secret generic redis-auth -n shoplite-dev \
    --from-literal=redis-password="$(openssl rand -hex 16)" --dry-run=client -o yaml \
    | kubeseal --format yaml > apps/web/overlays/dev/redis-auth.sealed.yaml
    grep -A2 encryptedData apps/web/overlays/dev/redis-auth.sealed.yaml
    ── expected output ──
    encryptedData:
    redis-password: AgBy3i4OJSWK+PiTySYZZA1rO43cGDEq...
  3. 3

    Connect ESO to AWS Secrets Manager

    A ClusterSecretStore says where secrets live and how to authenticate. On EKS the operator's service account is bound to an IAM role via IRSA or Pod Identity (AWS course, EKS topic), allowed secretsmanager:GetSecretValue only on shoplite/*. No access keys anywhere.

    shoplite-gitops/platform/external-secrets/cluster-store.yamlwhole fileyaml
    apiVersion: external-secrets.io/v1
    kind: ClusterSecretStore
    metadata:
      name: aws-secrets-manager
    spec:
      provider:
        aws:
          service: SecretsManager
          region: ap-south-1
          auth:
            jwt:
              serviceAccountRef:
                name: external-secrets
                namespace: external-secrets
  4. 4

    Reference the prod DB secret

    This file is safe to commit: it contains names, not values. ESO creates a normal Secret web-db that the Deployment mounts as usual. refreshInterval picks up rotations; pods need a restart (or Reloader) to see new env values.

    shoplite-gitops/apps/web/overlays/prod/db-secret.yamlwhole fileyaml
    apiVersion: external-secrets.io/v1
    kind: ExternalSecret
    metadata:
      name: web-db
    spec:
      refreshInterval: 1h
      secretStoreRef:
        kind: ClusterSecretStore
        name: aws-secrets-manager
      target:
        name: web-db
        creationPolicy: Owner
      data:
        - secretKey: DB_PASSWORD
          remoteRef:
            key: shoplite/prod/db
            property: password
        - secretKey: DB_USER
          remoteRef:
            key: shoplite/prod/db
            property: username
  5. 5

    Verify

    Add the new file to the prod overlay's resources, push, and check the ExternalSecret's status. SecretSynced means the Secret exists in the cluster.

    terminal
    $ git add . && git commit -m 'secrets: sealed (dev) + ESO (prod)' && git push
    kubectl -n shoplite-prod get externalsecret web-db
    kubectl -n shoplite-dev get secret redis-auth
    ── expected output ──
    NAME STORE REFRESH INTERVAL STATUS READY
    web-db aws-secrets-manager 1h SecretSynced True
    NAME TYPE DATA AGE
    redis-auth Opaque 1 2m

Checkpoint — you should now have

  • ✓git grep -i password in the config repo finds only references and ciphertext.
  • ✓The ExternalSecret is SecretSynced and the Sealed Secret has been unsealed into a Secret.
  • ✓You've backed up the sealed-secrets controller key somewhere safe (or chosen ESO for everything).

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

The operator can't read the secret

Reference shoplite/prod/db-v2, a key the IAM role isn't allowed to read (or that doesn't exist).

terminal
$ kubectl -n shoplite-prod describe externalsecret web-db | tail -3
── what you'll see ──
Warning UpdateFailed 12s external-secrets error retrieving secret at .data[0], key: shoplite/prod/db-v2, err: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/shoplite-eso/... is not authorized to perform: secretsmanager:GetSecretValue

Break #2

Lose the sealing key

Delete the kind cluster, create a new one, reinstall sealed-secrets, and let Argo sync the old SealedSecret.

terminal
$ kubectl -n kube-system logs deploy/sealed-secrets-controller | tail -1
── what you'll see ──
Error updating shoplite-dev/redis-auth, giving up: no key could decrypt secret (redis-password)

Part 5

Interview questions from this mission

01

How do you manage secrets in a GitOps workflow?

02

A developer committed a real password to the config repo. What do you do?

0/4 · 0%