Secrets Management in Kubernetes

In one line

A raw Kubernetes Secret is a reasonable starting point, but real production secrets management means access-controlling who can read them, never committing them to Git, and often integrating an external secrets manager entirely.

0/4 · 0%

Key ideas

01
Topic 2.1 covered the mechanics of a Secret object; this topic covers actually managing them SAFELY in a real production cluster — the exact same 'never commit a real credential' lesson from Git's own course (Phase 7.5), now applied specifically to Kubernetes manifests.
02
RBAC (Topic 5.1) should genuinely restrict WHO and WHAT can read Secrets — by default, anyone with broad get/list access to a namespace can read every Secret in it, including ones belonging to completely unrelated applications; a properly scoped Role should grant Secret access only to the specific ServiceAccounts and users that genuinely need it, never broadly.
03
Committing a Secret's YAML directly to Git — even with real values correctly placed in stringData — means that value sits in your Git history FOREVER (exactly the problem Git's Phase 7.5 covered in depth), discoverable by anyone with repository access, past or present. The standard fix: never commit real secret VALUES at all; commit only the Secret's STRUCTURE (as a template) or generate Secrets through an entirely separate, non-Git mechanism.
04
SEALED SECRETS is one popular approach: a controller and CLI tool that lets you ENCRYPT a Secret's values using a cluster-specific public key, producing a SealedSecret object that IS genuinely safe to commit to Git (since only the cluster's own private key, held nowhere in Git, can decrypt it) — the sealed, encrypted form is committed and deployed like any other manifest; the controller decrypts it into a real Secret only inside the cluster itself.
05
The EXTERNAL SECRETS OPERATOR takes a different, increasingly common approach: it SYNCHRONIZES secrets FROM an external secrets manager (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) INTO native Kubernetes Secrets automatically — the actual sensitive values live and are managed entirely in the external system (with its own real access controls, audit logging, and rotation), and Kubernetes only ever holds a synced, up-to-date copy for pods to consume normally.
06
SECRET ROTATION (regularly replacing a credential with a new one, exactly as Linux and Git's own courses covered for compromised credentials) is genuinely harder to do well with raw, manually-managed Kubernetes Secrets — an external secrets manager with native rotation support (many cloud providers' secrets managers support automatic rotation for supported credential types like database passwords) is one of the strongest practical arguments for adopting one over plain, manually-updated Secret objects.

Code & diagrams

ExternalSecretsFlowdiagram

The real, sensitive value lives and is managed in the external system — Kubernetes only ever holds a synced copy.

Rendering diagram…

external-secret.yamlmarkdown

This creates a real Kubernetes Secret automatically, synced from AWS — no real value ever touches this file.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials       # the resulting native Secret's name
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password   # the key in AWS Secrets Manager — no value here at all

secrets-rbac-audit.shmarkdown

Audit exactly who can read Secrets in a namespace — a genuinely important, easy-to-skip real check.

# Confirm the ExternalSecret is syncing correctly
kubectl apply -f external-secret.yaml
kubectl get externalsecret db-credentials
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

# Audit who can actually READ secrets in this namespace — genuinely worth doing
# on any real production namespace
kubectl get rolebindings,clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name | test("secret|admin|edit")) | .metadata.name'

# Directly check a specific identity's actual Secret access
kubectl auth can-i get secrets --as=system:serviceaccount:production:some-other-sa -n production

Explain it without notes

Why is committing a Secret manifest with real stringData values directly to Git genuinely dangerous, even for a private repository?

What's the core advantage of the External Secrets Operator approach over just manually creating and updating Kubernetes Secret objects directly?

Practice

Audit an RBAC setup you have access to (or the example from Topic 5.1) and identify exactly which Roles grant access to Secrets specifically, versus other resource types.

If you have access to a cloud secrets manager and a test cluster, set up the External Secrets Operator and confirm a real Kubernetes Secret gets created and kept in sync automatically, with no real value ever appearing in your own YAML files.

Trade-offs

↔
An external secrets manager integration is a genuinely more secure, more auditable, and more maintainable approach at real production scale, but it adds real infrastructure and setup complexity (the operator itself, cloud IAM permissions for it to read the external store, an additional moving piece to keep healthy) — for a genuinely small project or early-stage system, plain Kubernetes Secrets (managed carefully, never committed with real values) may be a reasonable, simpler starting point, with a clear upgrade path once the operational maturity to adopt an external manager properly is actually there.

Done when you can

I understand why a Secret's real values should never be committed to Git, even in a private repository.
I know what Sealed Secrets and the External Secrets Operator each solve, and how their approaches differ.
I can audit an RBAC setup to identify exactly who has access to read Secrets in a namespace.