Lab 1.4·Supply ChainHIGH

The image your cluster runs — is it the one CI built?

Anyone who can push to the registry can replace shoplite:1.4.0 with something else, and every node that pulls that tag will run it.

Weakness: SLSA · Build integrity · CICD-SEC-9 (improper artifact integrity validation)
Target: a local registry (localhost:5000) and the deploy step

tags vs digestscosign key-based and keyless signingcosign verifyadmission-time verification

Run these techniques only against the lab you own. Using them on systems you don't have permission to test is illegal.

Lab setup

01Run a local registry and push an image
terminal
$ docker run -d -p 5000:5000 --name registry registry:2
docker tag shoplite:good localhost:5000/shoplite:1.4.0 && docker push localhost:5000/shoplite:1.4.0
── output ──
1.4.0: digest: sha256:9b2e…41c7 size: 1784

The threat

How this weakness gets abused

01Tags are mutable pointers
A tag like 1.4.0 is just a name pointing at a digest, and a registry lets anyone with push rights move it. A leaked CI token, a compromised registry account, or an insider can re-point a trusted tag at a different image. Deployments that reference tags pull whatever it points to at that moment, and nothing in the cluster notices the change.

What's at risk

Arbitrary code running in production under a trusted name, with the service's credentials and network access.

Detect

How you'd catch it — before or during

01Sign images in CI, verify before deploying
cosign (Sigstore) signs an image DIGEST and stores the signature in the registry next to it. Verification checks that a valid signature exists from the expected key or identity. An image that CI didn't sign has no valid signature, and verification fails.
terminal
$ cosign generate-key-pair
cosign sign --yes --key cosign.key localhost:5000/shoplite@sha256:9b2e…41c7
cosign verify --key cosign.pub localhost:5000/shoplite:1.4.0 | jq -r '.[0].critical.image["docker-manifest-digest"]'
── output ──
Private key written to cosign.key
Public key written to cosign.pub
Pushing signature to: localhost:5000/shoplite

Verification for localhost:5000/shoplite:1.4.0 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
sha256:9b2e…41c7
02An image CI didn't sign fails verification
Simulate the problem by pushing a different, unsigned image to the same tag. Verification now fails, because the tag points at a digest nobody signed.
terminal
$ docker tag busybox localhost:5000/shoplite:1.4.0 && docker push -q localhost:5000/shoplite:1.4.0
cosign verify --key cosign.pub localhost:5000/shoplite:1.4.0
── output ──
Error: no matching signatures

Defend

Fix the cause, not just this instance

01Keyless signing in CI, deploy by digest

In CI, keyless signing uses the workflow's OIDC identity, so there's no private key to manage or leak, and the signature records WHICH repo and workflow signed it. Deployments reference the DIGEST CI produced, not the tag, so what was built, tested, and signed is exactly what runs.

Vulnerable

deploywhole filebash

image = "123456789012.dkr.ecr.ap-south-1.amazonaws.com/shoplite:1.4.0"   # whatever the tag points at today

Hardened

.github/workflows/build.ymladd to fileyaml

permissions: { id-token: write, contents: read, packages: write }
steps:
  - id: build
    uses: docker/build-push-action@<sha> # v6
    with: { push: true, tags: "${{ env.IMAGE }}:${{ github.sha }}" }
  - uses: sigstore/cosign-installer@<sha> # v3
  - run: cosign sign --yes "$IMAGE@${{ steps.build.outputs.digest }}"
# deploy with image = "$IMAGE@sha256:..." (the digest output)

02Enforce at admission: nothing unsigned runs

Verification in a deploy script can be skipped by someone deploying by hand. In Kubernetes, an admission controller (Kyverno, or Sigstore's policy-controller) refuses to create pods whose images aren't signed by the expected identity, whoever submits them.

Hardened

kyverno/verify-images.yamlwhole fileyaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-shoplite-images }
spec:
  validationFailureAction: Enforce
  rules:
    - name: signed-by-our-ci
      match: { any: [{ resources: { kinds: [Pod] } }] }
      verifyImages:
        - imageReferences: ["ghcr.io/acme/shoplite*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/acme/shoplite/.github/workflows/build.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"

Verify

Prove the fix with the same checks

01Only CI-built, signed digests pass
Re-push the signed image, re-verify successfully, then try deploying the unsigned one through the admission policy: the API server rejects the pod with the policy's message.
terminal
$ kubectl run bad --image=ghcr.io/acme/shoplite:unsigned
── output ──
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Pod/default/bad was blocked due to the following policies
verify-shoplite-images:
signed-by-our-ci: 'failed to verify image ghcr.io/acme/shoplite:unsigned: no matching signatures'

The concepts

Digests are content addresses

A digest (sha256:…) is a hash of the image manifest: change one byte and it changes. Referencing images by digest makes deployments immutable and reproducible. Tags are for humans; digests are for machines. Use tags to find images and digests to run them.

Keyless signing and transparency logs

Sigstore's keyless flow issues a short-lived certificate binding the signature to an OIDC identity (the CI workflow), and records the signature in a public transparency log (Rekor). Verifiers check the identity and the log entry, so there's no long-lived private key to steal, and any signing event is publicly auditable.

Your turn

How do you find the digest a tag currently points to without pulling the image?

Besides signing the image, what else can you attach and verify with cosign?

Interview questions

How do you ensure only trusted container images run in production?

0/4 · 0%

Command Palette