Command Palette

Search for a command to run...

Hectal
Lab 0.4·SecretsHIGH

Used four minutes after the leak

A key leaked despite everything. The question now is how fast you find out someone is using it — and what they did.

Weakness
MITRE ATT&CK T1078.004 · Valid accounts: cloud accounts
Target
an AWS account (a canary key you create yourself)
honeytokens (canary credentials)CloudTrail investigation by access keyGuardDuty findingscredential incident response

Run these techniques only against the lab you own. Using them on systems you don't have permission to test is illegal.

Lab setup

  1. 01Create a canary AWS key

    A HONEYTOKEN is a credential that nothing legitimate ever uses, so ANY use means compromise. Canarytokens.org (free, by Thinkst) issues AWS keys that belong to an account with no permissions and alert you by email the moment they're used. Plant one where attackers look, such as ~/.aws/credentials on a build server or a config file in a private repo, labelled to look valuable.

    ~/.aws/credentials (on the build server)add to fileini
    [prod-admin]
aws_access_key_id = AKIA...CANARY...     # from canarytokens.org — never used legitimately
aws_secret_access_key = ...

The threat

  1. 01An attacker finds the credentials file and tries it

    The first thing anyone does with a found key is ask AWS whose it is. This single, read-only call is enough to trigger the canary.

    terminal
    $ aws sts get-caller-identity --profile prod-admin
    ── output ──
    {
    "UserId": "AIDA...",
    "Account": "052310077262",
    "Arn": "arn:aws:iam::052310077262:user/canarytokens.com@@..."
    }

What's at risk

  • Here, nothing: the canary key has no permissions. A real key's impact depends entirely on its IAM policy, which is why least privilege (Chapter 4) matters even for 'internal' keys.
  • More importantly, you now KNOW the build server is compromised, before the attacker finds anything real.

Detect

  1. 01The canary alert

    Within minutes an email arrives with the source IP, user agent, and the AWS API call used. That IP and time become the starting point of the investigation on the machine where the token was planted.

    terminal
    $ (email from canarytokens.org)
    ── output ──
    Canarytoken triggered: "prod-admin creds on build-01"
    Event: AWS API key used
    API call: sts:GetCallerIdentity
    Source IP: 185.220.101.47
    User agent: aws-cli/2.17.0 Python/3.11 Linux/6.5
    Time: 2026-09-26 03:14:07 UTC

  2. 02For REAL keys: find every use in CloudTrail

    CloudTrail records every API call, including the access key used. Searching by AccessKeyId answers the two questions that matter: what did the attacker do, and when did it start? GuardDuty raises findings like UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration or Discovery:IAMUser/AnomalousBehavior for suspicious patterns automatically.

    terminal
    $ aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA2E7QXKZ3MNB4LP5W \
    --query 'Events[].[EventTime,EventName,CloudTrailEvent]' --output text | cut -c1-120
    ── output ──
    2026-09-26T03:21:44Z CreateUser {"sourceIPAddress":"185.220.101.47","userAgent":"aws-cli/2.17.0"...
    2026-09-26T03:19:02Z ListBuckets {"sourceIPAddress":"185.220.101.47",...
    2026-09-26T03:18:40Z GetCallerIdentity {"sourceIPAddress":"185.220.101.47",...
    CreateUser: the attacker is creating persistence. Your response must include finding and removing it.

Defend

  1. 01Respond: contain, investigate, eradicate

    1) Deactivate the key (not delete, yet: keep it for the investigation). 2) Attach an explicit deny-all policy to the user, which also stops its active sessions from doing more. 3) From CloudTrail, list everything created or changed by that key, such as new IAM users, access keys, roles, EC2 instances (crypto-miners), or Lambda functions, and remove it. 4) Rotate anything the attacker could have read. 5) Delete the key and the user once the investigation is complete.

    terminal
    $ aws iam update-access-key --user-name ci-deployer --access-key-id AKIA2E7QXKZ3MNB4LP5W --status Inactive
    aws iam put-user-policy --user-name ci-deployer --policy-name DenyAllIncident \
    --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'
    aws iam list-users --query "Users[?CreateDate>='2026-09-26'].UserName"
    ── output ──
    [
    "backup-svc-2"
    ]

  2. 02Prevent: remove long-lived user keys from the organisation

    The root cause is that a long-lived key existed on a server at all. Use roles (instance profiles, task roles), OIDC for CI, and SSO for humans. Then enforce it: an SCP denying iam:CreateAccessKey (Terraform course, Mission 7.3) makes new user keys impossible, and IAM credential reports find the old ones.

    Vulnerable

    build-01 ~/.aws/credentialswhole fileini
    [default]
aws_access_key_id = AKIA2E7QXKZ3MNB4LP5W
aws_secret_access_key = q9Tz4Vb1...

    Hardened

    build-01 (EC2 instance profile)whole filebash
    # No credentials file. The instance has a role:
$ aws sts get-caller-identity
"Arn": "arn:aws:sts::123456789012:assumed-role/build-01/i-0abc..."
# Temporary credentials, rotated automatically, usable only from this instance
# (GuardDuty alerts if they're used from elsewhere).

Verify

  1. 01The stolen key no longer works

    terminal
    $ aws sts get-caller-identity --profile stolen
    ── output ──
    An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

  2. 02No long-lived keys remain

    The credential report lists every IAM user and whether they have active keys. The goal is an empty list apart from documented, reviewed exceptions.

    terminal
    $ aws iam generate-credential-report >/dev/null && aws iam get-credential-report --query Content --output text | base64 -d | awk -F, '$9=="true"||$14=="true" {print $1}'
    ── output ──
    (no users with active access keys)

The concepts

Assume breach, optimise for detection time

Prevention eventually fails somewhere. What decides the damage is how long an attacker operates unnoticed. Honeytokens give near-zero false positives and near-instant detection for almost no effort. CloudTrail plus GuardDuty give visibility into real credentials. Both should feed the same on-call path as production alerts (Observability course, Chapter 4).

Persistence

Attackers with cloud credentials quickly create a way back in that survives the original key being revoked: new IAM users or keys, new roles with trust to their own accounts, Lambda backdoors, modified SSO settings. Incident response isn't finished when the key is disabled; it's finished when everything the key created or changed is reviewed.

Your turn

01

Write the CloudTrail lookup for all IAM changes (not reads) in the last 24 hours, by anyone.

02

Where else could you plant honeytokens?

Interview questions

01

An AWS access key was exposed publicly. Walk through your response.

02

What is a honeytoken and why are its alerts so reliable?

0/4 · 0%