Topic 6.4
X-Ray Tracing & CloudTrail Auditing
In one line
Distributed tracing with X-Ray (via OpenTelemetry) shows where time goes across services in a single request; CloudTrail records every API call made in your account, answering 'who changed what, and when?'
Think of it like this
X-Ray is a parcel tracking page showing every hub a package passed through and how long it sat at each. CloudTrail is the building's visitor log and CCTV — it doesn't tell you why the package was slow, but it tells you exactly who opened the storeroom at 2:14 am.
Key ideas
- 01
A TRACE follows one request across services; each hop is a SEGMENT/SPAN with timing, status, and metadata. The trace ID is propagated in headers so API Gateway, Lambda, ECS services, and downstream AWS SDK calls stitch together. The SERVICE MAP shows dependencies with latency and error rates on each edge.
- 02
Instrument with the AWS DISTRO FOR OPENTELEMETRY (ADOT) or OpenTelemetry SDKs and send to X-Ray (or any OTel backend). Enable active tracing on Lambda and API Gateway. Use SAMPLING — tracing every request is expensive and unnecessary.
- 03
CLOUDTRAIL logs management events (API calls that create, modify, or delete resources —
RunInstances,PutBucketPolicy,AttachRolePolicy) with who (identity ARN), when, from where (source IP), and the parameters. Event history keeps 90 days; create a TRAIL to deliver all regions' events to an S3 bucket for long-term retention, ideally an ORGANIZATION trail into a separate, locked-down log-archive account. - 04
DATA EVENTS (S3 object reads/writes, Lambda invocations, DynamoDB item operations) are NOT logged by default because of volume and cost — enable them selectively for sensitive buckets and tables.
- 05
Use CloudTrail with EventBridge to react in near-real time to dangerous changes — a security group opened to
0.0.0.0/0, root user login, CloudTrail logging stopped — and with Athena or CloudTrail Lake to investigate incidents after the fact.
Code & diagrams
# Multi-region trail with log file integrity validation
aws cloudtrail create-trail --name org-audit --s3-bucket-name acme-cloudtrail-logs \
--is-multi-region-trail --enable-log-file-validation
aws cloudtrail start-logging --name org-audit
# Who modified this security group in the last 90 days?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0abc123 \
--query 'Events[].[EventTime,EventName,Username]' --output table
# Enable active tracing on a Lambda function
aws lambda update-function-configuration --function-name checkout-api \
--tracing-config Mode=ActiveEventBridge pattern on CloudTrail events — route to an SNS topic that pages security.
{
"detail-type": ["AWS Console Sign In via CloudTrail"],
"detail": {
"userIdentity": { "type": ["Root"] }
}
}One checkout request as a trace: the span tree makes the 380 ms payments call the obvious bottleneck.
Explain it without notes
A request to your API takes 2 seconds. Logs and metrics from each service look normal. How does tracing help where they didn't?
Why send CloudTrail logs to a separate account rather than a bucket in the same account?
Practice
An S3 bucket's policy was changed last night and the bucket became public. How do you find out who did it?
Someone reports that an object in a sensitive bucket was downloaded by an unknown party. Why might CloudTrail not show it, and what should you change?
Trade-offs
- ↔
Tracing everything gives perfect visibility but costs money and adds overhead; sampling keeps cost low but may miss rare slow requests — tail-based sampling (keep all errors and slow traces) is the usual compromise. Similarly, CloudTrail data events give full access auditing at significant volume cost, so enable them only where needed.
Done when you can
I can instrument services with OpenTelemetry/X-Ray and read a trace.
I have a multi-region (ideally organization) CloudTrail trail in a separate log account.
I know management events vs data events.
I alert on high-risk CloudTrail events via EventBridge.