SQS: Queues, Visibility Timeouts & DLQs

In one line

SQS is a fully managed queue that lets a producer hand off work and move on while consumers process it at their own pace; visibility timeouts, dead-letter queues, and idempotent consumers are what make it reliable.

0/4 · 0%

Think of it like this

A restaurant's order spike. The waiter clips tickets on the rail (the QUEUE) and goes back to taking orders; cooks pull tickets as fast as they can. A dinner rush makes the rail longer, not the waiter slower. If a cook drops a ticket, it goes back on the rail.

Key ideas

01
A producer SendMessages; a consumer ReceiveMessages (use LONG POLLING — WaitTimeSeconds up to 20 — to avoid empty, billable polls), processes, then DeleteMessages. A message that's never deleted comes back.
02
VISIBILITY TIMEOUT: when a consumer receives a message, it's hidden from others for this period. If the consumer crashes or takes too long, it reappears and another consumer gets it. Set it longer than your worst-case processing time (for Lambda consumers, at least 6× the function timeout), or extend it while working.
03
STANDARD queues give nearly unlimited throughput with AT-LEAST-ONCE delivery and best-effort ordering — so duplicates happen and consumers must be idempotent. FIFO queues guarantee order within a MESSAGE GROUP ID and deduplicate within 5 minutes, with lower throughput limits.
04
DEAD-LETTER QUEUES: after maxReceiveCount failed attempts, a message moves to a DLQ instead of looping forever and blocking progress (a 'poison message'). Alarm on DLQ depth, investigate, fix, then REDRIVE the messages back to the source queue.
05
Queue depth (ApproximateNumberOfMessagesVisible) and age of the oldest message (ApproximateAgeOfOldestMessage) are the metrics that tell you whether consumers are keeping up — and are the right signals to scale workers on.

Code & diagrams

sqs.shbash

DLQ_URL=$(aws sqs create-queue --queue-name orders-dlq --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url $DLQ_URL \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

aws sqs create-queue --queue-name orders --attributes '{
  "VisibilityTimeout": "180",
  "ReceiveMessageWaitTimeSeconds": "20",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"'$DLQ_ARN'\",\"maxReceiveCount\":\"5\"}"
}'

aws sqs send-message --queue-url $QUEUE_URL --message-body '{"orderId":"789"}'
aws sqs receive-message --queue-url $QUEUE_URL --wait-time-seconds 20 --max-number-of-messages 10
aws sqs delete-message --queue-url $QUEUE_URL --receipt-handle "$HANDLE"

# After fixing the bug, move DLQ messages back to the source queue
aws sqs start-message-move-task --source-arn $DLQ_ARN

QueueWithDlqdiagram

Rendering diagram…

Explain it without notes

Why must SQS consumers be idempotent even with a correctly configured visibility timeout?

What problem does a dead-letter queue solve?

Practice

A consumer takes up to 4 minutes per message, but the visibility timeout is 30 seconds. What will you observe?

You need to process payment events for each customer strictly in order, but different customers can proceed in parallel. How do you configure SQS?

Trade-offs

↔
Queues absorb spikes and isolate failures, but introduce latency and eventual consistency: the caller no longer knows when (or whether) the work succeeded, so you need status tracking, DLQ monitoring, and idempotent consumers.

Done when you can

I configure long polling, a visibility timeout above worst-case processing time, and a DLQ.
I write idempotent consumers.
I know when FIFO queues are needed and what message group IDs do.
I alarm on DLQ depth and oldest-message age.