On-call report — 23 manual interventions last week; 14 were 'restart the worker when the queue backs up'
40% of the week on restarts
You're the on-call engineer this week, and also on the team deciding next quarter's priorities.
Briefing
The receipt worker's queue backs up a couple of times a day. The fix everyone applies is to restart the worker, which clears it. It takes ten minutes each time, including getting paged. It's been happening for two months.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
You're paged again: queue depth 4,000, oldest job 18 minutes old.
- [FIRING] ReceiptQueueBacklog queue_depth=4012 oldest_age=18m
What do you do this time?
Debrief
Toil is manual, repetitive, automatable work that scales with the service and has no lasting value. The best response is to understand and fix the cause, not just automate the workaround. Measure toil so its cost is visible, and treat reducing it as engineering work that earns its place in planning.
The postmortem
Blameless postmortem · draft
Summary. The receipt worker hung on SMTP connections with no timeout, requiring ~14 manual restarts per week for two months.
Impact. Delayed receipts (up to ~45 minutes) several times daily; ~2.5 engineer-hours/week of interrupts.
Timeline
- MonthsRecurring backlogs; restarts as standard fix
- MonEvidence captured before restart; SMTP hang identified
- ThuTimeout + retry with backoff deployed
Root cause. No connection or read timeout on the SMTP client, so a stalled connection blocked the worker indefinitely.
Contributing factors
- The restart workaround was effective enough that nobody investigated.
- Toil wasn't measured, so its cost was invisible in planning.
Action items
- preventTimeouts on all outbound client libraries (lint rule for missing timeouts)platform
- processWeekly toil log in the on-call handover; top items reviewed in planningsre
The practice behind it
Toil
Google's SRE book defines toil as work that is manual, repetitive, automatable, tactical, without enduring value, and that grows linearly with the service. Some toil is unavoidable; SRE teams commonly cap it at about 50% of their time so the rest can go to engineering work that reduces future toil.
Your turn
Classify each: (a) reviewing a design doc, (b) manually rotating certificates every 90 days, (c) investigating a new type of incident, (d) approving routine access requests by hand.
Interview questions
What is toil and how do you reduce it?