Change request — raise the ShopLite DB connection pool from 10 to 50 on all tasks, now, during the evening peak
A config change at peak hour
You're the on-call SRE asked to approve (or not) a change during an ongoing slowdown.
Briefing
Checkout is a bit slow (p99 1.3 s) during the evening peak. A developer believes the connection pool is too small and wants to raise it from 10 to 50 per task, across all 12 tasks, right now.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
The database's max_connections is 300. 12 tasks × 50 = 600.
What's your response?
Debrief
Changes cause most incidents, so the change process is a reliability tool. Check the math against shared limits, prefer small canaries with automated comparison, treat irreversible changes (schema, data, external contracts) with expand-and-contract, and invest in progressive delivery so safety doesn't depend on one person catching the problem.
The postmortem
Blameless postmortem · draft
Summary. No incident: an unsafe pool change was replaced by a canary and a smaller change; a column drop was rescheduled as expand-and-contract.
Impact. Checkout p99 improved from 1.3 s to 0.9 s after the canaried change.
Timeline
- 19:10Pool change request reviewed; canary at 20 on 2 tasks
- 19:30Canary healthy; rolled to all tasks
- 19:40Column drop rescheduled as expand-and-contract
Root cause. N/A
Contributing factors
- Configuration changes bypassed the canary process that code deploys use.
Action items
- preventRoute config changes through the same canary pipeline as codeplatform
- preventPre-deploy check: total pool connections across tasks vs DB max_connectionssre
The practice behind it
Progressive delivery
Release to a small slice first (canary tasks, a percentage of users, one region), compare its SLIs against the baseline automatically, and expand or roll back based on the result. Feature flags decouple deploying code from exposing features. It reduces blast radius for every change, not just the ones someone thought were risky.
Your turn
Describe expand-and-contract for renaming a column from email to email_address.
Interview questions
How do you reduce the risk of production changes?