The Troubleshooting Playbook: Real Incidents, Step by Step

In one line

Five incidents you will meet on real servers, each with the order of checks and the fix: can't SSH in, disk full, service won't start, port already in use, and out-of-memory kills.

0/5 · 0%

Think of it like this

A pilot's emergency checklist. Pilots don't improvise when an engine fails; they follow a practised list, in order. A troubleshooting playbook gives you the same calm under pressure.

Key ideas

01
CAN'T SSH IN: is the instance running and reachable (ping/cloud console)? Is port 22 allowed (security group, firewall, NACL)? Timeout means filtered or unreachable; 'Connection refused' means sshd isn't running; 'Permission denied (publickey)' means wrong key/user or bad permissions on ~/.ssh (700) and authorized_keys (600). Use the cloud serial console or SSM Session Manager to get in another way.
02
DISK FULL: df -h (which filesystem), df -i (inodes), du -xh / --max-depth=1 | sort -h to walk down, lsof +L1 for deleted-but-open files, then clean logs (journal: journalctl --vacuum-size=500M), old releases, Docker images (docker system df, docker system prune). Then prevent it: logrotate, alerts at 80%.
03
SERVICE WON'T START: systemctl status <unit> (exit code and last lines), journalctl -u <unit> -n 100 --no-pager, run the ExecStart command by hand as the service user to see the real error; common causes are a config typo, missing environment file, permission on a directory, or the port already in use.
04
ADDRESS ALREADY IN USE: ss -ltnp 'sport = :8080' or lsof -i :8080 shows the owner; often an old instance still running or a different service. OUT OF MEMORY: dmesg -T | grep -i -E 'killed process|out of memory', check the victim's memory growth in monitoring, then fix the leak, raise the limit (systemd MemoryMax, container limits), or add swap/RAM deliberately.
05
Always finish with: what changed recently (deploys, config, packages: /var/log/apt/history.log), write down the timeline, and add the alert or automation that would have caught it earlier (SRE course, postmortems).

Code & diagrams

service won't start: the sequencebash

systemctl status shoplite-api --no-pager
journalctl -u shoplite-api -n 100 --no-pager
sudo -u shoplite /usr/bin/java -jar /opt/shoplite/app.jar   # run it exactly as the service would
ss -ltnp 'sport = :8080'                                      # something already on the port?
namei -l /var/lib/shoplite                                    # permissions along the whole path
systemctl cat shoplite-api                                    # the unit as systemd sees it (with drop-ins)

Explain it without notes

What's the difference between 'Connection refused' and a timeout when SSH fails?

Practice

After a deploy, the service restarts in a loop every 5 seconds. Walk through your investigation.

Trade-offs

↔
Fixing forward on a live server is tempting but slow and risky; rolling back first and investigating calmly is usually faster for users.

Done when you can

I have a practised sequence for SSH failures, full disks, failed services, port conflicts, and OOM kills