strace, lsof & Debugging a Misbehaving Process

In one line

When logs say nothing, watch what the process actually asks the kernel to do. strace shows its system calls, lsof shows its open files and sockets, and together they explain most 'it just hangs' mysteries.

0/5 · 0%

Think of it like this

An employee who's 'working' but producing nothing. Instead of asking them, you watch the doorway: every time they go to the filing cabinet, the phone, or the printer (system calls). If they're standing at the printer for ten minutes, you've found the problem.

Key ideas

01
STRACE traces system calls: strace -f -tt -p <pid> attaches to a running process (and its threads with -f) and prints every call with timestamps. -e trace=network or -e trace=file filters; -c summarises time per call. A process stuck in connect( to one IP is waiting on a network dependency; stuck in futex( is waiting on a lock; repeating open(... ENOENT) means it's looking for a missing file.
02
strace slows the traced process significantly; use it briefly in production. For low-overhead tracing, eBPF tools (bcc/bpftrace: opensnoop, tcpconnect, execsnoop, biolatency) observe the whole system with little impact.
03
LSOF lists open files: lsof -p <pid> (everything a process has open), lsof -i :8080 (who's using a port: the classic 'Address already in use'), lsof +L1 (files deleted but still open, which still use disk space: a common reason df says full but du can't find the space; restart the process to release it).
04
Other tools: ss -tanp (sockets with states and owning processes, Topic 4.2), pstack/gdb -p or language tools for stack traces (jstack/jcmd for the JVM), perf top to see which functions burn CPU.

In your stack

→
For JVM services, pair strace/lsof with jcmd <pid> Thread.print (all thread stacks: find BLOCKED threads and what they wait on), jcmd <pid> GC.heap_info, and async-profiler for CPU flame graphs; jstack output three times, a few seconds apart, shows threads that never move.

Code & diagrams

find why it hangsbash

sudo strace -f -tt -p $(pgrep -f myapp) -e trace=network,file 2>&1 | head -20
# 10:14:02.118 connect(12, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.20.21.44")}, 16) = -1 EINPROGRESS
# 10:14:02.119 poll([{fd=12, events=POLLOUT}], 1, 30000   <-- waiting 30 s for the database

sudo strace -c -p $(pgrep -f myapp)      # Ctrl-C after 10 s: time per syscall
sudo lsof -i :8080                        # who holds the port
sudo lsof +L1 | head                      # deleted-but-open files still using disk

Explain it without notes

df shows the disk 100% full but du finds only 20 GB of 50 GB used. What's going on?

Practice

An app intermittently takes 30 s to respond, then works. Logs show nothing. How would you use these tools?

Trade-offs

↔
strace is universal but heavy; eBPF tools are light but need a modern kernel and privileges. Profilers and app metrics are better for continuous insight; strace is the flashlight for emergencies.

Done when you can

I can attach strace to a running process and read what it's waiting on
I can find port owners and deleted-but-open files with lsof

Command Palette