Debugging DNS Like an SRE

In one line

Most DNS incidents come from a handful of causes (stale caches, wrong resolver, search-domain surprises, split horizon, NXDOMAIN caching); dig with the right flags finds them quickly.

0/4 · 0%

Key ideas

01
Always ask WHICH resolver answered and WHAT it answered: dig @resolver name pins the resolver; the ANSWER SECTION shows records and remaining TTLs; status: shows NOERROR, NXDOMAIN (doesn't exist), or SERVFAIL (the resolver couldn't get an answer, often a DNSSEC or authoritative-server problem).
02
Compare the authoritative answer (dig @ns1.provider name) with your recursive resolver's answer. If they differ, you're looking at a cached old answer, not a wrong record.
03
SPLIT-HORIZON DNS: the same name resolves differently inside and outside a network (e.g. Route 53 private hosted zones or VPC endpoints with private DNS, AWS course Topic 3.3). 'Works from my laptop, fails from the server' is often this.
04
SEARCH DOMAINS and ndots: with search svc.cluster.local and ndots:5, a lookup for api.example.com (fewer than 5 dots) first tries api.example.com.svc.cluster.local, then the other search domains, and only then the real name. That's several wasted queries per lookup in Kubernetes (Topic 3.4); write external names fully qualified, with a trailing dot, to skip the search list.
05
Application caching: JVMs historically cached DNS forever (networkaddress.cache.ttl), and some HTTP clients resolve once at startup. After a failover that changes an IP, those apps keep calling the old address until restarted. Know your runtime's DNS caching behaviour.

Code & diagrams

dns-debug.shbash

dig api.example.com                      # status, answer, TTL, which server answered (SERVER: line)
dig @1.1.1.1 api.example.com +short       # a public resolver's view
dig @ns-123.awsdns-45.com api.example.com  # the authoritative truth
dig api.example.com +norecurse @127.0.0.53 # is it in the local cache?
resolvectl statistics                      # systemd-resolved cache hits/misses
resolvectl flush-caches                    # clear the local cache (your machine only)

Explain it without notes

dig shows the right new IP, but your Java service still connects to the old one after a database failover. What's going on?

Practice

You get SERVFAIL for app.example.com from your resolver, but the authoritative servers answer fine when queried directly. What are two possible causes?

Trade-offs

↔
Client-side DNS caching reduces latency and resolver load but slows failover; very low TTLs speed failover but increase lookups and dependence on the resolver's availability.

Done when you can

I can compare recursive vs authoritative answers with dig.
I can read NOERROR, NXDOMAIN, and SERVFAIL and know what each implies.
I understand split-horizon DNS, search domains, and application-level DNS caching.