Topic 4.6
MTU, Fragmentation & the 'Small Requests Work, Big Ones Hang' Bug
In one line
Every link has a maximum packet size, the MTU (1,500 bytes on most Ethernet). Tunnels and overlays (VPNs, VXLAN, some cloud links) add headers and lower the usable size. Too-big packets are fragmented or, with 'Don't Fragment' set, dropped with an ICMP message telling the sender to shrink. If that ICMP message is blocked, you get a black hole: pings and small requests work while large responses or TLS handshakes hang.
Think of it like this
A delivery van with a fixed cargo door. Most parcels fit. Send a parcel wider than the door to a depot with a smaller door, and either it gets split into smaller boxes, or (if it's marked 'do not split') the depot should phone you to say 'too wide'. If the phone line is cut, the parcel just disappears.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- MTU
- Maximum Transmission Unit: the largest IP packet a link can carry in one piece.
- MSS
- Maximum Segment Size: the largest TCP payload per packet, usually MTU minus 40 bytes.
- Fragmentation
- Splitting an IP packet into smaller pieces that the receiver reassembles.
- Don't Fragment (DF)
- An IPv4 header flag telling routers to drop, not split, a too-big packet.
- Path MTU discovery
- The process of finding the smallest MTU along a path using ICMP 'too big' messages.
- MSS clamping
- A router rewriting the MSS in TCP handshakes so segments fit the path.
- Jumbo frames
- Ethernet frames larger than 1,500 bytes (often 9,000), used inside data centres.
Step by step
01The symptom
Tiffin connects its Mumbai VPC to a kitchen partner's network over a VPN. Health checks pass and small API calls work, but the daily menu upload (a few hundred KB) hangs forever. It smells like a firewall, but nothing is denied in the logs.
02Finding the path MTU
Arjun pings with 'Don't Fragment' set and a growing payload. Payload + 28 bytes of ICMP and IP headers = packet size. 1,472 bytes of payload is a full 1,500-byte packet, and that fails. 1,392 (a 1,420-byte packet) gets through, so the VPN path only carries 1,420.
03Fix it two ways
The proper fix is letting the ICMP 'fragmentation needed' messages through, so path MTU discovery works. As a belt-and-braces fix for TCP, the VPN gateway clamps the MSS in every handshake so both sides send segments that fit.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Blocking all ICMP 'for security'
A hardening checklist says 'block ICMP'. The team drops all ICMP at the edge firewall.
Myth vs fact
Myth
ICMP is just ping, so blocking it is harmless.
Fact
ICMP also carries the error messages networks depend on, especially path MTU discovery. Block selectively, never all of it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Inside AWS, instances use a 9,001-byte MTU, but traffic leaving the VPC (internet, VPN, some peering) is limited to 1,500 or less.
ip linkshowingmtu 9001on an EC2 instance is normal, and path MTU discovery handles the rest, as long as ICMP isn't blocked.
Remember this
- 1
MTU = the largest IP packet a link carries: 1,500 bytes on standard Ethernet, often 9,001 (jumbo frames) inside AWS VPCs, and less over tunnels.
- 2
TCP's MSS (maximum segment size) = MTU minus IP and TCP headers (1,500 − 40 = 1,460). Both sides announce it in the handshake.
- 3
Encapsulation costs bytes: WireGuard adds about 60–80, IPsec 50–70, VXLAN 50, PPPoE 8. A 1,500-byte packet no longer fits on a link whose usable size is now 1,420.
- 4
Path MTU discovery: senders set 'Don't Fragment'. A router that can't forward a packet drops it and returns ICMP 'fragmentation needed' (IPv4) or 'packet too big' (IPv6) with the size that fits. IPv6 routers never fragment.
- 5
The black hole: firewalls that block *all* ICMP also block these messages. The sender keeps sending big packets that vanish. The classic symptom: SSH logs in but
lson a big directory hangs, or HTTPS handshakes stall when certificates are large. - 6
Fixes: allow ICMP type 3 code 4 (and ICMPv6 'packet too big'), set the right MTU on tunnel interfaces, or clamp MSS on the router so TCP never sends segments too big for the path.
Explain it without notes
Why do small requests work while large ones hang in an MTU problem?
What does MSS clamping do?
Practice
Find the largest packet that reaches a host without fragmenting.
Calculate the MSS for a 1,420-byte tunnel MTU.
Trade-offs
- ↔
Lower MTUs on tunnels avoid black holes but add per-packet overhead. Jumbo frames improve throughput inside a data centre but must match on every device of that network. MSS clamping fixes TCP only, not UDP (QUIC, DNS), which relies on correct MTUs.
Done when you can
I know where MTU shrinks (tunnels, overlays).
I can find the path MTU with ping -M do or tracepath.
I never block all ICMP.