IPsec Packet Builder 2D: AH vs ESP, Transport vs Tunnel, NAT Traversal

IPsec Protocol

Mode

Network Path

NAT router on path

Payload size 120 B

Travel speed 1.0×

Live Packet

Original payload

140 B

Wire size

164 B

IPsec overhead

+24 B (+17%)

Integrity check at receiver

n/a — no NAT on path

Legend

How it works

IPsec secures IP traffic with one of two headers, in one of two modes:

AH  (Authentication Header, RFC 4302)
    integrity + origin auth, NO encryption
    ICV = hash(shared key, IP header* + payload)
    * source/dest address are part of the hash

ESP (Encapsulating Security Payload, RFC 4303)
    encrypts the payload, optional auth trailer
    outer/new IP header is NOT covered by the ICV

Transport mode: [ orig IP | IPsec hdr | orig payload ]
                secures host-to-host traffic, original
                IP header stays on the wire unencrypted

Tunnel mode:    [ new IP | IPsec hdr | orig IP + payload ]
                the ENTIRE original packet becomes the
                payload of a fresh IP packet — used for
                gateway-to-gateway site-to-site VPNs

AH + NAT breaks: a NAT router must rewrite the source IP address (and port) to do its job. Because AH's ICV was computed over that same address, the receiver recomputes the hash, gets a different address, and the integrity check fails — the packet is dropped. This is why AH is considered NAT-incompatible in real deployments.
ESP survives NAT: ESP's ICV never covers the outer IP header, so NAT can freely rewrite it without breaking authentication. The only remaining problem is that plain ESP has no port number for NAT's translation table — solved by NAT-Traversal (NAT-T, RFC 3948), which wraps the whole ESP packet in an extra 8-byte UDP header (port 4500) purely so NAT devices have something to translate.
Wire size grows with every header layer: in transport mode the original IP header (20 B) stays on the wire alongside the IPsec header; in tunnel mode a whole new outer IP header (20 B) is added instead, wrapping the original IP header and payload. AH adds a 24 B header; ESP adds an 8 B header plus a ~2 B trailer and a 12 B authentication trailer; NAT-T adds a further 8 B UDP header when active.
Payload size and drag the wire: drag the packet left/right on the network path to scrub it to any point manually, and use the payload slider to see how a bigger original message shrinks the IPsec overhead percentage even though the header bytes stay fixed.

Real-world relevance: this exact AH-vs-NAT incompatibility is why almost every commercial VPN gateway (site-to-site or IKEv2/IPsec road-warrior client) uses ESP, never AH, the moment either endpoint might sit behind NAT.

drag packet to scrub ⟷