📦 How the Internet Actually Moves a Packet

How the Internet Actually Moves a Packet

Almost every network problem you will ever debug — a page that hangs at 99%, a VPN that works for SSH but not for large file transfers, a service that is reachable from the office but not from home — is a problem at one specific layer. If you can name the layer, you have already halved the search space. This module builds that map.

Two models: the one you get taught, and the one that ships

The OSI reference model (ISO/IEC 7498-1) has seven layers. The stack that every operating system on the internet actually implements has four, described in RFC 1122. OSI is a vocabulary, not an implementation: nothing in Linux, Windows or a Cisco router has a "session layer" module. It is still worth knowing because the industry talks in OSI numbers — "a layer 7 firewall", "layer 2 adjacency", "an L3 switch" — and because those numbers are unambiguous once you learn them.

TLS is the classic argument-starter: it sits above the transport and below the application, which is a place OSI has no number for. QUIC is worse — it merges transport, encryption and stream multiplexing into one protocol carried inside UDP. That is not a flaw in the protocols; it is a reminder that the seven-layer model is a teaching aid.

Encapsulation: what each layer bolts on

Sending data is a series of wrappings. Your HTTP bytes get a TCP header, that gets an IP header, that gets an Ethernet header and trailer. The receiver unwraps in the reverse order. The headers are small and fixed enough that you can do the arithmetic in your head.

MTU, MSS and fragmentation

The MTU is the largest payload a link will carry in one frame. Ethernet's default is 1500 bytes. Subtract a 20-byte IPv4 header and a 20-byte TCP header and you get the classic MSS of 1460 — the largest chunk of application data TCP will put in one segment. Over IPv6 the base header is 40 bytes, so the same link gives an MSS of 1440. A PPPoE DSL link has an MTU of 1492, which is where a lot of "small requests work, big uploads hang" bugs come from.

If an IPv4 packet is too big for the next link, a router may fragment it — unless the Don't Fragment bit is set, in which case the router discards it and returns ICMP type 3, code 4 ("fragmentation needed"). IPv6 routers never fragment; only the sending host may, and an oversized packet earns an ICMPv6 Packet Too Big (type 2). Both mechanisms are called Path MTU Discovery, and both break when someone blanket-blocks ICMP at a firewall. The result is a connection that completes its handshake and then stalls the moment a full-size segment is sent — a "PMTU black hole". IPv6 guarantees a minimum link MTU of 1280 bytes, which is why tunnels often clamp to that.

MAC versus IP, and ARP

A MAC address is 48 bits, burned in (or randomised, on modern phones), and only meaningful on the local link — it never survives a router hop. An IP address is globally routable and identifies where you are in the topology, not who you are. Every packet therefore carries two destinations at once: the final IP, and the MAC of the next hop, which is rewritten at every router.

To learn a neighbour's MAC, IPv4 uses ARP (RFC 826): a broadcast frame to ff:ff:ff:ff:ff:ff asking "who has 192.168.1.1?", answered unicast by whoever owns it. There is no authentication in ARP at all, which is the whole basis of local-network spoofing. IPv6 replaced it with NDP — Neighbor Solicitation and Neighbor Advertisement, ICMPv6 types 135 and 136, sent to a solicited-node multicast group rather than broadcast.

$ ip neigh show
192.168.1.1   dev eth0 lladdr 3c:37:86:0a:1b:22 REACHABLE
192.168.1.42  dev eth0 lladdr 8e:11:5f:c0:9d:04 STALE

TCP: the handshake, in real flags

TCP (current specification: RFC 9293) gives you an ordered, reliable, flow-controlled byte stream over an unreliable packet network. It opens with a three-way handshake that synchronises a sequence number in each direction:

$ sudo tcpdump -n -i eth0 'host 203.0.113.10 and port 443'
IP 198.51.100.7.51544 > 203.0.113.10.443: Flags [S],  seq 1829301,           win 64240, options [mss 1460,sackOK,wscale 7]
IP 203.0.113.10.443 > 198.51.100.7.51544: Flags [S.], seq 998273, ack 1829302, win 65160, options [mss 1460,sackOK,wscale 8]
IP 198.51.100.7.51544 > 203.0.113.10.443: Flags [.],  ack 1,                  win 502

In tcpdump notation [S] is SYN, [S.] is SYN+ACK, [.] is a bare ACK, [P.] is PSH+ACK (data), [F.] is FIN+ACK and [R] is RST. So the handshake is SYN → SYN-ACK → ACK, and after that third packet the socket is ESTABLISHED on both sides.

Teardown is four packets, because each direction is closed independently. In practice the FIN almost always rides along with an ACK, so you see [F.]:

IP client > server: Flags [F.], seq 517,  ack 4381   # client says "no more data from me"  → FIN_WAIT_1
IP server > client: Flags [.],  ack 518              # server acknowledges                  → CLOSE_WAIT
IP server > client: Flags [F.], seq 4381, ack 518    # server finishes too                  → LAST_ACK
IP client > server: Flags [.],  ack 4382             # final acknowledgement                → TIME_WAIT

The closing side then sits in TIME_WAIT for twice the maximum segment lifetime (60 seconds on Linux) so that late duplicate segments cannot be mistaken for part of a new connection on the same four-tuple. A pile of TIME_WAIT sockets in ss -tan is normal, not a leak. An abrupt RST skips all of this and destroys the connection immediately — that is what you get from a closed port, or from an application that crashed.

UDP, and when each is right

UDP adds exactly four things to IP: two ports, a length and a checksum. No handshake, no ordering, no retransmission, no congestion control. That is a feature when late data is worthless or when you want to build your own logic on top: DNS queries, NTP, VoIP and video, QUIC (and therefore HTTP/3), and most game traffic. Use TCP when every byte must arrive in order and a few hundred milliseconds of recovery is acceptable — which is nearly everything else.

Ports, sockets and the four-tuple

A port is a 16-bit number, so 0–65535. IANA splits the space into system ports 0–1023, registered ports 1024–49151 and dynamic/ephemeral ports 49152–65535. Linux actually draws its ephemeral source ports from net.ipv4.ip_local_port_range, which defaults to 32768–60999. A connection is identified by a four-tuple: source IP, source port, destination IP, destination port. That is why one server socket on port 443 can serve thousands of clients simultaneously — each connection has a different source port or address.

Scanning behaviour follows directly from the handshake: an open port answers SYN with SYN-ACK, a closed port answers with RST, and a firewalled port answers with nothing at all, so the scanner has to wait for a timeout. Run the port scanner → against a host you own and watch open, closed and filtered behave differently. To see the round-trip cost of just the handshake, measure TCP connect latency and jitter →.

NAT, and why nobody can reach you

Network address translation lets many private hosts share one public address by rewriting the source address and source port of outbound flows and keeping a translation table (RFC 2663 for the terminology; RFC 4787 sets out the required behaviour for UDP, RFC 5382 for TCP). The entry that makes return traffic work is created by the outbound packet. Nothing creates it for an inbound packet, so an unsolicited connection from the internet arrives at the NAT with no matching entry and is dropped. NAT is not a firewall, but this side effect behaves like one.

Workarounds all amount to creating the mapping first or having a third party relay: static port forwarding, UPnP IGD / NAT-PMP / PCP (RFC 6887) for automatic forwarding, STUN (RFC 8489) to discover your external mapping, hole punching so both sides send outbound at once, and TURN as the relay of last resort. Carrier-grade NAT (RFC 6598, using 100.64.0.0/10) puts a second layer of this above your own router, which is why port forwarding sometimes does nothing on mobile broadband. Check what address the internet sees → — if it differs from your router's WAN address, you are behind CGNAT.

Enter to pixels: the whole sequence

1. URL parsing and HSTS. The browser normalises the URL and checks its HSTS preload list; a preloaded domain is upgraded to https:// before any traffic is sent.
2. Name resolution. Browser cache → OS stub resolver → recursive resolver → root, TLD and authoritative servers. Usually one round trip, sometimes four.
3. Address selection. With both A and AAAA answers, Happy Eyeballs v2 (RFC 8305) starts a connection to the IPv6 address first and races IPv4 shortly after, taking whichever wins.
4. TCP handshake to port 443 — one round trip. (Or a QUIC handshake over UDP 443 if HTTP/3 is advertised via Alt-Svc or an HTTPS DNS record.)
5. TLS handshake. ClientHello carrying the SNI and a key share, ServerHello, certificate, Finished — one more round trip in TLS 1.3.
6. The request. GET / HTTP/1.1 with a Host: header, or an HTTP/2 HEADERS frame, sent inside the encrypted channel.
7. Response and render. Status line, headers, body; the parser then discovers subresources and repeats steps 2–6 for each new origin.

That is roughly three round trips of latency before the first byte of HTML — which is why a server 200 ms away feels slow no matter how fast it is, and why connection reuse, HSTS preloading and HTTP/3 all exist. Inspect the response and its redirect chain → to see steps 6 and 7 for any site.