Network Stack Primer

Our client runs curl https://api.example.com/user/42. Those bytes need to leave the curl process, cross the kernel, ride the wire, and surface inside the server process on the far end. Five layers between the two: HTTP (L7) → TCP (L4) → IP (L3) → Ethernet (L2) → physical (L1). Four sections walk down the stack: the 5 layers (what each header costs and protects); sockets and ports (the kernel'sstruct sock and the 4-tuple that bounds every connection); routing, ARP, MTU with an interactive packet-descent walkthrough; and NIC drivers — ring buffers, NAPI, RSS, XDP, kernel bypass. Then a quick reference.

The OSI fairy tale vs the TCP/IP reality — 5 layers people actually use

When curl https://api.example.com/user/42 runs, its bytes need to reach a server across the public internet. Five layers stand between the two processes; each one adds a header on the way out and strips it on the way in.

Universities teach the OSI 7-layer model: physical, data link, network, transport, session, presentation, application. It's a useful piece of history. In production you reason about 5 layers (the TCP/IP model) — the "session" and "presentation" layers have no real implementations; whatever they were supposed to do is absorbed by the application or TLS.

Layer  Name              Real-world example                Header size
L7     Application       HTTP, gRPC, DNS, SSH, SMTP        variable
L4     Transport         TCP, UDP, QUIC                    20 B (TCP) / 8 B (UDP)
L3     Network           IPv4, IPv6, ICMP                  20 B (IPv4) / 40 B (IPv6)
L2     Data link         Ethernet, Wi-Fi (802.11)          14 B + 4 B FCS
L1     Physical          Copper, fiber, radio              n/a (line coding)

Encapsulation — what each layer adds

Sending one HTTP GET request for /user/42:

HTTP request body                                       ~500 B
+ TCP header (src/dst ports, seq, ack, flags, window)    +20 B   → segment 520 B
+ IPv4 header  (src/dst IP, TTL, proto=6, checksum)      +20 B   → packet  540 B
+ Ethernet header (dst MAC, src MAC, EtherType=0x0800)   +14 B   → frame   554 B
+ Frame Check Sequence (CRC32, trailer)                  + 4 B   → wire    558 B

MTU cap on the wire: 1500 B for standard Ethernet. Anything bigger
gets fragmented at L3 or, with TCP, segmented below MTU via MSS.

On the receiving server, the NIC pulls bits off the wire, the driver hands a frame up, Ethernet strips its 14 B and dispatches by EtherType, IP strips its 20 B and dispatches by protocol number (6 = TCP), TCP strips its 20 B and routes by destination port to the listening struct sock, which the application reads via recv(). Every layer's sole job is "peel my header, hand the rest to the right upstairs neighbour."

The kernel subsystems that implement it

In the Linux kernel, the descent is implemented by a chain of subsystems, each owning one layer:

Socket layer (net/socket.c) — translates the BSD socket API into protocol-specific calls.
TCP / UDP (net/ipv4/tcp.c, net/ipv4/udp.c) — segmentation, sequence numbers, retransmit, flow control (TCP); just a length + checksum (UDP).
IP (net/ipv4/ip_output.c) — routing table lookup, TTL, fragmentation, ARP resolution for next hop.
qdisc (queueing discipline, net/sched/) — per-interface egress queue, decides packet order and applies traffic shaping (fq, htb, codel).
Driver / NIC — DMAs the bytes into the card's TX ring buffer; the card serializes them onto the wire.

On RX everything runs in reverse: NIC IRQ → driver NAPI poll → IP input → TCP input → socket receive queue → user recv() wakeup. We'll trace this descent end-to-end with the demo in section 03.

The takeaway. "Forget OSI's 7 layers in production code — it's 5: L7 application, L4 transport (TCP / UDP), L3 network (IP), L2 link (Ethernet), L1 physical. Each layer adds its header on egress, strips it on ingress, and hands the rest to the right upstairs neighbour. A typical ~500-byte HTTP request grows to ~558 bytes on the wire — 54 bytes of headers — and gets MTU-capped at 1500 bytes per frame."

Sockets, ports, and the kernel's struct sock

A socket is the user-space handle for a kernel object that owns send and receive buffers, sequence numbers, congestion state, and the 4-tuple that uniquely identifies a flow. Everything above the wire ultimately reads and writes through this struct.

When curl calls socket(AF_INET, SOCK_STREAM, 0), the kernel allocates a struct socket wrapping a struct sock (or struct tcp_sock for TCP, which embeds struct sock as its first member — classic Linux struct-inheritance). The syscall returns a file descriptor pointing at that object, the same way open() returns one for a file. Every send() / recv() / read() / write() goes through the VFS layer, dispatches to the socket file_operations, and lands on TCP-specific routines.

Send and receive buffers

Each socket owns two per-flow buffers in kernel memory:

SO_SNDBUF — bytes send() can hold while TCP waits for ACKs. Linux default ~200 KB; capped by net.core.wmem_max (~4 MB).
SO_RCVBUF — bytes arriving from the wire that the application hasn't yet read. Linux default ~200 KB; this is also the basis for TCP's advertised receive window.

These caps interact with TCP's bandwidth-delay product (BDP). A long-fat-network link — 1 Gbps × 100 ms RTT = 12.5 MB in flight — needs a window that size or throughput collapses to a fraction of the link rate. Linux's auto-tuning (tcp_rmem, tcp_wmem sysctls) usually handles this; setting SO_RCVBUF manually disables auto-tuning and is almost always a mistake.

bind, listen, accept — the server side

A server reserves a (local IP, local port) tuple with bind(), marks it passive with listen(fd, backlog), and then accept() pulls fully-established connections off the accept queue. Two queues are involved:

SYN queue   — half-open: SYN received, SYN-ACK sent, awaiting ACK
              capped by net.ipv4.tcp_max_syn_backlog
              under SYN-flood, kernel falls back to SYN cookies

Accept queue — fully established: 3WHS complete, waiting for accept()
              capped by min(backlog, somaxconn)
              overflow → SYN-ACK retransmits, eventually RST or drop

Overflow of the accept queue is one of the most-missed production bugs: the kernel quietly drops new connections, the load balancer sees timeouts, the application logs nothing. Watch ss -lnt's Recv-Q (current accept-queue depth) and Send-Q (the configured cap) on your listening sockets.

The 4-tuple uniqueness rule

TCP demultiplexes incoming segments by the 4-tuple:

(src IP, src port, dst IP, dst port)

Any two ACTIVE connections must differ in at least one of these four.
A server listening on (*, 443) can have millions of connections
because the (src IP, src port) varies per client.
A client connecting to one (dst IP, dst port) is bounded by its
ephemeral port range.

Linux's default ephemeral port range is 32768–60999 (net.ipv4.ip_local_port_range) — about 28K ports. That's also the maximum number of simultaneous outbound connections to a single (dst IP, dst port) from one source IP. A frontend hitting a single backend pool with more than ~28K concurrent connections will hit EADDRNOTAVAIL at connect() time. Fixes: widen the port range, use multiple source IPs, add more backend ports, or use a connection pool that reuses TCP connections via HTTP keep-alive.

UDP — the same idea, simpler

A UDP socket also has a 4-tuple, send buffer, and receive buffer — but no sequence numbers, no acks, no congestion control, no in-order guarantees. Each sendto() is one packet on the wire (atomic up to MTU); each recvfrom() returns exactly one packet. If the receive buffer fills, the kernel drops, silently. UDP gives you a packet pipe; TCP gives you a byte stream.

epoll, the readiness signal

A modern server with thousands of sockets doesn't read each one in turn. It registers all fds with epoll_create + epoll_ctl and then blocks in epoll_wait, which returns the subset that have data ready or buffer space free. The socket layer signals the epoll instance whenever its receive queue gains a packet or the send buffer drains. We cover the readiness model in depth in the I/O Models primer — for this primer it's enough to know that the kernel exposes per-socket readiness as a first-class notification, and that the alternative (a thread per socket) does not scale past tens of thousands of connections.

The takeaway. "A socket fd points at a kernel struct sock that owns send / receive buffers, TCP state, and the 4-tuple (src IP, src port, dst IP, dst port). Server-side: bind → listen → accept, with the SYN queue and accept queue as the silent capacity bottlenecks. Client-side: the ephemeral port range caps simultaneous outbound connections to one (dst IP, dst port) at ~28K. UDP is the same idea minus reliability; epoll is how you wait on thousands of these at once."

IP routing, ARP, MTU — how the packet actually finds its way

Once your packet leaves the socket layer it needs three things to reach the wire: an interface to exit on, a next-hop IP, and a destination MAC. Routing answers the first two; ARP answers the third; MTU bounds how big the result can be.

The kernel's routing table is a longest-prefix match keyed on the destination IP. ip route show prints it; for each outgoing packet the kernel finds the most-specific matching prefix and uses its (interface, next-hop) pair.

$ ip route show
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.42
169.254.0.0/16 dev eth0 scope link metric 1000

Destination 93.184.216.34 → matches "default" → next-hop 10.0.0.1 via eth0
Destination 10.0.0.55     → matches 10.0.0.0/24 → next-hop is the host itself

ARP — the kernel needs a MAC

The Ethernet header demands a destination MAC, not an IP. ARP (Address Resolution Protocol) bridges the gap: "who has 10.0.0.1? Tell 10.0.0.42." The reply ("10.0.0.1 is at 00:1b:21:3a:7c:9f") is cached in the neighbour table for ~60 seconds; ip neigh show prints it.

$ ip neigh show
10.0.0.1 dev eth0 lladdr 00:1b:21:3a:7c:9f REACHABLE
10.0.0.55 dev eth0 lladdr 52:54:00:12:34:56 STALE
10.0.0.99 dev eth0 INCOMPLETE              ← no ARP reply yet

ARP is unauthenticated by design — any host can claim to own any IP. ARP spoofing / poisoning exploits this: attacker on the same L2 segment announces "I am 10.0.0.1", the victim's cache updates, and the attacker becomes a silent man-in-the-middle. The mitigations live above L2: dynamic ARP inspection on the switch (drop suspicious replies), 802.1X port authentication, or just do not put untrusted hosts on the same L2 segment as your servers. Cloud VPCs solve this by isolating each tenant in its own virtual L2.

MTU and the 1500-byte ceiling

Ethernet's standard payload is 1500 bytes (MTU). With 20 B IP + 20 B TCP, the most payload TCP can carry per segment is 1460 bytes (MSS). Datacenters often enable jumbo frames (MTU 9000) to amortize per-packet overhead — useful for storage replication, internal RPC, NVMe-over-TCP. Public internet, you stay at 1500.

If a packet larger than the path MTU shows up at a router, two outcomes:

IPv4, DF=0 — router fragments the packet into MTU-sized pieces, the receiver reassembles. Avoided in practice because fragmentation kills performance: a single lost fragment forces the whole packet to be retransmitted, and reassembly buffers are a DoS target.
IPv4 with DF=1, or IPv6 — router drops the packet, sends an ICMP "Fragmentation Needed" back with the next-hop MTU. The sender shrinks future packets. This is Path MTU Discovery (PMTUD).

TCP avoids fragmentation entirely by setting DF=1 and discovering path MTU at handshake time, then using MSS to keep every segment at or below the path MTU.

PMTUD black holes

PMTUD relies on ICMP Type 3 Code 4 ("fragmentation needed and DF set") reaching the sender. A misconfigured firewall that blocks ALL ICMP — depressingly common in "security hardening" templates — silently swallows these messages. Symptoms: small responses arrive instantly, larger ones hang. Sender keeps retransmitting the same too-big packet forever. The fix is to allow at least ICMP Type 3, or rely on PLPMTUD (Packetization Layer PMTUD, RFC 4821) which infers MTU from probe loss instead.

Walk it end to end

The demo below shows the same packet descending the five layers on the client and re-ascending on the server, with the header added at each step on the way down and stripped at each step on the way up. Watch the size column — 500 B of HTTP becomes 558 B on the wire, then is unwrapped back to 500 B inside the server process.

1 / 8

Each layer's only job is to add its header on egress and strip it on ingress. Five layers on the way down at the client, the same five in reverse on the server. The wire only sees one bag of bytes; the structure is restored on the other side.

The takeaway. "IP routing picks the outgoing interface and next-hop gateway by longest-prefix match on the destination IP. ARP turns that next-hop IP into a MAC for the Ethernet header. MTU (1500 B standard, 9000 B jumbo) bounds frame size. TCP avoids fragmentation via Path MTU Discovery — and the most common bug is a firewall blocking the ICMP it depends on, producing a PMTUD black hole where small requests work and large ones hang."

Drivers, ring buffers, NAPI, RSS, XDP, and kernel bypass

The bottom of the stack is the part most engineers never see — and the part that decides whether your server tops out at 100K or 10M packets per second. Ring buffers, interrupt coalescing, per-queue CPU affinity, and the eBPF / kernel-bypass exits.

The NIC and the kernel meet at a pair of ring buffers in main memory:

RX ring — an array of descriptors, each pointing at a kernel-allocated buffer (typically 2 KB). The NIC owns "ready" descriptors; when a packet arrives, it DMAs the bytes into the buffer and advances its tail pointer.
TX ring — same idea in reverse: kernel writes descriptors pointing at outgoing packet buffers, the NIC DMAs and serializes them.

Typical ring sizes are 256 to 4096 entries; you tune them with ethtool -G eth0 rx 4096. When the RX ring fills because the kernel hasn't drained it fast enough, the NIC drops packets and increments a counter visible in ethtool -S eth0 as rx_dropped or rx_missed_errors. That counter going up under load is the smoking gun for "the box can't keep up."

From interrupt to NAPI

Pre-2003, every received packet raised a hardware IRQ. At 10 Gbps that meant ~800K IRQs/sec — interrupt overhead alone saturated the CPU. The fix is NAPI (the "new" API, now twenty years old):

First packet arrives → NIC raises an IRQ.
Driver's IRQ handler disables further IRQs on that queue and schedules a softirq.
The softirq runs napi_poll in process context, draining up to budget (default 64) packets per call.
When the ring goes empty, NAPI re-enables IRQs. If packets are still arriving fast, the loop stays in polling mode and IRQs stay off — no per-packet IRQ cost at all.

This is the same idea as epoll one layer up: switch from interrupt-per-event to amortized polling under load. The softirq count in top rising into the double digits is NAPI doing exactly its job; only worry if a single CPU is pegged at 100% %si.

RSS — fan out across cores

Receive Side Scaling is the NIC feature that makes a single fast NIC usable on a many-core machine. The NIC has multiple RX rings (one per queue, typically one per CPU); on receive it hashes the packet 4-tuple, masks the result to pick a queue, DMAs into that queue's ring, and IRQs the CPU that owns that queue. The kernel only ever runs napi_pollfor queue N on CPU N. Same flow → same queue → same CPU → warm L1/L2 cache, no inter-core synchronization.

Diagnose with cat /proc/interrupts | grep eth0 — if all packets pile up on one CPU, RSS is misconfigured (or you're seeing one elephant flow that the 4-tuple can't split). Software fallback for NICs without enough queues is RPS (Receive Packet Steering), where the kernel does the hashing in software and dispatches via inter-processor interrupts.

XDP — eBPF at the driver level

XDP (eXpress Data Path) lets you attach an eBPF program inside the driver, before any sk_buff allocation or protocol processing. The program runs on the raw packet buffer and returns a verdict:

XDP_DROP    — discard immediately (DDoS scrub, ACL drop)
XDP_PASS    — continue up the normal stack
XDP_TX      — bounce back out the same NIC (load balancer)
XDP_REDIRECT — send to a different NIC or AF_XDP socket

XDP_DROP at the driver costs ~50 nanoseconds per packet — orders of magnitude cheaper than dropping in iptables. Cloudflare's L3 DDoS scrubber, Facebook's Katran L4 load balancer, and Cilium's container networking all use XDP. Limits: no in-flight TCP state, no fragmented packets, only what fits in a BPF program (1M instructions in modern verifiers).

Kernel bypass — DPDK, AF_XDP

If even XDP isn't fast enough, you skip the kernel entirely. DPDK (Data Plane Development Kit) binds the NIC to a userspace driver via UIO or VFIO; the application polls the ring buffer in a busy loop on a pinned CPU. Per-packet latency drops from ~3 µs (kernel stack) to ~300 ns (DPDK), and one core can sustain ~14 Mpps of 64-byte packets — line rate for 10 Gbps. Trading systems, NFV appliances, and high-end firewalls live here.

AF_XDP is the kernel's answer: a special socket type where the application maps RX/TX rings into its address space and the driver DMAs directly into application memory, bypassing sk_buff entirely. Close to DPDK throughput with much less operational pain (no exclusive NIC ownership, no custom drivers).

Note: kernel bypass means you implement TCP, retransmit, congestion control, ARP. For storage replication and trading, worth it. For a typical web server, the kernel stack is already good for ~1 Mpps per core — far below most application bottlenecks.

The takeaway. "The NIC and kernel meet at RX/TX ring buffers in RAM; full rings show as rx_dropped in ethtool -S. NAPI amortizes interrupts by switching to polling under load. RSS hashes the 4-tuple to a queue and pins that queue to a CPU, turning a single fast NIC into N independent receive pipelines. XDP runs eBPF in the driver for <100 ns drops; DPDK and AF_XDP bypass the kernel entirely when even that is too slow."

Quick reference

Six questions worth being able to reason about cold, and five red flags to spot in a code review.

What are the 5 layers a packet traverses, and what does each add?

L7 application (HTTP, variable size, names and methods); L4 transport (TCP 20 B / UDP 8 B, ports + reliability); L3 network (IPv4 20 B, source / dest IP + TTL); L2 data link (Ethernet 14 B + 4 B FCS, dest / src MAC); L1 physical (line coding, 1500 B MTU). A ~500 B HTTP request is ~558 B on the wire. OSI's session and presentation layers have no real implementations — what they were meant for is absorbed by the application or TLS.

What is a 4-tuple, and why does it bound your connection count?

(src IP, src port, dst IP, dst port) — TCP demultiplexes incoming segments by this. Any two active connections must differ in at least one. A server listening on (*, 443) can have millions of inbound connections (clients vary src IP/port). A single client to a single (dst IP, dst port) is bounded by its ephemeral port range (Linux default 32768-60999, ~28K). Hit that and connect() returns EADDRNOTAVAIL. Fix with wider port range, more source IPs, more backend ports, or HTTP keep-alive.

What is path MTU discovery and what happens when it's broken?

TCP sets the DF (Don't Fragment) bit and discovers the smallest MTU on the path by listening for ICMP "fragmentation needed" messages. It then keeps every segment at or below that size using MSS. When a firewall blocks all ICMP (common "security hardening" mistake), PMTUD silently fails: small responses arrive instantly, larger ones hang because the too-big packet is dropped at the bottleneck router with no notification reaching the sender. Fix: allow ICMP type 3, or enable PLPMTUD (RFC 4821).

What is RSS and why does it matter for a high-throughput server?

Receive Side Scaling — the NIC has multiple RX queues, hashes each incoming packet's 4-tuple to pick one, and IRQs the CPU pinned to that queue. Same flow → same queue → same CPU → warm cache, no inter-core synchronization. Without RSS, all RX processing piles up on one core and a single 10 Gbps NIC can't be drained. Check cat /proc/interrupts | grep eth0 — a flat distribution means RSS is working; one column dominating means it isn't.

When would you reach for XDP or DPDK?

XDP when you need cheap drops or trivial L2/L3 processing ahead of the normal stack: DDoS scrubbing (XDP_DROP at ~50 ns per packet), L4 load balancing (Katran, Cilium), or per-packet ACLs. DPDK / AF_XDP when you need full kernel bypass and line-rate forwarding: trading systems, NFV appliances, high-end firewalls. The cost is that you implement TCP, retransmit, congestion control yourself — for a typical web server the kernel stack at ~1 Mpps per core is already past the application's actual bottleneck.

What signals indicate driver-level packet drops?

ethtool -S eth0 | grep -E 'drop|miss|error' shows the NIC's own counters; rising rx_dropped or rx_missed_errors under load means the RX ring filled before the kernel could drain it. Cross-check with /proc/net/softnet_stat (third column is dropped packets per CPU because the per-CPU input queue overflowed) and nstat -az TcpExtListenDrops TcpExtListenOverflows for accept-queue overflows at the socket layer. Tune with ethtool -G (ring size), ethtool -C (interrupt coalescing), or scale RSS queues.

5 red flags in a code review

SO_SNDBUF / SO_RCVBUF tuned without measuring BDP. Manually setting these disables Linux auto-tuning and usually hurts throughput. The right value is bandwidth × RTT for the link; for everyone else, leave the sysctl defaults alone.
App opens 100K outbound connections to one (dst IP, dst port). The ephemeral port range caps you at ~28K. You will get EADDRNOTAVAIL in production. Use connection pooling with HTTP keep-alive, or fan out to multiple destinations / source IPs.
UDP code without explicit message-boundary handling. Each recvfrom() returns exactly one datagram or drops the buffer-tail bytes (with MSG_TRUNC noted). If the code assumes "keep reading until I have N bytes" you have a TCP-shaped bug in UDP-shaped code.
Production tcpdump on hot path without a BPF filter. An unfiltered tcpdump copies every packet into userspace, easily 10× the per-packet cost. At minimum pass a BPF expression (tcp port 443) so kernel filters the copy.
ICMP fully blocked at the edge. Path MTU discovery silently breaks, neighbour reachability detection breaks, traceroute breaks. Allow at least ICMP type 3 (destination unreachable), type 4 (source quench), type 11 (time exceeded). "Block all ICMP" is a 1990s security trope that does more harm than good.