Anatomy of a Web Request Primer

The actors: your laptop (curl process, libc, kernel, NIC), the public internet in between (~30 ms RTT, a dozen intermediate routers, a CDN edge or two), and a server somewhere — nginx fronting a backend process, that backend talking to Postgres or MySQL, the database backed by NVMe through a filesystem. None of those pieces was designed in a vacuum; each was a deliberate response to a problem the layer below couldn't solve, or an abstraction the layer above demanded.

The four layers we covered

The CS Fundamentals series spent twenty primers building up the stack. Each was one slice of this same request:

• Hardware (5 primers). Bits and number representation, what one CPU core actually does, the six orders of magnitude of memory latency, multi-core coherence, the assembly the compiler emits. Start with CPU Architecture.
• Operating system (8 primers). Processes and threads, the scheduler that picks who runs, virtual memory and page faults, allocators, the syscall / interrupt boundary, the filesystem and page cache, I/O models from blocking to io_uring, concurrency primitives. Start with Process & Thread.
• Network (4 primers). The packet's journey off the NIC, TCP's reliability machinery, DNS and HTTP at the top, TLS wrapping it all in confidentiality. Start with Network Stack.
• Storage & data (3 primers). The physics of disks, B-trees and LSM-trees the database is built on, the transactional guarantees the application is allowed to rely on. Start with Disk Storage.

What this capstone does

Five sections, in order. Section 2 is the spine — twelve stages of one curl call, walked end to end, every primer cross-linked to its stage. The embedded waterfall demo lets you step through it visually. Section 3 takes the same trace and asks where the time actually goes — the latency budget. Section 4 asks what breaks where — every layer's favourite failure mode, mapped to its primer. Section 5 is a quick-reference Q&A.

The teaching contract is that every primer in the series gets linked from somewhere in the body — no exceptions. If you understood each individually, this primer ties them into a single story; if you skimmed some, the cross-links are an invitation to go deeper at the exact spot you found yourself losing the thread.

Stage 1 — Keystroke and process spawn

You hit Enter; bash forks a child and execs curl. The kernel builds a new address space, a fresh set of page tables, and loads the curl ELF binary's text and data segments. glibc runs its constructors; the heap starts at a fresh brk. Even before the first byte of network traffic, we've already spent ~5 ms doing OS-level setup. Detail: Process & Thread for the fork / exec machinery, Virtual Memory for the address-space construction and demand-paged loading, and Memory Allocation for what glibc's heap is doing under the hood.

Stage 2 — DNS resolution

curl calls getaddrinfo("api.example.com", ...). The stub resolver in glibc consults /etc/nsswitch.conf, usually hands off to systemd-resolved, which sends a UDP query to the configured upstream (often 1.1.1.1 or 8.8.8.8). If the resolver doesn't have it cached, it walks root → TLD → authoritative nameserver. ~30 ms is typical on a first miss; with a warm cache it's under a millisecond. Detail: DNS & HTTP.

Stage 3 — socket() and the first SYN

curl calls socket(AF_INET, SOCK_STREAM, 0) then connect(). The first syscall allocates a kernel struct sock; the second triggers the kernel to build an IP header + TCP SYN segment, queue it on the appropriate qdisc, and hand it to the NIC driver, which writes a TX descriptor into the device's ring buffer. The NIC DMAs the packet and signals completion. Detail: Syscalls & Interrupts for the user→kernel transition and Network Stack for the header construction and driver path.

Stage 4 — The 3-way handshake

SYN goes out, the server replies SYN-ACK from its listen() backlog, the client sends ACK. One full round trip — ~30 ms on a typical internet path. Both sides are now in ESTABLISHED; sequence numbers are agreed, window sizes negotiated, MSS and selective-ACK options exchanged. Detail: TCP Deep Dive.

Stage 5 — The TLS handshake

With https://, there's a second handshake on top of the TCP one. In TLS 1.3 the client sends a ClientHello containing a key share (an ephemeral ECDHE public key) plus the cipher suites it supports; the server replies with ServerHello, its X.509 certificate chain, and a Finished message — all in a single round trip. The client verifies the chain against the system trust store and derives the session keys. Another ~30 ms. Detail: TLS & Security.

Stage 6 — Sending the HTTP request

Now we send GET /user/42 HTTP/1.1 with the standard headers. The bytes are first encrypted into a TLS record, then wrapped in a TCP segment, an IP packet, an Ethernet frame, posted as a TX descriptor on the NIC, DMA'd to the wire, and routed across the internet to the server. Half an RTT — ~15 ms typical. Detail: DNS & HTTP for the request format, Network Stack for the encapsulation, and CPU Architecture for how the NIC's descriptor ring and DMA actually move bytes without involving the CPU more than necessary.

Stage 7 — Server NIC, IRQ, softirq

On the server, the NIC DMAs the incoming packet into its RX ring and raises an MSI-X interrupt. The driver's IRQ handler schedules a softirq — NAPI polling reads as many packets as it can in one batch. Each packet climbs the stack: IP layer (re)assembly, TCP segment processing, payload appended to the socket's receive buffer, the listening fd marked readable. Detail: Syscalls & Interrupts for the IRQ → softirq mechanics and Network Stack for the climb back up the layers.

Stage 8 — epoll wakes a worker

A server worker thread had been parked in epoll_wait(). When the socket flips to readable, the kernel walks epoll's ready list, marks the thread runnable, and the CFS scheduler puts it on a CPU. The wake-up itself uses a futex under the hood — cheap when uncontended, expensive when many threads pile up. Detail: I/O Models for epoll vs io_uring and why this is the architecture of every high-throughput server, CPU Scheduling for what "runnable" actually means to CFS, and Concurrency Primitives for the futex wake-up itself.

Stage 9 — Parsing HTTP and dispatch

The worker thread reads the request bytes, parses the HTTP request line and headers, and dispatches to the /user/:id handler. The parse loop is the kind of thing that lives entirely in L1/L2 — small hot buffers, mostly predictable branches, the kind of code where the difference between "naive" and "careful" is whether the branch predictor catches the inner loop. Detail: CPU Architecture for what the pipeline is actually doing, Memory Hierarchy for why staying in L1 is the difference between fast and slow, Cache Coherence for what happens once the connection table is touched from multiple cores, and Assembly & ISA for what the JIT or AOT compiler emitted for the hot path.

Stage 10 — The database query

The handler issues SELECT * FROM users WHERE id = 42 over the pooled connection to Postgres. The DB does a B+tree descent on the primary key index: root page, internal pages, leaf page; the leaf gives a heap tuple pointer; the buffer pool either has the heap page already (cache hit, ~50 μs total) or fetches it from disk through the filesystem and NVMe (cache miss, +50 μs to a few ms). The row is read under MVCC visibility rules. Detail: Database Storage for the B+tree and buffer pool, Memory Hierarchy for why the buffer pool exists at all, File System for the page-cache layer below the buffer pool, and Disk Storage for what NVMe actually does on a 4 KB random read.

Stage 11 — Serializing the response

The handler builds a response object, hands it to a JSON serializer, which asks the allocator for a 2 KB buffer and writes the UTF-8 bytes into it. The TLS library encrypts the resulting HTTP response into a TLS record; the kernel pushes that down through TCP, IP, the driver, the NIC. Detail: Memory Allocation for the allocator (jemalloc / tcmalloc in most servers), Binary & Number Systems for the UTF-8 encoding the serializer is producing, and TLS & Security for the AEAD encryption of each record.

Stage 12 — Round trip home

Packets traverse the internet (~30 ms RTT, so the response trip adds ~15 ms). On the client, the NIC RX path mirrors stage 7 in reverse: NIC DMA → softirq → TCP reassembly → socket receive buffer. curl reads from the socket, the TLS library decrypts the record, and curl writes the plaintext JSON to stdout. Meanwhile, the server's handler closes the database transaction with a COMMIT — the visible effect of which is one of N concurrent transactions on the server, ordered by the isolation level you chose. Detail: TCP Deep Dive for reassembly, Network Stack for the descend back through the layers, I/O Models for how curl reads from the socket, and Transactions for what the server's COMMIT actually guarantees.

A typical first-time HTTPS request to a single backend, ~30 ms RTT, warm DB cache, no connection reuse:

STAGE                              TYPICAL      DOMINATED BY
─────────────────────────────────  ───────      ──────────────────────────
1. Process spawn + glibc init      ~5 ms        fork, ELF load, page tables
2. DNS lookup (first hit)          ~10–50 ms    upstream resolver + auth NS
3. TCP 3-way handshake             ~30 ms       1 × RTT
4. TLS 1.3 handshake               ~30 ms       1 × RTT + cert verify
5. Send request (half RTT)         ~15 ms       wire + intermediate routers
6. Server NIC RX + dispatch        ~1 ms        softirq + epoll wake
7. Parse HTTP + route              ~1 ms        L1/L2 hot, mostly branches
8. DB query (warm)                 ~1–50 ms     buffer pool vs disk
9. Serialize + TLS encrypt         ~1–5 ms      allocator + AEAD
10. Response (half RTT)            ~15 ms       wire
11. Client NIC RX + TLS decrypt    ~1 ms        softirq + AEAD
12. curl write to stdout           ~50 μs       one write() syscall
─────────────────────────────────  ───────
TOTAL                              ~110–250 ms

Where headroom hides

Three numbers in that table are the entire game. DNS: cached after the first lookup, so amortized to zero on subsequent requests — but the first request always eats it. See DNS & HTTP for cache layers (browser, OS, resolver). TCP handshake: ~30 ms once, then zero for the lifetime of the connection if you keep-alive. TLS handshake: another ~30 ms once, again zero on a reused session. Reuse the connection and you save 60 ms per request — that's why HTTP/2 multiplexing and connection pools matter so much. HTTP/3 (QUIC) saves another RTT by fusing the TLS and transport handshakes. See TCP Deep Dive and TLS & Security for the keep-alive and 0-RTT machinery.

Two surprising ones

First-request DNS is often slower than you expect because the resolver chain is deep: your machine → systemd-resolved → upstream → root → TLD → authoritative. Each hop is a real UDP round trip, and the authoritative server may be on another continent. A first lookup of 80 ms is not unusual; 200 ms happens.

fsync on the WAL is often the longest single thing happening on the server — you don't see it in the trace above because we're doing a SELECT, not a write. Make this a POST that updates last_seen, and the server's COMMIT issues an fsync on the write-ahead log. That's 5–15 ms on NVMe, 50+ ms on cloud block storage with a degraded disk, and it dominates the latency of any durable write. See Database Storage for WAL, Transactions for what COMMIT actually waits on, and Disk Storage for what fsync makes the SSD do (flush the volatile cache).

P99 spikes — the mental playbook

Average latency is the table above; P99 latency tells a different story. The four usual suspects:

• GC pause or compaction. Most managed runtimes (JVM, Go, .NET) periodically stop the world to compact. Read Memory Allocation for why generational GCs exist and what they cost.
• Scheduler delay. Under CPU pressure your worker may sit in the runqueue for tens of milliseconds before the scheduler picks it. See CPU Scheduling.
• Page fault on a cold page. If the worker's stack or a hot data page was swapped out (or never paged in), the first touch is a disk round-trip. See Virtual Memory for major vs minor faults.
• fsync stall. The disk or its write cache hit a contended state; commit blocks. See File System for the journal commit path and Disk Storage for why SSDs occasionally pause everything to GC their NAND.

• DNS server unreachable. Query times out, glibc returns EAI_AGAIN after a few seconds, the application surfaces a confusing "name resolution failed". Often caused by a wedged systemd-resolved or a dropped UDP packet on a network with no retry budget. Fix: short resolver timeouts, retry once, never block a request handler on synchronous DNS. See DNS & HTTP.
• TCP SYN dropped silently. Firewall, path MTU discovery broken, or an over-aggressive cloud security group. Client retransmits SYN with exponential backoff (1s, 3s, 7s…) — which is what an unexplained "5-second hang" actually looks like. See TCP Deep Dive and Network Stack.
• TLS certificate expired or chain broken. The handshake fails before any HTTP is sent. Symptom: connection opens, then immediately closes with a TLS alert. The most common culprits in production are an expired leaf cert, a missing intermediate, or a system trust store missing the relevant root. See TLS & Security.
• Ephemeral port exhaustion. The client opens ~50K connections faster than they can be cleaned up; newconnect() calls return EADDRNOTAVAIL. Usually caused by per-request TCP connections (no pool) hitting a load balancer that keeps them in TIME_WAIT for 60+ seconds. See TCP Deep Dive and Network Stack for connection state lifetimes.
• Server NIC RX overrun. Packets arrive faster than the driver can drain the ring; visible as rx_dropped in ethtool -S. The TCP retransmits eventually mask it but tail latency goes through the roof. See Network Stack for ring buffers and Syscalls & Interrupts for the softirq budget that determines drain rate.
• App worker starved by CPU pressure. Under load (or a runaway neighbour on a shared host), your worker sits in the runqueue for tens of milliseconds. Tail latency rises uniformly across all handlers, not specific ones — a telltale sign. See CPU Scheduling.
• Page fault during the request. Memory pressure swapped out the worker's stack or a hot data page; first touch is a disk read (5-15 ms). Diagnosed via vmstat showing non-zero si/so. See Virtual Memory and Memory Allocation.
• Database row lock contention. Two transactions want to UPDATE the same row; one waits up tolock_timeout. Under serializable isolation, the loser gets a retry-able error. See Transactions for isolation and Concurrency Primitives for the underlying lock machinery.
• fsync stall on COMMIT. NVMe degraded, disk full, or volatile cache flush slow. COMMIT blocks for hundreds of milliseconds. Cloud block storage with throttled IOPS is the canonical environment to see this. See File System for the journal commit path and Disk Storage for why NAND occasionally pauses everything to garbage-collect.
• Cache coherence false sharing. Two cores repeatedly write neighbouring fields of the same 64-byte struct (counter, lock, hot-table slot). MESI bounces the line between cores; throughput silently drops by 2-4×. See Cache Coherence.
• HTTP/2 stream-ID exhaustion or HOL block.One connection runs out of stream IDs (2^31, but you can hit it on a long-lived connection with high churn), or one slow stream blocks the whole connection in TCP head-of-line. HTTP/3 fixes the HOL by moving to QUIC. See DNS & HTTP.

Production red flags

• "Just throw more replicas at it."Without measuring where the latency actually sits, scaling horizontally just multiplies the same wasted handshakes and fsync waits. Profile first; scale second.
• Treating an external HTTP call as cheap inside a request handler. Synchronous DNS + TCP + TLS handshake on every request burns 60–90 ms per call. If you're making N such calls per user request, you pay it N times. Pool connections, cache DNS, batch where possible.
• Per-request database connection without pooling.A new TCP + TLS handshake on every query is the kind of self-inflicted wound a connection pool fixes in one config line. The PgBouncer-or-equivalent rule.
• fsync skipped on commit because "the request was fast enough." Sometimes correct (caches, logs); often the start of a data-loss bug that surfaces only on the one machine that ever loses power. Decide explicitly which writes need durability.
• No distributed tracing. Debugging a slow request by guessing which of the 20 layers is to blame is the biggest single productivity sink in production engineering. Spans on every layer boundary — DNS, TCP, TLS, server entry, DB query, downstream call — turn "where is the time going?" from a multi-day investigation into a single screen.