DNS & HTTP Primer

When curl runs, the very first thing libc does is getaddrinfo("api.example.com", "443", ...). That function consults /etc/nsswitch.conf (which usually says "files dns" — try /etc/hosts first, then the DNS resolver), then the stub resolver in libc sends a UDP packet to whatever IP is in /etc/resolv.conf — often 127.0.0.53 on modern systemd-resolved hosts, or 1.1.1.1 / 8.8.8.8 if you've overridden it.

The recursive walk

That nearby IP is a recursive resolver. If it has the answer cached, you get a response in <1 ms. If not, it walks the DNS hierarchy on your behalf:

1. Root servers (the 13 labeled a.root-servers.net through m.root-servers.net, each actually anycast to hundreds of physical instances) tell it where to find .com.
2. TLD servers for .com (run by Verisign) tell it which nameservers are authoritative for example.com.
3. Authoritative server for example.com returns the A record for api.example.com.

That's three queries minimum on a cold cache, each with its own RTT. Recursive resolvers cache aggressively to amortise this — which is why your second DNS lookup is fast and your first one, from a fresh machine, can take 100 ms+.

$ dig +trace api.example.com
;; .                  86400 IN NS a.root-servers.net.   ; root
;; com.               172800 IN NS a.gtld-servers.net.  ; TLD
;; example.com.        86400 IN NS ns1.example.com.     ; authoritative
;; api.example.com.      300 IN A  203.0.113.42         ; final answer

Record types that matter

• A — IPv4 address. The most common record.
• AAAA — IPv6 address. "quad-A."
• CNAME — canonical name; an alias to another name. Cannot coexist with other records at the same name (RFC limitation). This is why apex records (example.com itself) can't be CNAMEs — you need ALIAS / ANAME or flattening from your DNS host.
• MX — mail exchanger. With a priority field for fallback ordering.
• TXT — arbitrary text. Used for SPF, DKIM, domain ownership verification ("put this TXT record to prove it's yours").
• NS — nameserver delegation. Says "ask ns1.example.com for anything under example.com."
• SOA — start of authority. Zone metadata, including the negative TTL — how long resolvers should cache NXDOMAIN responses.
• SRV — service location with port. Used by SIP, XMPP, Kerberos, and Kubernetes service discovery.
• CAA — Certification Authority Authorization. Lists which CAs are allowed to issue certificates for the domain. CAs are required by the CA/Browser Forum to check this before issuing.

TTLs and caching at every layer

Every record carries a TTL — "cache this for N seconds." And every layer caches: the browser (~minutes), the OS resolver (systemd-resolved or nscd), the recursive resolver (1.1.1.1 holds it for the TTL), and the authoritative server itself. A 300-second TTL means "the world might see your old IP for the next 5 minutes after you change DNS."

The negative TTL from the SOA record matters just as much. If you typo a hostname, NXDOMAIN gets cached too — often for 5+ minutes. The first sign of a misconfigured DNS deployment is "some users are seeing the wrong record and the rest see nothing" for an hour after you push the fix.

UDP vs TCP, EDNS0, and DoH / DoT

Classic DNS is UDP on port 53, with a 512-byte response limit per the original RFC. Anything bigger and the server sets the truncate flag (TC) and the client retries over TCP — until EDNS0 (RFC 6891), which lets clients advertise support for larger UDP responses (typically 4096 bytes). Most modern deployments use EDNS0 and never fall back to TCP for normal queries.

DNS over HTTPS (DoH) and DNS over TLS (DoT)wrap DNS queries in TLS so your ISP can't observe or rewrite them. DoH uses port 443 (looks like normal HTTPS traffic) and is what Firefox and Chrome speak natively to cloudflare-dns.com or dns.google. DoT uses port 853 (a dedicated port — easy to block at a corporate firewall, which is exactly why DoH won).

Anycast — why 1.1.1.1 is fast everywhere

Cloudflare's 1.1.1.1 and Google's 8.8.8.8 don't live in one datacenter. The same IP is announced from hundreds of points of presence via BGP anycast; whichever PoP is closest on the BGP topology wins. Your packets to 1.1.1.1 from Tokyo go to a Tokyo PoP; from London, to a London PoP. Same IP, different physical servers. This is why "public resolver" is fast enough to compete with your ISP's.

Production gotchas

• JVM caches DNS forever by default. Java'snetworkaddress.cache.ttl defaults to -1 (infinite) when a SecurityManager is set, and to 30 seconds otherwise. A long-running JVM resolving to a load balancer with round-robin DNS will pin itself to one IP and never notice that backend dying. Set it to the actual TTL.
• musl libc doesn't reload /etc/resolv.conf. Alpine-based containers cache the resolver IP at startup. If your DNS server's IP changes (or the container starts before /etc/resolv.conf is populated), you're stuck. Restart the process.
• TTL of 0 doesn't mean "don't cache." Browsers and some libraries cache for an internal minimum (often 60 seconds) regardless of what the record says.
• Hardcoded resolver IPs. Pointing apps at 1.1.1.1 directly bypasses your local cache and pins you to one external provider. When 1.1.1.1 had its 2020 outage, hardcoded stacks were the ones that died.

Once DNS gives us 203.0.113.42, curl opens a TCP connection to 203.0.113.42:443, does the TLS handshake (covered in the next primer), and then speaks HTTP. For HTTP/1.1 what goes on the wire after TLS is exactly this:

GET /user/42 HTTP/1.1
Host: api.example.com
User-Agent: curl/8.4.0
Accept: */*
Accept-Encoding: gzip, deflate
Cookie: session=abc123; csrf=xyz789

Two newlines (\r\n\r\n) end the request headers. If this were a POST, the body would follow with Content-Length: N telling the server how many bytes to read. The server then replies with a similar text-framed message:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 47
Cache-Control: private, max-age=60
ETag: "abc123"
Date: Wed, 27 May 2026 12:00:00 GMT

{"id":42,"name":"Ada","email":"ada@example.com"}

Keep-alive — reusing the TCP connection

In HTTP/1.0, every request opened a new TCP connection and closed it after the response. That meant a TCP handshake (1 RTT) and a TLS handshake (1-2 RTTs) per request — punishing latency. HTTP/1.1 made keep-alive the default: after the response, the connection stays open and the next request reuses it. Servers advertise a timeout (often 60-75 seconds) and a max request count per connection.

This is why curl https://a/x then curl https://a/y from the shell is much slower than curl https://a/x https://a/y — the first form opens two connections, the second reuses one.

Pipelining — and why nobody uses it

HTTP/1.1 technically allows pipelining: send request 2 before response 1 arrives, send request 3 before response 2, and the server must respond in order. In practice, almost nothing ships with it enabled because of two issues:

1. Head-of-line (HOL) blocking. If response 1 is slow (a 200ms DB query), responses 2 and 3 are stuck behind it even if they'd be instant individually. The client gets no improvement and the connection is wasted while the server thinks.
2. Broken proxies. Many transparent proxies in the wild misbehave on pipelined requests — they buffer, reorder, or drop responses. Browsers tried enabling pipelining in the mid-2000s and rolled it back universally.

Why browsers open 6 connections per origin

Since pipelining is dead and one HTTP/1.1 connection can carry only one outstanding request at a time, the only way to load multiple resources concurrently is to open multiple TCP connections to the same origin. The de-facto limit, enforced by every major browser, is 6 parallel connections per origin. That number is a compromise between "load assets fast" and "don't DDoS the server."

That's also why the old performance trick was "domain sharding" — serve assets from img1.example.com,img2.example.com, ... so the browser opens 6 × N connections. Modern HTTP/2 makes that trick actively harmful (one connection beats six), but the old advice still lurks in articles.

Chunked transfer encoding — streaming responses

Sometimes the server doesn't know the response size up front — it's generating output from a database query, a long-poll, or a server-sent-events stream. Instead of Content-Length: N, the server sets Transfer-Encoding: chunked and emits size-prefixed chunks:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/event-stream

5\r\n
hello\r\n
6\r\n
 world\r\n
0\r\n
\r\n            ← zero-length chunk = "end of stream"

The headers nobody escapes

• Host: — required since HTTP/1.1; the only way one IP can serve multiple virtual hosts. Without Host, name-based virtual hosting (everything modern) wouldn't work.
• Cookie: / Set-Cookie: — the original stateful add-on to a stateless protocol. Sent on every request to the matching domain.
• Cache-Control:, ETag:, If-None-Match: — how the browser asks "is my cached copy still good?" and how the server answers "304 Not Modified".
• Accept-Encoding: — "I can take gzip / br / deflate." The server picks one (or none) and announces with Content-Encoding:.
• Connection: keep-alive / Connection: close — vestigial but still around; keep-alive is the default and rarely sent explicitly anymore.

HTTP/1.1's "6 parallel connections per origin" was always a workaround. HTTP/2 (RFC 7540, 2015) replaces that with one connection per origin, internally split into many concurrent streams. Same HTTP semantics — methods, paths, status codes, headers — but the wire format changes from text to binary frames.

The binary framing layer

Every HTTP/2 message is a sequence of frames. Each frame has a 9-byte header (length, type, flags, stream id) and a payload. The important frame types:

+----+--------------------------------------------------+
| 24 | length (frame payload size, max 16 MB)            |
|  8 | type (HEADERS, DATA, SETTINGS, WINDOW_UPDATE...)  |
|  8 | flags (END_STREAM, END_HEADERS, ...)              |
|  1 | reserved                                          |
| 31 | stream id (0 = connection, odd = client streams)  |
+----+--------------------------------------------------+
|    | payload                                          |
+----+--------------------------------------------------+

• HEADERS — request/response headers (HPACK-encoded).
• DATA — request/response body bytes.
• SETTINGS — per-connection knobs (initial window size, max concurrent streams, max frame size, header table size).
• WINDOW_UPDATE — flow control credit. Sent both per-stream and per-connection.
• RST_STREAM — cancel a stream.
• PING / GOAWAY — keepalive and graceful shutdown.

Multiplexing works because each frame carries the stream id it belongs to. The server can interleave bytes from streams 1, 3, 5, 7 freely — the receiver demuxes by stream id and reassembles each stream's frames in order.

HPACK — why HTTP/2 headers are small

HTTP/1.1 sends the same Cookie:, User-Agent:, and Accept: on every request — often hundreds of bytes, often identical. HPACK (RFC 7541) compresses headers with a shared dynamic table.

Both peers maintain a synchronised table of (name, value) pairs. A common header gets sent once and then referenced by index — a single byte instead of 200. There's also a fixed static table of common headers (:method GET, :status 200) and Huffman coding for literal values. Typical savings on a cookie-heavy request: 80-95%.

Request 1: send "cookie: session=abc123" → adds to table at index 62
Request 2: just send "62" → one byte instead of 22
Request 3: just send "62" → still one byte

HPACK has a security wrinkle: the CRIME attack on TLS compression applies in spirit, so HPACK never compresses across stream boundaries that mix attacker-controlled and secret values. The spec is careful but practical implementations need to know the gotcha.

Flow control with windows

HTTP/2 has its own flow control on top of TCP's. Each stream starts with a 64 KB receive window. The receiver sends WINDOW_UPDATE frames to grant more credit as bytes are consumed. This matters in two ways:

1. For a fast response (a video segment), the default 64 KB window stalls the sender after one window-worth — you have to wait an RTT for the WINDOW_UPDATE. Clients that care for throughput grow the initial window via SETTINGS to several MB.
2. For a slow receiver, flow control lets the receiver throttle a fast sender without dropping data — essential when streams multiplex onto one TCP connection.

Server push — the feature nobody shipped well

HTTP/2 added server push: the server can preemptively send app.css in response to a request for index.html. In practice it was a flop. Servers couldn't know what the browser already had cached, so pushed a lot of waste; browser support was inconsistent; the rules around cancellation were complex. Chrome disabled push in 2022. The modern replacement is Link: rel=preload hints in the HTML response, which lets the browser decide.

The HOL blocking that didn't go away

HTTP/2 fixed application-level HOL blocking: streams are independent at the HTTP layer. But HTTP/2 still runs over TCP, and TCP delivers a single ordered byte stream. One lost packet pauses the whole connection while it waits for the retransmission — even if 5 of the 6 streams sharing the connection had their data already on the receiver. This is TCP-level HOL blocking, and it's the single biggest reason HTTP/3 exists. Under 1% packet loss, HTTP/2 can be slower than 6-parallel HTTP/1.1 because the parallel HTTP/1.1 connections at least isolate one stream's loss from another's.

HTTP/2 was a great rewrite of HTTP over the same TCP it had always used. HTTP/3 (RFC 9114) goes further: keep the application-layer framing similar, but throw out TCP and TLS-on-top as separate layers. Replace them with QUIC (RFC 9000) — a transport protocol designed end-to-end with multiplexing, TLS 1.3, and connection migration baked into the same handshake.

Why UDP, and how QUIC isn't "just" UDP

QUIC runs on top of UDP because every middlebox on the internet — carriers' NAT boxes, corporate firewalls, captive portals — understands UDP and TCP and nothing else. A brand-new transport protocol would be silently dropped. So QUIC pays the "UDP tax" (a tiny UDP header) and rebuilds reliability, congestion control, in-order delivery, and stream multiplexing in userspace.

Critically, QUIC's streams are independent at the transport level. Each stream has its own sequence numbers. A lost packet on stream 3 forces a retransmit of just that stream's bytes — streams 1, 2, 4, 5, 6 keep flowing the moment subsequent packets arrive. This is what kills HTTP/2's TCP-level HOL blocking dead.

TLS 1.3, baked in

QUIC doesn't layer TLS on top — it integrates TLS 1.3 into its handshake. The cryptographic exchange and the transport parameter exchange happen in the same flight of packets. Net effect: a fresh connection to a new server takes 1 RTT(vs TCP's 1-RTT + TLS's 1-RTT = 2 RTTs for HTTP/2). To a server you've talked to before, 0 RTT — the first request goes out in the same packet as the handshake, with the resumption secret derived from a previous session.

0-RTT has a subtle caveat: those early bytes are vulnerable to replay attacks. Servers must treat 0-RTT requests as idempotent (GET only) or have application-level replay protection. CDNs do this for you; raw QUIC servers need explicit care.

Connection migration

A TCP connection is identified by the 4-tuple (src IP, src port, dst IP, dst port). Your IP changes — moving from Wi-Fi to LTE, a NAT rebinding, a roaming handoff — and the connection dies. Every long-running HTTP/2 connection drops; the browser opens a new one and the user sees a stall.

QUIC identifies a connection by a Connection IDcarried in each packet. Your packets can come from a different IP and the server still associates them with your session. Your phone keeps streaming a video as it moves from cafe Wi-Fi to LTE without a reconnect. This is the killer feature for mobile.

QPACK — HPACK's out-of-order cousin

HPACK assumed in-order delivery of frames — fine for HTTP/2 on TCP. QUIC streams arrive out of order, so HPACK's dynamic table updates can't depend on previous frames having been processed. QPACK (RFC 9204) separates header encoding into a control stream (table updates) and the request streams (header references), with explicit synchronisation. Same compression ratio, fixed for QUIC's out-of-order reality.

ALPN — how the client ends up speaking h3

Curl runs curl https://api.example.com/user/42. How does the client know whether to speak HTTP/1.1, h2, or h3?

1. For HTTP/1.1 vs h2: during the TLS handshake, the client lists the protocols it supports in the ALPN (Application-Layer Protocol Negotiation) TLS extension. The server picks one and includes it in the ServerHello. By the time TLS finishes, both sides know which HTTP version to speak.
2. For HTTP/3: the client first connects with TCP+TLS and gets an Alt-Svc response header: Alt-Svc: h3=":443"; ma=86400. That tells the client "next time, try QUIC on port 443." The client caches that for the max-age. On subsequent visits, it races a QUIC handshake against TCP and uses whichever connects first.

Adoption — where you'll see it

Cloudflare's 2024 numbers: ~30% of HTTPS traffic across their network is HTTP/3. Google, Meta, Akamai, and Fastly all serve it. Every major browser supports it. Where you won'tsee it: most internal infrastructure. Your own backends behind a load balancer are almost certainly h2. The middleboxes between your servers don't know QUIC; your monitoring tools may not parse it; your tracing libraries assume TCP. HTTP/3 is the CDN-edge protocol, h2 stays for everything internal.

Red flags in code review

• JVM with networkaddress.cache.ttl=-1. Caches DNS forever. A round-robin DNS pointing at a load balancer gets pinned to one IP; when that backend dies, the JVM doesn't notice. Set it to the actual record TTL (typically 30-300 seconds).
• Hardcoded resolver IP. Pointing apps directly at 1.1.1.1 or 8.8.8.8 bypasses the local cache and pins you to one external provider. When that provider has an outage (which happens), your stack dies with it. Use the system resolver.
• App expecting HTTP/2 server push to work. Chrome disabled push in 2022; other browsers were inconsistent for years. Use Link: rel=preload hints in the HTML response instead — they let the browser decide what it actually needs to fetch.
• Large Cookie: header on every request, with a pre-HPACK code path or no HTTP/2 backend. A 4 KB cookie sent on every asset request adds up fast — 50 assets × 4 KB = 200 KB of overhead per page load on HTTP/1.1. On HTTP/2 with HPACK the same cookie costs ~1 byte after the first send.
• Custom protocol layered on HTTP/1.1 pipelining. Pipelining is not deployed because of proxy breakage and HOL blocking. A "clever" design that assumes you can send 1000 requests on one connection and stream back responses out of order will fail the moment a broken proxy is between you and the server. Use HTTP/2 streams or WebSockets.