Disk Storage Primer

Right now, two writes are going to disk for our curl request: the web server appends a ~200-byte line to access.log, and the database backing user/42 reads a few B-tree pages (~4 KB each) to satisfy the query. Whether those operations cost 5 ms each or 50 μs each is a function of which physical layer you provisioned.

HDD — the spinning platter

A hard disk drive is a stack of magnetic platters spinning at 7,200 RPM (consumer) or 15,000 RPM (enterprise), with a movable actuator arm holding one read/write head per surface. To read a block, the head has to seek to the right track and then wait for the right sector to rotate under it. The math is brutal:

seek time            ~3-15 ms   (full-stroke vs adjacent)
rotational latency   ~4 ms       (half a rotation at 7200 RPM)
transfer time        ~0.05 ms    (negligible for 4 KB)
                     ──────
random 4 KB read     ~8-15 ms

sequential throughput     ~150-250 MB/s
random IOPS               ~100-200

That's why a 200-IOPS HDD with queue depth 32 averages 160 ms of wait per request — your database's P99 latency is the physics of a moving metal arm. Sequential reads are 1,000× faster than random because the head doesn't move and the rotation is already lined up. This is the fundamental reason database engines obsess about sequential I/O — write-ahead logs, LSM-tree compactions, batch ETL. They're all trying to turn IOPS-bound work into MB/s-bound work.

SATA SSD — no moving parts, still slow wire

A solid-state drive replaces platters with NAND flash chips. No seek, no rotation. A 4 KB random read becomes a NAND program cycle (~50 μs) plus a SATA round-trip — call it 100 μs total. The SATA cable itself caps you at 6 Gbps = 600 MB/s, regardless of how fast the NAND is:

random 4 KB read         ~100 μs
sequential throughput    ~500-550 MB/s   (capped by SATA 3.0)
random IOPS              ~50,000-100,000

AHCI command queue       32 outstanding ops

That's a 500× IOPS improvement over an HDD, but the SATA protocol — designed for spinning rust — leaves significant NAND performance on the table. AHCI, the host controller spec, has a single 32-deep command queue. Modern NAND can serve hundreds of concurrent reads; SATA cannot ask for them.

NVMe — PCIe-attached, queue-per-core

NVMe sits on the PCIe bus directly. PCIe 4.0 x4 = 8 GB/s of raw bandwidth (PCIe 5.0 x4 = 16 GB/s). The NVMe protocol replaces AHCI's single 32-deep queue with up to 64K submission queues, each up to 64K deep — typically one queue pair per CPU core, so the kernel never contends on a global lock to submit I/O:

random 4 KB read         ~10-50 μs
sequential throughput    ~7 GB/s          (PCIe 4.0 x4)
random IOPS              500,000 - 1.5M+

queues                   up to 64K × 64K deep

That's a 100× latency improvement over HDD and a 10× IOPS improvement over SATA SSD. But you only see those numbers at queue depth 32 or higher — a single-threaded synchronous read loop achieves maybe 30,000 IOPS even on a million-IOPS device, because the device is idle waiting for the next request. This is why benchmarks like fio --iodepth=32 matter, and why modern I/O paths (io_uring, kernel-bypass SPDK) exist — to actually fill those queues.

The latency cliff — where storage sits in the hierarchy

Storage latency only makes sense relative to everything else the CPU might be waiting on:

CPU L1 cache               ~1 ns
CPU L2 cache               ~4 ns
CPU L3 cache               ~15 ns
DRAM (main memory)         ~80 ns
NVMe SSD (random read)     ~10-50 μs    ← 100,000× slower than L1
SATA SSD (random read)     ~100 μs
network SSD / EBS gp3      ~500 μs
HDD (random read)          ~5-15 ms     ← 10,000,000× slower than L1
S3 / cold storage (1st B)  ~10-100 ms

Two takeaways. First, the gap between a cache hit and a cold disk read is roughly the gap between "walking to your kitchen" and "walking to a different continent" — this is why every layer above storage (Postgres shared_buffers, Redis, the kernel page cache) is so aggressive about caching. Second, the gap between NVMe and HDD (1,000×) is bigger than the gap between DRAM and NVMe (~500×) — the network-attached HDD era is genuinely over for any latency-sensitive workload.

IOPS vs throughput — they're the same number at 4 KB

For tiny I/O (4 KB, the typical database page), IOPS times block size equals throughput. 100,000 IOPS × 4 KB = 400 MB/s. So a SATA SSD's "500 MB/s sequential" spec and its "100,000 random IOPS" spec describe roughly the same wall. For large I/O (1 MB ETL reads), throughput dominates — even an HDD pulling 250 MB/s sequential is fine. The reason 4 KB-random-read is the canonical benchmark is that it matches the most painful real-world workload: OLTP, where every user query touches a few index pages scattered around the data file.

NAND cells — SLC, MLC, TLC, QLC

At the bottom of an SSD is a grid of NAND cells, each storing 1 to 4 bits depending on the technology:

• SLC (1 bit/cell) — fastest, ~100K program/erase cycles, enterprise / cache only. Nearly extinct in standalone form.
• MLC (2 bits/cell) — ~10K P/E cycles. Most older enterprise SSDs.
• TLC (3 bits/cell) — ~3K P/E cycles. The dominant tech in consumer and most modern enterprise SSDs.
• QLC (4 bits/cell) — ~1K P/E cycles. Cheap and dense; used for read-heavy bulk storage (Samsung 870 QVO, Intel D5-P5316).

Higher density per cell means cheaper $/TB but fewer rewrites before the cell breaks. This is why endurance ratings(TBW — total bytes written) are vendor-published: a consumer 1 TB TLC SSD is rated ~600 TBW, an enterprise NVMe at ~3,000+ TBW. Over that lifetime the cell wear-leveling and over-provisioning hold the device together.

Pages and erase blocks — the asymmetry

Two granularities matter, and they're different:

• Page — the smallest unit you can program (write). Typically 4 KB to 16 KB.
• Block — the smallest unit you can erase. Typically 128 KB to 1 MB (64-256 pages per block).

You cannot rewrite a page in place. If a page has been programmed even once, the only way to put new data at that LBA is to (a) write to a different free page and (b) eventually erase the whole block. This is the source of every other SSD oddity.

FTL — the firmware that hides all this

The Flash Translation Layer is software running on the SSD's controller. It does four jobs:

1. LBA → physical-page mapping. A big in-RAM table (typically ~1 GB of DRAM per 1 TB of NAND). When the host writes to LBA 42, the FTL picks any free physical page and updates the map.
2. Garbage collection. When free pages run low, pick a block with mostly invalid pages, relocate the still-valid ones, then erase the source block. Visualized in the demo below.
3. Wear leveling. Spread program/erase cycles across all blocks so no single block hits its P/E limit while others sit unused. Cold data (rarely overwritten) gets relocated periodically to free up well-worn blocks for hot data.
4. Bad-block management. Cells eventually wear out and refuse to program. FTL retires them and substitutes from a spare pool, transparently.

Write amplification — the silent IOPS multiplier

Write amplification (WA) = physical bytes written / logical bytes written. Ideal is 1.0 (every host write costs one device write). In practice:

sequential writes, plenty of free space      WA ≈ 1.0
mixed random + 30% over-provisioning         WA ≈ 1.2 - 2.0
heavy random writes on nearly-full SSD       WA ≈ 5 - 10+
adversarial (4 KB random, 95% full, no TRIM) WA can exceed 20

High WA matters for two reasons: it burns through TBW endurance proportionally (a WA of 5 means your 600 TBW drive dies after 120 TB of actual host writes), and the GC IO competes with your application IO, raising P99 latency exactly when the device is most loaded.

TRIM, over-provisioning, and how to fight WA

Two levers reduce WA dramatically:

• TRIM / discard. A command the OS sends to the SSD saying "these LBAs no longer hold meaningful data." Without it, the FTL doesn't know which pages can be dropped during GC — it sees them as valid and keeps copying them around. Modern filesystems (ext4, XFS, APFS, NTFS) issue TRIM automatically on file delete (or via a weekly fstrim cron). Check with lsblk --discard.
• Over-provisioning. Reserve a chunk of NAND invisible to the OS. A 1 TB raw drive sold as 960 GB has ~7% OP; enterprise drives might reserve 20-50%. More OP means more free pages always available, so GC always has a low-cost block to reclaim, so WA stays close to 1. You can manually under-partition a consumer drive to get the same effect.

Interactive — one rewrite, twenty-one physical writes

SMART, wear indicators, smartctl

Every SSD exposes a SMART (Self-Monitoring, Analysis, and Reporting Technology) page. The fields that matter for SSD health:

# smartctl -a /dev/nvme0n1
...
Percentage Used:                    37%        ← P/E budget used
Data Units Written:                 312,500,000 [160 TB]
Power On Hours:                     8,760
Available Spare:                    98%        ← spare blocks left
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Percentage Used climbs from 0 to 100 over the drive's rated TBW; past 100 the vendor stops guaranteeing data integrity.Available Spare is the bad-block reserve — if it drops below ~10%, schedule a replacement. Plot these via Prometheus node_exporter (nvme_ metrics) for any production fleet.

SATA — the legacy spinner protocol

SATA 3.0 tops out at 6 Gbps (≈ 600 MB/s after 8b/10b encoding) and uses the AHCI host controller interface — a spec designed around the assumption that the device is an HDD. AHCI exposes one command queue, 32 entries deep, with no notion of parallelism between cores. That was fine when the device could only handle one IO at a time anyway. With NAND able to serve hundreds of concurrent reads, AHCI is the bottleneck.

SATA SSDs still ship for compatibility — cheap consumer drives, 2.5-inch enterprise bays — but no new performance-tier hardware is designed for SATA. If you see SATA SSDs in a database tier, it's either legacy or a cost decision.

SAS — enterprise serial

SAS (Serial Attached SCSI) was the enterprise answer to SATA, with 12 Gbps and 24 Gbps generations, dual-port connectivity for HA (each drive attached to two controllers), and much better command queueing (256 entries). Used heavily in enterprise spinning disk arrays (3PAR, NetApp FAS) and in some enterprise SSDs. NVMe has largely displaced SAS SSDs but SAS HDDs remain dominant for large-capacity nearline storage.

NVMe — designed for parallel NAND

NVMe (Non-Volatile Memory Express) is a fresh protocol on top of PCIe, designed in 2011 explicitly for SSDs. It throws out AHCI's assumptions and replaces them with:

• Up to 64K submission queues, each up to 64K commands deep.Typical configuration: one submission queue + completion queue pair per CPU core. The kernel's NVMe driver maintains per-CPU queues so submitting IO never contends on a global lock.
• Submission/completion ring buffers in host RAM.The host writes a Submission Queue Entry, then rings a doorbell (PCIe MMIO write) to tell the device new work is queued. The device DMAs the data, writes a Completion Queue Entry, and either raises an MSI-X interrupt or the host polls. The model mirrors a modern NIC almost exactly.
• ~13 mandatory commands (vs SCSI's sprawling spec). Lean protocol, lean drivers, lean firmware.

Wire speeds scale with PCIe: PCIe 3.0 x4 = 4 GB/s, PCIe 4.0 x4 = 8 GB/s, PCIe 5.0 x4 = 16 GB/s. A modern enterprise NVMe drive (Samsung PM9A3, Intel D7-P5520) delivers 1M+ random read IOPS and 7 GB/s sequential. The interface is no longer the bottleneck — the NAND itself usually is.

Queue depth — why your benchmark lies

Queue depth (QD) is the number of in-flight IO operations the host has handed to the device but not yet collected completions for. NVMe SSDs hit their rated IOPS only at QD ≥ 32, often QD ≥ 128:

# fio --rw=randread --bs=4k --iodepth=1 ...
read: IOPS=28.4k, BW=111MiB/s, lat avg 35.1 μs

# fio --rw=randread --bs=4k --iodepth=32 ...
read: IOPS=812k, BW=3.17GiB/s, lat avg 39.4 μs

# fio --rw=randread --bs=4k --iodepth=128 ...
read: IOPS=1.04M, BW=4.07GiB/s, lat avg 122 μs

A single-threaded sync read loop is QD 1: submit, wait, repeat. On a 1M-IOPS device it gets ~30K IOPS — leaving 97% of the device's capacity on the floor. To get the full number you need either threads (each at QD 1), libaio, orio_uring — anything that lets you issue many requests without blocking on each one. Benchmark numbers without QD annotations are meaningless.

io_uring — the modern Linux async path

io_uring (added in Linux 5.1, 2019) gives userspace a pair of shared ring buffers — a Submission Queue and a Completion Queue — that mirror the NVMe submission/completion model. The application writes SQEs directly into shared memory, optionally rings a doorbell with one syscall, and reads CQEs out without ever transitioning to the kernel for individual IOs. Combined with IORING_SETUP_SQPOLL (kernel polls the SQ instead of needing a syscall) and IORING_OP_READ /WRITE_FIXED (pre-registered buffers), the result is a zero-syscall, zero-copy IO path. Trading systems and modern databases (ScyllaDB, modern Cassandra) use it natively. Postgres 16+ adopted it for the I/O worker pool.

SPDK and the kernel-bypass extreme

SPDK (Storage Performance Development Kit, from Intel) goes one step further: it pulls the NVMe driver entirely into userspace via VFIO, polls completions on a dedicated core, and skips the kernel block layer. The win is ~1 μs of latency and ~10M IOPS per device that wasn't possible through the kernel. The cost is operational: you give up generic filesystems, generic permissions, and generic iostat. Used by storage vendors (Pure Storage), high-frequency trading, and a handful of cutting-edge databases. For 99% of services, io_uring through the kernel is the right tradeoff.

NVMe-oF — NVMe across the network

NVMe over Fabrics exports NVMe's submission/completion semantics across a network: TCP for compatibility, RDMA (RoCE / InfiniBand) for ~5 μs added latency. This is how modern shared-storage arrays (Pure Storage FlashBlade, NetApp ASA, DigitalOcean Block Storage) deliver NVMe-like performance over the wire. The host's perspective is essentially identical to a local NVMe — same driver, same queues, same fio numbers — except latency is 100-500 μs instead of 50 μs.

RAID 0 — striping, no redundancy

Split every write across N drives. Throughput and IOPS scale linearly with N; capacity is N × drive_size. Failure rate also scales linearly — losing any one drive loses everything. Used for scratch volumes, video editing, and benchmark numbers. Never for anything you can't reproduce from somewhere else.

RAID 1 — mirror

Two drives hold identical data. Reads can serve from either drive (~2× read throughput); writes go to both (no write speedup). Capacity = N / 2. Survives one drive loss. The simplest reliability increment, and still common for OS / boot volumes.

RAID 5 — stripe + 1 parity

Stripe data across N-1 drives, store XOR parity on the Nth (rotated per stripe so no single drive holds all parity). Survives one drive loss; on failure, rebuild reads the remaining N-1 drives and recomputes the missing data. Capacity = (N-1) × drive_size.

The hidden cost is the RAID 5 write penalty: a single random small write means read-modify-write of both the data block and the parity block — 4 physical IOs per logical write. For large sequential writes the penalty disappears (full-stripe writes compute parity in RAM and write everything in one go), but OLTP workloads pay it constantly.

RAID 6 — stripe + 2 parities

Two independent parity blocks per stripe (XOR plus a second function using Reed-Solomon math). Survives two simultaneous drive losses. Capacity = (N-2) × drive_size. Write penalty climbs to 6 IOs per small random write.

Why this matters now: a modern 10 TB or 18 TB drive takes 12 to 36 hours to rebuild after a failure, because the rebuild has to read every block of every surviving drive in the group at the drive's sustained throughput. During that rebuild window, the surviving drives are at 100% load — exactly when a marginal drive is most likely to die. A second failure during a RAID 5 rebuild is total loss. With 10+ TB drives, RAID 6 is the minimum responsible choice; RAID 5 on a JBOD of large drives is begging to be the next outage postmortem.

RAID 10 — mirror then stripe

Pairs of mirrored drives, striped. Capacity = N / 2 (same as RAID 1), write penalty is just 2 (write to both mirrors, no parity math), rebuild only needs to copy one drive's worth of data from its surviving partner. The default for OLTP databases — high IOPS, cheap rebuilds, predictable failure behaviour. The capacity overhead (50%) is the price.

ZFS and RAIDZ — copy-on-write meets redundancy

ZFS integrates the volume manager, filesystem, and RAID into one stack. RAIDZ1 / Z2 / Z3 are analogues of RAID 5 / 6 / 7, with two key improvements:

• No write hole. Classic RAID 5/6 has a window between writing a data block and updating its parity where a crash leaves the stripe inconsistent. ZFS's copy-on-write means every write goes to a fresh location atomically; there is no in-place update to be half-done.
• End-to-end checksums. Every block stores a checksum in its parent block (not next to itself). On read, ZFS verifies; on mismatch, it reconstructs from redundancy and rewrites the bad sector. Periodic zpool scrubbackground-reads every block and corrects silently.

btrfs has similar goals; both are the modern answer to "I want my filesystem to notice when the disk lies to me."

Bitrot — why checksums matter

Drive specs quote a unrecoverable read error rate of around 1 in 10¹⁴ bits for consumer HDDs, 1 in 10¹⁵for enterprise. 10¹⁴ bits = 12.5 TB. So statistically, scanning a 12 TB drive front to back is expected to produce one silently corrupted read. On a 60 TB RAID 6 array scrubbed weekly, you encounter ~5 silent corruptions a week — invisible without checksums, devastating if they land on a metadata block or a critical database row. ZFS catches them; ext4 over MD RAID does not.

Distributed storage — RAID at network scale

The same ideas at rack and datacenter scale:

• Replication — N copies of each block on N machines (commonly N=3). Like RAID 1 at machine granularity. Tolerates N-1 machine failures. Used by HDFS (3 replicas default), Ceph (replicated pools), Cassandra (replication factor).
• Erasure coding — Reed-Solomon (k, m) splits each object into k data shards + m parity shards, surviving m simultaneous shard losses with only (k+m)/k overhead. S3 uses this (Reed-Solomon with ~1.8× overhead for 11 nines of durability), as do Ceph erasure-coded pools, MinIO, Backblaze Vaults. Cheaper than triple replication; CPU cost on rebuild.
• Quorum — to read or write, contact a majority of replicas (R + W > N). Lets you tune the consistency / availability tradeoff per operation. The Dynamo paper's big insight; powers Cassandra, DynamoDB, modern Redis replication.

Same principles as RAID — N replicas instead of N drives, network link instead of SATA cable, machine failure instead of head crash. The math (parity, erasure coding, quorum) is the same; the failure domain is a rack or a region.

5 red flags in a code review

1. Benchmarking with dd if=/dev/zero. Many SSDs detect all-zero data and either compress or dedup it before writing — your "6 GB/s" number measures the compressor, not the NAND. Use fio with--rw=randwrite --refill_buffers or/dev/urandom-seeded data.
2. Capacity planning from fio --iodepth=1 numbers. A 30K-IOPS QD-1 measurement on a million-IOPS device tells you about syscall overhead, not the drive. Production traffic has natural queue depth from many concurrent clients — your benchmark needs the same.
3. RAID 5 in a JBOD of ≥10 TB drives. Rebuild time, URE math, and the load-on-survivors all conspire to make second-failure-during-rebuild a likely outcome. Use RAID 6 or RAIDZ2 minimum at that capacity tier.
4. Skipping fsync on commit because "we use an SSD." Storage media never determined durability — it's about the power-loss boundary between volatile DRAM caches (page cache, SSD controller cache) and persistent NAND. Without fsync, a crash loses the last 5-30 seconds of writes regardless of whether the disk is spinning rust or PCIe 5.0 NVMe.
5. Comparing SSDs from spec-sheet IOPS without QD or workload mix. Two drives both claiming "1M IOPS" can differ 10× in your actual workload depending on read/write ratio, block size, queue depth, fill level, and whether TRIM is wired up. Always benchmark with a workload that resembles production.