File System Primer

At the bottom of any Unix-style filesystem are blocks — fixed-size chunks of the underlying disk (typically 4 KB to match the page size). Above that, inodes are the per-file metadata structures: size, permissions, timestamps, owner, and pointers to the blocks holding the file's data. Above inodes, directories are themselves just special inodes whose data is a table of (name → inode-number) entries.

The inode in detail

A traditional Unix inode is ~128 bytes containing:

mode (file type + permissions, e.g. rwx)
uid, gid (owner)
size (in bytes)
atime, mtime, ctime (timestamps)
link count (how many directory entries point here)
direct block pointers (first 12 blocks of the file)
indirect block pointer (a block holding pointers to more blocks)
double-indirect, triple-indirect (for huge files)

With 4 KB blocks and 8-byte pointers, the direct pointers cover 48 KB; single indirect adds 2 MB; double indirect adds 1 GB; triple indirect adds 512 GB. Modern filesystems (ext4, XFS) use extents instead — a (start, length) pair that describes a contiguous run, so a large file might be representable in a few extents instead of thousands of pointers.

Path resolution

When you open("/var/log/access.log"), the kernel:

1. Looks up the root inode (inode 2 by convention on ext4).
2. Reads the root's directory data, finds an entry for var with some inode number N.
3. Reads inode N (confirms it's a directory).
4. Reads inode N's directory data, finds an entry for log with inode M.
5. ... continues until the final name.

Each step is potentially an inode read + a directory-data read. Without caching, opening a deeply-nested path would be slow. Thedentry cache (directory entry cache) in the kernel caches recent (path-component → inode) lookups; the inode cache caches recent inode contents. Both live in the page cache and are vastly more effective than the data cache for typical workloads — most path lookups hit zero disk.

Hard links and soft links

A hard link is just an additional directory entry pointing at the same inode. The inode's link count goes up; when it reaches zero the file is freed. ln file1 file2creates a hard link; you cannot distinguish the "original" from the "link." A soft link (symlink) is a special inode whose data is a path string; the kernel follows it transparently. Symlinks can cross filesystems and point at nonexistent paths; hard links cannot.

File descriptors revisited

An open fd in your process is an index into a per-process file-descriptor table. Each entry points to a kernelstruct file which contains the inode, the current offset, the open flags (O_RDONLY, O_APPEND), and the position. Two processes opening the same file get two different struct files but the same underlying inode (and same data). Two dup'd fds in one process share one struct file (and therefore one offset).

The page cache is the kernel's store of recently-accessed file data, indexed by (inode, offset). On read, the kernel checks the page cache first; if the page is there, copy it into the user buffer and return immediately. If not, fetch it from disk into the page cache, then copy. On write, the kernel copies into the page cache, marks the page dirty, and returns — the actual disk write happens asynchronously in the background.

Where it lives in memory

On a typical Linux system, the page cache takes whatever physical memory isn't being used by processes. free -hshows it under "buff/cache"; this number is large by design and gets reclaimed automatically when applications need memory. Cold-boot a server, run a workload, and the page cache grows to fill ~free RAM. This is why Linux is "always running out of memory" — it isn't; the cache is just doing its job.

Writes are not what you think

A successful write() means "data is in the page cache." That's it. The kernel may write it to disk in 5 seconds (the default dirty_writeback_centisecs), or much later under low memory pressure. On power loss between write() and disk-commit, the data is gone — even though write() returned success.

Three syscalls force durability:

• fsync(fd) — flushes all dirty pages for this file and its metadata to disk, returns when complete. Typical: 5–15 ms on NVMe, 5–50 ms on spinning disk.
• fdatasync(fd) — like fsync but skips metadata if the file size and modification time haven't changed. Marginally faster.
• sync() — kicks off writeback for the whole system; doesn't wait. Use sync_file_range for a no-wait single-file alternative.

O_DIRECT — bypass the page cache

Setting O_DIRECT on an open file tells the kernel: no page cache, do DMA straight from the user buffer to the device. Required buffer alignment (typically 512 bytes), user-buffer-size constraints, and you lose the cache benefits — every read truly hits the disk. Used by databases (Oracle, Postgres withdirect_io) and file-system benchmarks where the application has a better idea than the kernel about what to keep in memory.

readahead and prefetching

When the kernel notices sequential reads from a file, it speculatively reads ahead — fetching pages the application hasn't asked for yet. This converts what looks like 10 separate read calls into one large I/O operation, dramatically improving throughput. posix_fadvise lets you give the kernel hints (POSIX_FADV_SEQUENTIAL, POSIX_FADV_RANDOM, POSIX_FADV_WILLNEED) about your access pattern.

Without journaling, a power loss in the middle of a multi-block write would leave the filesystem in an inconsistent state — half the metadata updated, half not. fsck at boot would have to scan every inode and every block to find inconsistencies, taking minutes to hours on a large disk. Modern filesystems usejournaling (write-ahead logging) to avoid this.

How journaling works

A journal is a fixed-size circular log of pending transactions. Before applying any change to the filesystem proper, the kernel:

1. Writes a begin-transaction marker to the journal.
2. Writes the new contents of every affected block to the journal.
3. Writes a commit marker, with a checksum, after which the transaction is durably recoverable.
4. Applies the changes to the real filesystem blocks.
5. Marks the journal entry as complete.

On crash recovery, the filesystem scans the journal: any transaction with a commit marker is replayed to the filesystem blocks; any transaction without is discarded. Recovery is bounded by journal size (typically a few MB), so it's seconds not hours.

Three journaling modes (ext4)

• data=writeback — journal only metadata; data may land out of order. Fastest, but a crash can expose stale data in newly-extended files. Rarely used in production.
• data=ordered (default) — journal only metadata, but ensure data blocks are written before the metadata commit. The metadata never references unflushed data. Good balance of speed and safety.
• data=journal — data goes through the journal too. Doubles writes but offers the strongest crash semantics. Sometimes used by databases or for high-integrity filesystems.

fsync — the only durability promise

fsync(fd) is the only POSIX guarantee that data committed before the call survives a crash. Internally it:

1. Flushes all dirty pages of the file to the block layer.
2. Issues a hardware write cache flush to the storage device.
3. Waits for the device to acknowledge completion.

Step 2 matters. SSDs and HDDs have write caches that hold data in the device's volatile DRAM until the device decides to commit. A bare write to the device may return success while the data is still volatile. fsync issues a FLUSH_CACHE / FUA command that forces the device to commit before acknowledging.

fsyncgate and how Postgres learned

Postgres had a famous near-disaster in 2018 ("fsyncgate"): if an fsync fails, the kernel may discard the dirty pages from the page cache without retry. A subsequent fsync on the same fd returns success — but the data is gone. Postgres's WAL design assumed fsync failures were retryable; they weren't. The fix across the industry: take any fsync failure as a panic; restart the process, rebuild from the WAL.

Torn writes and POSIX's honest scope

POSIX guarantees that write of N bytes ≤ PIPE_BUF (512 bytes typically) is atomic. Above that, a write can be split — including crashing midway through. Databases handle this with full-page writes in the WAL (Postgres) or copy-on-write semantics (ZFS, btrfs). Filesystems do not generally promise more than what POSIX requires.

ext4

The default on most Linux distros. Successor to ext3 with extents, delayed allocation, journal checksums. Mature, well-tuned, supports files up to 16 TB and filesystems up to 1 EB. Conservative feature set — no snapshots, no transparent compression. The thing you reach for when you want "a filesystem" with no questions asked; also what most Docker images run on top of.

XFS

Originally SGI's, now the default for Red Hat / Fedora. Designed for very large files and very high parallelism. Extent-based, delayed allocation, allocation groups (per-CPU parallel allocation). Slightly faster than ext4 on big-file workloads; doesn't shrink (you can only grow); no online defrag historically (recent versions added it). The default for serious storage on RHEL.

ZFS

Originally Sun's; available on Linux via OpenZFS. Distinguished by: copy-on-write (every write goes to fresh blocks, the metadata pointer flip is the atomic commit); strong checksums on every block (catches bit rot the disk firmware silently allows); snapshots and clones in O(1) (just a metadata operation); built-in volume management (zpool replaces LVM + filesystem); transparent compression and deduplication.

Cost: high RAM use (ZFS's ARC cache is separate from the page cache and wants a lot of memory — 1 GB per 1 TB is a common rule); licensing complications on Linux (CDDL vs GPL means it ships out-of-tree); more complex than ext4 to administer well.

btrfs

Linux's answer to ZFS. Copy-on-write, snapshots, checksums, compression, subvolumes. In-tree (GPL clean). More flexible than ZFS in some ways (online resize, send/receive replication), historically less stable in some RAID configurations. Used by Facebook for its massive on-disk storage; default on Fedora 33+ and openSUSE.

tmpfs and special-purpose

tmpfs is RAM masquerading as a filesystem: write to /tmp and the data lives only in the page cache, no disk involvement. Fast, but volatile. procfs (/proc) and sysfs (/sys) are virtual filesystems whose contents are generated by the kernel on demand, exposing process and device state as files.

Picking one

• Default Linux machine → ext4 (or whatever your distro picked).
• Big-file / high-parallelism server (databases, backup targets) → XFS.
• Strong integrity and operations features(snapshots, checksums, send/receive replication) → ZFS or btrfs. ZFS in environments with the licensing flexibility, btrfs in GPL-only ones.
• Container images and ephemeral data → ext4 inside overlayfs. The Linux kernel's container infrastructure (overlayfs) layers a read-only base + a writable upper, both typically on ext4.

Red flags in code review

• write() in a loop without fsync, claiming durability.Data is in the page cache, not on disk. If durability matters, fsync after the last write of a logical transaction.
• Treating fsync failure as transient. See fsyncgate. Treat it as "data is gone, restart the process" — the kernel may have discarded the pages.
• Logging via printf without flushing. stdio buffers internally; a crash loses the last few KB of logs even if the file was opened for line-buffered mode. Use unbuffered stderr or explicit flushes at critical points.
• Storing many tiny files in one directory. ext4 and XFS handle this but directory operations slow significantly past ~100K files. Hash into subdirectories (aa/aabbcc-deadbeef) like git or any large object store.
• Assuming O_DIRECT is faster. Without the page cache, your reads always hit disk. O_DIRECT is faster only when you have your own cache and know better than the kernel. Otherwise the default page cache wins by a lot.