Process & Thread Primer

When you type curl ..., the shell calls fork() + execve(): the kernel allocates a fresh task_struct (Linux's name for the process control block, PCB), a fresh mm_struct (the address space), a fresh file-descriptor table, and assigns a new PID. The ELF binary's code is mapped, the entry point is loaded into the program counter, and execution begins. Everything the running program is — every register, every byte of stack and heap, every open socket — lives in those kernel structures, not in the binary file.

The address space

Every process has a private 64-bit virtual address space (256 TB on x86-64, of which the user half is 128 TB). The layout, low to high:

• text — the program's machine code, read-only, mapped from the ELF file.
• data / bss — global variables, initialised and uninitialised.
• heap — grown upward by brk() or via mmap. Where malloc lives.
• mmap region — shared libraries, large allocations, file mappings.
• stack — grows downward from a high address; one per thread.
• kernel space — the upper half (above 0xffff_8000_0000_0000 on x86-64 Linux), mapped into every process but accessible only in kernel mode.

The address space is virtual: every load and store goes through the page table to find which physical page (if any) backs that virtual address. Two processes can both hold the integer 42 at virtual address 0x7fff1234 in different physical pages. The virtual-memory primer covers the mechanism.

The rest of the PCB

Beyond the address space, a process owns:

• Open-file table — fd 0 (stdin), 1 (stdout), 2 (stderr), plus any sockets or files open()ed. fd numbers index into a per-process array of struct file pointers.
• Credentials — uid, gid, capabilities. Used by every kernel permission check.
• Signal disposition — which signals are blocked, which have custom handlers.
• Working directory, root, umask, rlimits — small per-process state that everything else relies on.
• Parent / children pointers — the process tree. Used by wait() and how zombies are reaped.
• Scheduling state — runnable, sleeping, zombie; priority; accumulated CPU time. The CPU-scheduling primer explains how the scheduler uses this.

That bundle — address space + open files + credentials + scheduling state — is the unit of isolation. The kernel guarantees no process can read another's memory, intercept its file descriptors, or impersonate its credentials, except through the IPC mechanisms it explicitly mediates (shared memory, pipes, sockets, signals, ptrace).

On Linux, pthread_create actually calls the clone() syscall with flags that ask for a new task_struct that shares its parent's mm_struct, fd table, signal handlers, and other state. The new task gets its own stack, its own thread-local storage, its own register state, and its own kernel scheduling slot — but reads and writes to global memory hit the same physical pages that every other thread in the process sees.

That sharing is both the appeal and the danger. Appeal: passing data between threads is free — they all see the same pointers and the same heap. Danger: unsynchronised access to shared state corrupts memory silently; the concurrency primer covers what to do about it.

Kernel threads vs user threads

On modern Linux, every pthread is a kernel thread — the kernel knows about it, schedules it, can suspend and resume it, can move it between CPUs. This is sometimes called the "1:1 model": one application-level thread per kernel thread. It's simple and lets a process actually use multiple CPUs in parallel, but kernel threads are heavy — each one costs a task_struct, a kernel stack (16 KB on Linux), and ~1 KB more of bookkeeping. Practical limit on a server: a few thousand.

User-space threads (green threads, fibers, coroutines) multiplex many application-level concurrent tasks onto fewer kernel threads. Go's goroutines, Java virtual threads (Project Loom, JDK 21+), Rust async tasks, Python asyncio tasks — all of these are user-level. The runtime maintains its own scheduler and parks/wakes tasks on top of an M:N or 1:N model over kernel threads. The benefit is scaling to millions of concurrent tasks (a goroutine weighs ~2 KB of stack); the cost is that any blocking syscall by an underlying-kernel-thread blocks all the user tasks scheduled on it (mitigated by the runtime moving them to other kernel threads or by using non-blocking I/O — the I/O primer covers epoll and io_uring).

Thread-local storage

Every thread shares the address space, so a global variable in C is shared by default. To give each thread its own value, declare it thread_local (C++11) or __thread (GCC), or use pthread_key_create for runtime-keyed storage. The compiler emits accesses through the FS segment register (x86-64) or TPIDR_EL0 (ARM64) so each thread reads from a different physical location.

Thread-locals are the standard way to avoid contention on shared state: the per-thread counter pattern from cache-coherence, the per-thread allocator cache in tcmalloc/jemalloc, the per-thread current-request context in tracing — all of these live in TLS.

What threads cost

• Memory. Each thread defaults to an 8 MB stack reservation (virtual; physical pages allocated on use). 1 000 threads = ~8 GB virtual address space reserved. You can shrink with pthread_attr_setstacksize but most servers don't bother.
• Scheduler load. The CFS scheduler decides who runs next in O(log N) of runnable threads. A few thousand is fine; a few million isn't (which is why those scenarios need user-space threading runtimes).
• Cache pressure. Each thread has its own working set; context-switching between them evicts each other's data from L1/L2. High-concurrency-low-throughput servers can spend more time on cache thrashing than actual work.

Modern CPUs implement multiple privilege rings (x86: 4 rings, only 0 and 3 used in practice; ARM: 4 exception levels EL0–EL3). For our purposes there are two:user mode (ring 3 / EL0), where every application runs, and kernel mode (ring 0 / EL1), where the OS runs. The hardware enforces the boundary; software cannot escape it.

What user mode cannot do

• Touch memory not mapped for it. The kernel half of every process's address space is mapped but inaccessible from ring 3 — attempting a read or write raises a fault that the kernel turns into a SIGSEGV.
• Use privileged instructions. Disabling interrupts (cli), loading new page tables (mov cr3, ...), reading model-specific registers, executing I/O port instructions (in/out), configuring the IDT — all reserved for ring 0.
• Talk to hardware directly. No reading the network card's buffers, no programming the timer, no spinning up DMA. User code asks the kernel via syscalls; the kernel does the actual hardware work.

What kernel mode can do (almost everything)

Once running in ring 0, the kernel can do anything the hardware can do. The kernel still respects its own policies — capability checks, namespace restrictions, cgroup limits — but those are software conventions, not hardware-enforced. A bug in the kernel can corrupt any memory, take down the machine, or grant root to a malicious user. This is why kernel code is reviewed with a paranoia level beyond anything in userspace.

How control crosses the boundary

Three mechanisms move the CPU between rings, each covered in its own primer:

• Syscall (syscall on x86-64, svc #0on ARM64): user code asks the kernel to do something. Voluntary, controlled, arguments in registers. See the assembly-isa primer.
• Interrupt: hardware demands the kernel's attention — the network card has a packet, the timer fired, the keyboard has input. Involuntary from user code's perspective. See the syscalls-interrupts primer.
• Exception: user code did something illegal — divided by zero, dereferenced an unmapped address, executed an undefined instruction. The CPU jumps to a fixed kernel handler; the kernel typically delivers a signal (SIGSEGV, SIGFPE, SIGILL) back to the process.

In all three cases the hardware atomically saves the user's instruction pointer and stack pointer, switches to a per-CPU kernel stack, raises the privilege ring, and jumps to a kernel entry point recorded in a model-specific register. The kernel does what it has to, then arranges for a return back to ring 3 at the saved (or modified) user instruction pointer.

Why two modes exist

Without privilege separation, every program could corrupt the kernel, every program could read every other program's memory, every program could read every disk file. Practical multi-user computing requires isolation. The kernel is the trusted intermediary that enforces it; the privilege boundary is how the hardware backs the kernel up.

The cost is real — every syscall, every page fault, every interrupt is a boundary crossing. Post-Meltdown KPTI made each crossing flush the TLB, which pushed the per-crossing cost from ~50 ns to ~150 ns. High-performance servers therefore go to great lengths to avoid crossings (the I/O primer covers io_uring; the syscalls primer covers vDSO).

A context switch happens whenever the kernel needs to stop running one thread and start running another. Triggers include: a timer interrupt expiring the current thread's time slice, the current thread voluntarily blocking on I/O, a higher-priority thread becoming runnable, or the current thread calling sched_yield.

What gets saved and restored

The kernel saves the outgoing thread's CPU state into its task_struct: general-purpose registers, instruction pointer, stack pointer, flags, segment registers, FPU and vector state (lazy on most modern CPUs — only restored when actually used). For x86-64 that's ~200 bytes of state; saving and restoring it costs tens of cycles.

Then the kernel decides what to run next (the scheduling decision — see the next primer), updates per-CPU pointers (current on Linux), and starts loading the incoming thread's state. If the incoming thread is in a different process, the kernel also swaps CR3 (the page-table base register), which triggers a TLB flush — every cached virtual-to-physical translation is thrown out and must be re-walked from the page tables on next access.

The cost asymmetry

Process → process switching is the expensive one. Beyond the direct register-save/restore (~100 ns), the TLB flush means the next ~50–500 memory accesses each cost a page-table walk instead of a cached translation. The L1 instruction and data caches also lose their content from the outgoing thread and need to re-warm from L2/L3/DRAM for the incoming thread — measurable as elevated IPC drops in the first ~10 μs after switch.

Thread → thread switching within the same process skips the CR3 swap and TLB flush entirely. Direct cost is ~100 ns; cache effects are minimal because the incoming thread's working set is likely still warm (it ran recently). This is why "a few hundred threads sharing a single process" outperforms "a process per request" for I/O-bound servers by 5–10× on the same workload.

Modern mitigations

• ASID / PCID-tagged TLB. Recent x86 (since Westmere, 2010) and all modern ARM tag TLB entries with a process ID, so a switch to a different process can preserve the outgoing process's TLB entries instead of flushing them. Reduces re-walk cost but doesn't eliminate cache effects.
• CPU affinity. Pinning a thread to a specific core (sched_setaffinity, taskset) keeps its cache state warm across switches. Used by low-latency systems (high-frequency trading, databases like ScyllaDB).
• User-space scheduling. Go's goroutine scheduler does almost everything in userspace, switching between goroutines without involving the kernel. Each switch costs ~50 ns vs the kernel's ~1 μs. Java virtual threads (Loom) and Rust's tokio work the same way.

How to measure

On Linux: perf stat -e context-switches,cs,migrations shows the rate; perf sched shows wall-clock latency between wakeup and run.vmstat 1's cs column is the easiest one-line check. A typical web server doing 50K req/s shows 100K–500K context switches per second; a busy database might do 10K–50K. Numbers in the millions per second suggest a thread pool that's too large or excessive lock contention.

Red flags in code review

• Spawning a process per request. Apache prefork-style. Works for low concurrency but the per-fork overhead and per-process memory footprint collapse it past a few hundred concurrent. Use threads or async.
• Unbounded thread pool. "Create a new thread for each incoming request" with no upper limit hits the ~few-thousand-thread ceiling, then either OOMs (stack reservation) or thrashes the scheduler. Use a bounded pool with a queue.
• Blocking calls inside an async / goroutine context. A blocking syscall pins the underlying kernel thread and starves every other task scheduled on it. Use the async variant, or hand it off to a worker thread.
• Shared mutable state without synchronisation. Threads share memory; nothing in the language makes you remember that. std::atomic,Arc<Mutex<...>>, channels, or message-passing — something must own the synchronisation.
• Tight CPU-bound loops without yielding. A goroutine or thread that doesn't hit a scheduling point (function call, channel op, syscall) for milliseconds will hold the CPU and delay everything else on the same runqueue.