Three Transformer Shapes

An encoder-only model is a stack of Transformer blocks with no attention mask. Every token attends to every other token — to its left and to its right. The output is one contextualized vector per input token, each one informed by the entire sentence. BERT (2018) is the canonical example, and it set the template for a whole family: RoBERTa, ALBERT, DeBERTa, and the sentence-embedding models that power semantic search.

How BERT is trained. You can't train a bidirectional model the way you train GPT — predicting the next token would be trivial, because the model can just look to the right and copy the answer. BERT instead uses masked language modeling: randomly hide 15% of the tokens, replace them with a special [MASK] symbol, and ask the model to reconstruct the originals using context from both sides. The demo shows exactly this — the [MASK] slot pulls signal from “the cat” on its left and “on the mat” on its right to recover “sat.”

What it's good at. Because every token sees the full context, encoder outputs are excellent representations. Tasks that map a sequence to a label or to spans inside it — sentiment classification, named entity recognition, extractive question answering, retrieval embeddings — are BERT's home turf. For years, “fine-tune BERT” was the default first move for any NLP classification problem.

What it can't do. Generate. An encoder produces all its outputs in one shot, conditioned on the full input — there is no notion of “the next token,” no left-to-right ordering to exploit, no way to sample a continuation. If you want a model that writes, bidirectional attention is exactly the wrong tool, and you need the next shape.

A decoder-only model is the same stack of blocks, but with a causal mask: each token may attend only to itself and the tokens before it, never to the future. The training objective is dead simple — predict the next token, at every position, simultaneously. This is the architecture of GPT-2, GPT-3, GPT-4, Llama, Qwen, DeepSeek, Mistral, Gemini — essentially every large language model in production today.

Why this shape won. Three reasons compounded. First, the objective is self-supervised and unlimited: any text is training data, because the “label” for each position is just the next token already present in the corpus. No human annotation, no task-specific datasets — just predict-the-next-word over the entire internet. Second, one model does every task: framed as text continuation, translation, summarization, Q&A, and code all become the same next-token problem. Third, it scales cleanly — the recipe that worked at 100M parameters kept working at 100B+.

The training/inference asymmetry. During training the model sees the whole sequence at once and predicts all positions in parallel — the causal mask is what makes this safe (position i can't cheat by peeking at i+1). During generation, though, it runs the loop the demo shows: predict one token, append it, feed the longer sequence back in, predict again. This is why generating a 1,000-token response takes 1,000 forward passes, and why the KV cache (storing past keys/values so you don't recompute them each step) is the central inference optimization.

One subtlety worth retiring. “Decoder-only” is a slightly misleading name — these models have no encoder to decode from, and no cross-attention. They're really just causal language models. The “decoder” label is a historical holdover from the original encoder-decoder Transformer, whose decoder half had both the causal mask and the cross-attention. Modern GPTs kept the causal mask, dropped the cross-attention, and kept the name.

The encoder-decoder shape is the original 2017 Transformer, and it combines the previous two. An encoder (bidirectional, like BERT) reads the entire input into a set of context vectors — a fixed “memory.” A decoder (causal, like GPT) generates the output one token at a time. The bridge between them is cross-attention: at every decoder layer, the generated tokens attend back into the encoder's memory. T5 and BART are the best-known examples.

Cross-attention, precisely. A decoder block actually has two attention sublayers. The first is ordinary causal self-attention over the tokens generated so far (Q, K, V all come from the decoder). The second is cross-attention: the queries come from the decoder, but the keys and values come from the encoder's memory. So each output token can ask “which parts of the source are relevant to what I'm writing right now?” For translation, the word being generated looks back at the source words it's rendering.

When it's the right shape. Encoder-decoder shines when the input and output are distinct sequences with a clear boundary — and the input is fully available up front. Translation (French in, English out), summarization (long document in, short summary out), and speech-to-text are the classic fits. The separate encoder gives the input a rich bidirectional representation that the decoder can repeatedly consult.

Why decoder-only ate its lunch anyway. You can do translation with a pure decoder too: just feed it “French: … English: …” and let causal attention handle both halves. This is clunkier in theory but wins in practice — one stack instead of two, one objective, and it scales to general instruction-following where there's no clean input/output split. By GPT-3's era, the field had largely consolidated on decoder-only. T5-style models remain strong for focused seq-to-seq tasks, but the frontier is decoder-only.

Everything in the three shapes above comes down to one question: can a token attend to the future? Encoders say yes; decoders say no. The mechanism that enforces “no” is the causal mask — and it is almost comically simple. Recall the attention score matrix from the self-attention primer: an n × n grid where entry (i, j) is how much query token i attends to key token j. The causal mask sets every entry where j > i (the key is in the future relative to the query) to −∞, before the softmax.

Why −∞, and why before softmax. Softmax exponentiates every score: e^(−∞) = 0. So a masked cell contributes exactly zero weight after normalization, and — crucially — it doesn't even participate in the denominator. The remaining (past + present) cells renormalize among themselves to sum to 1. The result is a strictly lower-triangular attention matrix: token i distributes all of its attention across tokens 0 through i, and none beyond. In code it's typically scores.masked_fill(mask == 0, float('-inf')) — one line.

Why this is exactly what generation needs. When the model predicts the token at position i, it must only use information from positions 0…i — because at inference time, positions i+1 onward don't exist yet. The causal mask guarantees this during training too, so the train-time and test-time information sets match exactly. Without it, the model would learn to predict token i by peeking at token i itself (or later) — trivially perfect on training data, useless at generation. The mask is what makes next-token prediction a real, non-cheating learning problem.

The unifying picture. The Transformer block is one fixed design. The mask is the dial that selects the architecture: no mask → encoder (bidirectional, BERT); causal mask → decoder (autoregressive, GPT); an encoder stack plus a causal decoder stack joined by cross-attention → encoder-decoder (T5). Same blocks, same attention math, same FFN — one triangular matrix decides whether you've built a model that understands or a model that writes. With that, the entire modern LLM is assembled: tokenize → embed + position → N masked blocks → final norm → next-token logits → sample → loop.