Three Transformer Shapes

An encoder-only model is a stack of Transformer blocks with no attention mask at all: every position may draw on every other position, left and right. Drag the highlighted token and watch it reach every other token in the row, both ways:

Notice that direction never matters here: a token early in the sentence reaches just as easily into what comes after it as a token near the end reaches into what came before. That symmetry is the entire definition of an encoder — the representation at every position is built from the complete sentence, never a prefix of it.

BERT is trained to exploit exactly this. Hide 15% of the tokens behind a [MASK] symbol and ask the model to recover them from context on both sides — a trick a one-directional model could not learn, because it could just copy the answer from a position it hasn't reached yet. Reveal what the [MASK] slot can draw on, one step at a time:

Watch the prediction only appear once both sides have been reached: at step 1 the slot has attended to “the cat” on its left and “on the mat” on its right, and only then does step 2 resolve it to sat. Masked-language modeling turns bidirectionality from a limitation into a training signal.

Nothing about this is thrown away: the whole n × n grid of query rows against key columns stays open, for any sentence length. Change n and watch every cell stay lit:

Because every cell survives, every output token is conditioned on the entire input at once. That is what makes encoder representations so good at classification, extraction and retrieval — sentiment labels, named-entity spans, semantic-search embeddings — tasks that map a sequence to a label or to a piece of itself. For years, fine-tuning a BERT was the default first move for exactly this family of problems.

What an encoder cannot do is generate: there is no notion of the next token when every position already sees the whole sentence, so there is nothing left to predict one step at a time. Restrict the same row to only look backward, and compare it with the version that already has, side by side:

Since the bottom row can no longer reach forward, it has thrown away exactly the information a generator is not allowed to have anyway. That restriction is not a defect — it is the entire mechanism the next shape is built from.

A decoder-only model keeps the same blocks but adds one rule: a token may attend only to itself and the tokens before it, never to what comes after. Drag the active token forward through the sentence and watch how far its reach grows:

Notice that the reach only ever grows, never shrinks — token 5 sees everything token 0 sees, plus everything in between, plus itself. That one-directional rule is the entire difference between this shape and the last one, and it is what makes the training objective almost embarrassingly simple: at every position, predict the next token.

The rule is not optional. Switch the mask off and the same row suddenly reaches into the future it is supposed to be predicting:

Since nothing about the shapes or the loss changes, nothing crashes — the model simply learns to copy the answer from a position it will never have access to at generation time. Training loss falls unusually fast, and the model then produces near-random continuations once it has to generate for real, because the shortcut it learned does not exist outside training. That gap between a suspiciously good training curve and a broken generation is the tell.

Because a token can only see the past, generation has to happen one token at a time: predict, append the new token to the input, and run the whole thing again. Step through one full generation loop:

Notice each step reprocesses everything that came before it, not just the new token — the model has no memory between calls, so the tokens already generated go back in as ordinary input on the next pass.

That reprocessing is expensive: recomputing attention over the whole prefix at every step costs O(n²) work at step n, and summed across a run of N steps that is O(N³) — one full attention pass' worth of squares, over and over. A cache that keeps yesterday's keys and values around needs only O(n) new work per step, for an O(N²) total. Watch the gap widen as the run gets longer:

Run the same two formulas out to N = 1,000 and the ratio is exactly 667× — not a rough order-of-magnitude, the same arithmetic above, just at a scale too large for the bars to draw. That is why every inference server keeps a KV cache: caching does not make one step cheaper, it stops the model from redoing N − 1 of the previous steps at every new one.

The cache has to live somewhere, though, and its size is not free either. Each token pins down its own key and value at every layer, so the cache grows linearly with how much context is still open. Drag the context length and watch it fill:

At Llama-2-7B's published shape — 32 layers, hidden size 4,096, stored in fp16 — that is 512 KiB for every single token, at every layer combined. A 4,096-token context therefore pins down exactly 2 GiB of memory before the model has generated a single new word, which is why context length is as much a memory-budgeting problem as it is a compute one.

The encoder-decoder shape is the original 2017 Transformer, and it is literally the previous two shapes stapled together: a bidirectional encoder stack reads the whole input, and a causal decoder stack generates the output one token at a time. Step through the layers of both stacks side by side:

Notice that, so far, nothing connects the two columns — each stack is exactly the shape it already was on its own. The only new part is the bridge between them.

That bridge is cross-attention: at every decoder layer, the token being generated reaches back into a single, fixed summary of the input — the encoder's finished output — rather than into anything the decoder produced itself. Step through the decoder's layers and watch where the bridge always lands:

Because the arc lands on the same memory whether you check the first decoder layer or the last, this is the detail worth being precise about: the queries for cross-attention come from the decoder, but the keys and values come from the encoder — computed once, reused by every layer, never recomputed as generation proceeds.

A decoder block actually runs two attention sub-layers back to back, not one, and they draw on completely different sources. Compare them side by side for one decoder position:

Since the top panel and the bottom panel share the same query, the only thing that changed is where the keys and values came from: its own past above, the encoder's memory below. Both run at every layer; neither one substitutes for the other.

And cross-attention carries no triangle at all — the input is fully available before generation even starts, so there is nothing to hide from a decoder position. Step through the decoder rows and watch every encoder column stay open:

That full rectangle is exactly what a translated pair needs: source and target are two distinct sequences with a clear boundary between them, and the whole source is known up front. Translation, summarization and speech-to-text all share that shape, which is why T5 and BART are still the standard fit for them.

Yet a plain decoder can do translation too — feed it “French: … English: …” and let the causal mask handle both halves inside one stack instead of two. Toggle between the two layouts:

Because the two-stack version needs a matched encoder and decoder, a fixed input/output split, and its own cross-attention wiring, while the one-stack version needs none of that — just a longer prompt — the field mostly consolidated on the simpler shape once decoders got large enough to carry both halves in-context. T5-style models stay strong for focused sequence-to-sequence work; the frontier moved to decoder-only.

Every mask in the three shapes above is really the same question, answered two ways: how far ahead of the diagonal is a query allowed to look? Drag the window and watch every row's weight redistribute over whatever keys are left open:

Notice that every row's share always adds up to one, however many keys are open — fewer open keys just means each one carries more of the weight. Now drag past exactly causal, down to −1: row 0 loses even itself, and the invariant has nothing left to distribute. A real implementation must never let the window go that far, which is why the diagonal is always included.

That redistribution is not hand-waving — it is a real softmax over real scores. Here are four raw scores for one small causal mask; step through the rows and read the actual weights the row settles into:

Notice row 0: with exactly one key allowed, its weight is 1.00 whatever its own score says — the first position has no choice to make. By row 3 every key is open and the weight has to split four ways.

The mechanism that keeps a disallowed key at exactly zero is setting its score to −∞ before the softmax, because exp(−∞) = 0. The classic bug swaps that for a finite substitute — masking with 0 instead — which looks harmless until a disallowed key happens to have a large raw score. Switch the rule and watch the masked future position stop being zero:

With the correct mask that position carries exactly 0% of the row, every time. With the buggy one it carries 64.4% — nearly as much as the row's best legitimate key gets under the correct mask — because exp(0) = 1 still competes fully in the sum. Nothing throws an exception and no shape is wrong; the model just quietly learns to read an answer it was never supposed to see, and training loss falls a little too fast to be suspicious of anything.

Every shape in this primer is this same triangular matrix, present or absent, in one place or two: no mask makes an encoder, a causal mask makes a decoder, and a causal mask plus cross-attention makes an encoder-decoder. Switch between all three on the one row:

Because the blocks, the attention math and the feed-forward layers never change, the entire architectural decision comes down to this one dial. Tokenize, embed, run the tokens through N masked blocks, take the final norm, read off next-token logits, sample, and loop — the mask is what decided which of the three machines you just built.

Start from a name you already know. Pick a model and watch the same mask pattern light up the row, with the numbers that go with it:

Notice BERT and GPT share almost the same block count and parameter budget — 12 layers, roughly 110–117M parameters — and still land on opposite masks: one an encoder, one a decoder, because the mask was always an architectural choice, not a consequence of scale. T5 pays for its encoder-decoder second stack with roughly double the parameters at the same per-layer width.

In practice the fastest way to pick a shape is to look at the output, not the parameter count. Choose what the output looks like and watch which shape answers:

Since a label or a span never has to be generated token by token, it never needs a causal mask at all — an encoder alone is enough, and often cheaper to run. The moment the output has to be produced one token after another, some causal mask enters the picture, whether or not there is a second stack behind it.

With that, every piece from this primer and the last one assembles into the same seven-stage pipeline every deployed model actually runs. Step through it once, end to end:

Notice that only one stage — the third — is where today's decision lives: swap its mask and the same seven stages describe a BERT classifier, a GPT-style generator, or a T5-style translator without changing a single other line. The block was never the hard part; the mask was.