Skip to main content

The Transformer Architecture

A from-scratch guide to the full encoder-decoder transformer architecture. An overview of components, assembling and training, and how to evaluate, built up piece by piece.
architectures
LLMs
attention
Author

Abbie Petulante

Published

November 17, 2026

The Basic Transformer Architecture

Transformers, introduced in the seminal paper “Attention Is All You Need” by Vaswani et al., are a major breakthrough in machine learning, especially for natural language processing (NLP). Today, they stand out as the architecture underneath essentially every modern language model - and, increasingly, as the core architecture for images, audio, and more.

So, what makes the transformer so special? Why is “attention” such a fundamental component of modern models? In this tutorial, we’ll try to break it down. We’ll cover the core architecture, build a small one and train it, and see what it actually learned. Along the way, we’ll talk all about how it learns, and why the architecture is so impactful to the field.

Open In Colab

Why Transformers?

Natural Language Processing (NLP) tasks especially, are a quintessential example of a sequence which is better treated as a whole rather than as a collection of words.

Consider the sentence: “Time flies like an arrow.” In this sentence:

  • “Time” is the subject,
  • “flies” is the verb,
  • “like an arrow” is a prepositional phrase.

Now consider: “Fruit flies like a banana.”

  • “Fruit flies” is the subject,
  • “like” is the verb,
  • “a banana” is the object.

Words here take on very different meanings depending on their context. Flies can be a verb or a noun. Like can be a verb or a preposition. Processing this as a sequence allows models to understand that the arrow is metaphorical in the first sentence, but the banana is a concrete object. And the meaning of words is fundamentally tied to how they are put together, not just what words are there. Any model that wants to understand language, then, has to be able to relate every word to the others around it - sometimes ones that are right next to it, and sometimes ones much further away.

What Came Before

Of course, transformers weren’t the first models built to handle sequences. Before 2017, the standard approach was the recurrent neural network (RNN), and its more robust variants like the LSTM. An RNN reads a sentence the same way a person does - one word at a time, left to right. As it does this, it carries a running summary that it updates with each new word. To translate a sentence, one RNN would read the whole input into that summary, and a second RNN would generate the translation from it, one word at a time.

This works pretty well, but has two main issues. First, it only works sequentially. With no way to compute all of the positions at once, training can’t take full advantage of hardware like GPUs that are built to do many things in parallel. That puts a hard ceiling on how big a model, and how much data, you can practically train on. The second issue is that information has to travel a long way. When something in position 100 needs to know about something in position 10, that information has to survive being passed through 90 updates of that running summary. In practice, information starts to fade, and gradients that would teach the model about that connection shrink as they travel back through every step. LSTMs (which have extra internal “gates” to selectively remember some information) helped a lot, but they didn’t eliminate the problem.

The first big breakthrough was attention, introduced by Bahdanau et al., 2014, in their implementation where they added it to the RNN. Instead of forcing the whole input sentence through one summary vector, attention let the decoder look back directly at every input word, and decide for itself which ones mattered for the word it was producing right now. It was a huge improvement, but the RNN underneath still had the same issues reading words one at a time.

The Transformer’s Big Idea

The transformer’s insight is right in the paper’s title: attention is all you need. The idea was just that - keep the attention part, but throw out the rest of the RNN entirely.

In a transformer, every word looks at every other word directly, in a single step. That solves both problems at once:

  1. Everything happens in parallel. No accumulating calculations, so a whole sequence gets processed at once. This allows the compute to scale massively, and is a big part of why transformers can scale up to the enormous models we have today.
  2. Every connection is one step long. All positions can look at each other directly. No information has to survive a long chain of updates to get where it’s needed - it’s all right there.

These were major upgrades, but they don’t come completely for free. In fact, while you can take advantage of high-performance compute much better, the compute you need is larger - comparing every word to every other word means the work grows with the square of the sequence length. And by looking at everything at once, the model loses any built-in sense of word order, so that needs to be added back explicitly. But both of those things are trades very much worth making, and we’ll discuss in this notebook how both problems are dealt with.

And though we’ll use text-based tasks as the primary example in this notebook, none of this is specific to language at all. The same architecture works on anything that can be framed as a sequence, and indeed, today, the transformer is popular for processing images split into patches, audio split into frames, and more.

In This Notebook

We’ll build the full transformer from the original paper, which includes both an encoder, which reads and understands an input sequence, and a decoder, which generates an output sequence from it, one token at a time. The core model architecture then just needs an input to process the data, those two parts, and an output to give the final predictions:

arch-simple

Of course, a lot happens inside those individual pieces, which we’ll fill as we go. But those 4 main components are the core of the architecture.

And while this encoder-decoder design is the original, complete form of the architecture, modern models also come in encoder-only and decoder-only flavors. We’ll discuss those once we fully understand the “complete” version.

As we go, we’ll use the original paper is our blueprint, but this notebook isn’t a line-by-line reproduction of it. In the time since 2017, the field has settled on a clearly better choice for some parts of the model, so where those choices are obvious and settled, we’ll choose them instead rather than sticking faithrfully to the paper and choosing something deprecated. When that happens, I’ll point out what the paper did differently and why things changed. The goal is a strong understanding of the core architecture, built the way you’d reasonably build it today.

We’ll build up to actually assembling and training a transformer on a small, simple task. The paper’s training task was translating between languages, which needs far more data and compute than we’d want here. So we’ll give our model a miniature version of translation: converting human-written dates into a standard format.

"15 june 2017" \(\rightarrow\) "2017-06-15"

It’s small enough to train quickly, but still has all of the same core considerations as the original task, which we’ll get to see really matter in how we make decisions about setting up our model.


Major Sections

This notebook is divided into 11 core sections, each covering some major block of the architecture, training, evaluating, or modern extensions.

  1. Inputs - how text becomes numbers: tokenization, byte-pair encoding, embeddings, and positional encodings.
  2. The Building Blocks - attention, multi-head attention, residual connections, layer normalization, and feed-forward networks.
  3. The Encoder - assembling the blocks into the half that understands the input.
  4. The Decoder - causal masking and cross-attention, for the half that generates.
  5. The Output - turning vectors back into tokens.
  6. Assembling the Full Transformer - putting it all together in one model.
  7. Training - teacher forcing, the loss, the optimizer, and learning-rate scheduling.
  8. Training on a Real Task - our date-translation model, trained from scratch.
  9. Using the Model - generating outputs, and testing how far it generalizes.
  10. Inspecting the Model - attention maps and embeddings: what did it actually learn?
  11. Transformers in the Wild - how modern models build on (and depart from) this architecture.

A handful of deeper tangents live in the Appendices, where I link to the appropriate one wherever it comes up.

📌 Note: This notebook assumes you’re comfortable with the basics of neural networks - layers, weights, and training by gradient descent. If you aren’t - I suggest a brief introduction there first!

Getting Set Up

In this tutorial, we’ll be making use of some packages to help us understand, visualize, and create a small transformer model. Before we write any code, we’ll import all of the necessary packages here.

We’ll start by making sure that our versions of torch and torchtext are compatible and stable. Run the below cell with the pip install’s, and when it finishes go to Runtime -> Restart session in the toolbar above to be sure that the changes are implemented properly.

# ------------- for plotting and display -----------
import os
# must be set BEFORE importing transformers / huggingface_hub:
os.environ["TRANSFORMERS_VERBOSITY"] = "error"        # kills the "LOAD REPORT / UNEXPECTED keys" table
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"      # kills the config.json/vocab.txt/model.safetensors bars
os.environ["HF_HUB_DISABLE_IMPLICIT_TOKEN"] = "1"

import warnings, logging
warnings.filterwarnings("ignore")                     # generic Python warnings
logging.getLogger("huggingface_hub").setLevel(logging.ERROR)  # kills the "unauthenticated requests" warning

# ------------- for building with PyTorch -----------
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

from transformers.utils import logging as hf_logging
hf_logging.set_verbosity_error()                      # reduce some logging

print("Version: " + torch.__version__)
Version: 2.14.1+cu130
# ------------- some general utilities -------------
import numpy as np
import json
import random
from datetime import date, timedelta # for our training task later
import base64
from collections import Counter
import gensim.downloader as api # for GloVe
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.decomposition import PCA
from transformers import AutoTokenizer, AutoModel     # to load existing hugginface models

# ------------- for plotting and display -----------
import matplotlib.pyplot as plt
from matplotlib.patches import FancyArrowPatch
import matplotlib as mpl
import imageio.v2 as imageio, json
from IPython.display import HTML

mpl.rcParams['font.family'] = 'DejaVu Sans'
TEXT, LINE, DOT = '#2b2b2b', '#5B8FB9', '#E0876A'

# ----------- set seed, for robust re-runs --------
torch.manual_seed(0)

1. Inputs - How to Process Words

Before we get started with the actual architecture, there’s actually quite a lot we need to cover on the basics of how we set up to do computations over words. Or really, what really does the input to the model need to be. Of course, computations must be done over numbers, so we’ll have to turn our words into numbers somehow, even though they don’t naturally or obviously map that way. So, we need to decide how to transform words into numbers.

As it turns out, there are a lot of different decisions that can be made there, and much that we can do to set ourselves up for success so that those numbers have as much meaning as possible.

1.1 Tokenization: How to Deal with Words

We just talked in the introduction about how sentences aren’t just really collections of words - it matters how they are ordered and structured. Taking a step back: are words just collections of letters? If I want to understand a word, especially one that’s new to me (like all words are new to a model at the start) - do we, and if so how can we, break that word up in a way that helps understand it better?

What this really is, is a question in how do we tokenize words. How to treat a word and turn it into numbers that can be processed by a computer really starts with the question: do I break it up or leave it whole?

1.1.1 How to Divide a Word

Take a word and assume you want to turn it into some numbers. Given that words have a smaller component, letters, we’re left with multiple options for how to do that.

  1. We could break it up into its letters - “dog” = [d, o, g] - and cleanly map each letter to a unique number (1 - 26 for the English language). Simple!
  2. We could treat “dog” as a complete word - it already is a small object with meaning in its own right. Then, we’d map each word to its own number - (1 - 10000 (or whatever size of English dictionary we decide to use)).
  3. We could try to break our words into roots that have meaning. “Dog” is its own word - “Dogs” becomes “dog” + “s” - since the dog part meant the same thing, but adding that s makes it pretty universally plural in English. We map each word root and ending to its own number (1 - ~7500, if we used the same English dictionary as last time but now have fewer new entries for repeated roots).

What this is, is a choice in tokenization - how a word is broken up. Each word can be tokenized by letters, can be its own token, or can be tokenized into subwords.

But of course, as with every choice, not all are equally good.

  1. In Option 1, the vocabulary is tiny, which can be useful - every word remains fairly small and there aren’t many numbers to work with. And nothing is ever unknown - any new word in the English language is constructed with letters - so we have a working model for any new word we try to give the model. But remember, that we’re still hoping to keep the meaning of our words, and letters don’t naturally have meaning. We’ve essentially thrown the meaning of the word away. The model will have to rediscover that d-o-g is one thing when it shows up in that order and learn to attribute meaning to that. New words that the model has never seen before, will be computationally passable - but those letters likely won’t mean much in their unseen order.

  2. Option 2 keeps words whole and meaningful, but the vocabulary is large, and without good reason. Consider walk/walks/walked/walking. They become four separate entries that each have to be learned independently by the model to have nearly identical meaning.

  3. Option 3 gives us the best of both worlds. First, it has a smaller vocabulary. Because the “walk” part is a single word, and the endings “s”, “ed”, “ing” are repeated constantly, we don’t need as many entries in our dictionary. Never before seen words by the model still have potential to decompose into useful parts (maybe you don’t have a word in the dictionary, but “ing” tells you it’s an active action). AND we get the added benefit that, broken up like this, the units now line up better with how words are actually built. “Walk” gets to mean the exact same thing every time. Adding “ed” or “ing” or “s” just gives extra context or detail to how the core thing is happening. We want our subwords to contribute the exact same starting meaning, every single time, in reusable pieces, to best later derive meaning from words.

1.1.2 Dictionaries and Vocabulary Size

From deciding on the tokenization scheme, turning words into numbers does become straightforward - and it goes back to the size of our vocabulary, or dictionary that we use.

For every token that we’ve turned our words into - assign them a number. Maybe “walk” is the 6800th word in the dictionary, so “walk” = 6800. Maybe “run” = 4300. Then all of our word endings, “s” = 1, “ing” = 2, “ed” = 3, … and so on, let us assign every subword its own unique number.

📌 Note: It’s worth asking, since we pointed it out above, why we’d care about vocabulary size at all. And there are two reasons, which I’ll say now but might make more sense a little later.

First, as we’re about to see, every entry in the vocabulary will eventually get its own learned meaning-vector of size n (the embedding dimension). The vocabulary size will directly set how big that table of vectors needs to be, meaning that a bigger dictionary is a bigger model.

The other reason is that at training time, rarer words get to learn from fewer appearances (the model is learning how to contextualize instances of words, after all). So smaller can be better, in the sense that every entry has more instances to learn from. Of course, only if those instances actually teach something. Letter-level dictionaries don’t have many entries, but it’s very hard to learn what a “w” means every time it comes up. Subwords, again, have the best of both worlds here, where you independently learn from all instances of “walk” and “ed”, and don’t have to learn “walked” independently from just its specific examples. In general, keeping vocabulary size small isn’t the point, but it is convenient if it can be kept a little tighter, if that tightness also tracks better to word meanings.

But if we’re decided on mapping subwords to a dictionary - what dictionary do we use? How do we know what parts become a subword? For that, we need to go a little deeper.

1.2 Byte-Pair Encodings (BPE): Making a Vocabulary

You may have been tracking, as we talked about tokenization, that we kinda glossed over something when we picked very nice examples of words and subwords in our discussion. “walk” + “ed”, “s”, “ing” all make nice, clean words by just breaking the existing word in two.

Unfortunately, not all words work that way. Consider “happy” and “happily” - do we split on the “happ” to have that same-meaning token root even though we would track “happy” as the simplest form of that word? If we do, is that “happ” the same one as we’d get if we break “happening” into “happ” and “ening” like we would want to do with the word “thickening”?

It’s not a purely grammatical choice to make, since grammar is not pure and clean like that. Rather, it’s really a fact about how words are used.

How exactly to split your words into subwords is a decision you could sit down and make for every word in the English dictionary - or - we could use byte-pair encoding to make these decisions for us.

1.2.1 The Idea

The basic idea is this. We can discover which words and subwords to make their own tokens, just by trying to split up words into all the little pieces that we can and seeing which little pieces show up often enough that they probably actually have meaning.

We can turn walking into “wa” + “lk” + “ing” - but can tell that’s probably bad immediately because “wa” and “lk” don’t really occur often across MANY words in the English language, so they’re likely not a root or ending or core word.

Conversely, “walk” + “ing” - both of those show up in many dictionary entries attached to each other, so they more likely do mean something. BPE just does this programmatically - building up a vocabulary by trying to stick letters together, and based on how often those letters tend to all come together, keeping them together only if that collection frequently comes up.

1.2.2 The BPE Loop by Example

Because we’re building this based off of frequency of actual use - we need to start with a corpus of data. Basically, lots of English text. The main advantage here is that what gets to be a token is based off of actual use. We also can better handle rare words this way. We could start with just the dictionary, but then every word gets equal importance from the get-go: a word that almost never shows up gets treated the same as words that are used constantly. Starting with a real corpus of text means that we can make decisions about if words need to be their own vocab word, if they have meaningful, reusable pieces, if they are important enough to get their own entry, or if they stay in smaller pieces that show up more often, based on how important it is that we know about them.

Let’s see how BPE proceeds for a small example.

  1. We have a corpus. Let’s say it contains the words:
  • low, with count 5
  • lower, with count 2
  • newest, with count 6
  • widest, with count 3
  1. Start by assuming that the vocab is every letter independently. For us, that means we have \([{l, o, w, e, r, n, s, t, i, d}]\) as the starting vocabulary.

  2. We try to merge letters together. The goal here is to determine what letters, if any, ought to have their own token when together, since they appear together often. To do this, we take every adjacent pair, try to stick it together, and count how often that pair appears. So we make every pair we can: \([{lo, ow, we, er, ...}]\) and count how many times that pair occurs in the corpus.

  • e + s appears in newest (×6) and widest (×3) = 9 times

  • s + t also appears in both = 9 times

  • l + o appears in low(×5) + lower(×2) = 7

  • And so on, for every pair.

    We choose, say because we found it first, to stick e and s together. (In a real corpus you really wouldn’t have ties so no arbitrary decision like this must be made - the most frequent pair is the most useful and we leave it at that). We add it to the vocab, so now the vocab is \([{l, o, w, e, r, n, s, t, i, d, es}]\).

  1. We repeat what we did in step 3, except now “es” counts as its own unit, permanently glued so that every instance of “e + s” acts as a single “es” unit, and all “pairs” made with it will now contain 3 letters. So new candidate pairs made out of the existing vocab are: \([{lo, ow, ... ew, wes, ne, wi, id, des, est]}\)
  • es + t appears in newest(×6) + widest(×3) = 9
  • l + o still appears 7
  • etc. So “es + t” wins, and we add that as a vocab word (vocab \([{l, o, w, e, r, n, s, t, i, d, es, est}]\)) and proceed.

We’d next attach l+o, and you’d see “low” come out shortly after. We keep going, until we reach a determined vocab size. If we say we wanted our vocab to have 12 entries, we’d already be done. If we said 20, we’d keep merging until we had 20 unique pieces in our vocab. For this reason, vocab size - which is a choice that is set beforehand - is a choice that has to be sensible, at least a little. 12 is obviously too small for this corpus, and 100 would be too big.

In practice, and for a real corpus, this number is fortunately much less sensitive than that, and there is a very wide range of numbers that all give good outcomes. It is an empirical/engineering decision, found from testing what works well, rather than a principled one. But fortunately in practice, the range is wide (typically ~30,000–50,000 for models like BERT and GPT-2 with how they tokenize, but much larger, ~100k-300k for large, modern models that tokenize with byte-level BPE). The is one that’s worth exploring, but not critical so long as it’s in a reasonable window that matches the tokenization algorithm.

1.2.3 A BPE Loop in Code

So, since the algorithm is so simple, and you may be more inclined to look at code than a text description, let’s actually write this into code and see it run quickly.

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
words = {tuple(w): count for w, count in corpus.items()}

vocab = sorted({char for word in words for char in word})
target_vocab_size = 14                          # vocab size is chosen, 14 here so it matches what we worked through above.
print(f"Starting vocabulary ({len(vocab)} tokens):", vocab, "\n")

merges = [] # keep track of the exact merges we did, in what order
while len(vocab) < target_vocab_size:
    # 1. count every adjacent pair, weighted by how often its word appears
    pairs = Counter()
    for word, count in words.items():
        for a, b in zip(word, word[1:]):
            pairs[a, b] += count

    if not pairs:                               # nothing left to merge (fully glued)
        break

    # 2. greedily pick the most frequent pair and glue it into one token
    (a, b), count = pairs.most_common(1)[0]
    merged = a + b
    vocab.append(merged)

    # 3. re-spell every word, replacing that pair wherever it occurs
    def glue(word):
        out, i = [], 0
        while i < len(word):
            if word[i:i + 2] == (a, b):
                out.append(merged); i += 2
            else:
                out.append(word[i]); i += 1
        return tuple(out)
    words = {glue(word): count for word, count in words.items()}
    merges.append((a, b))

    print(f"Merge: {a!r} + {b!r} -> {merged!r}   (seen {count} times)")

print(f"\nFinal vocabulary ({len(vocab)} tokens):", vocab)
Starting vocabulary (10 tokens): ['d', 'e', 'i', 'l', 'n', 'o', 'r', 's', 't', 'w'] 

Merge: 'e' + 's' -> 'es'   (seen 9 times)
Merge: 'es' + 't' -> 'est'   (seen 9 times)
Merge: 'l' + 'o' -> 'lo'   (seen 7 times)
Merge: 'lo' + 'w' -> 'low'   (seen 7 times)

Final vocabulary (14 tokens): ['d', 'e', 'i', 'l', 'n', 'o', 'r', 's', 't', 'w', 'es', 'est', 'lo', 'low']

In the end, BPE buys you multiple, important things, all at once, while constructing this final vocabulary.

  1. You choose the vocabulary size directly. The vocab starts as “all the characters” (~26 + punctuation) and grows by exactly one every merge until hitting the set size. So subword vocab size gets to be something you decide explicitly and can limit or increase freely as needed.
  2. The vocabulary will be able to cover all incoming words. Because we begin with characters and only ever add extra bits to that, every word has, at a minimum, its characters to be tokenized as. A long, rare word the tokenizer has never seen (like antidisestablishmentarianism), a typo, an uncommon proper noun - all of these can just be built from smaller, known pieces when they are not already in the vocab as-is. Nothing ever needs to be truly marked as “unknown” to the model - so long as it’s constructed of at least letters.
  3. The pieces still track meaning. Since frequent sequences get merged into units, and the most frequent sequences in English ARE the common words (the, walk, run) and common morphemes (ing, ed, s, er), we will recover those with BPE. So the simple algorithm, though it just looks for raw pairing frequencies, tends to land on units that happen to line up with roots and endings that are most frequently used “for free”. (Though… not always, of course. See Bostrom & Durrett, 2020 to dive deeper!)

1.3 Encoding and Decoding Words: Using a Vocabulary

So, we’ve now successfully built a vocabulary, based on some (ideally representative) corpus of text, because we know that we want to break words into subwords as our tokenization scheme. Now that we have that frozen vocabulary, the next step is to actually use it. Every piece of text that ever goes through the model must be mapped to numbers using this vocabulary, and the model’s numeric output must be turned back into text using the same vocabulary. We call this encoding, and then decoding those words.

1.3.1 Tokenizing New Text

Imagine that we want to process a new word. Say, “lowest”. After running the BPE loop, we have the vocabulary:

\[ \text{[d, e, i, l, n, o, r, s, t, w, es, est, lo, low]} \]

So how do we tokenize it?

You might say, obviously, “low” + “est”. But why not “lo” + “w” + “est”, or some other combination? This vocabulary, while a fixed reference, still doesn’t tell you how to break words into subwords for tokenization. Just having a vocabulary doesn’t automatically determine the best way to use it.

And just sticking the largest chunks together (like you might think naively makes the most sense) isn’t reliable either. Recall that in our example above, “es” was put together early on, and because of that, “st” was not chosen as a pair. In that case, “st” only showed up as belonging to “est”. But imagine that corpus also had 5 instances of the word styles, so our final vocab contained both “st” and “es”. Then, based on just that vocab, when the word “pester” comes in - how do you know to make the “es” a pair versus the “st”?

For this reason, we keep the merge order around, remembering it, and tokenizing new words by replaying the frozen merges we did to build our vocabulary. When we do this, we can know that “es” was found first in our merges, so it’s correct to merge the “es” first in “pester” - not the “st”. Re-performing our merges in the exact order they were learned ensures every incoming word gets the same treatment as was actually used to build the vocabulary, and keeps those presumptions of most common = best merge intact.

Let’s do a simple application of this to the word “lowest”, which we expect to compile well from our existing vocab, and some new words that won’t find merges as cleanly.

def tokenize(text, merges):
    tokens = tuple(text)                    # start from raw characters
    for a, b in merges:                     # apply each learned merge, IN ORDER
        merged = a + b
        out, i = [], 0
        while i < len(tokens):
            if tokens[i:i + 2] == (a, b):   # this position starts the pair -> glue it
                out.append(merged); i += 2
            else:
                out.append(tokens[i]); i += 1
        tokens = tuple(out)
    return list(tokens)

for w in ["lowest", "newer", "bees", "pester"]:
    print(f"{w:8} -> {tokenize(w, merges)}")
lowest   -> ['low', 'est']
newer    -> ['n', 'e', 'w', 'e', 'r']
bees     -> ['b', 'e', 'es']
pester   -> ['p', 'est', 'e', 'r']

1.3.2 From Tokens to Integer IDs

Great. So now, for any given word, we can properly tokenize it into component parts. But those tokens are all still comprised of strings - and we started this whole thing needing numbers. So we simply decide on a fixed order for our vocabulary of tokens, and let each token’s position be its numerical ID. So a simple lookup table tells us what number every token in our vocabulary maps to.

In doing so, this is also where we can reserve slots for special tokens. Special tokens aren’t words or word-components, but rather special entries that help the model orient itself, do something special, or just deal with anything else otherwise unaccommodated by the dictionary. Some very common ones include:

  • <pad> - a token that’s meant to be largely ignored, used as filler to make sequences in a batch the same length (we’ll see why this matters when we hit masking)
  • <unk> - a fallback for anything we genuinely can’t represent - something that doesn’t exist in our vocabulary at all. (In the above example we just worked, “p” and “b” would have been represented with this token - because even though BPE says they’re their “own” token - that simple vocabulary doesn’t actually have a token for them at all!)
  • <bos> / <eos> - “beginning” and “end” of sequence, so a generating model knows where a sequence starts and, crucially, when to stop (this comes back in the decoder)

These aren’t learned by BPE but are rather added by hand as expected needed features. More can be added to add other special functionalities to the model, or teach it about other special markers or types in text to enhance formatting, readability, etc. A real production model will use lots of extra special tokens to help it format text, call tools, understand different data types, etc.

This lookup table is used to turn tokens into numbers, and the other way around. The same way that [“low”, “est”] becomes [17, 15], when the model outputs [17, 15], we know that meant [“low”, “est”].

📌 Note: Ok, so we totally glossed over something actually, and a keen reader might be asking: how would I know that “low” and “est” are supposed to go back together, if the model is outputting these little word chunks one at a time?

Great question! The answer is that actually we oversimplified our BPE vocabulary. In reality, you split a corpus into a vocabulary in a way that preserves its spaces with special characters. Whether “est” occurs at the beginning, end, or middle of a word is meaningful, so spaces are denoted by a character (like in GPT-2’s tokenizer, Ġ) that keep where those spaces are. Then, in a real vocabulary, “Ġest” and “est” would likely be their own tokens (if we say we always attach a space to the front of a word), so that “low estate” vs “lowest ate” can easily resolve to [“low”, “Ġest”, “ate”] vs [“low”, “est”, “Ġate”].

But doesn’t that mean our vocab size needs to be much, much bigger to accommodate every word start/end having a version that carries a space? Yes, but not as much as you’d think. Most words will only need their space-prepended version, as they’ll mostly show up at the start or on their own. Only words or parts of words that frequently sit inside or at the end of other words need a “bare” version - precisely those pieces like “ate”, “est” - are the ones who will get repeats. And as we discussed before, vocabulary size isn’t the primary thing to consider when other factors matter more. In this case, making sure that you know where the boundaries are between your words is more important to get right!

So a real encoding using our simple vocabulary might encode like:

special = ["<pad>", "<unk>", "<bos>", "<eos>"]
full_vocab = special + vocab

token2id = {tok: i for i, tok in enumerate(full_vocab)}   # token -> integer
id2token = {i: tok for tok, i in token2id.items()}        # integer -> token (for decoding)

def encode(text, merges, token2id):
    ids = [token2id["<bos>"]]
    for tok in tokenize(text, merges):
        ids.append(token2id.get(tok, token2id["<unk>"]))  # unknown token -> <unk>
    ids.append(token2id["<eos>"])
    return ids

def decode(ids, id2token):
    toks = [id2token[i] for i in ids
            if id2token[i] not in ("<bos>", "<eos>", "<pad>")]
    return "".join(toks)

for w in ["lowest", "widest", "zzz"]:
    ids = encode(w, merges, token2id)
    print(f"{w!r:9} -> tokens {tokenize(w, merges)}")
    print(f"{'':9}    ids    {ids}")
    print(f"{'':9}    back   {decode(ids, id2token)!r}\n")
'lowest'  -> tokens ['low', 'est']
             ids    [2, 17, 15, 3]
             back   'lowest'

'widest'  -> tokens ['w', 'i', 'd', 'est']
             ids    [2, 13, 6, 4, 15, 3]
             back   'widest'

'zzz'     -> tokens ['z', 'z', 'z']
             ids    [2, 1, 1, 1, 3]
             back   '<unk><unk><unk>'

And where this leaves us, is that any incoming string of words can be tokenized into its parts, and then turned into numerical IDs - something our model can actually work with!

📌 Note: You may still have questions. Like, “what was the”byte” in BPE?” Or, “how do models handle all of the other, random things that can be formulated as”text”: the emojis, the code snippets, the non-alphanumeric characters?” And as it turns out, those have the same answer, which is that, while we in this example paired normal text characters, many modern BPE implementations actually operate over bytes. Working through an example in bytes is less intuitive and takes more background, so I’ve left that all for Appendix 1 - go there if you want to see how this really really operates typically!

1.3.3 The Tokenizer

If you’ve ever worked with a transformer model before, you’ve probably encountered the idea of a tokenizer as a specific, separate thing that has to go along with a model.

Well, a tokenizer, as a packaged object, is just the process that we just talked about, wrapped up into a reusable object. It’s the specific vocabulary, special tokens, and merge order that was used for the corpus for that specific model. Its job is just to re-perform, on any piece of text, the same merges in the same order, using the same dictionary, so that these token IDs match what the model was trained on and therefore expects. That’s it!

Since the tokenizer is generally specific to the model, you can see now why it’s crucial to use THE tokenizer for the given model. We’re using it to turn our words into numbers - the model needs “688” to mean the same thing as it expects, or everything going forward falls apart. So while a tokenizer might just look like “prepping the input” - it really is a core part of the model itself, on which all future understanding that the model has will rely.

1.4 Word Embeddings

So now we technically do have a number for every token, and we could just move ahead. But really, still what we’ve done is transformed words, something that does have meaning, into numbers - something that doesn’t. If “walk” is tokenized and looked up in the vocabulary to match ID 6800 - that number is similar to 6805 - which could correspond to “wash”. So similar numbers, but not similar words. “Stroll” might get assigned 4500 - numerically much farther away despite being a synonym. So turning our words into a number via a lookup table doesn’t mean those numbers carry any information about the original meaning. In fact, you don’t expect them to carry any of that information at all. Even though we’ve divided our words in a way that has the opportunity for meaning, tokenization alone doesn’t give us meaning. It IS true that tokenization means “walked” likely decomposes into “walk” + “ed” (so 6800 would be re-used on walked vs walk - a very useful feature!) - BUT, for more complex relationships, between non-related tokens, all of our words are still just numbers that don’t cleanly relate to one another.

1.4.1 Words as Vectors

This is the reason that we further convert words into embeddings. These embeddings represent words as vectors of real numbers, which carry with them meanings and relations that actually mean something.

Word embeddings are an important concept - unlike simpler representations that treat words as discrete symbols (such as one-hot encoding, where ‘dog’ is just a unique ID), embeddings represent words as vectors in a continuous space. (e.g. dog = \([.234, .033, -.394, .825, ....]\) - many numbers, not just one). These vectors not only place similar words closer together in the vector (or ‘latent’) space but also capture relationships through directionality. For example, not only will “king” and “queen” point to more similar places in the latent space than “apple”, but the directional difference between ‘king’ and ‘queen’ will also be similar to that between ‘man’ and ‘woman’. In this way, word embeddings reflect both semantic similarity and the relationships between words.

Here is a very-simplified visualization of this concept:

# Example embeddings for a few words, these are made up for now
word_embeddings = {
    'king': [0.8, 0.1, 0.3, 0.5],
    'queen': [0.75, 0.12, 0.35, 0.48],
    'apple': [0.15, 0.76, 0.3, 0.4],
    'man': [0.6, 0.2, 0.35, 0.5],
    'woman': [0.58, 0.22, 0.36, 0.52]
}

# Convert embeddings to 2D using PCA for visualization
pca = PCA(n_components=2)
words = list(word_embeddings.keys())
embeddings = np.array(list(word_embeddings.values()))
reduced_embeddings = pca.fit_transform(embeddings)

# Plot the embeddings in 2D space
plt.figure(figsize=(8, 6))
for i, word in enumerate(words):
    plt.scatter(reduced_embeddings[i, 0], reduced_embeddings[i, 1], label=word)
    plt.text(reduced_embeddings[i, 0] + 0.02, reduced_embeddings[i, 1], word, fontsize=12)

# Use FancyArrowPatch for king-queen and man-woman directional vectors
arrow_king_queen = FancyArrowPatch(posA=reduced_embeddings[0], posB=reduced_embeddings[1],
                                   arrowstyle="-|>,head_width=0.3,head_length=0.5", color='blue', mutation_scale=15, lw=1)
arrow_man_woman = FancyArrowPatch(posA=reduced_embeddings[3], posB=reduced_embeddings[4],
                                  arrowstyle="-|>,head_width=0.3,head_length=0.5", color='red', mutation_scale=15, lw=1)

# Add arrows to the plot
plt.gca().add_patch(arrow_king_queen)
plt.gca().add_patch(arrow_man_woman)

# Final plot adjustments
plt.title('Example Word Embeddings + Directional Vectors')
plt.xlabel('PCA Dimension 1')
plt.ylabel('PCA Dimension 2')
plt.grid(True)
plt.xlim([-.4,.8]) #set so apple's label doesn't get cut off, purely visual
plt.show()

1.4.2 What a Real Embedding Looks Like

So how do we get word embeddings, and make sure they’re meaningful like this? The goal is to transform a word into a vector representation with semantic meaning, so the vectors should contain more info than just their position in a dictionary.

In practice, multiple pre-trained embedding models exist whose whole job is to take a word and map it to a vector like this. Two popular choices are:

  • Word2Vec: A popular method that uses neural networks to learn word associations from a large corpus of text. It has two main architectures: Continuous Bag of Words (CBOW), where the model predicts a word given its context, and Skip-Gram, where the model predicts the context given a word. Through this process, Word2Vec captures semantic relationships between words.

  • GloVe (Global Vectors for Word Representation): Another widely used method that generates word embeddings by analyzing word co-occurrences in a corpus. The model trains on the aggregated global word-word co-occurrence statistics from a corpus, capturing both local and global contexts of words.

Because these models were trained with the specific goal of generating word embeddings, they do a great job of embedding semantic meanings of words. Though note, although we’re just coming off of BPE for tokenization, note that these were both tokenized at the word level, not using BPE.

That makes it easy for us to see how a whole word gets embedded. So let’s take a look at embeddings of the word “dog”, with GloVe:

Note! The below cell may take a while to run! Feel free to skip running this cell and just inspect the cell outputs without re-running.

# GloVe 6B, 100-dimensional
# (First run downloads ~130MB and caches it, so it may take a minute.)
glove = api.load("glove-wiki-gigaword-100")

# Build an embedding layer from the pretrained vectors, exactly as before
embedding_layer = nn.Embedding.from_pretrained(torch.FloatTensor(glove.vectors))

word_idx = glove.key_to_index['dog']
word_embedding = embedding_layer(torch.tensor([word_idx]))
print(word_embedding)
print(word_embedding.shape)
[==================================================] 100.0% 128.1/128.1MB downloaded
tensor([[ 0.3082,  0.3094,  0.5280, -0.9254, -0.7367,  0.6348,  0.4420,  0.1026,
         -0.0914, -0.5661, -0.5327,  0.2013,  0.7704, -0.1398,  0.1373,  1.1128,
          0.8930, -0.1787, -0.0020,  0.5729,  0.5948,  0.5043, -0.2899, -1.3491,
          0.4276,  1.2748, -1.1613, -0.4108,  0.0428,  0.5487,  0.1890,  0.3759,
          0.5803,  0.6697,  0.8116,  0.9386, -0.5100, -0.0701,  0.8282, -0.3535,
          0.2109, -0.2441, -0.1655, -0.7836, -0.4848,  0.3897, -0.8636, -0.0164,
          0.3198, -0.4925, -0.0694,  0.0189, -0.0983,  1.3126, -0.1212, -1.2399,
         -0.0914,  0.3529,  0.6464,  0.0896,  0.7029,  1.1244,  0.3864,  0.5208,
          0.9879,  0.7995, -0.3462,  0.1409,  0.8017,  0.2099, -0.8601, -0.1531,
          0.0745,  0.4082,  0.0192,  0.5159, -0.3443, -0.2453, -0.7798,  0.2743,
          0.2242,  0.2016,  0.0174, -0.0147, -1.0235, -0.3970, -0.0056,  0.3057,
          0.3175,  0.0214,  0.1184, -0.1132,  0.4246,  0.5340, -0.1672, -0.2718,
         -0.6255,  0.1288,  0.6253, -0.5209]])
torch.Size([1, 100])

You’ll notice that the word “dog” is now represented as a 100-dimensional vector. In fact, all words that are embedded with GloVe will be 100-dimensional vectors, that tell us vital information about the word’s meaning.

1.4.3 Training an Embedding Layer

While pre-trained embedding models are great, and one of these can be inserted frozen into the model, in practice, most modern models leave this as a trainable layer. In this way, the model can learn on its own what embeddings make the most sense (and it can easily adapt to vocabulary size).

So really, this would become the very first layer in the model. Even though we’re still talking “prepping for the transformer layers” - which we are! - it’s important to note that a full transformer-based model architecture does contain some non-transformer layers, like this one. So this layer will have trainable weights, hyperparameters, and all of the other typical things you’d find in a standard neural net layer.

That being said, embedding layers are fairly simple as far as layers go. It’s just a single matrix operation:

\[ X_{embed} = X_{IDs} W \]

where \(W\) is a matrix of size \(n_{vocab}\) x \(n_{embeds}\) - one row for every single word in the dictionary, with the length of our embedding dimension. And \(X_{IDs}\), our input, is size \(n_{words}\) x \(n_{vocab}\).

In this way, \(X_{IDs}\) functions as a matrix of one-hot encoded ID’s for every token. For every token, take its ID, put a 1 there, and a 0 everywhere else. So, if a vocab is size 10, ID “7” becomes:

\[ [0,0,0,0,0,0,0,1,0,0] \]

And a matrix constructed of these one-hot entries - one for every word - makes \(X_{IDs} W\) really just equivalent to a lookup operation. Every word corresponds to a row in \(X_{IDs}\) which has a 1 in only one location - it grabs that row from the W matrix, and returns the embedding at that row. (In practice during the computation, the literal matrix multiplication isn’t even performed since the \(X_{IDs}\) matrix is so sparse - just doing the simple lookup is easier). But conceptually, that simple matrix multiply is all this layer does. The W matrix - those learnable parameters - IS the lookup table.

To train, or not to train?

We showed that pretrained embedding layers like GloVe exist, so even though a fresh embedding layer is relatively small, and simple, why wouldn’t you just use those? Historically, when training data was scarce, this was the more common choice. And it will still give a model a running start on word meaning without having to learn it.

But, modern transformers almost always learn their embeddings from scratch instead. The reason is of course, that there are real advantages to doing so that will almost always make it worth it.

The first reason is about embedding quality. While pre-trained embeddings like those from GloVe are good, an embedding learned as part of the model gets to learn on the actual task, objective, and specific data being used - not by GloVe’s generic co-occurrence statistics. The vectors get to specialize for exactly what this model is being asked to do. When you have enough data, that beats a one-size-fits-all pretrained set.

The second is also relating to that vocabulary size coming back. The weight matrix \(W\) has a dimension that is exactly that vocabulary size - so first, here is where that vocabulary size charges a price! But more importantly, this vocabulary determines what is allowed into the model. If you want, or have, a different vocabulary for your dataset, you can’t use it with GloVe’s embedding layer if they don’t match, which they generally won’t. A pretrained embedding layer means you are stuck with the vocabulary it accommodates, and therefore the tokenizer it used, which offers less flexibility.

So, in this tutorial, we’ll be adding one of these embedding layers to our model and training it from scratch. We can build an embedding layer simply with PyTorch, and see what might come out of a randomly initialized embedding layer:

vocab_size = 10  # Small vocabulary size for demonstration
embedding_dim = 4  # Small embedding dimension for easy visualization

# Create the embedding layer
embedding_layer = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim) #the nn.Embedding layer handles this for us

# Sample input: indices of words
input_indices = torch.tensor([0, 2, 4, 5, 3])  # Example word indices from the vocabulary, each of these numbers might correspond to a word in the dictionary.
                                              # Notice with PyTorch, we don't need to worry about doing the one-hot encoding ourselves.

# Sample output: embedding vectors
embedding_vectors = embedding_layer(input_indices)
print(embedding_vectors)
tensor([[-1.1258, -1.1524, -0.2506, -0.4339],
        [ 0.3223, -1.2633,  0.3500,  0.3081],
        [-1.3527, -1.6959,  0.5667,  0.7935],
        [ 0.5988, -1.5551, -0.3414,  1.8530],
        [ 0.1198,  1.2377,  1.1168, -0.2473]], grad_fn=<EmbeddingBackward0>)

The resulting dimension of word-embedding a text example is an \(n_{words}\) x \(n_{embeds}\) matrix, which represents, for each word in our sentence, an \(n_{embeds}\) - dimensional vector. As the model trains, it’ll learn the best embeddings.

1.4.4 The Embedding Dimension

One more important thing to discuss a bit more. Above, GloVe used 100 as the embedding dimension. In our simple example above, we just chose 4. In reality, that embedding dimension is the first major architectural hyperparameter that we have control of. And it matters.

Firstly, it has huge implications for the size of the model. In a transformer, the width of a word’s vector here becomes the width that every subsequent layer must take in. This becomes d_model, literally, the dimension of the model, because all later layer dimensions depend on it! We’ll see it turn up everywhere in Section 2 as we go on to build the transformer components.

Second, because of that, it’s a fundamental choice in the model’s capacity - how many weights the whole model will have free to learn. While GloVe worked with 50-, 100-, 200-, and 300-dimensional vectors, the original transformer used 512. Large modern models run to 12,288 (GPT-3) and beyond. Bigger means more room to encode nuance and deeper meaning. But of course, every layer gets proportionally larger, so this is the key capacity vs cost choice getting made, right here before we even start.

1.5 Positional Encodings

We saw above how we can embed words into more useful meanings. But what we did above was turn a sentence into a collection of vocabulary indices, then each word into a vector embedding.

Transformers can process these embeddings as entire objects (as opposed to, for instance, recurrent neural networks (RNN’s), which go one element at a time), which means there’s no order of elements in the sequence represented anymore. This is where positional encodings come in. They add unique information to each element in the sequence, giving the model a way to consider the position of each element. Without positional encodings, “I walked my dog” can look the same as “my dog walked I,” losing the meaning conveyed by the order of words.

1.5.1 How Positional Encodings are Calculated

Positional encodings are vectors designed to represent the position of elements in a sequence, and they are added to the word embeddings to give the model information about the order of words. Each position in the sequence receives a unique positional encoding. The most commonly used method, introduced in the original Transformer paper, employs sine and cosine functions with different frequencies for each dimension of the encoding:

  • For even indices (2i): \[\begin{equation*} PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \end{equation*}\]
  • For odd indices (2i+1): \[\begin{equation*} PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \end{equation*}\]

where:

  • pos is the position in the sequence (i.e. the number of the word in the sentence or text),
  • i is the dimension in the positional encoding vector, (i runs from 0 -> \(\frac{n_{embeds}}{2} - 1\)) and
  • \(n_{embeds}\) is the dimensionality of the model (i.e., the size of the embedding).

It’s important to note here how this \(pos\) is not the same as \(i\) in more detail. \(pos\) is the literal position in the sequence. A sequence [“The”, “dog”, “ate”] has \(pos\) values: [0,1,2]. \(i\) is the frequency index that maps to pairs of dimensions. Every \(i\) has a value along both its sin and cosine dimension. For this reason, \(i\) ranges from 0 to \(\frac{n_{embeds}}{\bf{2}} - 1\) - there is one value for each the sin and cosine dimension for each i.

This approach ensures that each position is given a distinct encoding, while maintaining a level of similarity between encodings for positions that are close to each other. The use of sine and cosine functions allows the model to compute positional encodings for sequence lengths longer than those encountered during training.

The “10000” here is a scaling factor chosen as part of the design of the positional encoding mechanism. This number helps to create a range of wavelengths in the sine and cosine functions, ensuring that each position in the sequence gets a unique and smoothly varying encoding. The choice of 10000 is somewhat arbitrary, but it’s based on the intuition that it allows the encoding to generate a wide variety of frequencies.

1.5.2 Doing the Math for One Example

Let’s look into encoding, for instance, the sentence: “I love dogs”.

We’ll say that our word embedding layer embeds each word into a 4-dimensional representation. We’ll also say that our model operates on only 3-word sentences. So our maximum sequence length is 3.

From our word embedding layer, we’ll get a 3x4 matrix. Then, our positional encoding needs to also be a 3x4 matrix. Let’s calculate the first row.

Given:

  • The position pos = 1 (for the first word in the sequence).
  • The embedding dimension n_embeds = 4.

The positional encoding for each dimension i in the first row is calculated as follows:

  • For the first dimension, i = 0, the sin component, where \(2i\) is even: \[ PE_{(1, 0)} = \sin\left(\frac{1}{10000^{\frac{2i}{d_{\text{model}}}}}\right) = \sin\left(\frac{1}{10000^{0/4}}\right) = \sin(1) \]

  • For the second dimension, where i = 0 again, the cosine component \(2i+1\) is odd: \[ PE_{(1, 1)} = \cos\left(\frac{1}{10000^{\frac{2i}{d_{\text{model}}}}}\right) = \cos\left(\frac{1}{10000^{0/4}}\right) = \cos\left(1\right) \]

  • For i = 1, the sin component (column \(2i = 2\)): \[ PE_{(1, 2)} = \sin\left(\frac{1}{10000^{2/4}}\right) = \sin\left(\frac{1}{100}\right) \]

  • For i = 1, the cos component (column \(2i+1 = 3\)): \[ PE_{(1, 3)} = \cos\left(\frac{1}{10000^{2/4}}\right) = \cos\left(\frac{1}{100}\right) \]

Thus, the first row of the positional encoding matrix for the first position, with a 4-dimensional embedding, is calculated as: \[ \text{First Row} = [\sin(1), \cos(1), \sin\left(\frac{1}{100}\right), \cos\left(\frac{1}{100}\right)] \]

These values, derived from sine and cosine functions, provide a unique encoding that varies smoothly with position, which can then be element-wise added to the word embeddings to incorporate positional information into the model.

If we did this for the rest of the matrix, we should get:

\[ \begin{bmatrix} \sin(1) & \cos(1) & \sin\left(\frac{1}{100}\right) & \cos\left(\frac{1}{100}\right) \\ \sin(2) & \cos(2) & \sin\left(\frac{2}{100}\right) & \cos\left(\frac{2}{100}\right) \\ \sin(3) & \cos(3) & \sin\left(\frac{3}{100}\right) & \cos\left(\frac{3}{100}\right) \end{bmatrix} \]

You’ll note here, importantly, that none of these positional encoding values depended on the word embedding values. So, a positional encoding matrix can be calculated once, for the expected maximum sequence length, and applied to all examples.

1.5.3 Visualizing Positional Encodings

Let’s write some python code to calculate positional encodings, so that we can visualize them.

def get_positional_encodings(max_seq_length, d_model):
    positional_encodings = np.zeros((max_seq_length, d_model))

    for pos in range(max_seq_length):
        for i in range(d_model):
            denominator = np.power(10000, (2 * (i // 2)) / d_model)
            if i % 2 == 0:
                positional_encodings[pos, i] = np.sin((pos + 1) / denominator)
            else:
                positional_encodings[pos, i] = np.cos((pos + 1) / denominator)

    return positional_encodings

We’ll start by visualizing the values of a large matrix of position encodings.

max_seq_length = 300  # Example maximum sequence length
d_model = 100  # Example embedding dimension

# Generate positional encodings
pos_encodings = get_positional_encodings(max_seq_length, d_model)

# Plot the positional encodings
plt.figure(figsize=(18, 6))
plt.imshow(pos_encodings, cmap='viridis', aspect='auto')
plt.colorbar()
plt.title('Positional Encodings')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.xticks(np.arange(0, d_model,5))
plt.yticks(np.arange(0, max_seq_length,20))
plt.show()

This might look confusing.

Early on, in the “Dimension” axis, there’s high variability with regards to “Position”. These striations increase in size, and become more stripe-like as you move to the right. But remember - the “point” of the positional encoding is NOT in the individual values here! What we’re after, rather, is that the pattern of any given Position row is unique, and more similar to its neighbors than those far away. We want the Position 100 vector to be more different to the Position 0 vector than the Position 1 vector is to Position 0. This striation pattern ensures that is so.

Indeed, we can plot the cosine similarity between these positional encodings to inspect this further. Cosine similarity measures the cosine of the angle between two vectors in a multi-dimensional space. It provides a metric to assess how similar the directions of the two vectors are, regardless of their magnitude.

# Initialize a list to store cosine similarities
cosine_similarities = []
cosine_similarities_pairs = []

# Calculate cosine similarity relative to first
for i in range(1,max_seq_length):  # comparisons for positions 0, n
    sim = cosine_similarity([pos_encodings[0]], [pos_encodings[i]])[0][0]
    cosine_similarities.append(sim)
# Calculate cosine similarity between consecutive positions
for i in range(max_seq_length-1):  # comparisons for positions i, i+1
    sim = cosine_similarity([pos_encodings[i]], [pos_encodings[i + 1]])[0][0]
    cosine_similarities_pairs.append(sim)

plt.figure(figsize=(10, 6))
plt.plot(range(len(cosine_similarities)), cosine_similarities,label='relative to position 0')
plt.plot(range(len(cosine_similarities_pairs)), cosine_similarities_pairs, label='consecutive')
plt.title('Cosine Similarities Between Positional Encodings')
plt.xlabel('Position')
plt.ylabel('Cosine Similarity')
plt.grid(True)
plt.legend(); plt.show()

So, we can see that neighboring positions have typically high, and consistently high similarity scores - two positions next to each other have similar position encodings.

📌 Note: Actually, on further inspection, you may notice that the orange line is actually exactly flat. Not just similar - any two neighboring positional encodings have the exact same cosine similarity between them. It turns out, this is actually because of an important property of sinusoidal encodings. See Appendix 2 for all the nitty gritty details!

On the other hand, generally the farther away the position is from the first, the lower the similarity score and the more different that positional encoding vector is. This is the behavior we were hoping for!

1.6 Putting it Together

So, we talked about a lot in this section - the full picture of turning words into numbers is complete. Let’s finish by talking about how we put all of that together to get a robust input into the model.

1.6.1 Sequence Padding

As we’ve worked through this, we’ve talked about sequence length a bit, and how the size of these matrices fundamentally depends on it. We need to be able to construct input sequences in a consistent way (of a consistent size) if we’re going to create rectangular matrices from them. If you already know about how neural network-based models work, then it’s probably already obvious to you that especially during training, when we batch things, this is critical. So there’s a practical problem here that it’s time we address: how do we make sure that the input matrix is rectangular, every time, even though sometimes we’ll have 3 words, other times 50, other times hundreds.

The fix is padding. We pick a maximum sequence length, \(n_{seq}\), and force every sequence to be exactly that long.

Sequences shorter than \(n_{seq}\) get filled out with a special <pad> token (which we talked briefly about in Section 1.3, when we reserved our special tokens) until they reach the target length.

Sequences longer than \(n_{seq}\) just get truncated until they are only that long.

So “The dog ran,” with \(n_{seq}\) = 6, becomes:

\[[\text{The}, \text{dog}, \text{ran}, \texttt{<pad>}, \texttt{<pad>}, \texttt{<pad>}]\]

Do this to every sequence and you always have a fixed matrix size.

📌 Note: We haven’t talked about batching yet, but that’s really the reason why you do this. So in practice, the maximum sequence length, is a maximum allowable size for the model, but each batch is just padded to make sure all sequences are the same length, often to less than the maximum size. The maximum sequence length determines the largest size a sequence can be to properly pass through the embedding portion of the model. But as we’ll see later, the rest of the transformer can accept variable size input matrices - the only rule is they need to be of a rectangular shape, not a mish-mash of sizes.

It’s worth being clear on the exact order of operations here. We pad the token IDs first (while they’re still just integers), then embed the whole padded sequence. This means that the <pad> tokens go through every step like any other token. They also get an embedding, and as we’ll talk about in a moment, they also get a positional encoding. They’re just meaningless filler occupying the leftover slots.

Of course, you might wonder, if <pad> is meaningless, won’t it interfere with learning? Especially in cases where most of the input is padding? And it absolutely would if we did nothing about it. Fortunately though, it’s easy to skip computations over these pad tokens, which we’ll talk about when we get to attention in the next section, by making sure that the model doesn’t pay attention to those <pad>’s. Just to not leave you in too much suspense - the fix is something called a padding mask, which tells attention to ignore those positions entirely. So padding here is truly just a computational convenience that introduces junk positions but allows the math to work, and we’ll make sure we ignore them when it comes time for them to potentially matter.

📌 Note: It’s true that padding is a little wasteful. We need to spend computation on positions that we then later have to actively ignore. It’s what we’ll use here because it’s the simple, standard, foundational approach. But one way that modern models are able to adapt to gigantic sequence lengths is to do this a little bit smarter. In practice, high-throughput systems use cleverer tricks (like packing multiple short sequences together, or using variable-length attention kernels) to avoid the wasted work. But those things are optimizations on top of this same basic idea, and we won’t cover any of them here in detail.

1.6.2 Adding Embeddings and Encodings

So now, we have a matrix of word embeddings, and a matrix of positional encodings. What do we do with them?

You’ll notice that these two matrices are of the same size. This was, as we discussed, by design. The input text is turned into a matrix of size n_seq x d_model (the sequence length of the model x the hidden dimension of the model, or the size of the embedding for each token).

The positional encoding ends up as a matrix of the same size. n_seq x d_model, because for every position in our maximum sequence length, we assign this unique encoding.

Since they’re the same size, we can combine them the simplest way possible: we just add them, element-wise.

\[ X_{\text{input}} = X_{\text{embed}} + X_{\text{pos}} \]

And that’s it. Each token’s meaning-vector gets its position-vector added directly on top of it, with a result that’s still the \(n_{seq}\) x \(d_{model}\) that we started with.

… but there’s an obvious objection here. Namely, won’t just adding whatever the positional encodings are corrupt the meaning of the word embeddings? Aren’t those supposed to really mean something, which we took a lot of effort in 1.1-1.4 to ensure, and won’t that be changed by adding other, irrelevant values?

The short answer is that this does end up being ok. But it’s worth spending some time talking about why this ends up being ok, since it’s such an unintuitive thing to do.

The key insight is that addition in high-dimensional space doesn’t destroy information the way it would with single numbers. If meaning and position were each a single number, adding them would fuse them irreversibly. Given the sum 7, you can’t recover whether it was 3+4 or 5+2.

But our vectors have \(d_{model}\) dimensions, often hundreds or thousands, and the positional encodings additionally have a very structured form. They are comprised of sinusoids of different frequencies, which have their own pattern that looks nothing like the pattern of a typical word embedding. So, the model can actually learn to pull them apart. It can learn which part of an incoming vector is the recognizable, positional wave pattern and which part is the word’s semantic content, and use each for what it’s good for. In fact, the model typically learns to do this in the very first set of learned transformations that it does. Since its job in that layer is already to understand what to do with the input, part of what it learns is just what part is semantic and what part is positional. So really, instead of like the simple addition example above, it’s more like two signals that are superimposed, with enough room in high dimensions for the network to separate it back out.

Still, even though the model can re-separate, it’s not as clean as doing something like concatenating them - just keeping two stacked matrices of the same size, so the information never gets muddled. And while that would be advantageous in keeping them totally trivially separable, it doubles the width of the input (makes it \(n_{seq}\) x 2\(d_{model}\)), which would require doubling the effective size of everything in the model. In practice, simply adding them has worked so well that it’s been the default since the original paper - the model has plenty of room to keep the two kinds of information apart on its own. That said, it’s not a settled question, and there is active work in the field looking into things like this.

You might notice, this does lean on there being spare capacity for the model to do that separation. But at basically any real transformer scale, it’s a safe assumption that the capacity is there. At very small, capacity-starved sizes, the cost of disentangling is more real, though concatenation doesn’t cleanly recover it either, since its extra width eats into the same budget, and something like a layer dedicated to separation just does what the first layer would anyway. So addition holds up even in more extreme cases.

So truly, while it might seem strange, addition here is the go-to. It’s not just a compromise - the models are fully equipped to handle the information coming in this way, and have no problems with it.

1.6.3 Summary

So, putting everything we’ve talked about together. When the model gets an incoming word or sequence of words, in order to prep it for the main transformer block, for each word we:

  1. Tokenize it. Break it up into sub-words using clever, principled tokenization that turns it into reusable components that naturally carry the structural meaning.
  2. Encode it. Using our vocabulary, which now operates as a lookup table, map each token to an integer ID that the model can ingest.
  3. Pad it. With the special <pad> token, make sure the sequence length is equal to the maximum one - \(n_{seq}\). Truncate it instead if need be.
  4. Embed it. We create for each token, an embedding, that makes use of the tokenization scheme to further embed meaning in a way that arithmetic can operate over.

Do it for the whole sequence, and then you:

  1. Add it. Do element-wise addition with the word embeddings and the positional encoding matrix - which is calculated once and then can be added to every sequence thereafter!

The result is a single \(n_{seq}\) x \(d_{model}\) matrix that contains what every token means, and where it was in the sequence. And we can now expand the “input” section of our starting diagram, to realize it’s really this:

arch-expanded-input

But notice that each of these steps is still setting up one word at a time. We’ve done a lot to transform our words, but this matrix is still fundamentally just a list of the incoming words, with no relations between them.

This is where the transformer really shines. So let’s see next how we take those words and start to relate them to one another via the core transformer architecture!

2. The Building Blocks of Transformers

Transformers rely on a few fundamental concepts, which are re-used across the architecture. So before we talk about what a complete encoder/decoder block looks like, let’s cover what each of these pieces is doing in detail, so we can see how they’ll all fit together.

2.1 Self-Attention Mechanisms

Self-attention mechanisms are a groundbreaking feature of transformer models that allow the model to focus on different parts of the input sequence when making predictions.

For example, in the sentence “The cat chased the mouse because it was hungry,” to understand what “it” refers to, you need to relate it to “the cat” rather than “the mouse.” Self-attention allows the model to make these connections, weighing the importance of each word when considering another word.

2.1.1 Queries, Keys, and Values

For each word in a sentence, we can assign more meaningful information about its relevance to other words through the query, key, and value vectors.

Semantic embeddings have already transformed each word in our sentence into a d_model vector that captures its meaning. And positional encodings have captured where the word was in the text. Now, each word in the text will additionally have Q, K, and V vectors, which are learned during training, which keep track of how a word relates to other words. The meaning of these vectors draws inspiration from the process of information retrieval, focusing on finding relevant information for a given query.

  1. Query (Q): A Query signifies what information is being looked for. In self-attention, it’s a vector representation of what relevant information across the entire input sequence is being sought.

  2. Key (K): Keys serve as labels or identifiers for the information contained within an input. By comparing a Query to all Keys, the mechanism assesses the relevance of information throughout the input, relative to the Query.

  3. Value (V): Values hold the actual information that the Query aims to find. After determining the relevance between a Query and all Keys, the corresponding Values are aggregated in a weighted manner, prioritizing the most relevant information for the output.

These transformations, from the embedding vector into the Q, K, and V vectors, are learned transformations, learned through weights of the model and obtained by applying trainable linear transformations (usually fully connected layers) to the input embedding vectors:

Mechanism: Given an input sequence with embedding vectors \(X\), and aiming to generate Queries, Keys, and Values, the model applies three different linear transformations to \(X\):

  • \(Q\) = \(X\) \(W_{Q}\)
  • \(K\) = \(X\) \(W_{K}\)
  • \(V\) = \(X\) \(W_{V}\)

Here, \(W_{Q}\), \(W_{K}\), and \(W_{V}\) are weight matrices that the model learns during training. These matrices start with random values and are updated to minimize the training loss, thus learning to capture and utilize relevant information efficiently.

Let’s delve into a simplified numerical example to better grasp the concept:

  • Assume each word in your input sequence starts off represented by an embedding vector of size d_model (for a very simple example, let’s say this is 4). This embedding encapsulates the semantic meaning of each word.
  • Learnable weights transform these embeddings into Query (Q), Key (K), and Value (V) vectors for each word. Assuming d_model equals 4, then post-transformation, each Q, K, and V vector for a word will also typically span 4 dimensions (though the dimensions can vary if the model architecture dictates so).

We’ll go back to our example above of a model that accepts sequence length 3 and has a d_model of 4.

The input sentence is: \[ input (x) = \begin{bmatrix} a & b & c \end{bmatrix} \]

We transform this into its embedding, with one 4-dimensional vector for each of the 3 words:

\[ embed (X) = \begin{bmatrix} a_1 & a_2 & a_3 & a_4 \\ b_1 & b_2 & b_3 & b_4 \\ c_1 & c_2 & c_3 & c_4 \end{bmatrix} \]

Then, each weight matrix, \(W_{Q}\), \(W_{K}\), and \(W_{V}\), transforms \(X\) into \(Q\), \(K\), and \(V\) through matrix multiplication:

\[ Q = X \cdot W_{Q} = \begin{bmatrix} a_1 & a_2 & a_3 & a_4 \\ b_1 & b_2 & b_3 & b_4 \\ c_1 & c_2 & c_3 & c_4 \end{bmatrix} \cdot \begin{bmatrix} w_{Q11} & w_{Q12} & w_{Q13} & w_{Q14} \\ w_{Q21} & w_{Q22} & w_{Q23} & w_{Q24} \\ w_{Q31} & w_{Q32} & w_{Q33} & w_{Q34} \\ w_{Q41} & w_{Q42} & w_{Q43} & w_{Q44} \end{bmatrix} = \begin{bmatrix} q_{11} & q_{12} & q_{13} & q_{14} \\ q_{21} & q_{22} & q_{23} & q_{24} \\ q_{31} & q_{32} & q_{33} & q_{34} \end{bmatrix} \]

\[ K = X \cdot W_{K} = \begin{bmatrix} a_1 & a_2 & a_3 & a_4 \\ b_1 & b_2 & b_3 & b_4 \\ c_1 & c_2 & c_3 & c_4 \end{bmatrix} \cdot \begin{bmatrix} w_{K11} & w_{K12} & w_{K13} & w_{K14} \\ w_{K21} & w_{K22} & w_{K23} & w_{K24} \\ w_{K31} & w_{K32} & w_{K33} & w_{K34} \\ w_{K41} & w_{K42} & w_{K43} & w_{K44} \end{bmatrix} = \begin{bmatrix} k_{11} & k_{12} & k_{13} & k_{14} \\ k_{21} & k_{22} & k_{23} & k_{24} \\ k_{31} & k_{32} & k_{33} & k_{34} \end{bmatrix} \]

\[ V = X \cdot W_{V} = \begin{bmatrix} a_1 & a_2 & a_3 & a_4 \\ b_1 & b_2 & b_3 & b_4 \\ c_1 & c_2 & c_3 & c_4 \end{bmatrix} \cdot \begin{bmatrix} w_{V11} & w_{V12} & w_{V13} & w_{V14} \\ w_{V21} & w_{V22} & w_{V23} & w_{V24} \\ w_{V31} & w_{V32} & w_{V33} & w_{V34} \\ w_{V41} & w_{V42} & w_{V43} & w_{V44} \end{bmatrix} = \begin{bmatrix} v_{11} & v_{12} & v_{13} & v_{14} \\ v_{21} & v_{22} & v_{23} & v_{24} \\ v_{31} & v_{32} & v_{33} & v_{34} \end{bmatrix} \]

Here, each \(q_{ij}\), \(k_{ij}\), and \(v_{ij}\) represents the transformed values for the \(i\)-th word’s \(j\)-th dimension in the respective Query, Key, and Value spaces. Each of the \(w_{ij}\)’s is a learnable parameter of the model.

Typically, of course, the dimensions of the model are much bigger, both in the \(d_{\text{model}}\) size, and in the maximum sequence length.

2.1.2 Calculating and Using Attention Scores

Queries, keys, and values are useful because they can be interpreted together into an attention score, which allows us to re-embed our initial input with new contextual information.

The attention score tells us how much words relate together. It is calculated by comparing each Q vector with all K vectors. It quantifies, for each word in a sentence, how relevant every other word is. A higher score means high relevance, and a lower score means low relevance.

The full attention equation, as presented in the original paper, is:

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]

The process of combining Q, K, and V into a meaningful output involves a few steps:

  • Calculate Attention Scores: For each Query, calculate a score with every Key. This is often done using the dot product, as it’s a simple way to measure how aligned or similar two vectors are.

  • Scaling: Scale the attention scores by \(\sqrt{d_k}\) to ensure stable training.

  • Normalization: Use the softmax function to normalize these scores across each row (for each Query). This converts the scores into a probability distribution, ensuring they all add up to 1.

  • Weighted Sum: Multiply each normalized score with its corresponding Value vector. This step weights the information (Value vectors) by how relevant it is. Then, sum these weighted Values up to get a single vector for each Query.

There’s a lot packed into each of those steps, and a lot going into that simple attention calculation line. Let’s break it down step by step.

We’ll do this by continuing with our simple 3x4 example.

Given the Query (Q), Key (K), and Value (V) matrices for our simple example sentence above, we’ll walk through the steps to calculate the attention scores and how they’re used to create the output.

Step 1: Calculating Raw Attention Scores. The raw attention scores are calculated by performing the dot product of the Q matrix with the transpose of the K matrix:

\[ Attention\ Scores\ (S) = QK^T = \begin{bmatrix} q_{11} & q_{12} & q_{13} & q_{14} \\ q_{21} & q_{22} & q_{23} & q_{24} \\ q_{31} & q_{32} & q_{33} & q_{34} \end{bmatrix} \cdot \begin{bmatrix} k_{11} & k_{21} & k_{31} \\ k_{12} & k_{22} & k_{32} \\ k_{13} & k_{23} & k_{33} \\ k_{14} & k_{24} & k_{34} \end{bmatrix} = \begin{bmatrix} score_{11} & score_{12} & score_{13} \\ score_{21} & score_{22} & score_{23} \\ score_{31} & score_{32} & score_{33} \end{bmatrix} \]

Each element in this matrix represents the raw attention score between two words in the sentence, with each score\(_{ij}\) indicating the attention score between the \(i\)-th word and the \(j\)-th word. Each score\(_{ij}\) results from the dot product of the \(i\)-th Query vector and the \(j\)-th Key vector, and signifies how much attention the \(i\)-th word should pay to the \(j\)-th word, based on their relevance. A higher score\(_{ij}\) suggests that the \(j\)-th word is more relevant or important for understanding the context or meaning of the \(i\)-th word.

  • Diagonal Elements (score\(_{ii}\)) represent how each word is self-relevant. The importance of self-interaction varies and is learned by the model based on the training data and task.

  • Off-Diagonal Elements (score\(_{ij}\)) where \(i \neq j\) reflect the inter-word relevance as determined by the model’s training. Words that occur in similar contexts, or inform each other, or otherwise relate in some way the model needs to understand, would have higher attention scores relative to each other.

Step 2: Scaling by \(\frac{1}{\sqrt{d_k}}\).

Before we apply softmax, we must divide every score by \(\sqrt{d_k}\), where \(d_k\) is the dimension of the Key (and Query) vectors:

\[S_{\text{scaled}} = \frac{QK^T}{\sqrt{d_k}}\]

In our 3×4 example, \(d_k = 4\), so we’d divide every element of the score matrix by \(\sqrt{4} = 2\). This might seem like a minor detail, but it turns out to be essential for training stability.

This is because each element of \(QK^T\) is a dot product between a Query vector and a Key vector. A dot product is a sum of element-wise products:

\[\text{score}_{ij} = \sum_{m=1}^{d_k} q_{im} \cdot k_{jm}\]

That’s a sum of \(d_k\) terms. If the individual entries of Q and K are roughly mean-zero with variance 1 (which they tend to be, especially at initialization), then each product \(q \cdot k\) has mean 0 and variance 1, and the sum of \(d_k\) such terms has variance \(d_k\).

This means that, if left unscaled, as \(d_k\) grows, the raw attention scores get larger in magnitude. But attention is meant to tell us how much words are related, so an artificial increase due to summing more terms is no good. The standard deviation of the scores grows as \(\sqrt{d_k}\). For \(d_k = 4\), scores might typically land in the $$2 stdev range of [-4, 4]. But for \(d_k = 512\), the $$2 stdev range scores could easily reach magnitudes of 45 or more.

The Effect on Softmax

Softmax is extremely sensitive to the magnitude of its inputs, and it’s going to be the next step in the attention calculation. Recall that the softmax function is formed: ($(z_i) = $). When values are low, say:

\[\text{scores} = [1.0, \quad 2.0, \quad 3.0]\]

Applying softmax:

\[\text{softmax}([1, 2, 3]) = [0.09, \quad 0.24, \quad 0.67]\]

Probabilities remain reasonable, so this is fine. Gradients can flow through all three paths during backpropagation. But if instead we didn’t scale, so scores were able to be 10x higher:

\[\text{softmax}([10, 20, 30]) = [0.0000, \quad 0.0000, \quad 1.0000]\]

Probabilities become extreme, because the \(e^z\) operation in the softmax is so extreme. This is by design, but means that the softmax is easy to saturate - to put all probability on the highest score and zero on everything else. This is a problem for two reasons:

  1. Information loss: The output becomes just a copy of a single Value vector, losing all the contextual blending that makes attention useful.
  2. Vanishing gradients: In the saturated regime, the gradients of softmax with respect to its inputs become nearly zero for all entries. This means the model can barely learn - the gradient signal needed to update \(W_Q\) and \(W_K\) effectively disappears.

Dividing by \(\sqrt{d_k}\) is a simple but powerful fix that rescales the scores so that their variance is approximately 1, regardless of the dimension:

\[\text{Var}\left(\frac{\text{score}}{\sqrt{d_k}}\right) = \frac{\text{Var}(\text{score})}{d_k} = \frac{d_k}{d_k} = 1\]

This keeps the softmax inputs in the “well-behaved” range where it produces meaningful probability distributions and healthy gradients, no matter how large the model’s dimensions are. This is why the mechanism is formally called Scaled Dot-Product Attention, because the scaling is critical to make this work.

Step 3: Softmax Normalization. Normalization then transforms the scaled attention scores into a distribution that indicates the relative importance of each word in the sequence. This is done typically with the softmax function, which converts a vector of numerical values into a probability distribution. The values produced by softmax are in the range (0, 1) and add up to 1, making them directly interpretable as probabilities.

The softmax function is applied row-wise, so after it is applied, each row in the [seq_length x seq_length] matrix is transformed into a probability distribution.

Given the scaled attention scores matrix:

\[ S_{\text{scaled}} = \frac{1}{\sqrt{d_k}} \begin{bmatrix} score_{11} & score_{12} & score_{13} \\ score_{21} & score_{22} & score_{23} \\ score_{31} & score_{32} & score_{33} \end{bmatrix} = \begin{bmatrix} s_{11} & s_{12} & s_{13} \\ s_{21} & s_{22} & s_{23} \\ s_{31} & s_{32} & s_{33} \\ \end{bmatrix} \]

The softmax function ($ (z_i) = $) is applied to each row individually:

\[ \text{Attention Weights}\ (A) = \begin{bmatrix} \frac{e^{{s}_{11}}}{e^{{s}_{11}} + e^{{s}_{12}} + e^{{s}_{13}}} & \frac{e^{{s}_{12}}}{e^{{s}_{11}} + e^{{s}_{12}} + e^{{s}_{13}}} & \frac{e^{{s}_{13}}}{e^{{s}_{11}} + e^{{s}_{12}} + e^{{s}_{13}}} \\ \frac{e^{{s}_{21}}}{e^{{s}_{21}} + e^{{s}_{22}} + e^{{s}_{23}}} & \frac{e^{{s}_{22}}}{e^{{s}_{21}} + e^{{s}_{22}} + e^{{s}_{23}}} & \frac{e^{{s}_{23}}}{e^{{s}_{21}} + e^{{s}_{22}} + e^{{s}_{23}}} \\ \frac{e^{{s}_{31}}}{e^{{s}_{31}} + e^{{s}_{32}} + e^{{s}_{33}}} & \frac{e^{{s}_{32}}}{e^{{s}_{31}} + e^{{s}_{32}} + e^{{s}_{33}}} & \frac{e^{{s}_{33}}}{e^{{s}_{31}} + e^{{s}_{32}} + e^{{s}_{33}}} \end{bmatrix} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ \end{bmatrix} \]

Each element in this matrix represents the normalized weight of each word’s contribution to every other word in the sequence.

Step 4: Calculating the Weighted Sum. The final output is calculated by multiplying the normalized attention weights by the V matrix:

\[ \text{Output}\ (O) = A \cdot V \]

\[ O = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ \end{bmatrix} \cdot \begin{bmatrix} v_{11} & v_{12} & v_{13} & v_{14} \\ v_{21} & v_{22} & v_{23} & v_{24} \\ v_{31} & v_{32} & v_{33} & v_{34} \end{bmatrix} = \begin{bmatrix} o_{11} & o_{12} & o_{13} & o_{14} \\ o_{21} & o_{22} & o_{23} & o_{24} \\ o_{31} & o_{32} & o_{33} & o_{34} \end{bmatrix} \]

This new matrix, O, of the same size as the input to the attention layer, encapsulates the aggregated, contextually enriched information for each element in the input sequence. Each row now represents a weighted combination of all input information, with weights reflecting the relevance or attention each word should pay to every other word in the sentence.

  • O serves as the input to subsequent layers within the model, carrying forward both the intrinsic information of each input element and the contextual relationships learned through the attention mechanism.
  • This dynamic incorporation of context from the entire sequence is a key factor behind the effectiveness of models employing self-attention, enabling them to achieve remarkable performance across diverse tasks in natural language processing, computer vision, and other domains.

When all is said and done, it comes together like this:

single-self-attention

2.1.3 Attention Mechanism in Code

Now, let’s write a simple attention layer and see how we can apply these concepts in code.

In this code, we’ll follow the process we just outlined above. We’ll

  1. Define Q, K, and V matrices by applying the torch.nn.Linear() layer to perform our linear transformations. This layer will keep track of the learnable weight matrices \(W_Q, W_K,\) and \(W_V\) for us.
  2. Calculate attention scores from Q and K
  3. Apply the softmax to get attention weights
  4. Use V to get the final output.
class SimpleSelfAttention(torch.nn.Module):
    def __init__(self, embed_size):
        super(SimpleSelfAttention, self).__init__()
        self.embed_size = embed_size

        # Linear transformations for Q, K, V
        self.query = torch.nn.Linear(embed_size, embed_size, bias=False) # Q = X*W_Q
        self.key = torch.nn.Linear(embed_size, embed_size, bias=False)
        self.value = torch.nn.Linear(embed_size, embed_size, bias=False)

    def forward(self, x):
        # Generate Q, K, V matrices
        Q = self.query(x)
        K = self.key(x)
        V = self.value(x)

        # Step 1: Calculate raw attention scores, S = Q*K^T
        raw_attention_scores = torch.matmul(Q, K.transpose(0, 1))

        # Step 2: Scale by sqrt(d_k)
        attention_scores = raw_attention_scores / (self.embed_size ** 0.5)

        # Step 3: Apply softmax to get attention weights
        attention_weights = F.softmax(attention_scores, dim=-1)

        # Step 4: Generate the output matrix O by applying attention weights to V
        O = torch.matmul(attention_weights, V)

        return O

Now, for a very simple 3x4 example embedding, let’s ensure that this layer transforms the input as expected by confirming that the dimensionality of the output matches the input.

embed_size = 4  # Embedding size (d_model)
seq_length = 3  # Sequence length

# Simulate input embeddings: 3 words, each represented by a 4-dimensional vector
x = torch.rand((seq_length, embed_size))  # Random example input

# Initialize the self-attention model and apply it to the input
attention_model = SimpleSelfAttention(embed_size=embed_size)
O = attention_model(x)

print("Input Embeddings (X):", x.numpy())
print("Output Matrix (O):", O.detach().numpy())
Input Embeddings (X): [[0.593172   0.11234725 0.15345693 0.24170822]
 [0.7262365  0.7010802  0.20382375 0.65105355]
 [0.774486   0.43689132 0.5190908  0.61585236]]
Output Matrix (O): [[ 0.34115726 -0.6421402   0.10102071 -0.13509695]
 [ 0.3400095  -0.6441524   0.10018728 -0.13508819]
 [ 0.3405209  -0.64448154  0.10057425 -0.13494176]]

2.1.4 Visualizing the Attention Weights

It can be helpful to visualize the attention weight matrix to understand how relations are being made between words. This matrix contains weights that indicate how much focus or “attention” each element in the input sequence should give to every other element.

Visualizations of A typically take the form of heatmaps, where the intensity of the colors indicates the strength of the attention weights. This allows us to:

  • Trace how the model distributes its attention across the input.
  • Identify potential biases or shortcomings in the model’s focus.
  • Understand which parts of an input sequence are important to a specific word or token, revealing the model’s internal “reasoning” and how it processes linguistic structures or dependencies.

We’ll come back to this visualization once we have trained a model below (in Section 10), and have an A matrix that has some real meaning!

2.2 Multi-Head Attention

Imagine trying to understand a complex novel. You might focus on understanding the plot. But you may separately want to better understand a character’s development arc. Or maybe the book’s thematic elements. Real, complex text has many different, simultaneously occurring aspects. In machine learning models, especially in tasks as nuanced as language processing, there’s a wealth of information to be extracted from the data. Multi-head attention allows the model to look at its input from different perspectives (or “heads”), each focusing on different subsets of information, thereby enriching the model’s understanding and capabilities.

The essence of multi-head attention is to run several self-attention mechanisms (heads) in parallel, each with its own set of learnable weights, thereby allowing the model to simultaneously attend to information from different representation subspaces at different positions.

2.2.1 How Multi-Head Attention Works

Multi-head attention simply applies the attention mechanism that we just talked about multiple times, using multiple different “heads”, then takes an extra step to combine them. Each head \(i\) follows the steps of self attention that we just outlined. Namely, for each head, we:

  1. Apply separate linear transformations to the Query, Key, and Value matrices.
  2. Perform the scaled dot-product attention.

Then, we:

  1. Concatenate the attention outputs from each head.
  2. Apply a final linear transformation to the concatenated output.

We can write this as:

\[ \text{MultiHead}(Q's, K's, V's) = \text{Concat}(O_1, ..., O_h)W^O \]

Where each O here is as it was calculated above:

\[ \begin{align} O_i &= \text{Attention}(XW^Q_i, XW^K_i, XW^V_i) \\ &= \text{Attention}(Q_i, K_i, V_i) \\ &= A_i \cdot V_i \\ &= \text{softmax} (\frac{S_i}{\sqrt{d_k}}) \cdot V_i \\ &= \text{softmax}(\frac{Q_i K_i^T}{\sqrt{d_k}}) \cdot V_i \end{align} \]

as we saw above. Here,

  • \(W^Q_i, W^K_i, W^V_i\) are the learned weight matrices for the \(Q_i, K_i, V_i\) of the \(i\)-th head.
  • \(W^O\) is the learned weight matrix applied after concatenation.

So, multi-head adds this new matrix, \(W^O\), into the mix, whose job is to learn how to relate all of our different attention heads. This final transformation with \(W^O\) also ensures that the output from the multi-head attention mechanism has the same dimensionality as the input embeddings, so it can be passed on to subsequent layers in the transformer model.

This mechanism allows the model to look at the input from different perspectives simultaneously, improving the model’s ability to capture complex relationships in the data.

2.2.2 The Dimensionality of Heads

Ok so you might be thinking “multiple heads, and the attention calculation already relies on multiple matrices (Q,K,V) - surely this is getting to be an insurmountable amount of compute!”

And it would be, if we just ran \(\text{h}\) full-size attention operations (each on \(d_\text{model}\)-dimensional vectors). Instead though, each head operates on a slice of the embedding: \(d_k = d_\text{model} / \text{h}\) dimensions. So if \(d_\text{model} = 512\) and we use \(\text{h} = 8\) heads, each head works with 64-dimensional Q, K, and V vectors. The total work across all heads is roughly equivalent to a single attention operation on the full 512-dimensional space. The key is that we partition the representation instead of duplicating it.

This is why implementations enforce that \(d_\text{model}\) is evenly divisible by \(\text{h}\). It must be so that the embedding dimension can split cleanly into \(\text{h}\) equal pieces.

📌 Note: It’s important to keep the dimensions of everything straight here, and you might need a refresher for some things. Check Appendix 0 for a quick reference guide!

How it actually works in practice

The formulas above suggest that each head has its own separate weight matrices \(W^Q_i, W^K_i, W^V_i\), each of size \(d_\text{model} \times d_k\). And conceptually, that’s exactly right. But in practice, implementations do something mathematically equivalent that’s more efficient:

Step 1: Project with one big matrix, just the same as single-head. We apply a single \(W^Q\) of size \(d_\text{model} \times d_\text{model}\) to get a full-size Q matrix, and the same for K and V. This is identical to what we did in the Self Attention implementation above:

\[Q = XW^Q \quad \text{shape: } (n_\text{seq}, d_\text{model})\] \[K = XW^K \quad \text{shape: } (n_\text{seq}, d_\text{model})\] \[V = XW^V \quad \text{shape: } (n_\text{seq}, d_\text{model})\]

Step 2: Reshape to split into heads. We then reshape each of these matrices to separate the \(d_\text{model}\) dimension into \(\text{h}\) heads of \(d_k = d_{\text{model}}/\text{h}\) dimensions each:

\[Q: \quad (n_\text{seq},\ d_\text{model}) \longrightarrow (n_\text{seq},\ \text{h},\ d_k)\]

No new computation needs to happen here, it’s just a reinterpretation of the same numbers. The first \({d_k}\) values of each row become head 0’s query, the next \({d_k}\) become head 1’s query, and so on.

This works because the single big \(W^Q\) of size \(d_{model} \times d_{model}\) is mathematically equivalent to \(\text{h}\) separate \(W^Q_i\) matrices of size \(d_{model} \times d_{k}\) stacked side by side. The reshape just makes this partitioning explicit.

Let’s make this concrete. Say we have \(d_\text{model} = 6\) and \(\text{h} = 3\) heads, so \(d_k = 2\). After projecting, one word’s Query vector might look like:

\[q = [0.5, \quad 0.8, \quad | \quad 0.1, \quad 0.3, \quad | \quad 0.9, \quad 0.2]\]

When we reshape from \((d_\text{model})\) to \((h, d_k)\), it simply slices this into contiguous chunks:

\[\begin{bmatrix} 0.5 & 0.8 & 0.1 & 0.3 & 0.9 & 0.2 \end{bmatrix} \longrightarrow \begin{bmatrix} 0.5 & 0.8 \\ 0.1 & 0.3 \\ 0.9 & 0.2 \end{bmatrix}\]

Head 0 looks at \([0.5, 0.8]\). Head 1 looks at \([0.1, 0.3]\). Head 2 looks at \([0.9, 0.2]\).

Does the order matter? At initialization, no. The weights are random, so which dimensions end up in which head is arbitrary. But during training, yes, the model learns to organize information so that dimensions within the same head are useful together. Head 0’s slice of \(W^Q\) might learn to produce queries about syntactic relationships, while head 2’s slice learns to produce queries about semantic similarity. The model figures this out on its own.

Step 3: Attention per head. Each head now independently computes scaled dot-product attention using its own \(d_k\)-dimensional slice of Q, K, and V - exactly the same self-attention we already learned, just on smaller vectors.

Step 4: Concatenate and project. We reshape back from \((n_\text{seq},\ h,\ d_k)\) to \((n_\text{seq},\ d_\text{model})\), which is just concatenating each head’s output side by side. Then we apply \(W^O\) (size \(d_\text{model} \times d_\text{model}\)) to blend the heads’ outputs together.

To summarize the full dimension flow (ignoring batch size for clarity):

\[X \xrightarrow{W^Q, W^K, W^V} Q, K, V \xrightarrow{\text{reshape}} \underbrace{Q_1, K_1, V_1}_{\text{head 1}}, \ldots, \underbrace{Q_h, K_h, V_h}_{\text{head } h} \xrightarrow{\text{attention}} O_1, \ldots, O_h \xrightarrow{\text{concat}} O_\text{cat} \xrightarrow{W^O} \text{Output}\]

\[(n_\text{seq}, d_\text{model}) \rightarrow (n_\text{seq}, d_\text{model}) \rightarrow h \times (n_\text{seq}, d_k) \rightarrow h \times (n_\text{seq}, d_k) \rightarrow (n_\text{seq}, d_\text{model}) \rightarrow (n_\text{seq}, d_\text{model})\]

The input and output are the same shape. The “multi-head” part lives entirely in the middle - the embedding is split, attended to from \(\text{h}\) different perspectives, and recombined.

multihead-attn

2.2.3 Why Does Multi-Head Attention Help?

At this point you might be wondering - if we don’t add any new parameters (aside from \(W^O\)), and the total amount of computation is roughly the same as single-head attention, what’s the point? We’re doing the same work, just split into smaller pieces. How does multi-head learn anything that a single large attention matrix can’t?

The answer lies in a fundamental limitation of single-head attention: the softmax bottleneck.

Single-Head Attention: One Softmax for Everything

Say we have a 3-word sequence with \(d_\text{model} = 4\). In single-head attention, we compute one attention weight matrix \(A\) of shape \((3 \times 3)\) - one weight per word pair:

\[A = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) = \begin{bmatrix} 0.1 & 0.7 & 0.2 \\ 0.5 & 0.3 & 0.2 \\ 0.3 & 0.3 & 0.4 \end{bmatrix}\]

Each row is a probability distribution. Now, the output for word 1 is:

\[o_1 = 0.1 \cdot v_1 + 0.7 \cdot v_2 + 0.2 \cdot v_3\]

Here’s the thing to notice: \(v_1\), \(v_2\), and \(v_3\) are 4-dimensional vectors. Every single dimension of the output gets the same weights - 10% of word 1, 70% of word 2, 20% of word 3. Writing it out explicitly for each output dimension:

\[o_1[0] = 0.1 \cdot v_1[0] + 0.7 \cdot v_2[0] + 0.2 \cdot v_3[0]\] \[o_1[1] = 0.1 \cdot v_1[1] + 0.7 \cdot v_2[1] + 0.2 \cdot v_3[1]\] \[o_1[2] = 0.1 \cdot v_1[2] + 0.7 \cdot v_2[2] + 0.2 \cdot v_3[2]\] \[o_1[3] = 0.1 \cdot v_1[3] + 0.7 \cdot v_2[3] + 0.2 \cdot v_3[3]\]

The same 0.1, 0.7, 0.2 everywhere. All four output dimensions are forced to use the same blend of input words.

Multi-Head Attention: Multiple Softmaxes

Now take the same \(d_\text{model} = 4\) but split into \(\text{h} = 2\) heads, so \(d_k = 2\). Each head computes its own attention weight matrix from its own \(d_k\)-dimensional slice of Q and K. Critically, each head runs its own softmax:

\[A^{\text{head}_0} = \text{softmax}\left(\frac{Q_0 K_0^T}{\sqrt{d_k}}\right) = \begin{bmatrix} 0.1 & 0.7 & 0.2 \\ \ldots \end{bmatrix}\]

\[A^{\text{head}_1} = \text{softmax}\left(\frac{Q_1 K_1^T}{\sqrt{d_k}}\right) = \begin{bmatrix} 0.6 & 0.1 & 0.3 \\ \ldots \end{bmatrix}\]

These will generally be different distributions, because they’re computed from different slices of the projected Q and K. Now the output for word 1 is:

From head 0 (dimensions 0-1): \[o_1[0] = 0.1 \cdot v_1[0] + 0.7 \cdot v_2[0] + 0.2 \cdot v_3[0]\] \[o_1[1] = 0.1 \cdot v_1[1] + 0.7 \cdot v_2[1] + 0.2 \cdot v_3[1]\]

From head 1 (dimensions 2-3): \[o_1[2] = 0.6 \cdot v_1[2] + 0.1 \cdot v_2[2] + 0.3 \cdot v_3[2]\] \[o_1[3] = 0.6 \cdot v_1[3] + 0.1 \cdot v_2[3] + 0.3 \cdot v_3[3]\]

Dimensions 0-1 are mostly attending to word 2 (weight 0.7). Dimensions 2-3 are mostly attending to word 1 (weight 0.6). Different parts of the output vector are blending information from different words.

What’s Gained by Dividing Attention

In single-head attention, softmax acts as a bottleneck. No matter how rich the Q, K, and V projections are, everything gets squeezed through one probability distribution, and some of that richness is lost in that action.

You might think that having more dimensions contribute to that single softmax (the full \(d_\text{model}\) is much bigger than our \(d_k\)’s) would be an advantage. And it does give each score more information to work with. But one distribution can still only express one attention pattern. If word 1 needs to strongly attend to word 2 for some semantic purpose and simultaneously strongly attend to word 3 for some syntactic purpose, a single distribution can’t express that, and that’s a much bigger problem than having a full \(d_\text{model}\) of dimensionality is a potential advantage.

In multi-head attention, where each head’s softmax can specialize in a different type of relationship and each head controls its own slice of the output, different aspects of the representation can be built from different attention patterns. What you lose in “focus” on one aspect by having only \(d_k\) dimensions instead of \(d_\text{model}\) is more than made up for by getting to put that focus in more places.

What This Looks Like in Practice

In trained transformers, different heads demonstrably learn different roles. Some attend to the immediately previous word. Some attend to syntactically related words (like a verb attending to its subject). Some attend to semantically similar words. Some learn specific positional offsets. This specialization emerges naturally from training - the model discovers it’s useful to partition its attention into different purposes.

A single head couldn’t express all of these patterns simultaneously. Multi-head allows the same number of parameters to act much more usefully.

2.2.4 Multi-Head Attention Code

Below is an implementation of the multi-head attention mechanism in PyTorch. It reuses some of what we already wrote in the SimpleSelfAttention class that we built above to apply multi-head attention instead, so many lines here will be repeated from there.

class MultiHeadAttention(nn.Module):
    def __init__(self, embed_size, heads):
        super(MultiHeadAttention, self).__init__()
        self.embed_size = embed_size
        self.heads = heads
        self.head_dim = embed_size // heads

        assert (self.head_dim * heads == embed_size), "Embedding size must be divisible by heads"

        # Linear layers for to get query, key, value, and out
        self.values = nn.Linear(embed_size, embed_size, bias=False)
        self.keys = nn.Linear(embed_size, embed_size, bias=False)
        self.queries = nn.Linear(embed_size, embed_size, bias=False)
        self.fc_out = nn.Linear(embed_size, embed_size)

    def forward(self, values, keys, query):
        N = query.shape[0]
        value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

        # learned projections
        values  = self.values(values)
        keys    = self.keys(keys)
        queries = self.queries(query)

        # split into heads
        values  = values.reshape(N, value_len, self.heads, self.head_dim)
        keys    = keys.reshape(N, key_len, self.heads, self.head_dim)
        queries = queries.reshape(N, query_len, self.heads, self.head_dim)

        # move the head axis up so each head holds a (seq_len, head_dim) matrix
        values  = values.transpose(1, 2)    # (N, heads, value_len, head_dim)
        keys    = keys.transpose(1, 2)      # (N, heads, key_len,   head_dim)
        queries = queries.transpose(1, 2)   # (N, heads, query_len, head_dim)

        # scores: Q @ K^T, per head -> (N, heads, query_len, key_len)
        attention_scores = queries @ keys.transpose(-2, -1)
        attention = torch.softmax(attention_scores / (self.head_dim ** 0.5), dim=-1)

        # weighted sum of values -> (N, heads, query_len, head_dim)
        out = attention @ values

        # merge heads back -> (N, query_len, embed_size), then mix with W^O
        out = out.transpose(1, 2).reshape(N, query_len, self.heads * self.head_dim)
        return self.fc_out(out)

Now, though a lot of this is similar to code we’ve already seen, multi-head attention is really THE key mechanism of the transformer, and as we’ll see through sections 3 and 4, gets re-used repeatedly when building the full architecture. So we’ll take a beat here to make sure we understand precisely what each line does.

The setup (__init__).

We start by setting up the dimensions of our MHA implementation. We need to define the embed_size (the dimension of the embedding actually going into the model - which is n_embeds, but as we discussed also d_model as this number is held constant) and the number of heads we want.

Then, we calculate the dimension of the heads from this number, and of course, must check that they are actually divisible.

def __init__(self, embed_size, heads):
    super(MultiHeadAttention, self).__init__()
    self.embed_size = embed_size
    self.heads = heads
    self.head_dim = embed_size // heads

    assert (self.head_dim * heads == embed_size), "Embedding size must be divisible by heads"

Then, we define the learned projections - W_Q, W_K, W_V, and the output projection W^O. Two things are worth noticing:

  1. We do a single matrix multiplication, embed_size → embed_size, not embed_size → head_dim We are doing multi-head attention, so you might expect one projection per head. But as it turns out, since it’s mathematically identical to do all heads’ projections at once, in a single embed_size × embed_size matrix, then split the result afterward, and much more computationally efficient to do so, that’s what we’ll do. We’ll see this split happen later.
  2. bias=False on all of Q/K/V These are meant to be pure linear projections. We use the nn.Linear built in to do these, which accepts a bias. But we just need the “regular” matrix multiplies here. A bias would add a constant offset that, in practice doesn’t buy anything, so it’s conventionally dropped. (fc_out, by contrast, keeps its bias, the default, which we’ll get to at the end.)
    self.values  = nn.Linear(embed_size, embed_size, bias=False)
    self.keys    = nn.Linear(embed_size, embed_size, bias=False)
    self.queries = nn.Linear(embed_size, embed_size, bias=False)
    self.fc_out  = nn.Linear(embed_size, embed_size)

The forward signature.

Next, we need to define what a forward pass will do. The forward function takes values, keys, and query as three separate inputs, each shaped (N, seq_len, embed_size). N, the batch size, is something relatively new that we’re introducing here, which we’ll talk more about later. But we make sure we know what it is, and need it because we want to process multiple inputs at once.

Another thing to note here - this code tracks the sequence lengths: value_len, key_len, and query_len separately. While in our self-attention implementation, n_seq was always the same, because Q, K, and V were always being constructed for the same sequence, nothing requires that this be true. In fact, later in this tutorial, we’ll see an implementation of attention where that is explicitly not true - so this definition of different lengths just keeps that open. But more on that later!

def forward(self, values, keys, query):
    N = query.shape[0]  # batch size
    value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

Applying the projections.

The next step is then to apply the projections that we defined above. The first step of this, is then to simply use the Q,K,V projections on the queries, keys, and values we’ve passed in.

    values  = self.values(values)    # (N, value_len, embed_size)
    keys    = self.keys(keys)        # (N, key_len, embed_size)
    queries = self.queries(query)    # (N, query_len, embed_size)

Next, we do the “multi-head” split, which as we discussed, happens with pure reshaping. We take the embed_size vector, as if it were heads contiguous chunks of size head_dim. Then, head 0 gets the first head_dim numbers, head 1 the next, and so on. After these lines, each of values, keys, and queries have a new heads axis, so that for the remaining computations, the head slices travel through the computations on their own.

    values  = values.reshape(N, value_len, self.heads, self.head_dim)
    keys    = keys.reshape(N, key_len, self.heads, self.head_dim)
    queries = queries.reshape(N, query_len, self.heads, self.head_dim)

Computing the attention scores.

Next, we need to compute attention scores. This is the first time that we’re doing more “real” math on our matrices, so let’s pause on two key syntax pieces first.

  1. @ is matrix multiplication. For two 2D matrices it’s the ordinary matrix product: an (m × k) times a (k × n) gives an (m × n), and the inner dimensions, k, must match. For tensors with more than two axes, @ treats everything except the last two axes as a batch: it lines those leading axes up and does an independent matrix multiply on each (last two × last two) slice. So,
(N, h, q, d) @ (N, h, d, k)  ->  (N, h, q, k)

For every one of the N × h combinations, multiply the (q × d) matrix from the first input by the (d × k) matrix of the second input. The (N, h) just come along unchanged, and only the last two axes actually get multiplied. (The leading (N, h) axes must match on both sides - or be broadcastable - but here they’re always identical. It’s the inner matrix dimensions, the shared d, that have to line up for the multiply itself.)

  1. transpose(a, b) swaps two axes. The two arguments are which axes to swap, by their index, with every other axis staying as-is. So for a tensor of shape (N, s, h, d):
x.transpose(1, 2)   # swaps axes 1 and 2  ->  (N, h, s, d)

As with all indexing, negative indices count from the end, so -1 is the last axis, and -2 the second-to-last.

So, at this point in the calculation, each of queries, keys, values has shape (N, seq_len, heads, head_dim). If we want to matrix multiply along the correct axes (the last two), we first move the heads axis to the front via a transpose. This means swapping the two middle dimensions (we ultimately want (N, heads, seq_len, head_dim)), so we start with a transpose.

    values  = values.transpose(1, 2)    # (N, heads, value_len, head_dim)
    keys    = keys.transpose(1, 2)      # (N, heads, key_len,   head_dim)
    queries = queries.transpose(1, 2)   # (N, heads, query_len, head_dim)

Now, we can easily calculate the scores with a single batched matrix multiply:

    attention_scores = queries @ keys.transpose(-2, -1)
    # (N, heads, query_len, head_dim) @ (N, heads, head_dim, key_len)
    # -> (N, heads, query_len, key_len)

The transpose that happens here is a “regular” transpose - the math asks that we swap the final two dimensions. Then, we use the “@” operator to matrix multiply, over just the final two dimensions of those larger, batched matrices.

So, for a fixed batch item and head, queries is a (query_len × head_dim) matrix, with one row per query position. keys is transposed to (head_dim × key_len), so that multiplying gives a (query_len × key_len) matrix whose entry (i, j) is the dot product of query row i with key row j. Because the leading (N, heads) axes are just kept, treated as a batch, this matrix multiply happens independently for every item in the batch and every head, but gets to occur all at once.

Scaling and softmax.

Softmax is a very common function in machine learning, and thus also has an easy built-in with torch. We first divide by √head_dim, as that’s the dimension the dot product is really acting over and thus needs to be scaled by. Then, we apply the softmax, making sure to pass dim=-1, as we want to softmax over the last axis, the key_len axis. That turns each query’s row of scores across the keys into weights that sum to 1.

    attention = torch.softmax(attention_scores / (self.head_dim ** 0.5), dim=-1)

Applying the weights to the values.

The next step is to apply the attention weights to the values.

    out = attention @ values
    # (N, heads, query_len, key_len) @ (N, heads, value_len, head_dim)
    # -> (N, heads, query_len, head_dim)

Per head, this multiplies the (query_len × key_len) weight matrix by the (value_len × head_dim) value matrix.

Note that here, key_len and value_len are the same number, so the inner dimensions line up. This actually is a generally true property of attention - because this multiply has to happen, K and V, at least, must be asking about the same sequence. (We’ll see in Section 4, that Q doesn’t - and talk more about the implications there!)

Recombining the heads.

Now, the heads have been treated separately during most of the attention math. We recombine them here by putting the “original” dimension back together.

This starts with a transpose to “fix” the dimensions back to normal, returning them to (N, query_len, heads, head_dim). Then, we reshape, returning the heads x head_dim dimensions back into a single dimension of size embed_size.

    out = out.transpose(1, 2).reshape(N, query_len, self.heads * self.head_dim)  
    # (N, heads, query_len, head_dim) -> (N, query_len, heads, head_dim)
    # -> (N, query_len, embed_size)

Then, finally, fc_out is that multiply by the final matrix, W^O. This of course is the final of the projections that we defined in __init__.

    out = self.fc_out(out)

So, what does this all mean for how we actually call and use this MultiHeadAttention class? A few things.

  1. Because the function takes values, keys, and query as three independent inputs, self-attention is just the case where we pass the same sequence as all three, so that all of Q, K, and V operate on the same thing.

  2. Writing it in this way, using the pytorch built-ins means that pytorch handles a lot of the math behind the scenes for us.

Number (1) we’ll touch more on in the decoder section. But (2) is worth discussing in a little more detail here, because it can make the math and code a little more difficult to line up with one another.

If you’re not used to reading class-based code like this, the key thing to note is that each nn.Linear isn’t a function - it’s an object that holds a weight matrix inside it. That’s why when we defined the projections in __init__ like this:

self.values = nn.Linear(embed_size, embed_size, bias=False)

We could apply them later. Those definitions told torch to create an object, filled with a randomly-initialized embed_size × embed_size matrix, and stash it under the name self.values. That matrix is W_V, and we can look at it with self.values.weight.

So in the forward pass, when we do self.values(values), we’re asking to apply that matrix stored in self.values to the values matrix we’re feeding in. These objects also keep track of which weights belong to which layer - since each layer is a separate object, each object owns its own weights. Each distinct nn.Linear object holds an independent matrix, and each MultiHeadAttention object we’ll define when we build the encoder holds its own independent instances of those calls. So these objects are keeping everything separate for us “for free”.

It’s also of course what will make training very easy for us. Because we built MultiHeadAttention as an nn.Module and assigned those layers as attributes, PyTorch automatically registers every inner matrix as a learnable parameter. So when it comes time to actually do backpropagation and tweak those weights, PyTorch also handles figuring out how those weights all need to move.

Of course, if you want to make every single bit of the math totally transparent, you can write every single thing yourself. In practice, even a “highly custom” architecture won’t do that, and will use PyTorch instead.

Finally, we can see how we’d actually use this MultiHeadAttention we just built, and be sure that it gives us the output we expect in the self-attention case below:

embed_size = 4   # Embedding size (d_model)
seq_length = 3   # Sequence length
heads = 2        # (with embed_size=4, heads=2 gives head_dim=2)

# Simulate input embeddings - NOTE the batch dimension: (batch=1, seq_length, embed_size)
x = torch.rand((1, seq_length, embed_size))

attention_model = MultiHeadAttention(embed_size=embed_size, heads=heads)

# self-attention: Q, K, and V all come from the same input, so we pass x three times
O = attention_model(x, x, x)

print("Input shape: ", tuple(x.shape))
print("Output shape:", tuple(O.shape), " <- same shape in and out")
print("\nInput Embeddings (X):", x.numpy())
print("Output Matrix (O):", O.detach().numpy())
Input shape:  (1, 3, 4)
Output shape: (1, 3, 4)  <- same shape in and out

Input Embeddings (X): [[[0.37555784 0.5225613  0.57295054 0.61858714]
  [0.69621414 0.5299501  0.25603563 0.7365945 ]
  [0.02037555 0.20364666 0.37483507 0.25644332]]]
Output Matrix (O): [[[-0.44758716  0.14991242 -0.48490107  0.05346444]
  [-0.44845763  0.15030378 -0.48452932  0.05426714]
  [-0.4486941   0.14936079 -0.48329026  0.05332798]]]

2.3 Residual Connections

A residual connection (also called a skip connection) is an incredibly simple but important idea that ensures that information gets passed through a model without too much loss of what it originally contained. Instead of just passing input through a layer and using the output:

\[\text{output} = f(x)\]

You add the original input back to the output:

\[\text{output} = x + f(x)\]

In the case of the attention sublayer, this means concretely:

\[X_1 = X + O\]

where \(X\) is the input to the attention layer (our embedded + positionally encoded tokens) and \(O\) is the attention output we just computed. \(X_1\), the sum of both, is what gets passed forward.

That’s it. The layer’s output gets added to its input, element-wise, so that even though you do a bunch of useful transformations to the input via the attention section, you also keep what you already knew. In a transformer, this is applied around each sublayer - both the attention mechanism and the feed-forward network (which we’ll cover shortly). So after multi-head attention, for example, the actual output is:

\[\text{output} = x + \text{MultiHeadAttention}(x)\]

where \(x\) is the input that went into the attention sublayer.

2.3.1 Why Residual Connections Help

This idea comes from the 2015 paper “Deep Residual Learning for Image Recognition” (He et al.), which was originally developed for very deep convolutional networks. But the problem it solves is so universal and fundamental to training deep networks of any kind, including transformers, that it’s applicable here as well.

The Degradation Problem

You might expect (and indeed, it’s a typical assumption) that making a network deeper always makes it more powerful. More layers means more capacity to learn complex functions, more parameters to create more complex relationships, etc. But in practice, researchers found the opposite: beyond a certain depth, adding more layers actually made performance worse, even on the training data. This wasn’t overfitting (which would hurt test performance but NOT training performance). The deeper network was genuinely worse at learning.

This is strange if you think about it. A deeper network contains the shallower one - in theory, the extra layers could just learn to be identity functions (pass the input through unchanged), and you’d match the shallower network’s performance at worst. But of course, in practice, how a network learns is not so obvious, and learning the identity through a stack of nonlinear transformations is surprisingly difficult for gradient-based optimization.

What Residual Connections Change

With a residual connection, the layer doesn’t need to learn the full desired transformation. It only needs to learn the difference between the desired output and the input - the “residual”:

\[f(x) = \text{desired output} - x\]

If the best thing to do is pass information through unchanged (which is common - not every layer needs to dramatically transform every token to do something useful), the layer just needs to output something close to zero. Learning “do approximately nothing” is much easier than learning “reproduce the input exactly through a chain of matrix multiplications and nonlinearities.”

This connects to initialization, too. At the start of training, each block’s \(f(x)\) often starts out small relative to \(x\). With a residual connection, that means the whole network begins as approximately an identity function, with the input passing through relatively unchanged. Each layer then gradually learns to make useful refinements from that starting point, rather than having to reorganize from a random transformation.

And, there’s an additional intuition here specific to attention. Attention creates a weighted average of value vectors, where each position becomes a blend of information from other positions. After many layers, without residual connections, this repeated blending could cause individual token representations to converge toward a generic “average of everything,” losing token-specific information. The residual connection ensures that each position always retains its original representation, with contextual information added on top rather than replacing it.

Gradient Flow

Finally, residual connections also provide a direct path for gradients to flow backward through the network during training. Consider backpropagation through \(\text{output} = x + f(x)\). The gradient of the output with respect to \(x\) is:

\[\frac{\partial \text{output}}{\partial x} = 1 + \frac{\partial f(x)}{\partial x}\]

That “\(1\)” does a lot of work. It enforces that, no matter what \(f\) does - even if its gradient is very small (the vanishing gradient problem) - the gradient through the residual is always exactly 1. This gives the training signal a “highway” to travel back through many layers without degrading. If \(f\)’s gradient gets very small, you still have a signal, which is essential for training the deep stacks of layers that transformers use.

2.3.2 Residual Connections in a Transformer

In a transformer, residual connections wrap around every sublayer that makes up a transformer block. Each transformer block has two of such sublayers - attention and a feed-forward network - and each gets its own residual connection:

\[x_1 = x + \text{MultiHeadAttention}(x)\] \[x_2 = x_1 + \text{FeedForward}(x_1)\]

The intuition here is simple. Since each sublayer makes its own incremental refinement to the representation, we allow each of those refinements to be a difference via residuals. The attention sublayer adds contextual information from other positions. The feed-forward sublayer adds nonlinear transformations. The original information is preserved through both via the skip connection, and what exactly the attention and feed-forward need to change can be kept as separate components.

2.3.3 Residual Connections in Code

The implementation is as simple as the concept, no new parameters or layers, just a simple addition. It’s typically implemented inside an attention or feedforward block because it’s so straightforward it’s not really worth separating out.

Inside a transformer block’s forward pass, each sublayer is wrapped in a residual connection, just adding the sublayer’s input back onto its output. For that reason, this is the pseudocode for it, but this won’t run on its own because, as we said, it really just makes sense to be within the layer, not its own thing.

# Attention sublayer with a residual connection
attention_output = self.attention(x)
x = x + attention_output          # keep the input, add the change

# Feed-forward sublayer with a residual connection
ff_output = self.feed_forward(x)
x = x + ff_output                 # residual again, around the FFN

2.4 Layer Normalization

Layer normalization is a very standard technique (established even a year before the advent of the transformer architecture) that stabilizes training by normalizing the values flowing through the network. The idea is straightforward: for each token’s representation, we normalize across the feature dimensions so that the values have a mean of 0 and a standard deviation of 1.

For token \(t\) with representation \(X_t\) (a vector of \(d_\text{model}\) values), layer normalization computes:

\[\text{LayerNorm}(X_t) = \gamma \cdot \frac{X_t - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}} + \beta\]

where:

  • \(\mu_t = \frac{1}{d_\text{model}} \sum_{i=1}^{d_\text{model}} X_{t,i}\) is the mean across token \(t\)’s feature vector

  • \(\sigma_t^2 = \frac{1}{d_\text{model}} \sum_{i=1}^{d_\text{model}} (X_{t,i} - \mu_t)^2\) is the variance across token \(t\)’s feature vector

  • \(\epsilon\) is a tiny constant (like \(10^{-6}\)) to prevent division by zero

  • \(\gamma\) and \(\beta\) are learnable parameters (both of size \(d_\text{model}\))

Each token gets its own mean and variance, computed only from its own features, so this operation happens on a row by row basis.

The \(\frac{X_t - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\) part centers and scales the values to have mean 0 and standard deviation 1. Then \(\gamma\) and \(\beta\) let the model learn to shift and scale the normalized values to whatever range is actually useful. This means normalization doesn’t force the values to be mean-zero, it just makes mean-zero the default starting point that the model can adjust from.

2.4.1 Why Normalize?

As values propagate through many layers of a deep network, their magnitudes can drift - some values might grow very large while others shrink toward zero. This is a problem because:

  1. Unstable gradients: If activations become very large, gradients can explode. If they become very small, gradients can vanish. Either way, training becomes unreliable.

  2. Sensitivity to initialization and learning rate: Without normalization, the behavior of deeper layers depends heavily on the exact scale of activations coming from earlier layers. This makes the network fragile - small changes to the learning rate or weight initialization can cause training to diverge, making optimal values for those much harder to find.

  3. Internal covariate shift: Each layer’s inputs are the outputs of the previous layer, which are constantly changing during training. Normalization ensures that regardless of what the previous layer is doing, each layer receives inputs in a consistent, well-behaved range.

Layer normalization addresses all of these by repeatedly resetting the scale of activations. It’s a simple operation, but it’s one of the things that makes training deep transformers stable and practical.

2.4.2 Normalization in Deep Learning

As I mentioned when we first introduced this building block - normalization isn’t unique to transformers. It shows up in virtually every modern deep learning architecture. The underlying motivation is always the same - we want to keep the values flowing through the network in a stable, well-behaved range so that training converges reliably. What varies is which dimensions you normalize across, and the choice depends on the structure of your data.

  • Input Normalization / Feature Scaling: Before data even enters a model, it’s common practice to normalize input features to have zero mean and unit variance (or scale them to a fixed range like [0, 1]). Here, you just compute statistics across your dataset and rescale. This prevents the model’s gradients from being dominated by the largest-scale features.

  • Batch Normalization: Introduced for CNNs (Ioffe & Szegedy, 2015), this normalizes across the batch dimension. For a given feature (in this case, one channel), it computes the mean and variance of that feature across all examples in the current mini-batch, then normalizes. This works well because convolutional filters are applied identically across spatial positions and across images, so normalizing a filter’s activations across a batch is normalizing “measurements from the same instrument,” keeping them on a consistent scale.

  • Group Normalization, Instance Normalization, RMS Normalization: Other variants exist that normalize across different subsets of dimensions, each designed for specific architectural needs. RMS Norm, for instance, is a simplified version of layer norm (it skips the mean-centering step) that has become popular in recent LLMs like LLaMA for its computational efficiency.

However normalization is done, the common thread and purpose remains the same: wherever activations might drift in scale during training, insert a normalization step to reset them. The differences are just about what you average over to compute the statistics.

Why Layer Norm for Transformers?

Like filters in a CNN, transformers also share weights across positions. But attention creates a complication: each token’s representation after attention becomes a content-dependent blend of information from every other token in the sequence. Two tokens at the same position in different examples can have wildly different representations because they had wildly different context.

What a transformer’s downstream computations - the dot products in the next attention layer, the matrix multiplications in the feed-forward network - care about is whether the features within a single token’s representation are well-scaled relative to each other. These \(d_\text{model}\) numbers are about to interact with each other through matrix multiplications, and if some dimensions have drifted to large magnitudes while others have shrunk, those interactions become dominated by whichever dimensions happen to be largest, regardless of whether they’re the most informative.

So, layer normalization addresses this by normalizing each token’s feature vector independently, resetting all \(d_\text{model}\) dimensions to a common scale before the next computation. It doesn’t matter what other tokens in the sequence are doing, or what other examples in the batch look like. Each token’s internal geometry is kept well-behaved on its own terms.

There are practical advantages too. Layer normalizations don’t care about variable sequence lengths, work identically regardless of batch size, and behave the same at training and inference time. So they are the choice.

2.4.3 Where Layer Norm is Applied

In the original transformer paper, layer normalization is applied after the residual addition (“post-norm”):

\[x_1 = \text{LayerNorm}(x + \text{MultiHeadAttention}(x))\]

However, most modern implementations use “pre-norm”, where normalization is applied before the sublayer:

\[x_1 = x + \text{MultiHeadAttention}(\text{LayerNorm}(x))\]

Pre-norm tends to train more stably, especially for very deep models, because the residual path stays completely true to its original purpose. The raw \(x\) flows through without any modification. With post-norm, the normalization sits on the residual path itself, which can interfere with the gradient highway we discussed in Section 2.3. In our implementation, we’ll use pre-norm, which is the more common modern choice.

So after the attention sublayer with pre-norm, the full computation is:

\[X_1 = X + \text{MultiHeadAttention}(\text{LayerNorm}(X))\]

That is: we normalize our input \(X\), run it through attention to get \(O\), then add the original (un-normalized) \(X\) back. \(X_1\) then becomes the input to the feed-forward sublayer (which we discuss in detail next!), where the same pattern repeats:

\[X_2 = X_1 + \text{FFN}(\text{LayerNorm}(X_1))\]

But, one consequence of that residual \(X\) never getting normalized is that at the end of an encoder layer, the final output actually won’t be normalized. To fix this, we’ll add one final LayerNorm at the very end of each, to make sure what comes out is properly normalized. When it comes time to actually build the encoder, we’ll revisit this.

2.4.4 Layer Normalization in Code

# PyTorch provides LayerNorm out of the box:
layer_norm = nn.LayerNorm(d_model)

# Usage inside a transformer block (pre-norm style):
# Attention sublayer
attention_output = self.attention(self.norm1(x))
x = x + attention_output

# Feed-forward sublayer
ff_output = self.feed_forward(self.norm2(x))
x = x + ff_output

Notice how with pre-norm, the pattern becomes very clean: normalize, transform, add back. The residual connection always adds the un-normalized input, keeping the skip path simple.

2.5 Position-Wise Feed-Forward Networks

Each transformer block contains, in addition to the multi-head attention sublayer, a feed-forward network (FFN) that is applied independently to each token’s representation. It’s a simple architecture - two linear layers with a nonlinearity in between:

\[\text{FFN}(x) = \text{ReLU}(X \cdot W_1 + b_1) \cdot W_2 + b_2\]

This introduces two new weight matrices:

  • \(W_1\) is of size \(d_\text{model} \times d_\text{ff}\)
  • \(W_2\) is of size \(d_\text{ff} \times d_\text{model}\)

Here, \(d_\text{ff}\) is the inner dimension, typically \(4 \times d_\text{model}\) (so for \(d_\text{model} = 512\), \(d_\text{ff} = 2048\)), but of course can be different.

The input and output are both \(d_\text{model}\)-dimensional (so the residual connection still works), but internally the representation expands to a larger space and then contracts back down.

The paper calls this ‘position-wise’ to emphasize that it operates on each token independently - but that’s just the natural behavior of matrix multiplication, not a special mechanism. Naturally, in the operation above, you get token 1’s output depending only on token 1’s input, token 5’s depending only on token 5’s, etc. It’s more that an FFN was chosen because it’s position-wise, not that anything special needs to be done to make this FFN position-wise.

2.5.1 Why Is the FFN Needed?

The paper is called “Attention is All You Need”. The attention mechanism has learnable weights. It has a nonlinear function. So it might not be obvious why you’d bother with a “plain” feedforward network in the mix. We just did attention, which is the main mechanism of the transformer.

The answer is that attention, despite its power, is fundamentally still a linear operation over the value vectors. Look at what attention actually computes for each token:

\[o_i = \sum_j a_{ij} \cdot v_j\]

That’s a weighted sum - a linear combination of the value vectors, with the attention weights choosing how to blend them. The softmax introduces nonlinearity in computing the weights \(a_{ij}\), but the operation that produces the output (multiplying those weights by V) - is linear. If you stacked multiple attention layers without anything else in between, the composition of linear operations over V is still fundamentally doing weighted averaging.

The FFN adds the nonlinear transformation capacity that attention lacks. The ReLU (or GELU, in many modern implementations) between \(W_1\) and \(W_2\) means the FFN can compute nonlinear functions of its input. It can suppress some features, amplify others, and create entirely new combinations that a linear transformation simply cannot produce.

The two sublayers have complementary roles:

  • Attention gathers information from across the sequence, deciding what information each token should collect from other tokens.
  • FFN processes that information, transforming each token’s now context-enriched representation through a nonlinear function.

You might wonder - if it’s just about nonlinearity, why not apply a ReLU directly to the attention output? You could, but a pointwise ReLU on a \(d_\text{model}\)-dimensional vector is limited - all it can do is zero out negative values.

The FFN’s power comes from the expand-then-contract structure: \(W_1\) projects into a \(4\times\) larger space, ReLU selectively activates features in that larger space, and then \(W_2\) combines those selectively-activated features back into \(d_\text{model}\) dimensions. The expansion is what gives the FFN the capacity to compute rich, nonlinear transformations - not just “add nonlinearity” but “create new feature combinations that didn’t exist before.”

Similarly, adding more parameters to attention itself wouldn’t fill this role. Attention’s parameters control which tokens to gather information from and what to extract. Making those bigger improves the gathering, but the output is still a weighted average of value vectors - and still linear. The FFN’s job is fundamentally different: not gathering information across positions, but transforming it within each position.

2.5.2 The Expansion Factor

The inner dimension \(d_\text{ff}\) is typically \(4 \times d_\text{model}\). This expand-then-contract pattern might seem arbitrary, but there’s intuition behind it.

\(W_1\) projects from \(d_\text{model}\) up to \(d_\text{ff}\), then ReLU zeroes out the negative values. If roughly half the neurons are zeroed out (which is typical for ReLU), you’re left with about \(2 \times d_\text{model}\) active features - a richer representation than the original \(d_\text{model}\) dimensions but computed sparsely. \(W_2\) then compresses this back to \(d_\text{model}\) for the rest of the network.

The \(4\times\) factor was chosen empirically by the original authors and has proven to be a reliable default. Some modern architectures adjust this (LLaMA uses \(\frac{8}{3} \times d_\text{model}\) with a different activation function), but the expand-then-contract pattern is essentially universal.

2.5.3 Feed-Forward Network in Code

Simple as it gets - two linear layers, a ReLU in the middle. Applied identically and independently to every token position.

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff):
        super(FeedForward, self).__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(d_ff, d_model)

    def forward(self, x):
        # x shape: (batch_size, seq_len, d_model)
        x = self.linear1(x)   # (batch_size, seq_len, d_ff)
        x = self.relu(x)      # (batch_size, seq_len, d_ff)
        x = self.linear2(x)   # (batch_size, seq_len, d_model)
        return x

2.5.4 Where the Parameters Live

Here’s a fact that might surprise you: the FFN contains the majority of a transformer’s parameters. For a standard configuration with \(d_\text{model} = 512\) and \(d_\text{ff} = 2048\):

  • \(W_1\): \(512 \times 2048 = 1,048,576\) parameters
  • \(W_2\): \(2048 \times 512 = 1,048,576\) parameters
  • Total FFN: ~2.1 million parameters (plus biases)

Compare that to multi-head attention with \(\text{h} = 8\):

  • \(W^Q, W^K, W^V\): \(3 \times (512 \times 512) = 786,432\) parameters
  • \(W^O\): \(512 \times 512 = 262,144\) parameters
  • Total attention: ~1.05 million parameters

The FFN has roughly twice as many parameters as the attention mechanism. The part of the transformer that gets all the attention (pun intended) is actually the smaller component. The FFN is where most of the model’s learned knowledge is stored. It’s something to keep in mind as you appreciate the evolution from “regular” neural networks to transformers. The attention mechanism is, don’t get me wrong, a brilliant addition - it’s what allows information to flow between tokens and sequences to be efficiently treated as a whole. But the majority of a transformer’s parameters, and much of its representational power, still lives in the humble feedforward network!

3. The Encoder

So that’s all of the core theoretical components that go into building a transformer!

Now, it’s time to start seeing it in action - how these things come together to form the full transformer architecture, we’ll start with the Encoder.

We’ve basically already seen a single encoder block come together as we went over the last few of our building blocks in Section 2. As we discussed, a transformer layer is really just comprised of two sublayers in sequence, each wrapped with a residual connection and layer normalization (using pre-norm):

\[X_1 = X + \text{MultiHeadAttention}(\text{LayerNorm}(X))\] \[X_2 = X_1 + \text{FFN}(\text{LayerNorm}(X_1))\]

That’s it. Normalize, attend, add back. Normalize, feed forward, add back.

📌 Note on terminology: A single unit of these two sublayers together is > called both a “block” and a “layer” interchangeably in the literature. Within > each block/layer, attention and the FFN are each called a “sublayer.” The full encoder is a stack of N of these blocks. In the original transformer, every block is identical in structure and size. This means same \(d_\text{model}\), same number of heads, same \(d_\text{ff}\). The only things that differ between blocks are the learned weight values.

Reminder: you can see all of the dimension details in Appendix 0. But here, let’s do a quick recap tracing through the full computation for an input \(X\) of shape \((n_\text{seq} \times d_\text{model})\):

  1. Layer Norm: Normalize each token’s features. Each row of \(X\) is normalized independently using its own mean and variance (no mixing between tokens). → shape \((n_\text{seq} \times d_\text{model})\)

  2. Multi-Head Attention: This is where tokens interact. Each token’s output row is a weighted combination of information from every token’s value vector. This is the only step where information flows across rows of the matrix! → shape \((n_\text{seq} \times d_\text{model})\)

  3. Residual Add: Add the original \(X\) back to the attention output, element-wise. → \(X_1\), shape \((n_\text{seq} \times d_\text{model})\)

  4. Layer Norm: Normalize each row of \(X_1\) independently, same as step 1. → shape \((n_\text{seq} \times d_\text{model})\)

  5. Feed-Forward Network: Transform each token’s row independently through the same two-layer network - no mixing between tokens. → shape \((n_\text{seq} \times d_\text{model})\)

  6. Residual Add: Add \(X_1\) back, element-wise. → \(X_2\), shape \((n_\text{seq} \times d_\text{model})\)

encoder-block

So, the information is transformed many times, but the input and final output to the encoder block always have the same shape. This means we can additionally stack encoder blocks, making the output of one the input into the next.

The original transformer used 6 encoder blocks, but of course, this is a hyperparameter, and deeper, larger networks will stack many more!

3.1 The Encoder in Code

Now, let’s build an encoder block in code.

This will look incredibly simple! That’s because we already built these components, and all that’s left is just assembly.

3.1.1 A Single Encoder Block

class EncoderBlock(nn.Module):
    def __init__(self, d_model, heads, d_ff, dropout=0.1):
        super(EncoderBlock, self).__init__()
        self.attention = MultiHeadAttention(d_model, heads)
        self.feed_forward = FeedForward(d_model, d_ff)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

    def forward(self, X):
        # Sublayer 1: Multi-Head Attention
        norm_X = self.norm1(X)
        attention_output = self.attention(norm_X, norm_X, norm_X)  # Q, K, V all come from same input
        X_1 = X + self.dropout1(attention_output)

        # Sublayer 2: Feed-Forward Network
        norm_X_1 = self.norm2(X_1)
        ff_output = self.feed_forward(norm_X_1)
        X_2 = X_1 + self.dropout2(ff_output)

        return X_2

You’ll notice we’ve snuck in dropout - this is a standard regularization technique that randomly zeroes out a fraction of values during training. It’s applied after each sublayer (before the residual addition) to prevent overfitting. We won’t dwell on it since it’s not specific to transformers, but it’s worth noting that it’s typically there.

Also note that in the attention call, we pass norm_X as all three arguments (Q, K, V). This is what makes it self-attention - the queries, keys, and values all come from the same source. In the decoder, we’ll see a case where Q comes from one source and K, V come from another.

3.1.2 Tracing the Shapes

We can confirm that this block does what we expect by actually choosing some dimensions, and looking at what each part of this encoder block does to the input shapes.

# Using the classes we built in Section 2:
d_model = 64
heads = 4
d_ff = 256
n_seq = 5
batch_size = 1

# Create an encoder block
encoder_block = EncoderBlock(d_model, heads, d_ff)

# Simulate input: 5 tokens, each a 64-dimensional vector
X = torch.randn(batch_size, n_seq, d_model)
print(f"Input X:              {X.shape}")

# Trace through each step manually:
norm_X = encoder_block.norm1(X)
print(f"After LayerNorm:      {norm_X.shape}")

attn_out = encoder_block.attention(norm_X, norm_X, norm_X)
print(f"After Attention:      {attn_out.shape}")

X_1 = X + attn_out
print(f"After Residual Add:   {X_1.shape}")

norm_X_1 = encoder_block.norm2(X_1)
print(f"After LayerNorm:      {norm_X_1.shape}")

ff_out = encoder_block.feed_forward(norm_X_1)
print(f"After FFN:            {ff_out.shape}")

X_2 = X_1 + ff_out
print(f"After Residual Add:   {X_2.shape}")

print(f"\nInput shape:  {X.shape}")
print(f"Output shape: {X_2.shape}")
print(f"Same shape:   {X.shape == X_2.shape}")
Input X:              torch.Size([1, 5, 64])
After LayerNorm:      torch.Size([1, 5, 64])
After Attention:      torch.Size([1, 5, 64])
After Residual Add:   torch.Size([1, 5, 64])
After LayerNorm:      torch.Size([1, 5, 64])
After FFN:            torch.Size([1, 5, 64])
After Residual Add:   torch.Size([1, 5, 64])

Input shape:  torch.Size([1, 5, 64])
Output shape: torch.Size([1, 5, 64])
Same shape:   True

The input is a sequence of \(d_\text{model}\)-dimensional token vectors (already embedded). The output is a sequence of \(d_\text{model}\)-dimensional vectors - one per token - each enriched with contextual information from the entire sequence, processed through a round of attention and transformation.

This output is sometimes called the encoder representation or encoder hidden states. For tasks like classification, you might take just one token’s output (often the first token). For sequence-to-sequence tasks like translation, the full output matrix gets passed to the decoder - which is where things get more interesting.

3.2 The Encoder’s Role

So we’ve built it! And it comes together easily enough after we spent so much time building up the components. But let’s take a step back here to talk about the role of the encoder - what, exactly, is its function. In Section 2, we spent a lot of time explaining why each step is important for things like ensuring training stability, preserving information, making sure our model can learn, and yes, for attention, how it relates words.

But it’s worth pausing here to look at what the encoder does in isolation, to understand why a full word generation model has both encoder and decoder components, and how exactly our words are getting transformed and related as they go through this step.

Recall that, fundamentally, the goal of a word generation model is next token prediction. Or plainly, returning which word is likely to come next.

The thing is, which we’ll talk about a little later, the encoder isn’t necessarily where generation happens. Actually, as we’ll talk about in a bit, plenty of modern models have no encoder at all.

So then, what’s the point of the encoder? If later parts of the model, which we’ll discuss next, are where that literal next word is predicted, then why have an encoder at all? The answer is that the encoder is fundamentally understanding how the words in a sequence relate to one another to set up a decoder for success (and sometimes…. just for the sake of understanding in and of itself!). The key thing to understand is that, after the encoder stack, what comes out is rich, highly contextualized relationships between all of the words in the input sequence.

So to see exactly how this comes out, and why it can be so useful and important, let’s look a little more deeply at what we actually get out of the encoder.

3.2.1 Attention’s Output

Consider the following sentence moving through the encoder:

\[ \text{She poured water from the pitcher into the glass until it was} \]

And let’s say it tokenizes conveniently at one token per word to:

\[ \text{a} \space\space \text{b} \space\space \text{c} \space\space \text{d} \space\space \text{e} \space\space \text{f} \space\space \text{g} \space\space \text{h} \space\space \text{i} \space\space \text{j} \space\space \text{k} \space\space \text{l} \]

(note, it would be true that \(\text{e} = \text{h}\) - let’s keep them separate for clarity).

At the start, this sequence passes through the input funnel, and we end up with an embedding of size, say 13 \(\times\) 5. Imagine that this model’s maximum sequence length is 13 - our sentence has 12 words so far - and our embedding size and therefore \(d_{\text{model}}\) is 5.

\[ X_{in} ( \in \space \mathbb{R}^{13 \times 5}) = \space \begin{matrix} a_1 & a_2 & a_3 & a_4 & a_5\\ ... \\ k_1 & k_2 & k_3 & k_4 & k_5\\ ... \\ m_1 & m_2 & m_3 & m_4 & m_5\\ \end{matrix} \space \begin{matrix} \space ← \textit{embedding for "She"} \\ \space \\ \space ← \textit{embedding for "it"} \\ \space \\ \space ← \textit{padding token at end} \\ \end{matrix} \]

At the start, “it” is completely context-free: the vector \([k_1 \space k_2 \space k_3 \space k_4 \space k_5]\) is the same “it” that would be in any sentence ever written when input through the input portion of the model. As far as just this vector is concerned, “it” could be standing in for the water, the pitcher, the glass… there’s no way to know because right now, it exists completely independently from those words.

The next major transformation that comes is attention. During attention, this understanding of “it” becomes a combination of every other word in the sentence.

The Projections

Attention’s first move is to produce the \(Q\), \(K\), and \(V\) matrices - query, key, and value - which are constructed by multiplying the input by three learned weight matrices:

\[ Q = X_{in} W_Q, \qquad K = X_{in} W_K, \qquad V = X_{in} W_V \]

Each of the resulting \(Q\), \(K\), and \(V\) matrices is a 13 x 5, the same shape as the input, and also, importantly, contains one vector within that pertains to “it”. I will leave it to the reader to work through the complete math if you want (re-reference the “attention” section for exactly how these matrices are calculated), but here, let’s just look at exactly what is computed for “it.” “It”’s query vector \(Q_k\) is the row \([k_1\ k_2\ k_3\ k_4\ k_5]\) times the whole matrix \(W_Q\):

\[ Q_k = \begin{bmatrix} k_1 & k_2 & k_3 & k_4 & k_5 \end{bmatrix} \begin{bmatrix} w_{Q11} & w_{Q12} & w_{Q13} & w_{Q14} & w_{Q15} \\ w_{Q21} & w_{Q22} & w_{Q23} & w_{Q24} & w_{Q25} \\ w_{Q31} & w_{Q32} & w_{Q33} & w_{Q34} & w_{Q35} \\ w_{Q41} & w_{Q42} & w_{Q43} & w_{Q44} & w_{Q45} \\ w_{Q51} & w_{Q52} & w_{Q53} & w_{Q54} & w_{Q55} \\ \end{bmatrix} \]

Written out fully, entry by entry, that query vector is:

\[ Q_k = \begin{bmatrix} k_1 w_{Q11} + k_2 w_{Q21} + k_3 w_{Q31} + k_4 w_{Q41} + k_5 w_{Q51} \\[4pt] k_1 w_{Q12} + k_2 w_{Q22} + k_3 w_{Q32} + k_4 w_{Q42} + k_5 w_{Q52} \\[4pt] k_1 w_{Q13} + k_2 w_{Q23} + k_3 w_{Q33} + k_4 w_{Q43} + k_5 w_{Q53} \\[4pt] k_1 w_{Q14} + k_2 w_{Q24} + k_3 w_{Q34} + k_4 w_{Q44} + k_5 w_{Q54} \\[4pt] k_1 w_{Q15} + k_2 w_{Q25} + k_3 w_{Q35} + k_4 w_{Q45} + k_5 w_{Q55} \\ \end{bmatrix}^{\top} \]

Note the transpose: I wrote this as a column vector so it would fit, but note that this is a row of the resulting Q in the end.

And \(K_k\) and \(V_k\) are the exact same operation with \(W_K\) and \(W_V\):

\[ K_k = \begin{bmatrix} k_1 & \cdots & k_5 \end{bmatrix} W_K, \qquad V_k = \begin{bmatrix} k_1 & \cdots & k_5 \end{bmatrix} W_V \]

So you’ll notice, “it” is still context-free at this point. Every single term in every one of those sums is some \(k_i\) times a weight. Every row in \(Q\), \(K\), and \(V\) serves to contextualize a word against its own embedding - not yet at all against the other words. They are three different linear reshufflings of the same standalone token, knowing nothing yet about the rest of the sentence.

The Attention Calculation

Now. This might prompt a very fair question: if all three are just re-projecting “it” into itself, why do we need three of them? Wouldn’t one be enough? And the answer is in that interpretation that \(Q\) (query), \(K\) (key), and \(V\) (value) are “looking for different things”. But it’s worth being really clear about what that means as we see the next steps in the calculation of attention.

The main idea is that we need the model to be free to make these roles differ. We know a query (\(Q\)), asks “what do I refer to?”, a key (\(K\)), says “here’s what I offer to anyone looking”, and a value (\(V\)) says “here’s what I’ll actually contribute if attended to.” But the important thing is that those can be naturally reflected in how these matrices are used.

Attention starts by multiplying \(Q\) and \(K^\top\). Immediately, look at what that does. To compute how much “it” should attend to “glass,” we take “it”’s query row and dot it with “glass”’s key row. Writing \(Q_k\) for “it”’s query (remember - “it” was token “k”) and \(K_i\) for “glass”’s key (glass is token \(i\)), that single score is:

\[ S_{k,i} \;=\; Q_k \cdot K_i \;=\; Q_{k,1}K_{i,1} + Q_{k,2}K_{i,2} + Q_{k,3}K_{i,3} + Q_{k,4}K_{i,4} + Q_{k,5}K_{i,5} \]

This is one number - a single score measuring how well “it” - as a seeker - lines up with “glass” - as a target. The dot product here does this specifically - it is exactly a measure of alignment between two vectors. It’s large when \(Q_k\) and \(K_i\) point the same way, small (or negative) when they don’t. So this operation is literally asking “does what ‘it’ is looking for match what ‘glass’ is offering?” - and it answers with a number.

So \(Q\) and \(K\) can’t be the same matrix. The dot product \(Q_k \cdot K_i\) needs to be able to compare two different words in two different roles. Here, “it” as the one doing the looking, “glass” as the one being looked at. You’ll note that a different part of this calculation does the opposite - \(S_{i,k}\) gets to ask how much “it” matches what “glass” is looking for. If \(W_Q\) and \(W_K\) were identical, then a word’s “looking” vector would always equal its “being looked at” vector. The score of “it” attending to “glass” (\(Q_k \cdot K_i\)) would be forced to equal the score of “glass” attending to “it” (\(Q_i \cdot K_k\)), because both would just be the same dot product of the same two projected vectors. So attention would be symmetric, and every relationship mutual. This is of course not how real language works. “It” really needs to see “glass” to understand what it means, while “glass” doesn’t really get any extra contextualization from “it”. Separating \(W_Q\) and \(W_K\) is precisely what lets the query-side and key-side of a word diverge, and that’s what makes asymmetric attention possible at all.

When we look at how this plays out fully for “it”, we get a complete row of scores out of this operation. “It” gets to play seeker against every single other word in the sequence:

\[ S_k \;=\; \big[\; Q_k \cdot K_a \;\;\; Q_k \cdot K_b \;\;\; \cdots \;\;\; Q_k \cdot K_i \;\;\; \cdots \;\;\; Q_k \cdot K_l \;\big] \]

These scores get scaled and softmaxed into final attention weights, which say, for each word, how much “it” aligns with it.

Attention’s Output

So \(Q\) and \(K\) are “used up” in computing attention scores. We now just have the artifact of scores that says how much a word needs to pay attention to all the others. That’s why we need that third matrix - \(V\) - the value - to come in and let us actually weight the content. Recall that the next step in the attention calculation will be to calculate \(O\):

\[ O_k \;=\; A_{k,a}V_a + A_{k,b}V_b + \cdots + A_{k,i}V_i + \cdots + A_{k,l}V_l \]

Each \(V\) there is itself a 5-dimensional row. So written out fully, dimension by dimension, the first component of “it”’s new vector is:

\[ O_{k,1} = A_{k,a}\,V_{a,1} + A_{k,b}\,V_{b,1} + A_{k,c}\,V_{c,1} + \cdots + A_{k,i}\,V_{i,1} + \cdots + A_{k,l}\,V_{l,1} \]

So now, finally, the portion of \(O\) that is relevant to \(k\) is a combination of \(a\)’s, \(b\)’s, \(c\)’s, \(i\)’s - it’s fully contextualized by other words’ value vectors, weighted by how much attention it needs to give them. All of “she”, “poured”, “glass” are now literally, arithmetically part of what “it” is. So exactly where our understanding of “it” stops being context-free is in this calculation.

So where \(Q\) and \(K\) exist to decide relevance, \(V\) is what gets delivered once relevance is decided. The “key” says what about a word is relevant for matching (stuff like, this is a concrete, referable noun). That can, and should, be completely different from the information it contributes once matched - its full rich meaning.

So the three roles in constructing this final \(O\) aren’t three arbitrary labels - they really do reflect where and how the math of attention relates them. \(Q\) and \(K\) are used in a dot product, which compares them. \(V\) gets used in a weighted sum, so it’s the content. All three matrices, though they begin without any cross-context, are used in a way that ends up with a rich, weighted understanding between each word, and every other word.

The Values

Let’s look concretely at how some made up values for these steps might actually affect how a word gets understood. Let’s say we have the following attention weights for “it” (remember, this is our result for softmax, scaled, \(Q \times K^\top\) - it says “how much do I care about every other word”):

\[ A_k = \big[\, \underbrace{0.04}_{\text{She}}\ \ \underbrace{0.08}_{\text{poured}}\ \ \underbrace{0.14}_{\text{water}}\ \ \underbrace{0.03}_{\text{from}}\ \ \underbrace{0.03}_{\text{the}}\ \ \underbrace{0.05}_{\text{pitcher}}\ \ \underbrace{0.04}_{\text{into}}\ \ \underbrace{0.03}_{\text{the}}\ \ \underbrace{\mathbf{0.42}}_{\textbf{glass}}\ \ \underbrace{0.03}_{\text{until}}\ \ \underbrace{0.06}_{\text{it}}\ \ \underbrace{0.05}_{\text{was}} \,\big] \]

In this example, most of “it”’s attention (0.42) is pointing to “glass”.

The weights \(A_{k,j}\) decide how much of each word becomes a part of this word’s understanding. With “it”’s weight concentrated on “glass” (\(A_{k,i} = 0.42\)) (with smaller amounts on “water” and “poured”), the term \(V_i\) (glass) dominates the sum. What this means, is that \(O_k\) comes out close to glass’s value vector, with a little bit of “water” and a tiny bit of “poured”, and not much else. In vector terms, that means “it” has been rebuilt to point mostly to where “glass” points. Its new representation is, in effect, a stand-in for glass. And that is exactly how “it” is functioning in this sentence.

If you change the weights, you change what “it” becomes. Had the attention instead put 0.42 on “pitcher”, the identical formula would rebuild “it” toward the pitcher’s vector, and “it” would come to mean the pitcher. The attention weights \(A\) decide the “recipe” for rebuilding a word from its context. And learning what weights are right to get this right is a large part of what the model spends training figuring out.

So, coming out of the attention calculation, we have words as re-contextualized vectors, carrying with them combinations of all other words in the sequence, and better aligning themselves in vector space with other words that are close to their meaning. But of course, the encoder isn’t done!

But one quick addition before we move on. We walked through that whole attention calculation as if there were a single set of \(Q\), \(K\), \(V\). Of course, we’ve established that attention is typically multi-head, so several of these run in parallel, and we’d have many \(O\)’s at the end instead. The idea is that “it” gets to ask many questions at once about what it refers to. One head might put its weight on “glass” to catch the object being referred to, while another tracks “water” as what’s being poured into it. Each head produces its own rebuilt version of “it,” and \(W_O\) then combines them, letting the model weigh which of those many questions mattered most for this particular “it,” in this particular sentence.

I won’t go through that math again here (the multi-head section covers it). And importantly, it doesn’t change the interpretation of what comes out: “it” is still a representation rebuilt from its context, just assembled from several complementary views of that context instead of one. When \(W_O\) mixes the heads together, it is an additional learned transformation, so the model does get an opportunity to learn how to flexibly combine information here as well. But importantly, as we’re about to see, it’s still a linear combination, which is why the next portion is needed.

3.2.2 The MLP’s Output

Coming out of attention, “it” is now a rich, contextualized vector, which carries information about “glass”, “water”, “pitcher”, all weighted by relevance. So it’s fair to ask: couldn’t we stop there? What’s left to do to a vector that already knows what it means?

The issue is that the \(O\) that comes out of attention:

\[ O_k = A_{k,a}V_a + A_{k,b}V_b + \cdots + A_{k,i}V_i + \cdots + A_{k,l}V_l \]

Is some combination of the existing value vectors, scaled by a weight, and the weights are positive and sum to 1. So \(O_k\) is really interpreted as a weighted average of value vectors the sentence already contained. A weighted average, by definition, is only ever a mix of the things you’re averaging - it can’t produce anything genuinely new. So while attention is an excellent information retriever - deciding which information to gather and pull together - it never actually does anything to that information once gathered. It can only hand off a combination of what was already there.

(Even with multi-head, this stays true in spirit: \(W_O\) mixes the heads with a learned linear map, which is more than an average, but still linear.)

The feed-forward network (FFN) or multi-layer perceptron (MLP) - these are the same exact thing with interchangeable names - is instead where the learning from the information really happens. And it’s because it’s a non-linear transformation.

Recall the shape of the MLP from Section 2:

\[ \text{FFN}(x) = \text{ReLU}(x \cdot W_1)\,W_2 \]

That ReLU is the first time the network applies an elementwise non-linear transformation to a token’s own features. Attention did contain a non-linearity in the softmax, which decides how much of each value vector to blend together, but the result from that, \(O_k\), is a weighted average of value vectors that were already there. Without the MLP, every token is just a blend of existing value vectors. The network creates genuinely new features in the MLP.

📌 Note: Why non-linearity is non-negotiable. You may recall that non-linearity is the single property that lets a neural network learn anything that a plain matrix multiply couldn’t. If you’re already comfortable with why neural nets need activation functions, then read ahead. But if you need a refresher, we go into all of the details on why neural nets in general must contain non-linearities in Appendix 3 below.

There is a second thing the FFN does, that is also very important, which is that it actually runs on tokens independently. Each vector for each token is transformed alone, with no re-referencing the other tokens. This is exactly what we want - a division of labor. Attention already contextualized this token using the words around it. The MLP’s job isn’t to repeat that process - it’s to transform tokens into a space where we can do useful things with them.

3.2.3 The Residual Connection

Once a token passes through the MLP and comes out as a new, nonlinear function of its context, with features that can represent more than meaning, the residual connection then adds this reshaped token back to the version of it that entered the MLP - the one that came right out of attention.

We already discussed in depth in the “Residual Connections” section why this is something that we do. But keep in mind that the interpretation here, after the residual connection, is really the “full” re-mapped input in its “useful” space, as the interpretation of a residual connection is that it allows the MLP to learn the difference - or what it needs to change - instead of what the whole new output has to be.

So only coming out of this stage do you really have the “new, hidden space representation” of the input token.

3.2.4 Stacking the Blocks

You might have a nagging objection from the MLP description above, where we said that the MLP transforming each token in isolation is a good thing. And it is, for the reasons we said there. But it does seem to throw away the whole point. Yes, attention learns rich meaning and contextualization, but now we’re just going to treat it in isolation?

This is of course where it matters that a single block is not the whole story. The full encoder is a stack of \(N\) of the encoder blocks, so this process is happening repeatedly in that stack. Every time we go through, attention mixes information across tokens, and the FFN transforms each token.

But when the second block runs attention, the thing that it’s mixing is exactly those new, nonlinear features that the first block’s FFN just produced. The tokens get to interact all over again, and this time what’s mixing is the rich features. So the FFN’s per-token independence doesn’t mean that those features won’t get to mix. It’s just that that’s left as attention’s job in the next block.

For this reason, depth in a network holds a lot of its power. It allows repeated iteration and refinement of features. And it allows it to happen with increasing amounts of abstraction and complexity. Where a first block might resolve surface relationships based on just word meanings, the second block could build an understanding of a word based on how another word in the sentence was contextualized by a different word. And so on, the relationships continue to get complex the deeper you go! Each block builds more abstract structure on top of the understanding the previous ones already formed, building up richer and richer understanding of all of the nuances a token holds as it goes.

Back to our architecture, let’s flesh out the encoder, which we now understand to really look like:

arch-expanded-encoder

And that, really, is the encoder’s output and the value it brings. The encoder’s job isn’t to produce any language or make any predictions. It’s just to create one vector per token that carries deep, contextualized understanding of that token and its meaning, in the context of the entire sequence.

3.3 A Real Encoder: BERT

So we’ve seen all of the pieces come together. But let’s confirm it in a real, trained encoder.

To do this, we’ll look at what comes out of the encoder-only model BERT. BERT (“Bidirectional Encoder Representations from Transformers”) was introduced by Google in late 2018, and it’s one of the most influential models in NLP history. For a few years it was the backbone of an enormous amount of language understanding work, including Google Search itself. For our purposes, we’re choosing it because it’s small and its architecture was simple. It is essentially exactly the encoder stack we just worked through.

The specific model we’re loading, bert-base-uncased, is very small by today’s standards, but at the time was quite large. It contains:

  • ~110 million parameters
  • 12 stacked encoder blocks
  • 12 attention heads per block
  • \(d_{model} = 768\)
  • vocabulary of ~30,000 subword tokens
  • maximum sequence length of 512

(The “uncased” just means it lowercases text before tokenizing.)

BERT was pre-trained on a huge pile of unlabeled text (English Wikipedia plus a large corpus of books, roughly 3 billion words). Its training objective was masked language modeling, which essentially means trying to “fill in the blank”. Random words in each sentence were replaced with [MASK], and the model learned to predict what word was supposed to be there. Because it always got to see the full surrounding context - the whole sequence both before and after the masked word - it is very good at building those rich, context-aware representations. (We’ll talk more about model flavors (encoder only, decoder only, encoder-decoders) after we’ve gone through the decoder section below, but for now just note that BERT was never trained to generate new text at all - it only ever learned to understand so that it could fill in. It was trained, precisely, just to be a good encoder.)

Just to note, a few things do differ from the exact architecture we laid out. All of these are small tweaks, which we already discussed as we went along that these common alternatives exist.

  1. BERT uses learned positional encodings (a trainable table), rather than the sinusoidal version that we discussed in depth from the original paper.
  2. It uses post-norm (LayerNorm after the residual add - like the original paper), where in this tutorial we chose to present the pre-norm that is more common in modern times.
  3. Its feed-forward networks use GELU rather than ReLU (the smoothed version we mentioned in Appendix 3).
  4. It tokenizes with WordPiece rather than BPE. WordPiece is a similar subword tokenizer, that proceeds in a similar way as BPE, but optimizes for some slightly different things as it does its merges.

None of these significantly change the major points of what we just discussed - they’re all examples of small, practical changes and choices that exist in the landscape of building an encoder. All of the building blocks are exactly the same.

We’ll use BERT to look at the following two sentences:

\[ \textit{The river bank was flooded after the storm.} \]

and

\[ \textit{I deposited my paycheck at the bank downtown.} \]

An encoder that’s doing its job should have very different interpretations of the word “bank” in those two sentences - as they have entirely different meanings that can only be determined by looking at how they’re used in the sentence as a whole. Let’s look at what BERT gives us to confirm if that’s true.

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
model.eval()   # (this downloads the model the first time - it may take a minute)

def bank_vectors(sentence):
    enc    = tokenizer(sentence, return_tensors="pt")
    tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
    pos    = tokens.index("bank")               # 'bank' is a single token in BERT's vocab
    with torch.no_grad():
        before = model.embeddings.word_embeddings(enc["input_ids"])[0][pos]  # context-FREE
        after  = model(**enc).last_hidden_state[0][pos]                      # context-AWARE
    return before, after

river_in, river_out = bank_vectors("the river bank was flooded after the storm")
money_in, money_out = bank_vectors("i deposited my paycheck at the bank downtown")

print("BEFORE the encoder (raw embedding lookup):")
print("  cosine of the two 'bank' vectors:", round(F.cosine_similarity(river_in, money_in, dim=0).item(), 4))
print("AFTER the encoder (contextual representation):")
print("  cosine of the two 'bank' vectors:", round(F.cosine_similarity(river_out, money_out, dim=0).item(), 4))
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
BEFORE the encoder (raw embedding lookup):
  cosine of the two 'bank' vectors: 1.0
AFTER the encoder (contextual representation):
  cosine of the two 'bank' vectors: 0.4063

Before the encoder, the two “bank” vectors are identical - same token, same row of the embedding table. After the encoder, they’ve been pulled apart, because one absorbed “river…flooded” and the other “deposited…paycheck.” Same word in, two different meanings out. That is the whole job of the encoder in one measurement: it turns context-free lookups into context-aware representations.

We can go one step further and check the split is meaningful - the river-“bank” should sit closer to water words, the money-“bank” closer to finance words:

def contextual(sentence, word):
    enc = tokenizer(sentence, return_tensors="pt")
    tks = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
    with torch.no_grad():
        return model(**enc).last_hidden_state[0][tks.index(word)]

water = contextual("the boat drifted down the river", "river")
cash  = contextual("i keep my cash in the account", "cash")

print("river-'bank' vs 'river':", round(F.cosine_similarity(river_out, water, dim=0).item(), 3))
print("river-'bank' vs 'cash' :", round(F.cosine_similarity(river_out, cash,  dim=0).item(), 3))
print("money-'bank' vs 'river':", round(F.cosine_similarity(money_out, water, dim=0).item(), 3))
print("money-'bank' vs 'cash' :", round(F.cosine_similarity(money_out, cash,  dim=0).item(), 3))
river-'bank' vs 'river': 0.429
river-'bank' vs 'cash' : 0.342
money-'bank' vs 'river': 0.314
money-'bank' vs 'cash' : 0.428

4. The Decoder

So, great! The encoder has done its job. It understands the input, and turns our sequence of words into a set of rich, contextualized representations. But that rich embedding of the input sequence isn’t really a usable output as-is. If we wanted to say, output the next word we think will come, or translate that sequence into a different language, nothing the encoder produced is actually a word, a next token, or an answer. It’s “just” a pile of meaning-laden vectors. If we want to do any of the tasks that require us to actually produce language, we need another half of the architecture: the decoder.

The good news is that most of the building blocks here are largely reused, so we don’t need to define and understand a bunch more components. But, there are two new ideas on the decoder side that the encoder didn’t need, and those are causal masking and cross-attention. So despite the decoder being the “second half” of the model, the new conceptual work we’ll need to do is small, and these new concepts build strongly on the existing foundation we already have.

4.1 Why Two Halves?

Before we get to those new building blocks, it’s worth building the context as to why a full generation model is split cleanly into these encoder and decoder parts at all. And it’ll be useful to think of them as distinct parts, and what those distinct parts do differently, as actually many modern models are just one or the other, not two parts together.

The insight is that there are really two distinct things that have to happen to successfully model language:

  1. Understanding. To know how a language is built, what the words mean, how they relate, you have to be able to fully understand how sentences are constructed, why words go together and what order they do, what words can mean in different contexts, etc. This is the encoder’s job. Given a sequence, it looks at how the words relate in both directions - every token must attend to every other token, because comprehending a sentence requires seeing the whole thing. In the example at the end of the encoder section, “bank” is disambiguated by words on either side. Before you can do anything with language, you start by just knowing what it means. So its output is just rich representations.
  2. Generating. If you wanted to produce a new sequence, you have to do it one word (token) at a time, conditioned only on what came before. This is the decoder’s job. It needs to be able to look in one direction only - only at the words that come before - because when language is being produced, the words ahead don’t exist yet. So its output is actual tokens.

While a lot of the same concepts will be re-used, these two jobs are different enough that they can be separated entirely, and it mostly comes down to that “what do you get to look at” question. If to understand we must attend forward and backward, but to generate we can only attend backwards, we naturally have two fundamentally different ways that processing words can behave. But as it turns out, each half is useful on its own, and that’s why you’re likely to encounter three main flavors of transformer out in the world:

  • Encoder-only models (like BERT): Sometimes, the task only requires that you understand a sequence of words. For tasks like classification, sentiment analysis, searching, the point is just to get a good representation of the words, then use it. So you can run an encoder, take its representations, and attach a small task-specific head. If you assume you already “have” the words you’re working with, then there’s no generation, and no need for a decoder, at all.
  • Decoder-only (like GPT): Other times, you only need to do the generation, with each token conditioned on the past. As it turns out, most modern models are this, as this can also cover an enormous range, even including tasks that sound like understanding, so long as they’re reframed as generation tasks - (“[REVIEW]. The sentiment of this review is ___“). The job of a decoder-only model is really just to make more tokens, so you don’t need to understand something first, then generate, the understanding just needs to happen about the text you’re generating more of.
  • Encoder-decoder (the original Transformer, like T5): The full transformer architecture, both pieces, is really needed in tasks where you need to understand one thing completely, then generate a different thing from it. The classic example is translation - you need to read and understand the entire French sentence, then produce the English version one token at a time. This is the architecture we’re building in this tutorial, and translation is exactly the example we’ll use.

A decoder-only model like GPT might seem like it’s “doing both” jobs - it surely must see, process, and understand all text before deciding what to generate. In a sense, it is, but not by having two halves. It’s a single stack of masked self-attention blocks, trained from scratch without the separate encoder and cross-attention we’ll come to see. Most frontier models today are exactly this decoder-only.

And encoder-only models like BERT, which are trained in a “fill in the blank” type task (masked language modeling) also end in a token prediction, but are not generator models. The key difference, again, being that the context in all encoders is bidirectional, so the task is still about learning good representations.

At the end of the tutorial, we’ll talk more about exactly how these different model types are trained, so you can see exactly how these differences show up when building a real, modern model. For now, just keep in mind that the two halves do wind up being useful on their own, even though we’ll continue to think about the full encoder and decoder as doing separate things but connected into one architecture.

So in this tutorial, we’ll build the original, full encoder-decoder from Attention Is All You Need. All three of the above architecture types come out of it, so once you understand it fully, understanding exactly how the parts come to stand on their own becomes much simpler.

4.2 Causal Masking

The first new component we need to understand the full decoder architecture is causal masking, which will literally prevent the decoder portion of the model from giving any attention to certain parts of the sequence. This is an important feature that protects the decoder’s text generation from “cheating”, by making sure it only has access to and understanding of its previous tokens.

4.2.1 Autoregression

Before we get into exactly the math, let’s build some intuition about why we need this new component. The core idea is that a decoder must produce its output autoregressively - one token at a time, with every new token being a function of all those before it. The model’s own previous outputs become the input for producing the next token, over and over again.

For instance, say the decoder’s job is to produce a French-to-English translation. Then, production of the English sentence “I love cats”, will proceed one word at a time:

step 1:  [INPUT] <bos>                 -> [OUTPUT] "I"
step 2:  [INPUT] <bos> I               -> [OUTPUT] "love"
step 3:  [INPUT] <bos> I love          -> [OUTPUT] "cats"
step 4:  [INPUT] <bos> I love cats     -> [OUTPUT] <eos>   (stop)

Where <bos> and <eos> are special tokens that indicate the start/end of a generation sequence. The output is always a single next word, but the input grows with every word produced - the entire sequence so far goes back into the model to determine which next word to produce. It keeps generating, with every new word conditioned on everything generated so far (plus the encoder’s understanding, through cross-attention, which we’ll get to), until it produces the “end of sequence” token <eos> and stops.

The key difference between how the decoder and encoder get to process words is that the decoder, since its job is to generate one at a time, cannot attend to later positions since at generation time they simply don’t exist yet. And if we only ever ran the model in generation mode, producing one token at a time, this would always be the case and we wouldn’t need to worry about it. At step t, the input literally is only the tokens we’ve produced so far, so there’s nothing to “look ahead” at anyway.

The problem shows up at training time. Training one token at a time (constantly checking if every single individual token produced is the one we wanted) would be wastefully slow. Instead, we feed the decoder the entire known target sequence at once and check its prediction of the next token at every position in a single pass. The issue is that this means, when we go to see at position t, what token we predict at t+1, well the answer is already sitting right there at position t+1. If we didn’t do anything to cover up the later tokens, the model would just learn to go look at it.

Causal masking is what prevents this. It blocks each position from attending to anything after it, so even though the full sequence is present, every position only ever sees what’s behind it - exactly the information it would have during real generation. We get to process the whole sequence at once, without worrying about muddying up the whole point and premise of the decoder.

4.2.2 What the Mask Looks Like

It turns out that enforcing this is remarkably clean and easy to do - no complicated extra math. It winds up just being a triangular mask applied to the calculated attention that says “actually, don’t pay any attention to that”.

Recall that, coming out of the attention calculation, we have attention weights that come from a softmax over each query’s row of scores. For a sequence of length T, the score matrix: \[ S = QK^\top/\sqrt{d_k} \] is a \(T × T\) matrix, where row i says how the ith entry in the sequence should attend to each other position j. If we run this in the decoder, it’ll be completely filled in, including with how much attention a token at position 3 should attend to the token at position 5, even though that one is “in its future.”

Then, to prevent that, we can simply build a mask that says, for a token at position i, just ignore all of the attention scores to tokens for positions j > i. This is another matrix, the causal mask, which is also of size \(T × T\), but where entry (i, j) is 1 where i is allowed to see key j (that is, j ≤ i), and 0 otherwise. It ends up just being a lower-triangular block of ones:

          key1  key2  key3
query1:    1     0     0
query2:    1     1     0
query3:    1     1     1

Then, before we softmax the attention scores, we apply this mask to the attention score matrix, taking every disallowed entry of the score matrix and setting it to \(−\infty\):

scores = scores.masked_fill(mask == 0, -∞)

Why \(−\infty\), and why before softmax? Recall that softmax is:

\[\text{softmax}(s)_j = \frac{e^{s_j}}{\sum_k e^{s_k}}\]

To force a future position’s weight when the attention is actually applied to be exactly zero, \(e^{s_j}\) must be 0. This is only met when \(s_j = −\infty\). That part is easy to see. So why make sure we apply before softmax? The answer is to make sure the normalization is still right and the probability distribution still sums to 1. If we instead do softmax to get attention weights, then try to apply our mask and zero out the future weights, the weights will no longer sum to 1. But as we discussed in the attention section, this is important, so we’d have to re-normalize to make sure that they do. But that’s an extra step, and extra compute. Instead, setting this \(−\infty\) before the softmax means that the re-normalization happens as a part of that calculation, with the zeros getting no contribution already in the calculation of the probability distribution, to save us some work.

masked-mha

4.2.3 New Multi-Head Attention Code

Since this is so remarkably simple, it’s actually typically implemented right into the multi-head attention code, not as something independent. So while we already have MultiHeadAttention code above, we need to rewrite it with a small edit to reflect that it can also be, optionally, masked. That way this same code piece is reusable every time we need MHA.

And a quick note you’ll see as we’re about to implement this: in actual code we have to use a large negative number like -1e9 rather than literal -inf. This is because a literal -inf can produce a NaN in some cases, which will cause big problems, and e^{-1e9} is already zero to floating point precision anyway.

class MultiHeadAttention(nn.Module):
    def __init__(self, embed_size, heads):
        super(MultiHeadAttention, self).__init__()
        self.embed_size = embed_size
        self.heads = heads
        self.head_dim = embed_size // heads

        assert (self.head_dim * heads == embed_size), "Embedding size must be divisible by heads"

        self.values = nn.Linear(embed_size, embed_size, bias=False)
        self.keys = nn.Linear(embed_size, embed_size, bias=False)
        self.queries = nn.Linear(embed_size, embed_size, bias=False)
        self.fc_out = nn.Linear(embed_size, embed_size)

    def forward(self, values, keys, query, mask=None):              # <- NEW! We must have a mask option, default None
        N = query.shape[0]
        value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

        values  = self.values(values)
        keys    = self.keys(keys)
        queries = self.queries(query)

        values  = values.reshape(N, value_len, self.heads, self.head_dim)
        keys    = keys.reshape(N, key_len, self.heads, self.head_dim)
        queries = queries.reshape(N, query_len, self.heads, self.head_dim)

        values  = values.transpose(1, 2)
        keys    = keys.transpose(1, 2)
        queries = queries.transpose(1, 2)

        attention_scores = queries @ keys.transpose(-2, -1)
        attention_scores = attention_scores / (self.head_dim ** 0.5)

        if mask is not None:                                         # <- NEW! If we need the mask, apply it here
            attention_scores = attention_scores.masked_fill(mask == 0, -1e9)

        attention = torch.softmax(attention_scores, dim=-1)
        self.attention_weights = attention.detach()                  # <- (NEW - not mask related - save a copy to look at later)

        out = attention @ values

        out = out.transpose(1, 2).reshape(N, query_len, self.heads * self.head_dim)
        return self.fc_out(out)

One thing you might notice, is that we build MultiHeadAttention to take a mask instead of building one. This is really just for simplicity’s sake in the code. We want to be able to re-use this MultiHeadAttention definition in the encoder’s self-attention (with no mask), the decoder’s masked self-attention (with the causal mask we just discussed), and in the decoder’s cross-attention (which we’re about to see, is just another small edit to how the attention we already know is done). In each of these cases, the masking behavior is different. Passing the mask, whatever it needs to look like, is the simplest way of being able to use this definition in all 3 cases.

For causal masking, making the mask itself is done with a single line:

torch.tril(torch.ones(seq_len, seq_len))

Which is constructed once and passed down into each decoder block. We’ll come back to this, and see exactly where it’s defined and passed in, when we assemble a full decoder later in this section.

4.3 Cross Attention

So then, as we just alluded to, cross-attention will also just be a straightforward change to how we use the multi-head attention concepts that we already defined and already know. But let’s of course stop first to talk about what it’s meant to do.

For a task like French-English translation, and indeed any task that really requires the encoder-decoder architecture, there is a core insight: the whole point is that there’s something other that we had to understand in order to be able to generate this thing.

If we just started running the decoder part of a French-English translation model, as we saw above, it just starts with <bos>, and starts writing English text. How would it know to make “I love cats” over “I love dogs”? To start the sentence with “I” at all?

The answer, of course, is that we must connect it to the encoder and its understanding of “J’aime les chats”, and the way that is done is through cross-attention, which allows the tokens being produced in English to attend to the tokens that went through the encoder pipeline, and their rich, interconnected contextualization to each other. The decoder must, in order to produce “I”, be able to see how the encoder represents “J’”, which up until now has just gone through the encoder and sat there.

4.3.1 Back to Q, K, V

So what actually changes? Recall that in self-attention, the query, key, and value all come from the same sequence - every token asks a question about itself (Q), advertises what it offers (K), and carries its own content (V). Cross-attention uses those same interpretations of Q, K, and V, but now pulls its inputs from two different places:

  • The query comes from the decoder - the English sequence being generated. Each English position forms a query meaning, roughly, “given what I’m trying to produce next, what do I need from the French?”
  • The key and value both come from the encoder’s output - the contextualized French representations. Each French position offers a key (for matching against the decoder’s query) and a value (the content that gets pulled in if it matches).

Everything else is identical to the attention we already know. The scores are still computed as \(Q \cdot K^{\top}\), then scaled and softmaxed into a set of weights that sum to 1. The output is still a weighted sum of the values. The only difference is that the questions are being asked by one sequence (English) and answered by another (French). When the decoder is about to produce “cats,” its query lands most of its attention weight on the encoder’s representation of “chats,” and pulls that content into its own stream.

Now, there are two other major things to keep in mind that are a little different from how we’ve talked about attention so far, which we’ll see come to fruition in a real example later.

  1. The attention matrix will no longer be square. When we did self-attention, the attention scores were always square because seq_len was always the same - we were always working with the same sequence. But here, the English sequence is growing while the French sequence is a fixed size. So the attention matrix will come out to seq_len_o (output, English) × seq_len_i (input, French) .This isn’t a problem - nothing ever required attention to be square - self-attention was just a special case where the two sides happened to be the same sequence. The only consequence is that we’ll have two different sequence lengths that need to be tracked to make sure all of our sizes are right.
  2. We will not mask this at all. Causal masking is used at training time to be sure the decoder can’t look ahead. But here, the growing sequence is attending to something that will always be fully known - the French sequence is complete both at training time and at generation time. So even though this is another case of MHA in the decoder, it won’t need to be masked like the self-attention that comes just before it.

There’s one interesting thing to point out here, which might help hammer in the interpretations of Q, K, and V as distinct matrices which probe distinct things, that becomes a bit more clear when you see it in cross-attention.

We already alluded to it in the original MHA section, but notice that keys and values are a necessarily bonded pair. Whereas the query is free - it’s not required to match any other shape.

Look at the shapes. In the attention calculation, we begin with: \[ Q \ \ \in \ \ n_{\text{seq, 1}} \times d_{\text{model}} \] \[ K \ \ \in \ \ n_{\text{seq, 2}} \times d_{\text{model}} \] \[ V \ \ \in \ \ n_{\text{seq, 3}} \times d_{\text{model}} \]

Then, when we do \(Q K^{\top}\), the \(d_{\text{model}}\) becomes the interior dimension, so they are compatible regardless of what \(n_{\text{seq, 1}}\) and \(n_{\text{seq, 2}}\) are. The result is \(n_{\text{seq, 1}} \times n_{\text{seq, 2}}\).

But later, this result is multiplied by V: \[ O = A \times V \]

Then, this is a \(n_{\text{seq, 1}} \times n_{\text{seq, 2}}\) times an \(n_{\text{seq, 3}} \times d_{\text{model}}\) calculation. If \(n_{\text{seq, 3}} \neq n_{\text{seq, 2}}\), the math fundamentally fails, right there. So it’s mathematically forced that the keys and values must describe the same set of items, while the queries can come from somewhere else entirely. This is exactly what cross-attention exploits.

But it goes back to our intuitive interpretation of attention as a soft dictionary lookup as well. If our keys tell us “how you’d find an item”, and your values tell you “what you get from an item”, then they naturally are a bonded pair. Like a dictionary definition, and the word it belongs to (though attention allows us to quantify how much every potential word relates to every potential definition).

Still, the interpretation is that the key \(j\) and value \(j\) are two views of the same source token - as its searchable handle and its retrievable content. Though we produce them with different learned projections (\(W_K\) and \(W_V\)), they are bound to the same item. The query, by contrast, is flexible to ask about something else, which is why it’s so useful to do so in cross attention. We can ask about a different sequence, with a different length, or in a different language, and still get the right information that we need from the other two.

cross-attention

4.3.2 Cross-Attention in Code

As we already mentioned, our MultiHeadAttention class is actually already set up to support this change to cross attention without changing it at all - just by giving it different inputs.

Recall that when we did multi-head attention in the encoder, we passed the same value in 3 times to construct Q, K, and V. And in masked self-attention, we do the exact same thing with a mask.

# inside the decoder block's forward (x = decoder's own sequence, enc_out = encoder output):

# 1. how we did self-attention in the encoder: Q, K, V all = x, no causal mask
self_attn = self.attention(x, x, x)
# ...residual + norm...

# 2. masked self-attention in the decoder: Q, K, V all = x, with the causal mask
self_attn = self.attention(x, x, x, mask=trg_mask)
# ...residual + norm...

Well, in cross attention, we simply define the different sources that we need for Q, K, and V:

# 3. cross-attention: V and K come from the encoder, Q comes from the decoder
cross_attn = self.cross_attention(enc_out, enc_out, query)
#                                 └- V -┘  └- K -┘ └- Q -┘

In this case, \(V\) and \(K\), which remember must always be from the same source, come from the output of the encoder. We call this with no mask (for now, we’ll discuss a nuance in a bit), because remember - there’s no “cheating” about the known encoder information.

4.4 Building the Decoder

Now, we can assemble the decoder. The fortunate thing is that, the same way that the few extra pieces we went over were really just tweaks to things we already understood well, the decoder block is really the same as the encoder block, plus one extra sublayer.

Where an encoder block was constructed of two sublayers - 1) self-attention, 2) feed-forward, the decoder block has the extra sublayer for cross-attention. So it looks like:

  1. Masked multi-head self-attention. The decoder attending to its own tokens so far, with the causal mask so no position sees the future.
  2. Cross-attention with the encoder output. The decoder attending to the encoder’s output, the step that finally connects the two halves.
  3. The feed-forward network. Identical to the encoder, we end with a feedforward network to finally transform the result.

Additionally, each of these operations is wrapped in the same normalization and residual adding as the encoder before. So for each sublayer, we normalize, go through the sublayer, and add the un-normalized input back.

Putting it all together, this looks like:

decoder-block

4.4.1 A Decoder Block, in Code

Now, let’s write the code for what this decoder block looks like when we put it all together. You’ll notice it’s very similar to the encoder block we put together in the last section.

class DecoderBlock(nn.Module):
    def __init__(self, d_model, heads, d_ff, dropout=0.1):
        super(DecoderBlock, self).__init__()
        self.attention       = MultiHeadAttention(d_model, heads)   # masked self-attention
        self.cross_attention = MultiHeadAttention(d_model, heads)   # cross-attention (its own, separate weights)
        self.feed_forward    = FeedForward(d_model, d_ff)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.norm3 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)
        self.dropout3 = nn.Dropout(dropout)

    def forward(self, X, enc_out, src_mask, trg_mask):
        # Sublayer 1: MASKED self-attention - Q, K, V all from the decoder's own input
        norm_X = self.norm1(X)
        self_attn = self.attention(norm_X, norm_X, norm_X, mask=trg_mask)
        X_1 = X + self.dropout1(self_attn)

        # Sublayer 2: cross-attention - Q from the decoder, K & V from the encoder output
        norm_X_1 = self.norm2(X_1)
        cross_attn = self.cross_attention(enc_out, enc_out, norm_X_1, mask=src_mask)
        X_2 = X_1 + self.dropout2(cross_attn)

        # Sublayer 3: position-wise feed-forward network
        norm_X_2 = self.norm3(X_2)
        ff_output = self.feed_forward(norm_X_2)
        X_3 = X_2 + self.dropout3(ff_output)

        return X_3

The differences from the encoder block are really just:

  1. Two separate attention objects. self.attention and self.cross_attention are both MultiHeadAttention, but different objects so each carries its own independent W_Q, W_K, W_V, W_O.

  2. forward takes four inputs, not one. The encoder block only needed X. The decoder block also needs:

  • enc_out: the encoder’s output for cross-attention
  • trg_mask: The causal triangular mask for self-attention
  • src_mask: a mask we actually DO sometimes need in cross-attention to hide padding for different sequence lengths in batches. We’ll talk about this more later as it only matters for batching, which we haven’t yet covered.
  1. The extra sublayer. Sublayer 2, which is cross-attention, has one extra thing worth pointing out. We normalize X_1, which is the decoder’s current running representation, and give that as the query. We don’t re-normalize enc_out, which becomes K and V, since this is exactly as it was produced, normalized, then held fixed from the encoder side.

As with the encoder block, we have a residual connection add the input back after every sublayer. So, putting that all together, the decoder comes together to look like:

arch-expanded-decoder

4.5 The Decoder’s Role

We started this section by talking about how the role of the decoder is to autoregressively generate the next token, one at a time. But, actually nothing in the decoder block that we’ve talked about so far really does that. So how, actually, does that little dotted arrow that “runs until <eos>” actually operate? Where would I possibly get the next token from?

The answer is that, despite our “high-level” graphic showing: \[ \text{Input} → \text{Encoder} → \text{Decoder} → \text{Output} \]

as four very distinct blocks, the relation between \(\text{Decoder}\) and \(\text{Output}\) is where that clean pipeline starts to break down a bit.

The decoder’s role is, generally, to generate the next token. But in isolation the decoder stack is actually still an understanding sort of stack - meant to prepare us to generate the next token. It’s not like all of a sudden the same attention mechanisms and feed-forward networks are now doing generation simply because they exist in a decoder block - how they’re used has to be to help with generation rather than to help purely with understanding.

And indeed, everything we’ve built so far - and the decoder block if it sits entirely on its own - still only turns vectors into vectors. \(X_{out}\) is an \(n_{\text{out}} × d_{\text{model}}\) matrix - exactly the same kind of representation the encoder produced, now just relating the words that exist so far in the output to each other via self-attention, and the input via cross-attention. The decoder doesn’t actually produce a token yet. What we have been calling the “output” is what produces the token.

We’re going to get to that next. The important part to understand for the decoder’s role is that it’s fundamentally tied to the output in a way that the input and the encoder weren’t. Getting the input is work you do once. Similarly, getting the encoder’s representation of the input is work you do once.

The decoder, instead, runs repeatedly on a growing sequence, where its result is fed into the output layer to get an actual next token, that next token is appended to the existing sequence, and that becomes the input to run through the decoder again.

The decoder’s role is to “generate” insofar as its role is to produce a rich, highly contextualized representation of the output so far, that informs what the next token will be, but is not literally yet what the next token will be.

So the two run in tandem, over and over again.

\[ 1. X_{\text{output}} = [\text{<bos>}] \rightarrow DECODER \rightarrow X_{\text{out}} = \begin{bmatrix}{x_1 \ x_2 \ ... \ x_n}\end{bmatrix} \rightarrow OUTPUT \rightarrow ["\text{I}"] \\ \] \[ 2. X_{\text{output}} = [\text{<bos>}, \text{I}] \rightarrow DECODER \rightarrow X_{\text{out}} = \begin{bmatrix}{x_1 \ x_2 \ ... \ x_n \\ y_1 \ y_2 \ ... \ y_n}\end{bmatrix} \rightarrow OUTPUT \rightarrow ["\text{love}"] \\ \] \[ 3. X_{\text{output}} = [\text{<bos>}, \text{I}, \text{love}] \rightarrow DECODER \rightarrow X_{\text{out}} = \begin{bmatrix}{x_1 \ x_2 \ ... \ x_n \\ y_1 \ y_2 \ ... \ y_n \\ z_1 \ z_2 \ ... \ z_n}\end{bmatrix} \rightarrow OUTPUT \rightarrow ["\text{cats}"] \]

Then, that output is the final missing piece we need to discuss.

5. The Output

So, the final job we have to do is to turn an \(X_{out}\), and a growing one at that, which is coming out of the decoder, into a final prediction on a next token.

This is the job for the output layer, which fortunately, is by far the simplest portion of the entire architecture.

5.1 From Vector to Vocabulary

When we get \(X_\text{out}\) from the decoder, it’s still just representations of the current words in the sequence. It’s a size \(n_{\text{seq}} \times d_{\text{model}}\) matrix that says, “For every single word in the sequence so far, here’s a rich, vector representation of it, contextualized by all the other words so far, and the input we ran through the encoder”.

To turn that representation into a word, we need to somehow get to a score for every token in the vocabulary. Basically, if we want to pick the next word, we need to ask, “well, how likely is it that this is the next word?”, for every single word that we know.

Fortunately, that can just be done with the simplest type of layer we know in deep learning: the simple linear projection.

self.output_projection = nn.Linear(d_model, vocab_size)

This layer takes our current representation, and multiplies it by a set of learnable weights that transform that output to represent a prediction for each word in the vocabulary.

Applied to \(X_\text{out}\), it maps each row’s \(d_\text{model}\) vector to a \(\text{vocab\_size}\) vector (plus the batch dimension in code, as always):

logits = self.output_projection(X_out)   # (n_out, d_model) -> (n_out, vocab_size)

Each row coming out of this is a set of logits. A raw, unnormalized score per vocabulary token.

Of course, a softmax can turn each set of logits into a probability distribution over the entire vocabulary:

\[P(\text{next token}) = \text{softmax}(\text{logits})\]

So we get, for every token in our vocabulary, a probability that that token should be the next one in the sequence.

5.2 Which Row to Use

Of course, as we just said, the projection produces a distribution for every row. And every row represents a word in the current sequence, which grows every time we add a new word. So we end up with a vector of probabilities for the next token, for every single word in the sequence. Which one do we use?

The answer is that at generation time, we only read the last row - and determine the next word based on the next token that the final row in the decoder’s output representation represents.

This might be confusing. The key thing is to think about each row at position \(i\) predicting what the token at position \(i+1\) will be. With [<bos>, I, love] fed in, row 1 corresponds to <bos> and predicts what will be in position 2. This ends up being “I” at row 2. Now row 2 predicts what’s at position 3, and row 3 will predict what’s at position 4. Each row in the output is predicting what the token in the next position will be, so once we use the output at row 2 to get row 3, we don’t need it anymore. It tells us about a token we’ve already decided.

📌 Note: But we don’t have to just interpret it that way - it’s a mathematical truth that the last row, the new one that runs through the decoder, is actually all that is going to be different in the final output. So this means it’s not only the correct one to use, but that actually this is something that can be exploited at generation time to speed things up, with something called a KV Cache. Read Appendix 4 for more details!

5.3 How to Use a Probability Distribution of Tokens

Let’s say that the sequence so far is:

\[ \text{She poured water from the pitcher into the glass until it was \_\_\_} \]

You probably want the next token prediction to be “full”, or at least, to be likely to be “full”. You probably also want it to be likely to say “filled”. You probably don’t want it to say “potato”.

As it turns out, getting a probability distribution, over all tokens in the vocabulary, is quite an important concept for a task such as language modeling, where there are certainly “more right” and “more wrong” answers, but not necessarily always “perfect” answers you’d want every single time. The output is more important than just predicting one token, it’s useful because a good distribution gives us a good idea of how well every possible word fits, and thus lets us choose one with a little flexibility.

5.3.1 A Theoretical Distribution

The final output we get ends up being a probability distribution over all possible tokens. It might look something like this:

mpl.rcParams['font.family'] = 'DejaVu Sans'
TEXT = '#2b2b2b'

# A plausible next-token distribution for:
# "She poured water from the pitcher into the glass until it was ___"
tokens = ['full', 'filled', 'overflowing', 'nearly', 'empty', 'yellow', 'potato']
probs  = [0.54,   0.17,     0.1,          0.065,     0.03,    0.01,  0.001]

BLUE, ORANGE = '#A9CCE3', '#F5CBA7'          # sensible words vs. the nonsense one
colors = [BLUE]*(len(tokens)-1) + [ORANGE]

fig, ax = plt.subplots(figsize=(8, 4.2))
y = range(len(tokens))
ax.barh(y, probs, color=colors, edgecolor='#5b5b5b', linewidth=0.6)
ax.set_yticks(list(y)); ax.set_yticklabels(tokens, fontsize=12, color=TEXT)
ax.invert_yaxis()

for i, p in enumerate(probs):
    ax.text(p + 0.006, i, f'{p:.2f}' if p >= 0.01 else '≈ 0',
            va='center', fontsize=10.5, color=TEXT)

ax.set_xlim(0, 0.52)
ax.set_xlabel('probability the model assigns to this next token', fontsize=11, color=TEXT)
for s in ['top', 'right']: ax.spines[s].set_visible(False)
for s in ['left', 'bottom']: ax.spines[s].set_color('#bdbdbd')
ax.tick_params(colors=TEXT)
ax.grid(axis='x', color='#e8e8e8', linewidth=0.8); ax.set_axisbelow(True)
plt.tight_layout()
plt.show()

Since it’s a probability distribution, the values must sum to 1 (over the whole vocabulary - here assume we only show the top choices). This means we’ll have words that are very likely, words that are less likely, and words that are very unlikely.

5.3.2 Sampling the Distribution

A good token distribution is likely peaked, but not spiked. Here, “full” is the clear favorite, but there are other genuinely reasonable words that carry a decent amount of probability as well. Only truly nonsensical things should sit at essentially zero.

The simplest way to turn a distribution like this into a literal choice of next token would be to just take the most likely token every time. This is called greedy decoding. You’d simply do:

next_token_logits = logits[-1]                 # last row only
next_token = torch.argmax(next_token_logits)   # the single most likely token

But, importantly, greedy decoding isn’t the only option. We could also do sampling, which works well precisely because we expect to get a good distribution out. If the probability mass sits on reasonable words, we can afford to not always choose the single most likely thing, and the result will still make sense. And indeed, this is more in line with how language works. In the above example, both “full” and “filled” are completely reasonable things to say - they don’t even change the meaning. Both are a good choice, so sampling sometimes one, sometimes the other, is a genuine reflection of how language can be used in many different ways, even still to convey the exact same meaning.

If we do our sampling in proportion to the probabilities, we’ll usually get “full”, but sometimes get “filled” or “overflowing”. We will still basically never get “potato”. That’s how we give generated text variety without letting it stray too far or become incoherent. A good distribution is what makes non-greedy decoding safe. To sample the distribution, we’d instead do something like:

# sampling instead of greedy
probs = torch.softmax(logits[-1], dim=-1)
next_token = torch.multinomial(probs, num_samples=1)

This is what the temperature that you’ve probably heard of refers to. It scales the logits before the softmax, so that lower temperature sharpens the distribution toward the top token and higher temperature flattens it to allow more variety. Then when you sample, you’re more or less likely to spread out from the “best” next token. We won’t heavily tune or explore temperature here, but understand that this is exactly where that knob comes into play.

One more decoding strategy worth knowing, because it’s what the original paper actually used, is beam search. Greedy decoding and sampling both commit to one token at a time, which can be short-sighted. The single most likely next token can lead into a sentence that ends up less likely overall than one that started with the runner-up. So in beam search, instead of keeping one sequence, you keep the top \(k\) partial sequences (the “beam”). At each step, you try extending every one of them by every possible next token, then score each candidate by its total log-probability so far, and keeps only the best \(k\) to continue.

When it’s done, it returns the best complete sequence. Though, since every token you add brings in another negative log-probability, this scoring naturally favors shorter outputs. To fix this, a length penalty is usually added to even that out. Beam search works well for tasks like translation, where there’s essentially one right answer and you want the most likely one. But it’s rarely used for open-ended generation, like chatbots, because chasing the single most probable sequence tends to produce bland, repetitive text, and that’s exactly where the variety from sampling helps.

So there are many options, and the best one depends on the task. For the simple date translation task we’ll have, where there’s only one correct output and the model will be very confident, plain greedy decoding is a great choice, and the one we’ll go with.

5.4 The Input-Output Connection

Before we talk about what the full assembly of all of this looks like, there’s a couple more things to be clear about in how outputs become inputs, which is the loop that the decoder is constantly running. What does the output feeding back into the decoder actually look like?

5.4.1 What to do With New Tokens

So, we’ve chosen the next token. Now, we stick it onto the sequence. But there’s a few more nuances to talk about before this loop is really fully complete.

First, let’s talk about what we do with a token. That final output gives us a token ID - a single integer which is the index of the winning word in the vocabulary. The decoder, of course, can’t just take a token ID. Like every other transformer block, it expects embeddings. So before this new token can re-enter the loop, it has to become a vector again.

We do this, of course, by re-embedding it. The key thing here is that we do it with the same input embedding layer we used for every other token. The chosen token ID goes through the full process that any of our original “input” ids went through: the token-embedding lookup, adding the positional encoding for its position, and becoming a \(d_\text{model}\) vector. Then, we can append it as the new final row of the decoder’s input. Then the whole (now one longer) sequence can run through the stack again.

So every new token that’s generated by the decoder is generated from just a regular input embedding of the whole sequence so far - we don’t keep anything from the decoder stack to “inform” that token in any kind of a special way. (Conceptually, at least - a reader coming off Appendix 4 will note that we do store some information for computation’s sake, but in a way that’s computationally equivalent to a fresh pass.) So at least from an information standpoint, the pass that makes the next token is an independent pass through the full decoder.

Putting it all together now, understanding the job of the output, we see a complete view of how this comes together into a pipeline:

arch-expanded-complete

5.4.2 The Relation Between Input and Output Layers

One more thing, is that there’s a symmetry between the input and output layers that we haven’t fully appreciated yet. Look at the two ends of the model:

  • the input embedding takes a token ID and produces a vector,
  • the output projection takes a vector and produces scores over token IDs.

They’re mirror images. Where one’s job is to turn a word into a vector, the other’s is to turn a vector back into a word. And in fact, because a word should mean the same thing whether it’s going in through the embedding or being chosen on the way out, it makes sense for the two to be sort of “opposites” of one another.

In fact, they’re so closely related that many models explicitly tie them - the original Transformer included. Tying then just means the output projection literally is the embedding matrix, transposed: the logit for each token is just the dot product between the decoder’s final vector and that token’s embedding. In other words, the model predicts whichever token’s embedding points most nearly the same direction as what it computed. This takes advantage of the same “direction means meaning” idea from our word embeddings. The main benefit is efficiency: the embedding matrix of a 100k-token vocabulary is huge, and tying means storing it once. But the two jobs aren’t perfectly identical, and many large modern models opt keep them separate to allow them to be as different as they need be. For the tiny vocabulary we’ll have for our task and model in this notebook, the difference is negligible, so we’ll keep them as two separate layers.

6. Assembling the Full Transformer

We’ve now built every individual piece of the core transformer architecture:

  1. the input pipeline that turns text into vectors
  2. the encoder with its all-knowing multi-head attention
  3. the decoder with its masked attention and cross attention to the encoder
  4. the output that turns a vector back into a token

Each component, on its own, we understand in detail. And even (most of) how they come together. But of course, when assembling all of this into a full, final pipeline, there are a few extra things to consider, in how exactly those arrows between these segments are passing information.

So in this section, we’ll talk about those first. And then go ahead and proceed with our complete assembled transformer, that we can actually try to train.

6.1 The Input to the Encoder vs. to the Decoder

Hopefully by now, you’ve come to understand why our initial, simplified architecture is actually probably better drawn like this:

arch-simple-v2

And there’s something that we need to discuss a little bit more before we go on, and that’s what I’ve labeled in this diagram as Input 2.

In the last 2 sections, when we were talking about how we append the next token back to the input and feed it through again, we talked about how that means re-embedding it and adding back its positional encoding. But we didn’t answer the question: embedding it with what embedding layer?

Well, we already have the input pipeline for the encoder. Except it’s not necessarily true that we can re-use that same pipeline for our decoder step. And actually, in many cases, like the French-to-English example we keep referencing, we can’t.

The reason and decision really comes down to if your input and output share the same vocabulary. In a case like French-to-English translation, the encoder needs to understand the French input sequence, while the decoder needs to understand a growing English sequence. The languages share letters, sure, but tokenization as we’ve come to understand it will not apply to both in the same way. Nor would the same tokens come to mean the same thing, even if we could share them (“chat” in French vs “chat” in English mean very different things).

But nothing really requires it be separate, either. Imagine the task of summarization: the input is a long article and the output is a short summary of it, both in English. The two are completely different sequences, but they draw from the same vocabulary. An encoder-decoder is a great fit for the task, where the encoder reads and fully understands the whole article, and the decoder generates the summary conditioned on that understanding. But since the source and target share a vocabulary, the input pipeline that prepares text for the encoder can be re-used for the decoder just fine.

In fact, in the small task we’re about to train on (translating a date from one format into another), we will re-use the exact same input. Our input and output will both be made of digits, letters, and some common symbols, so they’ll share a single vocabulary. That means that when we assemble the model, we can use one embedding for both the input into the encoder and the input into the decoder, rather than the two separate ones like a French-to-English model would need.

6.2 The Other Mask

Back when we first wrote our DecoderBlock implementation, we gave it two different mask arguments in the forward call - the causal trg_mask for masked self-attention, and a src_mask that we said for cross-attention but otherwise left almost entirely unexplained. All we said was that it was “a mask we sometimes need to hide padding,” and that we’d come back to it later. Well, now is later.

6.2.1 When to Use the src_mask

The reason that we waited until now to talk about it, is that this is another one of those training time / data organization nuances, not really a critical feature of the architecture. If you only ever ran through one sequence at a time, as we’ve mostly thought of, we wouldn’t need it at all. But in practice, especially during training time, we process many sequences at once - as a batch.

Of course, a tensor has to be rectangular, meaning every sequence in the batch must be the same length. In basically any real application, that won’t be the case. There are specific cases where it wouldn’t be necessary at all, like if we were translating dates in the reverse from what we’ll actually do - from a fixed format (like 2026-06-05) to their full format (June 5th, 2026). If all input sequences were always exactly the same length, we’d never need a src_mask, so it is situational. It’s just that in most real situations, inputs will have mis-matched lengths. And to fix mis-matched lengths, we pad the shorter ones, tacking on a special <pad> token, over and over, until each sequence reaches the length of the longest in the batch.

But padding tokens aren’t tokens we want to affect anything - i.e, we don’t want attention on them. So we need to ignore them, much like we do in masked self-attention, with a mask of their own.

So the mask mechanism is the same, the one we already know well from the triangular mask in masked self-attention.

  • We mark which positions are padding and set their attention scores to −∞
  • We do it before the softmax so they get zero weight

The difference is just how we construct it.

  • the causal mask, because it depends only on sequence length, is baked into the architecture. The decoder must act autoregressively no matter what data you feed it.
  • the padding mask, because it depends entirely on the data (which positions happened to be padding in this batch), has to be constructed for the specific batch.

This padding mask is still passed into the MultiHeadAttention calculation. But it’s coming from the source sequence that gets padded on the encoder side, so the source padding mask (src_mask) is needed anywhere the source is attended to. That means that we need it in:

  • the encoder’s self-attention to make sure source tokens don’t attend to source padding
  • the decoder’s cross-attention to make sure the decoder doesn’t attend to source padding that was used in the encoder.

You might wonder: we batch the target sequences too, so they get padded as well. Won’t that mean the decoder’s self-attention needs to mask that target padding too?

And fortunately, it’s already handled for free by the causal mask. Since we always pad at the end of a sequence, the padding is always “in the future.” And the causal mask already handles masking out the future. So a real target token never reaches the padding. The causal mask, which we added for a totally different reason (autoregression), also happens to totally take care of this issue.

(One thing, the pad positions themselves will still compute outputs, just junk, since they do attend back to real tokens. But when it comes time to calculate the loss, we can just ignore those positions. We can let them generate and be whatever, we know we aren’t going to actually use them in the end anyway.)

6.2.2 Seeing a Real Example

We’ve said that the mask operates just the same as the causal mask, it’s just not triangular. But it’s the exact same idea: just 1’s and 0’s for “attention is allowed” vs “attention is not allowed”. And constructing it is super simple: all we need to know is what we padded.

Say a batch has two source sequences of different lengths:

  • “le chat dort” → [le, chat, dort] (3 tokens)
  • “il pleut” → [il, pleut] (2 tokens)

To batch them, we pad the shorter up to length 3:

[ le,  chat,  dort  ]
[ il,  pleut, <pad> ]

Then the src_mask just records which positions are real (1), and which are padding (0):

[ 1, 1, 1 ]
[ 1, 1, 0 ]

After that, all of the same advantages of masking apply and are handled exactly the same as the causal mask.

This is also why, though we’d need it for differing sequence lengths in a batch, we can’t make it ahead of time. We only know what our batches will look like during training. And for that reason, a padding mask is also just a training-time necessity. At generation time, running one sequence, there’s nothing to pad and nothing to mask.

But actually building it requires knowing exactly what the data looks like, so we’ll revisit the concept one more time when we get to actually setting up training. The example task that we’re going to use will be training on a date-format translation task, except unlike the exception I pointed out above, our inputs will be variable lengths, and so we will need to handle this padding. But more on that when we get there!

6.3 The Full Model in Code

Ok, done with the few extra nuances, it’s time to write the thing! We have already built many of these parts as we’ve gone along. Actual assembly is now mostly just plumbing, writing four classes that each handle a major component, arranging all of the pieces to come together into a real working model. Those major parts will be:

  1. an Input - this will cover the source input pipeline (embedding + positional encoding)
  2. an Encoder - will first use the input, followed by N stacked EncoderBlocks,
  3. a Decoder - this will include the same input pipeline, followed by N stacked DecoderBlocks,
  4. a Transformer will put the two pieces together, and handle the few extra outstanding things. It will be the wrapper that holds both encoder and decoder, handles building the masks, runs the source through the encoder and the target through the decoder, and applies the output projection.

We’ll build them in that order.

6.3.1 The Input

First, we create an InputEmbedding layer as its own thing. This is the general case - we can either call this same layer in both the Encoder and Decoder parts to have them explicitly share their embedding layer, OR we could call two separate instances of this if we didn’t want them to share.

In our case, with the shared vocabulary, we want to use this same input twice. But in the French-English translation example, you would make two separate instances of this. It’s all up to what task your architecture is meant for.

The code itself will reuse all of the pieces we already built back in Section 1. So this code is just a small wrapper that takes token IDs, looks up their embeddings, adds positional encodings, and applies dropout.

class InputEmbedding(nn.Module):
    """The 'Input' block from the diagram: token IDs -> vectors + positional encodings."""
    def __init__(self, vocab_size, d_model, max_len, dropout=0.1):
        super(InputEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        pe = torch.tensor(get_positional_encodings(max_len, d_model), dtype=torch.float)
        self.register_buffer("pos_encoding", pe) # store the positional encodings
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):                       # x: token IDs, (N, seq_len)
        seq_len = x.shape[1]
        x = self.embedding(x) + self.pos_encoding[:seq_len] # add the fixed PE to the embedding
        return self.dropout(x) # dropout at the END - right before the next layer

A few things to point out here:

  • The positional-encoding function is implemented here with the get_positional_encodings that we wrote before, but stored with register_buffer. This makes it a part of the module - moved to the GPU and stored there alongside it without being a learnable parameter, since sinusoidal positional encodings have nothing to learn.

  • nn.Embedding is a torch built-in layer, like the nn.Linear we discussed before. This special kind of layer is exactly the lookup table we described back when we first talked about embeddings. It contains a learnable weight matrix of shape \(n_{\text{vocab}} \times d_{\text{model}}\), with one row per token in the vocabulary, that returns row \(i\) for a token index \(i\). It does the exact same thing as the “one-hot vector times the embedding matrix” calculation that we talked about in Section 1, just done as a direct row-lookup instead of an actual matrix multiply. The nn.Embedding layer is a special layer that knows to do this instead of the full matrix multiply.

  • nn.Dropout is slipped into here again, just like we did with every encoder and decoder block so far. It might be obvious why dropping random activations helps the model from over-relying on certain features, but dropping parts of an embedding might seem like a stranger choice. But it’s still for the exact same reason. Regularizing here means that the next layer, the first layer of the encoder or decoder, doesn’t get every single feature of the embedding vector, and thus can’t learn to over-rely on certain features of that vector.

6.3.2 The Encoder

Next, we’ll assemble the encoder. But first, actually, we need to rewrite our original EncoderBlock code. At the time that we wrote that code, we didn’t yet understand the need for masking, and so we left it out. Now that we’ve covered the role of the source mask, we need to just make a tiny edit to that code to be sure that it can take in and handle that.

The two line changes are pointed out with comments.

class EncoderBlock(nn.Module):
    def __init__(self, d_model, heads, d_ff, dropout=0.1):
        super(EncoderBlock, self).__init__()
        self.attention = MultiHeadAttention(d_model, heads)
        self.feed_forward = FeedForward(d_model, d_ff)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

    def forward(self, X, mask=None):                                   # <- added mask=None by default
        norm_X = self.norm1(X)
        attention_output = self.attention(norm_X, norm_X, norm_X, mask=mask)  # <- attention is already wired for a mask, but pass it through here
        X_1 = X + self.dropout1(attention_output)

        norm_X_1 = self.norm2(X_1)
        ff_output = self.feed_forward(norm_X_1)
        X_2 = X_1 + self.dropout2(ff_output)
        return X_2

Then, building up the encoder is just a few lines, since it just means stacking encoder blocks, which contain all of the relevant operations, on top of each other.

class Encoder(nn.Module):
    def __init__(self, d_model, num_layers, heads, d_ff, dropout=0.1):
        super(Encoder, self).__init__()
        self.layers = nn.ModuleList(
            [EncoderBlock(d_model, heads, d_ff, dropout) for _ in range(num_layers)]
        )
        self.norm = nn.LayerNorm(d_model)       # <- new addition: final LayerNorm (needed with pre-norm)

    def forward(self, x, mask):                 # x: (N, src_len, d_model), already embedded
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)                     # <- normalize once more on the way out

So as you can see, really all this does is just puts num_layers encoder blocks all in a row. Just a few things to point out:

  • nn.ModuleList is a new piece of PyTorch. It’s meant for cases exactly like stacking our already made encoder modules: it’s really just a list that holds sub-modules, but unlike a regular Python list, everything gets registered with PyTorch so all weights are properly tracked, and will show up in model.parameters(). It just tells PyTorch “here’s a bunch of trainable things”.

  • We do wrap it around a “normal” list, with the list comprehension: python [EncoderBlock(...) for _ in range(num_layers)] simply making num_layers separate EncoderBlock objects. Since each is a distinct object, it’ll have its own \(W_Q, W_K, W_V, W_O\) and \(W_{\text{ff}}\) weights.

  • In forward, the loop is where the “stacked” behavior really comes to fruition. We go through each layer in that list, run x through it, take the x that comes out of it, and put it into the next one. The same mask is handed to every block, because the same batch passes through all blocks to get the final output.

  • Remember here that the first x is going to arrive already embedded. We’ll see when we build the full transformer object.

  • num_layers is the stack depth N, a hyperparameter - the original transformer used 6; we’ll pick something small for our toy task.

  • We snuck in an “extra” nn.LayerNorm. Remember that with pre-norm, each sublayer normalizes its input, but the residual stream that runs through the whole stack never gets normalized itself - it’s just added onto, block after block. So the last block hands back an un-normalized \(x\). This would leave one “missing” at the end - our final encoder output wouldn’t be normalized, if we didn’t add one more norm to the end.

6.3.3 The Decoder

The Decoder structurally matches the encoder almost exactly - a lot of the work already went into building our DecoderBlock object back in the Decoder Section, so now all that’s left is stacking a bunch of them together.

class Decoder(nn.Module):
    def __init__(self, d_model, num_layers, heads, d_ff, dropout=0.1):
        super(Decoder, self).__init__()
        self.layers = nn.ModuleList(
            [DecoderBlock(d_model, heads, d_ff, dropout) for _ in range(num_layers)]
        )
        self.norm = nn.LayerNorm(d_model)

    def forward(self, x, enc_out, src_mask, trg_mask):   # x: (N, trg_len, d_model), already embedded
        for layer in self.layers:
            x = layer(x, enc_out, src_mask, trg_mask)
        return self.norm(x)

So what is different from the encoder?

  • It stacks DecoderBlocks instead of EncoderBlock’s, of course.

  • forward must take four arguments instead of two, for the same reasons that our DecoderBlock’s forward needed 4 arguments. On top of x (the target sequence, already embedded), it also receives:

    • enc_out - the encoder’s output.
    • src_mask - the mask for the encoder output, for handling padding of different sequence lengths, and
    • trg_mask - the causal mask to block the future.

    These masks will get defined in the Transformer object.

  • Just like the encoder, we loop through the layers in the forward definition. Same as the encoder, x changes from one block to the next. But those extra inputs: enc_out, src_mask, and trg_mask stay fixed, handed identically to every block.

  • Also just like the encoder, we end with a nn.LayerNorm on the output, because pre-norm doesn’t normalize the final output as-is.

So structurally, they’re really the same, and the differences just come down to how we had to construct the encoder vs decoder block differently, which we already covered extensively.

6.3.4 The Full Transformer

Now, finally, we really put it all together.

class Transformer(nn.Module):
    def __init__(self, vocab_size, d_model, num_layers, heads, d_ff, max_len,
                 src_pad_idx, trg_pad_idx, dropout=0.1):
        super(Transformer, self).__init__()
        # Define all of the sections we just made
        self.input_embedding = InputEmbedding(vocab_size, d_model, max_len, dropout)
        #ONE Input block, that we'll share since our source and target share a vocabulary. Otherwise, we'd define two here!
        self.encoder = Encoder(d_model, num_layers, heads, d_ff, dropout)
        self.decoder = Decoder(d_model, num_layers, heads, d_ff, dropout)
        self.output_projection = nn.Linear(d_model, vocab_size)   # the "Output" / prediction layer is so simple, it just gets defined right here
        self.src_pad_idx = src_pad_idx # padding for the SOURCE - input
        self.trg_pad_idx = trg_pad_idx # padding for the TARGET - output

    def make_src_mask(self, src):
        # 1 where the token is real, 0 where it's padding
        # shape (N, 1, 1, src_len) so it broadcasts over heads and query positions
        # we get this DIRECTLY from the source padding!
        return (src != self.src_pad_idx).unsqueeze(1).unsqueeze(2)

    def make_trg_mask(self, trg):
        N, trg_len = trg.shape
        # lower-triangular causal mask, shape (N, 1, trg_len, trg_len)
        trg_mask = torch.tril(torch.ones((trg_len, trg_len), device=trg.device))
        return trg_mask.expand(N, 1, trg_len, trg_len)

    def forward(self, src, trg):
        # 1) build the masks
        src_mask = self.make_src_mask(src)       # (N, 1, 1, src_len)
        trg_mask = self.make_trg_mask(trg)       # (N, 1, trg_len, trg_len)

        # 2) embed both inputs, with the SAME embedding object
        src_emb = self.input_embedding(src)      # (N, src_len, d_model) for the input (source)
        trg_emb = self.input_embedding(trg)      # (N, trg_len, d_model) for the output (target)

        # 3) pass through the encoder
        enc_out = self.encoder(src_emb, src_mask)                     # X_enc: (N, src_len, d_model)

        # 4) pass through the decoder
        dec_out = self.decoder(trg_emb, enc_out, src_mask, trg_mask)  # X_out: (N, trg_len, d_model)

        logits = self.output_projection(dec_out)  # (N, trg_len, vocab_size)
        return logits

All things considered, it’s relatively brief. And of course that’s one of the big advantages to building up these building blocks as their own objects. Now that we have a full encoder, we just have to say “go through the encoder” - we can look to that specific object for the encoder architecture.

The forward pass is the entire architecture - the full flow of an input through the entire process, so let’s make sure we understand every line there to see how everything we’ve talked about finally comes together.

📌 Note: Recall in the above code, and what we’re about to discuss that \(N\) is referring to batch size. Where before we’ve talked about “N” layers in the encoder/decoder stacks, in code, that has become num_layers to be more explicit.

The main components to this forward pass are:

  1. The first thing we do is build the masks.
  • make_trg_mask is the causal torch.tril which simply constructs a triangular mask from the target length. Remember that this kind of triangular mask, and the size it needs to be, is just exactly determined by what the current sequence length is, so this needs no other information than “what is the current target sequence”.
  • make_src_mask is the padding mask we make from the source. Remember that, similarly, this mask just needs to assign a 1 where a real token was placed, and 0 where a padding token was placed. So this also needs no extra information other than what the current source is - it marks which source positions are real (src != pad_idx), and it’s shaped to broadcast across heads and query positions so a single mask covers the whole score matrix. All that we’ll need to do is tell it what the padding token looks like, what its ID is, which we’ll pin down when we get to actually passing in data.
  1. Then, we embed the sequences with the same InputEmbedding object. Both self.input_embedding(src) and self.input_embedding(trg) call the same object, because as we discussed, in this case, the vocabulary is shared - with tokens always meaning the same thing on both sides. After embedding, both come out as (N, seq_len, d_model).
  2. We run the full encoder first. It gives us enc_out, which is what we’ve been calling X_enc - the fixed, fully contextualized output from the encoder, that we hold fixed after the forward pass and use in the decoder. Its shape coming out is (N, src_len, d_model). It’s computed once, then handed to the decoder in the next step.
  3. The decoder runs after, using that enc_out. This gives us dec_out, which is what we’ve been calling X_out, the contextualized target representation that we use to get the next token. It has shape (N, trg_len, d_model).
  4. The output projection picks the next token. Finally, we give that X_out to the output layer, which projects each position from d_model up to vocab_size, giving logits of shape (N, trg_len, vocab_size)

Then, even though we aren’t quite ready to train yet, we can put this together, and make sure that it all runs forward, giving back the correct shape, at least with dummy numbers:

# a small toy config
vocab_size = 15          # something small - e.g. digits, '-', eos, bos, pad, unk
d_model, num_layers, heads, d_ff, max_len = 64, 3, 4, 256, 32
src_pad_idx = trg_pad_idx = 0

model = Transformer(vocab_size, d_model, num_layers, heads, d_ff, max_len,
                    src_pad_idx, trg_pad_idx)

N, src_len, trg_len = 2, 18, 11       # batch of 2; a padded source, a target
src = torch.randint(1, vocab_size, (N, src_len))   # fake token IDs
trg = torch.randint(1, vocab_size, (N, trg_len))

logits = model(src, trg)
print("src:   ", tuple(src.shape))
print("trg:   ", tuple(trg.shape))
print("logits:", tuple(logits.shape))
src:    (2, 18)
trg:    (2, 11)
logits: (2, 11, 15)

7. Training the Transformer

So we’ve done the full assembling - we literally have the Transformer defined. But looking at the forward, a few things probably feel conspicuously absent. Some of these are things we’ll cover right now: how to make sure that our targets really only learn from positions they should be able to see, what signal actually determines if we’re getting the answer right or wrong, the actual learning of the weights, and how to do it all when data comes in as a batch.

Other things that we had discussed - like the autoregressive loop that produces one token at a time, appends it, and feeds it back in - will come down to how we use the model, after it’s trained. We’ll cover those as their own set of topics, in Section 9, once we’ve trained this model on a real task in Section 8.

So this Section is just all about the theory and practicalities of training. We have a full Transformer architecture, we can pass in a tensor, it’ll get multiplied by all of the weights, and we get a set of logits back out. How, then, do we set it up with data so those weights shift to produce correct outputs? How do we measure correctness? How do we know how much to change the model by? These are all training time considerations.

7.1 Teacher Forcing

The first important concept that we need because things are different at training time vs. at generation time is something called teacher forcing. Basically, it’s how we mimic autoregressive generation - making just one token at a time, conditioned by what comes before - when during training, we actually already have the whole answer.

7.1.1 The Core Idea

Maybe the most obvious way to train a model that wants to run autoregressively is to train it autoregressively: just give it the input, have it generate token 1, feed that back in to generate token 2, and so on, and every time you look at the correct answer and ask if it was right or wrong.

That would work, but it’s a bad idea and not done in practice for two huge reasons:

  1. Sequential generation like that is painfully slow. Generating the output one step at a time means doing an entire forward pass, through the whole big decoder, with a whole new pass per token. You can’t parallelize, run through whole batches at once, or take real good advantage of knowing what the whole sequence should be ahead of time.
  2. Early in training, the model is terrible. If you rely on what the model generates at step \(i\) to determine step \(i+1\), it’s likely to be wrong at \(i+1\) if it was wrong at \(i\). It will compound its own errors over the sequence and likely just produce garbage, especially early on. But as those errors compound, the model can struggle to even get started on actually good trajectories.

Teacher forcing fixes both. And the core idea is to exploit the feature that we’ve already mentioned here a few times: during training, you have an entire correct sequence, and you know what the next token should be given the truth so far at every step. Teacher forcing just does exactly this - instead of running the “real” autoregressive loop, where the model gets its own previously generated tokens back, we only ever give it the ground-truth for a given position. We say “what would you generate for j…. given this is what {a….i} actually was”. The teacher “forces” the correct history at every step.

And remember, this is actually the whole reason we built the causal mask! As we discussed in Section 4 and then proved in Appendix 4, causal masking keeps previous tokens in the sequence frozen, not allowing them to look ahead. We did this so that the current position we’re looking at is the only one that has “full access” to see everything. But if we trained autoregressively, where we pick off the last row to predict the next token anyway, then who cares if a previous position’s embedding changes - we’re only looking at the most recent one anyway. It’s precisely because we want to feed the model the entire target sequence at once, and see how every token position does with the true prefix, that we need the causal mask - it allows us to predict the next token at every position at once in a single parallel forward pass.

(I encourage you to visit Appendix 4 if you haven’t already and need to see this play out with the exact matrix math. We work out there that, with masked self attention, for a sequence of length 2 vs. length 3, the rows that correspond to the 1 and 2 position don’t change. So they don’t change for an n-length sequence, either. Masked attention lets us put in the whole sequence at once, and still get a prediction “as if” the tokens ahead didn’t exist)

7.1.2 How it’s Done

Ok, so conceptually, it’s simple. And the masked self attention almost gives it to us without any extra effort. Mechanically, the one thing that we have to add is just a one-position shift.

Say the full target we want is:

\[\text{target} = [\texttt{<bos>}, \,y_1,\ y_2,\ \dots,\ y_n,\ \texttt{<eos>}\,]\]

Then to see if the decoder actually produces that, we need to give that sequence as input. Except because there’s nothing to predict at the end of the sequence, we drop the <eos>:

\[\text{decoder input} = [\,\texttt{<bos>},\ y_1,\ y_2,\ \dots,\ y_n\,]\]

And then the label, the thing we actually want to predict, is simply the target sentence, shifted one to the left with the start <bos> removed:

\[\text{truth label} = [\,y_1,\ y_2,\ \dots,\ y_n,\ \texttt{<eos>}\,]\]

Think about what this gives us and why this works. When we pass an input into the decoder, we get, as each row of the output, a prediction of what the next token will be.

\[\text{input} = [\texttt{<bos>},\ y_1] \\ \\ X_{\text{out}} = \begin{bmatrix} X_{\text{out, 1}} \\ X_{\text{out, 2}} \end{bmatrix} \begin{matrix} \leftarrow \text{prediction for} \ y_1 \\ \leftarrow \text{prediction for} \ y_2 \end{matrix} \]

Then for a given input, what we want out is the next token for every token of that input. It’s why we just need to do that single shift to get labels. Both are constructed from the truth, but the relative shift means then we get to grade what the input says will come next.

This is also why the causal mask gets to be so simple - since we pass in the whole sequence like this, the causal mask is just sized to remove the later tokens, decreasing what’s blocked one-by-one. That means exactly the triangular mask we talked about.

In code, trg is the full target sequence. Then the actual input to the decoder is just the target with the last token removed. The label, as we said, is just the target, shifted by one after removal of the <bos> token:

decoder_input = trg[:, :-1]   # everything except the last token
labels        = trg[:, 1:]    # everything except the first (<bos>)

7.1.3 An Aside: Exposure Bias

There is one more thing to mention here, just quickly before we move on. Teacher forcing means that the model always trains on correct prefixes. At each step, all of the tokens coming before are from the ground truth. So the model only makes mistakes for one token at a time - we can correct it towards the right token, but only given it messes up based on something previously correct.

Of course, at generation time, the model will have to build on its own outputs. But these can be wrong, meaning mistakes can compound - and each one can lead to the whole downstream trajectory of the generated sequence being wrong.

This mismatch is called exposure bias, and it’s a known limitation of teacher forcing. Basically, the model can become out of distribution - in a state that it doesn’t know what to do with, that it’s never seen before, just because of a mistake it made earlier on, that it never would’ve encountered during training. That’s where the name comes from: the model was only ever exposed to clean, correct histories, so the distribution of prefixes it trained on is biased away from the messier, sometimes-wrong ones it actually faces when generating.

In practice, this is often tolerable and a fairly small effect, given you train with enough data to generalize and are starting with a capable model. If so, the model will likely mostly produce its own good prefixes anyway, and errors will rarely cascade catastrophically. So teacher forcing is still the standard approach, and the one we’ll use here. But just know that this is a potential concern, and indeed, one that people in the field are thinking about. (There are some tricks that deliberately feed the model some of its own predictions during training to close the gap, like scheduled sampling, but we will not cover that here!)

7.2 The Loss

So, we use teacher forcing to generate our outputs all at once:

  • run the decoder_input through the model
  • get logits of shape (N, trg_len, vocab_size), which contains a score for every vocabulary token at every position
  • look at labels, which has shape (N, trg_len), and contains the single correct token ID at each position

Then, we just need to compare logits to labels.

Compare…. how? That, of course, is the job of the loss function - it quantifies just how wrong the model was compared to what we wanted it to produce, in a way that we can optimize for, to try to make it smaller next time.

7.2.1 Choosing a Loss Function

Before we talk about the specific one that we use, let’s take a minute to remind ourselves what a loss function is in general, and what it needs to do.

A loss function’s core job is to take the model’s output and the correct answer and returns a single number that says how wrong it was. Even though that might seem like a relatively simple task, it’s the thing that ends up determining all learning. Training the model is just literally tweaking the model’s weights to make the loss smaller.

So while its role might be pretty simple, that doesn’t mean the loss function itself can just be anything. It needs to satisfy some properties:

  1. It must produce a single scalar. That is a core property. Whatever the model outputs - a whole grid of logits in the case we’re discussing here, the loss needs to boil down to one number that gives an unambiguous notion of the current model’s “quality”.
  2. It needs to be lower when the model is better. The loss should be such that it’s smallest when the prediction matches the truth and larger as the prediction gets more wrong. Training’s job is to minimize it, so “small loss” has to mean “good model.”
  3. It has to be differentiable. This is where we start to get into the weeds of the math, which, for all we’ve done here, is going to be beyond the scope of this notebook. But training only works by computing the gradient of the loss with respect to every weight, so that we can determine how the weights need to move to make the loss smaller. That means the loss must be a smooth function of the model’s outputs.
  4. It should be graded, not just right/wrong. A good loss which gives good training signal should reflect how wrong the model is, not merely whether it’s wrong. This is crucial for stable learning and for our gradients to lead to meaningful improvement.

Those last two points are precisely why we don’t just train on pure accuracy, literally if the output is right or wrong. Accuracy is a step function: 1 if right, 0 if wrong, and flat everywhere in between. Its gradient is zero almost everywhere, so it can’t tell you how to improve a wrong answer. Instead, we choose a smooth, graded stand-in that rewards getting closer to the right answer even before it’s exactly right.

7.2.2 The Cross Entropy

The standard choice of loss function in language modeling tasks is the cross-entropy, which you may actually know as the standard choice in classification tasks. It’s powerful, but incredibly simple, and we’ll see why it’s the perfect choice here.

Consider what the model actually gives us as its output. For every single token that we put into the decoder, we get a score of likelihood for every token in the vocabulary, that that token is the one that comes next.

\[ X_{\text{out}} = \begin{bmatrix} X_{\text{out, 1}} \\ X_{\text{out, 2}} \end{bmatrix} \begin{matrix} \leftarrow \text{prediction for} \ i \\ \leftarrow \text{prediction for} \ i+1 \end{matrix} \]

\[ \text{output}(X_{\text{out, 1}}) = \begin{bmatrix} p_1 \ \quad \quad p_2 \ ... \ \quad p_n \end{bmatrix} \\ \quad \quad \quad \quad \quad \quad\begin{matrix} ⇑ \ \quad \quad ⇑ \ \quad \quad ⇑ \end{matrix} \\ \quad \quad \quad \quad \quad \quad \begin{matrix} \text{prob}(1) \ \ \ \text{prob}(2) \ \ \ \text{prob}(n)\end{matrix} \]

We get a probability, for every token in the vocabulary. But we also have exactly one correct answer. So it really is a classification problem in the end, for which the standard loss is cross-entropy. The reason that cross-entropy is so good for a case like that, is that we can also think of the correct answer as a probability distribution with all of its mass on the correct token, then comparing the distributions comes down to answering the question: “how much does the probability mass you gave to the correct answer match the correct probability mass for that answer - 1?”

So cross-entropy for any position works like this. First, take the output for that position coming out of the decoder, and pass it through the output layer to get the probability distribution over the whole vocabulary. Then, the loss is just the negative log of the probability the model assigned to the correct token:

\[\text{loss} = -\log\, p(\text{correct token})\]

The “negative log” part of this does some real work.

  • If the model gives the correct token a probability of 1, perfectly correct, then \(-\log(1) = 0\) - no loss, nothing to fix.
  • If the model gives the correct token a probability near 0, then \(-\log(0) \to\) \(\infty\) (in practice, we don’t let it be literally 0 because of this), and you get a massive penalty for being confidently wrong
  • If the model is in a reasonable range, the loss is a nice, reasonable range too.

The log helps punish being confidently wrong more severely. The difference between predicting 0.8 and 0.9 for the correct token is: \[ -\log(0.9) + \log (0.8) = -0.118 \] a small amount more loss for the slightly less confidence in the right answer. But the difference between predicting 0.2 and 0.3 for the correct token is: \[ -\log(0.3) + \log (0.2) = -0.405 \] The same lost probability on the right answer costs far more when you were already wrong. The \(−log\) ensures that: the penalty accelerates as the correct answer’s probability heads toward zero.

This is also of course, evident in the very shape of the function, which we can see here:

p = np.linspace(0.02, 1.0, 500)
loss = -np.log(p)

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(p, loss, color=LINE, linewidth=2.6)

ax.set_xlim(0, 1.02); ax.set_ylim(0, 4.2)
ax.set_xlabel("probability on correct token", fontsize=12, color=TEXT)
ax.set_ylabel("loss   =   \u2212log(p)", fontsize=12, color=TEXT)
ax.tick_params(colors=TEXT); ax.grid(color='#ececec', linewidth=0.8); ax.set_axisbelow(True)
plt.tight_layout(); plt.show()

So, then to turn each of these predictions into total loss over all of our data, we simply compute this:

  • at every position,
  • for every example in the batch,

and average them into a single scalar to summarize how wrong the model was across the whole batch.

The last thing that we need to do though, is just make sure that we ignore the padding. We talked about this during the decoder section, but when we batch, all of our sequences need to get padded to deal with the differing sequence lengths. Those pad positions mark those positions with meaningless filler. But as we also discussed in Section 6.2, the easiest thing to do is still let the model produce logits for them, and then just be sure to ignore them later. Well here, in the loss calculation, is the “later” where you need to ignore them.

It’s easier to do here - we just tell the loss to skip them. And in code, PyTorch’s built-in nn.CrossEntropyLoss handles that for us. We pass ignore_index, which just says what the indices were that we padded on the target sequences, and it keeps them out of the loss calculation:

loss_fn = nn.CrossEntropyLoss(ignore_index=pad_idx)

# logits: (N, trg_len, vocab_size)  ->  (N * trg_len, vocab_size)
# labels: (N, trg_len)              ->  (N * trg_len,)
loss = loss_fn(logits.reshape(-1, vocab_size), labels.reshape(-1))

The things to note here about CrossEntropyLoss:

  1. It expects a 2D (num_items, vocab_size) tensor of logits and a 1D (num_items,) tensor of labels, so we flatten the batch and position axes together. Every (example, position) pair becomes one classification item.
  2. It takes raw logits, not softmaxed probabilities, because it applies the log-softmax internally. So we actually just feed it the output of output_projection directly. Our built Transformer already outputs logits, so this is just a step we need to be sure to avoid, not something extra we’ll need to do
  3. ignore_index=pad_idx drops every item whose label is the pad token, so they contribute nothing to the average. We simply remember what we padded, and pass it in!

7.2.3 A Different Loss?

Cross-entropy is the natural fit here because of what our output is: at each position the model produces a distribution over a discrete vocabulary, with exactly one correct token, which as we’ve already discussed is a classification problem. Cross-entropy is the canonical loss for that, and for next-token language modeling it’s nearly universal. And it isn’t an arbitrary pick, either. Minimizing cross-entropy is the same thing as maximizing the probability the model assigns to the true data, so it forces the correct tokens to be as likely as possible, which means it’s exactly the behavior we want.

But, it’s worth asking: is there another loss that could be a good fit? Is there ever a time we’d want a different one? The answer is really just that the right loss depends on the task, and specifically, on what the model outputs.

  • Classification means the model chooses one of K discrete options - exactly our case here, and the general case for language modeling. Cross-entropy is a good, universal choice.
  • Regression, though, would mean the model predicts a continuous value (a price, a coordinate, a temperature, a pixel value), and there, you’d have to choose a different loss, maybe something like MSE (mean squared error), the squared distance to the true value.

So while everything we’ve talked about here has focused on language specifically, the basic transformer architecture is not required to output probabilities over tokens. And in a case where your transformer produces a continuous values per position instead, you’d choose an entirely different loss.

All of that being said, though cross-entropy is the standard choice, we didn’t strictly have to pick it. Remember, all we had to make sure of was that the loss satisfied the key rules we laid out above.

So even within next-token prediction, what else could you choose? Well, actually in the original Transformer, they used label smoothing, where instead of asking the model to put all the probability on the single correct token, you actually do build a probability distribution for the correct answer that has most, but not all of its mass on the correct token. Compared to what we just did, it might sound counterproductive, but it can keep the model from becoming overconfident and help it generalize better.

Of course, some setups change the objective entirely, for instance if you train your model with reinforcement learning which brings its own, very different loss. But that would be WAY too much to dig into here, so we’ll leave it at that.

The critical point is that the loss is a flexible thing to choose, but should always be chosen in a way that genuinely encodes what “good” means, as best as it can, for the specific task. Cross-entropy is a good choice here, and usually a good choice for language modeling, so it’s definitely the one to know for this task.

7.3 The Optimizer

So, we have a loss function, and we use it to calculate how “wrong” the model is in its current state. We glossed over it, but we know that the loss is something that’s differentiable such that it produces gradients that tell us how to tweak every weight. The last thing is to decide how much to tweak by.

The loss does tell us this - the gradients have values which are larger when the model needs to change more and smaller when it needs to change less. But in learning, stability is of critical importance. If we just listened to how much the loss told us to change the weights by, for some weights that might be HUGE and swing the model to a very different place, sometimes tiny and barely change at all. It’s too much burden on just the loss to decide all of that. So instead, we define an optimizer, which tells you what to do with the signal coming from the loss, to update in the right direction, but keep things more stable.

7.3.1 Weight Updates

The loss you get, while calculated from comparing the model’s produced answer to the truth, in reality depends on every weight in the model, since those all work to produce the output. In a real model, that can be billions of individual numbers.

For each one, we can compute a gradient of the loss with respect to that weight. The gradient for every weight points uphill, toward higher loss. So to “correct” that weight in a way that reduces the loss, you take a step the opposite way - downhill in slope - for every weight at once:

\[\text{weight} \leftarrow \text{weight} - (\text{its gradient})\]

If you keep doing this every time you see data, the loss will trend downward. This is gradient descent, the engine under essentially all neural-network training.

(Computing all those gradients efficiently, in one backward sweep through the whole network, is backpropagation. PyTorch handles this for us, so we won’t cover it in any depth here. But if you want the full mechanics, see my Basic Neural Network tutorial. The exact same principles apply to the transformer, so we just need to understand at a high-level here that that’s what happens.)

7.3.2 The Learning Rate

The thing about gradients though, is that they can be very large. If you just subtract the gradient, you might take a huge step downhill for that weight - maybe too far of a step. So in reality, what we do is set a learning rate, which just lessens the amount that we update by:

\[\text{weight} \leftarrow \text{weight} - (\text{learning rate}) \times (\text{its gradient})\]

The learning rate is really just a step size. The direction of update, and even the relative strength, is still coming from the gradient, which directly comes from the loss. But the learning rate lets you keep updates more gentle. It’s just one number, but because it literally determines how much the model can update, it’s one of the most important knobs in all of training:

  • If you make it too large, you can overshoot the corrections every time. The loss can bounce around trying to fix itself with huge updates, sometimes even diverging (the loss explodes).
  • But, if you make it too small, training can slow to a halt - effectively barely changing the model at all.

Just setting a learning rate, and letting the model go based off of that is “plain” gradient descent. It’s a little crude, but it will work ok if you get the learning rate right. In practice though, optimizing weights to reduce a loss is a hard problem, especially when the model is massive. So an optimizer does a few extra things that modulate this learning rate and help the model learn better over its whole training trajectory.

7.3.3 The Optimizer: Adam

For a large model with many weights, the optimizer is just an algorithm that essentially helps determine how much to actually update the weight, still based off of the gradients, based on how training is going. This is especially important because training is not an inherently stable thing.

  • When you just start training, the model has been initialized with random weights - it will be terrible at the task, and so updates will be massive because the model will be so far from the optimum.
  • But later in training, when you’re close to a really good model, a lot of the weights just need to move a little to make the model a little bit better. Sometimes a boost can help ensure they move enough.

The standard choice for transformers is the Adam optimizer, which tries to tackle these issues with a few additional parameters:

  1. Momentum. Instead of using only the current gradient, Adam keeps a running average of recent ones. This means that updates “build up speed”. If a lot of your recent updates have been moving in one direction, you’re allowed to take a bigger step, since likely that weight really does need to move significantly. But if your updates have been bouncing around, the update isn’t allowed to be as large - there isn’t a firm direction that we know we need to move in.
  2. A per-parameter learning rate. Adam also tracks how large each weight’s gradients have typically been, and scales the weight’s step accordingly. Weights that are consistently getting large updates have those updates dampened a little - making sure the steps they’re taking aren’t too big. Weights that are only getting tiny tweaks get boosted a bit - making sure that the weight is really getting an update instead of sitting there unchanged. Every weight gets an appropriately-sized update automatically, instead of forcing one global learning rate to suit all of them.

Concretely, Adam keeps two running averages per weight, one of the gradient (m, the momentum) and one of the squared gradient (v, the typical magnitude), governed by decay rates β₁ (typically ≈0.9) and β₂ (typically ≈0.999).

\[ \begin{aligned} m &\leftarrow \beta_1\, m + (1-\beta_1)\, g && \text{(1st moment: momentum)}\\ v &\leftarrow \beta_2\, v + (1-\beta_2)\, g^2 && \text{(2nd moment: typical magnitude}^2\text{)}\\ \end{aligned} \]

Then,

\[ \begin{aligned} \hat{m} &= \frac{m}{1-\beta_1^{\,t}}, \quad \hat{v} = \frac{v}{1-\beta_2^{\,t}} \end{aligned} \]

And our final weight update actually looks like this: \[ \begin{aligned} \text{weight} &\leftarrow \text{weight} - (\text{lr})\cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \varepsilon} \end{aligned} \]

With some small \(\varepsilon\) to prevent divide by 0. So those two properties that we mentioned above are showing up as:

  • Momentum is the numerator \(\hat{m}\). It’s the running average of the actual gradient, updated at each step to include the current gradient. \(\beta_1\) (default 0.9) is its decay rate, how much to weight past-gradient memory it keeps. A high \(\beta_1\) means a long memory.
  • Per-parameter learning rate is the division by \(\sqrt{\hat{v}}\) in the denominator. \(\hat{v}\) is the running average of the squared gradient, which is an estimate of how big this weight’s gradients typically are. Dividing by it shrinks steps for consistently-large-gradient weights and boosts quiet ones. Its decay rate is \(\beta_2\) (default 0.999)

7.3.4 The Optimizer in Code

In PyTorch, the optimizer is one object, created over the model’s parameters:

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

This does take values for all of the parameters we just talked about - \(β_1\), \(β_2\), and \(ɛ\) - but if you don’t set them, it uses the defaults we discussed above (.9, .999. \(10^{-8}\)), which in practice are universally good parameters for Adam, so you don’t usually change them.

📌 Note: The original Transformer paper actually used .9, .98, and \(10^{-9}\) as its Adam parameters. It’s a pretty small change, and the defaults are known to be good, so we’ll go with defaults even though that’s not technically what the original did exactly.

Notice, again, that this just needs to take model.parameters(). The optimizer understands both to use and to update every weight registered through the module tree.

Then each training step on a batch can happen in just three lines:

optimizer.zero_grad()   # 1. clear the gradients left from the last batch
loss.backward()         # 2. backprop: get the gradient of the loss from every weight
optimizer.step()        # 3. update every weight using those gradients

The things to note:

  • zero_grad() comes first because PyTorch accumulates gradients by default. If we don’t have this, this batch’s gradients pile on top of the last batch’s.
  • loss.backward() handles all backpropagation, and gives back a .grad for every parameter.
  • optimizer.step() runs the update we just discussed above. Using the gradient the loss gives us, and the actual weight value, calculate the internal Adam parameters, and update the weight.

7.4 Learning Rate Scheduling

One more thing. Even with Adam, which helps smooth how the gradients behave, we still had a fixed learning rate. This, actually, is a whole separate thing that is usually dynamic over training via learning rate scheduling.

The idea goes back to what we already mentioned: at the very start, the weights are random and the gradients are wild, and a large learning rate can make updates too large and destabilize training. Not even Adam totally fixes this, as especially at the beginning, there isn’t a big gradient history to use.

So, the standard thing to do is to warm up the learning rate - essentially, take a few hundred to a few thousand of the first steps to grow the learning rate from something very small to its “full size”, so that the first few sets of updates aren’t huge. It also will then slowly decay it once it reaches its full size, making sure that tweaks are smaller when you get closer to a good model, as you also don’t want to risk big updates messing it up.

7.4.1 How to Schedule a Learning Rate

The exact schedule the original Transformer used is:

\[\text{lr}(\text{step}) = {d_{\text{model}}}^{-0.5} \cdot \min\!\left(\text{step}^{-0.5},\ \text{step} \cdot \text{warmup}^{-1.5}\right)\]

So a few things to unpack there.

First, you might be wondering: Why is the learning rate a function of \(d_{\text{model}}\)? The job of that factor is just to set the overall scale. Wider models tend to produce larger-magnitude activations and gradients (actually, we do this for the same reason attention divides by \(√d_k\) - because sums over more dimensions grow like \(√d_{\text{model}}\)), so the learning rate is scaled down to compensate. Baking it into the formula just means the same schedule works across model sizes without needing to re-tune what the peak will be by hand.

The other two terms inside the \(min\) are the two phases made explicit:

  • while \(\text{step}\) is small, \(\text{step}\)· \(\text{warmup}^{-1.5}\) is the smaller term, and grows linearly,
  • once \(\text{step}\) passes \(\text{warmup}\) , \(\text{step}^{-0.5}\) takes over and the rate decays as \(1/√\text{step}\).
  • they are equal exactly at \(\text{step}\) = \(\text{warmup}\) , which is the peak.

We can see exactly what this looks like:

d_model, warmup = 512, 4000
steps = np.arange(1, 20001)
lr = d_model**-0.5 * np.minimum(steps**-0.5, steps * warmup**-1.5)

plt.figure(figsize=(8, 4.5))
plt.plot(steps, lr, color="#5B8FB9", linewidth=2.2)
plt.xlabel("training step")
plt.ylabel("learning rate")
plt.title("Transformer learning-rate schedule (warmup then decay)")
plt.show()

7.4.2 Learning Rate Scheduling in Code

In PyTorch, of course, you don’t need to compute this by hand for each step, as there’s a way to simply set what the learning rate scheduler will be. LambdaLR takes a function of the step number and uses it to scale the learning rate:

d_model      = 512     # same as the model's d_model
warmup_steps = 4000

def lr_lambda(step):
    step = max(step, 1)                               # avoid step=0 blowing up
    return d_model ** -0.5 * min(step ** -0.5, step * warmup_steps ** -1.5)

optimizer = torch.optim.Adam(model.parameters(), lr=1.0)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

If we set the optimizer’s base lr=1.0, the schedule is the learning rate - LambdaLR multiplies the base lr by whatever lr_lambda returns.

Then the training loop needs one more addition, where after each weight update, you step the scheduler too.

optimizer.zero_grad()
loss.backward()
optimizer.step()
scheduler.step()      # <- advance the learning rate

7.4.3 Other Scheduling Options

Of course, as with all decisions made in model training, we don’t have to pick that exact scheduler. And while we’ve chosen inverse square root for our training for its simplicity and because it’s what the original transformer did, modern training has mostly moved to different decay shapes. The warmup part, though, has stuck around almost universally - nearly every other modern choice still ramps the learning rate up at the start.

So, what are people usually using now instead? The most common modern choice is cosine decay, which warms up, then has the learning rate follow the shape of a cosine curve. Not oscillating - just gliding smoothly from its peak down to near zero as a cosine does. It’s a bit smoother than the \(1/√\text{step}\) decay and reliably ends very low, which tends to help the model settle into a good minimum. It’s what GPT-3 and many modern LLMs use.

But, there are also other common shapes:

  • Linear decay is a straight ramp down to zero after warming up. Simple and very common (BERT and many Hugging Face defaults use this).
  • Constant with warmup just holds the rate flat after warming up. Very simple, and often does work perfectly fine.

So whatever shape you choose, there are really only a few knobs to set:

  • Peak learning rate - whatever the maximum value you’ll ever reach is. It’s still the single most important number.
  • Warmup length - how long it will take you to reach peak. Usually a few hundred to a few thousand, or a small fraction of total training.
  • Final learning rate - whether you’ll decay all the way to zero or stop at some small floor.

But almost all of them will warm up in some way, reach some peak, and then decide by how much to decay after. This general shape is where you get the benefits, and exactly how you implement it is a choice you can make.

7.5 Modern Training

Across this section we’ve made a complete, working set of training choices, and flagged where some alternatives exist. Nothing that we’ve set up here, which is how we’ll proceed with training, is wrong, but still, modern training has some genuinely new additions, and some minor tweaks, to the choices we’ve made that are worth at least mentioning so that you’re aware of them, even though we won’t cover them in any depth here.

So before we go on, here’s a quick rundown of a handful of things that modern, large-scale training does differently.

  • Weight decay. A common additional regularizer is added to the weights themselves with something called weight decay. Its job is to gently shrink every weight toward zero on each step. The idea is that large weights tend to signal an overfit, brittle model, so nudging them all slightly downward each update discourages that - again, requiring no single weight to ever be too large or too important. How it’s actually done is by subtracting a small fraction \(\lambda \cdot \text{weight}\) from each weight every update, where \(\lambda\) sets the strength. It’s an optional addition to the optimizer step that’s fairly standard.

  • Adam vs. AdamW. However, if you do want weight decay, how you add it into the optimizer step matters. Our implementation will not use weight decay, but standard Adam adds in weight decay like this: when you go to update the gradient, folds it in right there, by using \(g + \lambda w\) in place of \(g\). You just apply it right to the gradient you’re about to run Adam on. But that means that the \(\lambda w\) term rides through everything, including the division by \(\sqrt{\hat v}\). That ends up distorting it - weights with large, active gradients get decayed less and quiet ones more, instead of all uniformly. AdamW just “corrects” this by applying weight decay in a place where it’s more stable. It applies the decay directly to the weight update, outside of the \(\sqrt{\hat v}\) scaling: \[\text{weight} \leftarrow \text{weight} - (\text{lr})\left(\frac{\hat m}{\sqrt{\hat v} + \varepsilon} + \lambda\,\text{weight}\right)\] so all weights get an identical decay rate. It’s a one-line change, but because of it, AdamW is the default optimizer for essentially all transformers.

    (Note that with no weight decay, Adam and AdamW are actually identical, so the choice only matters if you’re doing weight decay at all.)

  • Gradient clipping. Gradients can still be overly large during training. To prevent this, gradient clipping simply caps the overall size allowed (via the norm of the gradient) at some threshold, commonly 1.0, to scale it down if it’s too big. This is done before the optimizer, and just stops unlucky batches with enormous gradients from making a single catastrophic update that wrecks training. This is also now nearly universal.

  • Mixed-precision training. Because modern models are large, rather than 32-bit floats everywhere, most modern training runs the bulk of the math in 16-bit. 16-bit floats are smaller and faster, but they have far less precision and (for fp16) far less range than 32-bit. “Mixed” precision means running the parts that can tolerate that in 16-bit while keeping the parts that can’t in 32-bit. The big matrix multiplies of the forward and backward pass run in 16-bit - it’s where nearly all the compute is, so that’s where the speed and memory savings come from. Other crucial parts, that can benefit from more precision, stay in 32-bit. These are:

    • The master copy of the weights, and the optimizer step that updates them. The update \(\text{lr}\cdot\hat m/\sqrt{\hat v}\) is often tiny next to the weight itself (maybe \(10^{-7}\) added to a weight of order \(10^{-1}\)). In 16-bit, a number that small is below the precision available at the larger weight’s magnitude, so it can round away to nothing. So a “real” copy of the weights is kept in 32-bit and updated in full precision, with a 16-bit copy used for the forward/backward compute.
    • Any reductions or sums over many values. Operations like softmax, LayerNorm, and the loss require adding up many numbers. Doing this in 16-bit loses precision as small terms get swallowed by a large running total, so these accumulate in 32-bit. Adam’s moment estimates are in 32-bit as well, for the same reason.

    (fp16 actually has one extra wrinkle: its range is so small that gradients can underflow straight to zero, so training scales the loss up before the backward pass and scales the gradients back down after. bf16 has the same range as 32-bit, just with less precision, so it skips that step. This is why it’s often preferred when the hardware supports it!)

  • Gradient accumulation. Often in compute-limited situations, the batch size you want won’t fit in memory, but dropping a batch size has consequences for generalization and stability. In this case, you instead run several smaller batches, not changing the model after each but instead add up their gradients, and only step the optimizer once. It simulates a large batch without paying its memory cost.

It’s worth noting that so much of what modern models do also has a lot to do with scale. Frontier models run their training loops across many GPUs at once (data and model parallelism), over vastly more data, and often in multiple stages. So a lot of what has been developed in modern modeling has been to help in cases where the training scale and model size is truly massive, and designed to be multi-step.

7.6 Assembling the Training Loop

Ok, so now that we’ve talked about all of the pieces of the training loop, let’s put it together. The fortunate thing, is that we basically already built the core workhorse of it as we’ve gone through this section.

Here’s what it ends up looking like. We define a few components at the start. A couple of these will still depend on our data, so we won’t write them into code quite yet.

model     = Transformer(...).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1.0)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)
loss_fn   = nn.CrossEntropyLoss(ignore_index=pad_idx)

But once those are defined, the loop that runs the training just takes those as arguments:

def train_epoch(model, data_loader, optimizer, scheduler, loss_fn, device,
                global_step, target_set, save_ckpt):
    model.train()                                  # put the model in training mode (dropout ON)
    total_loss = 0.0

    for src, trg in data_loader:                   # src: (N, src_len), trg: (N, trg_len), the data
        src, trg = src.to(device), trg.to(device)

        # --- teacher forcing: shift the target by one ---
        decoder_input = trg[:, :-1]                # [<bos>, y1, ..., yn]
        labels        = trg[:, 1:]                 # [y1, ..., yn, <eos>]

        # --- run the forward pass from the model we built before ---
        logits = model(src, decoder_input)         # (N, trg_len-1, vocab_size)

        # --- compute the loss: flatten as input, give to loss_fn ---
        loss = loss_fn(logits.reshape(-1, logits.size(-1)),  # (N*(trg_len-1), vocab_size)
                       labels.reshape(-1))                    # (N*(trg_len-1),)

        # --- go through all of the update steps ---
        optimizer.zero_grad()                      # clear last batch's gradients
        loss.backward()                            # backprop to compute new gradients
        optimizer.step()                           # update every weight
        scheduler.step()                           # advance the learning rate

        total_loss += loss.item()

        # --- save a checkpoint off, in STEPS not EPOCHS - we'll use later ---
        global_step += 1                           # global_step keeps track of where we are
        if global_step in target_set:
            save_ckpt(global_step)                 # save if it's a predefined saving step

    return total_loss / len(data_loader), global_step         # average loss over the epoch

Notice that this loop just runs for one epoch. One epoch is one pass over all of the data, which we do in batches that are much smaller than the size of all of the data. So we accumulate the loss and then average it over the number of batches that we have at the end. One epoch takes many steps. (Inside this function, we’ve snuck in a way to save checkpoints on steps not epochs, because we want to use that granularity later.)

Running for many epochs, then, just means wrapping this in another loop, which again, I still won’t write into a real code cell yet, since some of this is waiting to be defined.

for epoch in range(num_epochs):
    avg_loss = train_epoch(model, train_loader, optimizer, scheduler, loss_fn, device)
    print(f"Epoch {epoch+1:2d} | loss {avg_loss:.4f}")

But that’s it! A few decisions to be made, but it still comes together into something very brief and readable.

The training loop is really just a few key steps:

  • The model is defined, and model.train() just puts it into training mode, turning dropout on (model.eval() by contrast, turns it off so we can actually use the model)
  • We shift for teacher forcing, using exactly the same code we called out earlier in this section.
  • model(src, decoder_input) runs the forward pass of the whole architecture we built in Section 6. And notice we don’t build any masks here: the Transformer’s forward constructs src_mask and trg_mask internally from the inputs, because it’s well understood how to structure them based on that.
  • The loss is exactly the code we used earlier in the loss section. We must flatten batch and position together, set the ignore_index to drop the padding, and make sure we give it raw logits.

It all ends in the optimizer update that we defined. Clear out old gradients, calculate the new ones, give them to the optimizer, then advance the learning rate scheduler.

8. Training on a Real Task

Ok. So everything we’ve set up so far has actually been fairly task-agnostic - we said what we would do with data when it came in, and how that data would matter, but so far everything about this setup could still apply to a variety of different encoder-decoder tasks. We’ve made some choices based on the type of task (like whether or not the input to the encoder vs decoder share a vocabulary), but that’s still broad, not based on specific task we’re training for.

In this section, we’re going to make it real. Choose a real task, build data, make our vocabulary, batch it, and watch our transformer train to do something it actually is good at learning.

8.1 Our Task: Date Format Translation

We’ve talked about how the encoder-decoder architecture is good for tasks like translation, where really, you want to produce an outgoing piece of text, based on your understanding of some separate, other sequence. Date formatting is a good example of a (very simple, so we can train it quickly) version of this task.

Our goal is going to be to construct a small dataset of dates in standard variations of “human” formatting:

\[\text{sept 04 2015 ; 22 may 2026 ; october 19, 1950}\]

And we’ll want to translate them into a standardized YYYY-MM-DD (ISO) formatting:

\[ \text{2015-09-04 ; 2026-05-22 ; 1950-10-19}\]

That’s it! Now, could this task be done more simply with a classic lookup table? Absolutely, yes. But a transformer can also learn to do it perfectly, which means we’ll get to see real training happen. So no, this isn’t some kind of breakthrough task, but we can easily generate large amounts of data here, and train a model in a few minutes, so it’s what we’ll go with.

So, let’s write a quick function to generate the data. We pick random dates, render each into one of several human-readable formats for the input, and pair it with its standardized target string:

INPUT_FORMATS = [
    "%d %B %Y",        # 03 march 2019
    "%B %d, %Y",       # march 03, 2019
    "%d %b %Y",        # 03 mar 2019
    "%b %d %Y",        # mar 03 2019
    "%A, %B %d, %Y",   # sunday, march 03, 2019
    "%d/%m/%Y",        # 03/03/2019  (always day/month/year)
]

def random_date(start_year=1950, end_year=2049):
    start, end = date(start_year, 1, 1), date(end_year, 12, 31)
    return start + timedelta(days=random.randint(0, (end - start).days))

def make_example():
    d = random_date()
    human = d.strftime(random.choice(INPUT_FORMATS)).lower()
    iso   = d.strftime("%Y-%m-%d")
    return human, iso

random.seed(0)
dataset = [make_example() for _ in range(50000)]

Then we can just look at a few examples to be sure it worked:

dataset[1:10]
[('07 apr 1957', '1957-04-07'),
 ('sep 26 2041', '2041-09-26'),
 ('28 aug 2022', '2022-08-28'),
 ('08 jul 2035', '2035-07-08'),
 ('sunday, march 12, 1989', '1989-03-12'),
 ('28 dec 1974', '1974-12-28'),
 ('28 january 1975', '1975-01-28'),
 ('tuesday, december 13, 1994', '1994-12-13'),
 ('15 may 1976', '1976-05-15')]

A few small design choices worth calling out:

  • The inputs vary in length, the outputs don’t. Across the dataset the human inputs run from about 10 to 29 characters (15 may 1976 vs tuesday, december 13, 1994). Every ISO target, though, is exactly 10 (YYYY-MM-DD). This is deliberate because it makes the need for padding real - we’ll have to pad the shorter inputs when we batch them. The fixed-length target means no target padding, though that was the less interesting case (it’s all handled internally for us), so that’s ok.
  • We’ll lowercase everything. In my above example, I capitalized months. But keeping both capital and lowercase is extra vocabulary, for no benefit - march is the same as March, and both get turned into 03 anyway, so it’s the kind of choice that just makes sense for a task like this. We’ll see this lowercasing pay off in the next part when we make the vocabulary.
  • Every format is unambiguous. We chose a variety of input formats to make the task a little more interesting, but none of them are missing information, and none of them could mean anything else. In practice for dates especially, that means choosing one purely numeric format (DD/MM/YYYY, day-first). If we’d also included MM/DD/YYYY, then a string like 03/04/2019 would have two valid answers and no way to know the intended formatting. So we choose just one so that every input→output mapping is well-defined.

8.2 The Vocabulary

So, the next step is to build a vocabulary from all of our data. Since we’re going to work at the character level (the model needs to produce one character at a time for dates), and the source and target share an alphabet, we just need one, pretty small vocabulary.

(For that reason… we aren’t going to build it up from BPE like we had talked about at the start of this notebook. BPE makes a lot of sense when you’re building up from a large corpus of data where you don’t know, or it’s impossible to know, exactly what’s in it and what pieces are most common. The task we’ve chosen here is a rare case where we want to work on the character level, so making the vocabulary is just “what are all the different characters that show up”. We don’t need to build statistics-based bindings of tokens - our total vocabulary is going to have like 40 items to begin with.)

# every character that appears, in inputs or targets
chars = set()
for human, iso in dataset:
    chars.update(human)
    chars.update(iso)

# special tokens FIRST, so that <pad> gets ID 0
special = ["<pad>", "<bos>", "<eos>"]
vocab = special + sorted(chars)

char_to_id = {c: i for i, c in enumerate(vocab)}
id_to_char = {i: c for i, c in enumerate(vocab)}

pad_idx = char_to_id["<pad>"]   # 0
bos_idx = char_to_id["<bos>"]   # 1
eos_idx = char_to_id["<eos>"]   # 2
vocab_size = len(vocab)

print("VOCAB SIZE:", vocab_size)
print("TOTAL VOCAB:", vocab)
VOCAB SIZE: 39
TOTAL VOCAB: ['<pad>', '<bos>', '<eos>', ' ', ',', '-', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'l', 'm', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'y']

So, just a tiny 39-token vocabulary. And it’s a nice sanity check actually that k, q, x, or z, never appear, because no month or weekday name ever uses them. So it still is genuinely built from the data, as it should be.

Two things just got decided here, which all through sections 6 and 7 we said would “need to come from the data”:

  • pad_idx is now 0. We listed all of the special tokens first, and <pad> was the first of them. That means that the token ID 0 will always mean “padding”.
  • vocab_size is 39. We of course need to give this number to the output projection and the shared embedding table, and now we know it.

Then tokenizing is just a lookup to the right index, and we tokenize the source and target the same, except that the target will need the special start/end tokens:

def encode_src(s):
    return [char_to_id[c] for c in s]                          # just the characters

def encode_trg(s):
    return [bos_idx] + [char_to_id[c] for c in s] + [eos_idx]  # needs to also wrap in <bos> and <eos>


test_src, test_target = dataset[0]
print(test_src, "=>", test_target)
print(encode_src(test_src), "=>", encode_trg(test_target))
feb 15 2019 => 2019-02-15
[22, 21, 18, 3, 8, 12, 3, 9, 7, 8, 16] => [1, 9, 7, 8, 16, 5, 7, 9, 5, 8, 12, 2]

So we see it works exactly how we want. The target sequence has <bos>=1 at the front, and <eos>=2 at back. Besides that, the sequences match their number of characters since we encode character-by-character. The ISO-format is exactly the length we expect (12: <bos> + 10 ISO chars + <eos>). So these functions give our tokenized sequences, ready to go.

8.3 Batching and Padding

Now, it’s time to put the data together into a batch, and this is where padding also comes into play. Remember, we need to pad the shorter sources in our batch out to match the length of the longest one in the batch, filling with <pad> (which is ID 0).

In PyTorch, we build our dataset to do this with a Dataset object. This handles the fact that batches constructed of different sources will need different amounts of padding (you always pad to the longest in the batch, not globally, to save compute). This will also handle the tokenizing via those functions we just defined.

class DateDataset(Dataset):
    def __init__(self, pairs):
        self.pairs = pairs
    def __len__(self):
        return len(self.pairs)
    def __getitem__(self, idx):
        human, iso = self.pairs[idx]
        return (torch.tensor(encode_src(human), dtype=torch.long), # use the functions we just wrote
                torch.tensor(encode_trg(iso),  dtype=torch.long))

# finish constructing the batch by padding when needed
def collate(batch):
    srcs, trgs = zip(*batch)
    # pad each side to the longest sequence in this batch, filling with pad_idx
    src = torch.nn.utils.rnn.pad_sequence(srcs, batch_first=True, padding_value=pad_idx)
    trg = torch.nn.utils.rnn.pad_sequence(trgs, batch_first=True, padding_value=pad_idx)
    return src, trg

train_loader = DataLoader(DateDataset(dataset), batch_size=16,
                          shuffle=True, collate_fn=collate)
  • DateDataset’s job is just to turn (human, iso) pairs into the two token-ID tensors
  • collate is where the padding happens. The DataLoader is what gathers batch_size examples and hands them to collate to fill them as needed. Within collate, batch_first=True just says to put the batch dimension first, giving the \((N, n_{\text{seq}})\) shape our model expects.
  • DataLoader ties it together. It shuffles the data each epoch and yields padded (src, trg) batches.

Now, our example here was really simple, so we also could have done this without the DataLoader, without too much trouble. But the Dataset/DataLoader pair is the standard PyTorch way that you’d realistically use and see in almost every implementation. The only hard requirement for using it is that the Dataset must implement two methods:

  • __len__ (how many examples), and
  • __getitem__(i) (get the i-th one)

That’s why our DateDataset implements exactly those two. The collate function is technically optional, but you do almost always have to supply it, because PyTorch’s default batching will assume every example is already the same size.

8.4 Configuring the Model

So the data is ready. The final thing is to construct the model with the specific choices for its configuration. This includes all things we get to choose and could tune: the model sizes, the optimizer parameters, and feeding in the information we collected while generating our data.

We’ll define them in the cell below, first thing, so you can see plainly all of the choices we get to make.

# ---- Define hyperparameters ----
d_model      = 128      # the original paper used 512 - we scale down to keep small
num_layers   = 3        # the original paper used 6 -  we keep small with 3
heads        = 8        # this is what the original paper used, confirm d_model is divisible by heads
d_ff         = 512      # the FFN inner size (we choose 4 x d_model, as in the paper)
dropout      = 0.1      # 0.1 is fairly standard
max_len      = 40       # most the model is allowed to generate. must exceed our longest sequence (source max ~29)
warmup_steps = 3000      # scaled down from the paper's 4000 - we expect to train fast
num_epochs   = 10

# will be blazing fast if you can get a GPU!
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ---- Set up the model----
# this is where all of those choices above come to actually shape what this model looks like!
model = Transformer(
    vocab_size  = vocab_size,     # 39
    d_model     = d_model,
    num_layers  = num_layers,
    heads       = heads,
    d_ff        = d_ff,
    max_len     = max_len,
    src_pad_idx = pad_idx,        # 0, decided when we built the vocabulary
    trg_pad_idx = pad_idx,        # 0, decided when we built the vocabulary
    dropout     = dropout,
).to(device)

# ---- Put together the training pieces ----
# we've seen this code before, but only wrote it within markdown
# here, we define it so it's usable.
def lr_lambda(step):
    step = max(step, 1)
    return d_model ** -0.5 * min(step ** -0.5, step * warmup_steps ** -1.5)

# define the optimizer, scheduler, and loss
optimizer = torch.optim.Adam(model.parameters(), lr=1.0, betas=(0.9, 0.999), eps=1e-8) #defining them here to be clear, but these are exactly the defaults for beta/eps.
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)
loss_fn   = nn.CrossEntropyLoss(ignore_index=pad_idx)

A few of the choices I’ll point out:

  • Our model is a scaled-down version of the original. Where the original paper used d_model=512, 6 layers, and d_ff=2048, we use 128 / 3 / 512. Our task is easy, and our data is small, so a small model will learn it quickly.
  • max_len must exceed our longest sequence. It determines our positional encoding, so it needs to be at least as big as the longest sequence we’ll ever feed. Our sources top out around 29 characters, so 40 is just comfortable headroom.
  • warmup_steps is scaled down from 4000 to 3000. The original paper used 4000. But warmup should be chosen based on the number of training steps you’ll do, because it’s determined by the data size, batch size, and number of epochs. For us, ≈ 50,000 examples ÷ 16 per batch ≈ 3125 steps per epoch, × 10 epochs ≈ 31,250 steps. So we need to warm up for a number of steps reasonable for our model size - here, over the first ~epoch.

And now that it’s really built, we can take a closer look at it, by just inspecting the PyTorch built-ins around it:

num_params = sum(p.numel() for p in model.parameters())
print(f"model parameters: {num_params:,}")

print(model)
model parameters: 1,395,623
Transformer(
  (input_embedding): InputEmbedding(
    (embedding): Embedding(39, 128)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (encoder): Encoder(
    (layers): ModuleList(
      (0-2): 3 x EncoderBlock(
        (attention): MultiHeadAttention(
          (values): Linear(in_features=128, out_features=128, bias=False)
          (keys): Linear(in_features=128, out_features=128, bias=False)
          (queries): Linear(in_features=128, out_features=128, bias=False)
          (fc_out): Linear(in_features=128, out_features=128, bias=True)
        )
        (feed_forward): FeedForward(
          (linear1): Linear(in_features=128, out_features=512, bias=True)
          (relu): ReLU()
          (linear2): Linear(in_features=512, out_features=128, bias=True)
        )
        (norm1): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
        (norm2): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )
    )
    (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
  )
  (decoder): Decoder(
    (layers): ModuleList(
      (0-2): 3 x DecoderBlock(
        (attention): MultiHeadAttention(
          (values): Linear(in_features=128, out_features=128, bias=False)
          (keys): Linear(in_features=128, out_features=128, bias=False)
          (queries): Linear(in_features=128, out_features=128, bias=False)
          (fc_out): Linear(in_features=128, out_features=128, bias=True)
        )
        (cross_attention): MultiHeadAttention(
          (values): Linear(in_features=128, out_features=128, bias=False)
          (keys): Linear(in_features=128, out_features=128, bias=False)
          (queries): Linear(in_features=128, out_features=128, bias=False)
          (fc_out): Linear(in_features=128, out_features=128, bias=True)
        )
        (feed_forward): FeedForward(
          (linear1): Linear(in_features=128, out_features=512, bias=True)
          (relu): ReLU()
          (linear2): Linear(in_features=512, out_features=128, bias=True)
        )
        (norm1): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
        (norm2): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
        (norm3): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
        (dropout3): Dropout(p=0.1, inplace=False)
      )
    )
    (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
  )
  (output_projection): Linear(in_features=128, out_features=39, bias=True)
)

Inspect it, and you’ll see every single choice we made show up in determining the architecture. Our transformer is wired together, ready to train!

8.5 Train the Transformer!

Now it’s time to actually set the training going! It’s what we’ve been building up to, but at this point the code to actually do it is anticlimatic: it’s actually the exact same code we saw earlier this section, just now made “real” instead of in markdown since there’s now a real model behind it, and with one additional saving step.

So the last little piece of setup we need to do first is get set up to save checkpoints of our model. We already added some infrastructure needed for this into our train loop, but here we’ll define when to actually save. For this model, we’re going to save checkpoints often (you’ll see why later - we are going to use them for something), but in a normal implementation you would not save so frequently, especially not early on.

In general, though, saving some checkpoints is always a good idea. It helps us have multiple points to reload our model from (saving the last few is a good idea, since in general sometimes one or two before the last is actually the best model), and allows any results you get to be exactly reproducible.

This model is small enough, that simply retraining only takes a few minutes. But still, every version of the newly trained model will have its own nuances and behaviors, so saving your model protects from that.

CKPT_DIR = "checkpoints"
os.makedirs(CKPT_DIR, exist_ok=True)

steps_per_epoch = len(train_loader)
total_steps     = steps_per_epoch * num_epochs

# log-spaced targets (dense early) + every epoch boundary + step 0
targets, k = {0}, 0
while int(1.5 ** k) < total_steps:
    targets.add(int(1.5 ** k)); k += 1
targets.update(range(0, total_steps + 1, steps_per_epoch))
target_set = {t for t in targets if t <= total_steps}

manifest = []
def save_ckpt(step):
    p = f"{CKPT_DIR}/ckpt_{step:06d}.pt"
    torch.save(model.state_dict(), p)    # weights only - robust across restarts
    manifest.append((step, p))

One thing first! Up until now, if you’ve been following along on colab, you have been able to run every cell so far on your own without consequences. This is where that will change. If you run the training cell, then try to re-run the cells in Sections 9 and 10 - you will likely have different results than I did, and the specific behaviors I point out won’t match exactly (in broad strokes, this model performs about the same every time it’s trained, but many specific references to the values we get out won’t line up anymore). So, you may want to: - run this training and then stop, and just read all the Section 9/10 cells, or - not run this training, and just read everything from here on out, or - uncomment and run the cell below - which reloads a checkpoint I have saved and is in the github repository for this notebook. If you reload this model, and skip the training, all analysis in Section 9 and 10 should match.

if False:      # change to True to load my trained model instead of training your own
    url = "https://raw.githubusercontent.com/apetulante/Tutorials/master/Architectures/Basic_Transformer_Tutorial/date_model_final.pt"
    state_dict = torch.hub.load_state_dict_from_url(url, map_location=device)   # downloads (and caches) the file
    model.load_state_dict(state_dict)                                            # copy those weights into our model
    model.eval()
    print("model restored from checkpoint")

Then, let’s set it going!

One note, is if you do have access to a GPU - even a weak one - that will speed this up a lot. It should still run on CPU only, but will likely take 10 or 15 minutes. On a GPU, you should see this train very quickly.

global_step = 0
save_ckpt(global_step)                             # frame 0: untrained

for epoch in range(num_epochs):
    avg_loss, global_step = train_epoch(model, train_loader, optimizer, scheduler,
                                        loss_fn, device, global_step, target_set, save_ckpt)
    print(f"epoch {epoch+1:2d} | loss {avg_loss:.4f} | step {global_step}")
json.dump(manifest, open(f"{CKPT_DIR}/manifest.json", "w"))

# save the final trained weights to a single file, so they can be shared and reloaded
torch.save(model.state_dict(), "date_model_final.pt")
epoch  1 | loss 0.2928 | step 3125
epoch  2 | loss 0.0193 | step 6250
epoch  3 | loss 0.0067 | step 9375
epoch  4 | loss 0.0053 | step 12500
epoch  5 | loss 0.0036 | step 15625
epoch  6 | loss 0.0022 | step 18750
epoch  7 | loss 0.0016 | step 21875
epoch  8 | loss 0.0016 | step 25000
epoch  9 | loss 0.0017 | step 28125
epoch 10 | loss 0.0012 | step 31250

And there you have it - our date translation model is trained!

There are a few things we can glean from that loss trajectory.

  1. It starts “high”. Even if you’ve re-run this notebook, you probably see this start at around 0.3. An untrained model, with no idea which token comes next, spreads its probability roughly evenly over all 39 vocabulary tokens, so it gives the correct token about 1/39 probability, and the cross-entropy loss should start near −ln(1/39) ≈ 3.66. So by the end of even the first epoch, when that 0.3 reports, it’s already learned how to do the task much better than random. A big part of that is that this task is deterministic and easy, so the model picks up the pattern fast.

  2. By epoch 3, it’s basically trained. The loss falls steeply and then flattens out near zero, with loss values of \(0.00xx\). That flattening is the model essentially nailing the mapping, and like we said, it gets there pretty fast because the task is so easy.

But that second point is actually worth digging into a little more, because you might be thinking, if the task is literally deterministic - trivial - why does it bounce around down near 0 at all? Why not just hone in on the answer and stay there - being literally perfect every time? And it’s worth pointing out, because really it’s a feature of training, coming from several different places, that means we never truly get perfect.

  1. The loss literally can’t be zero. Cross-entropy is −log(p) on the correct token, which is zero only if the model gives that token probability exactly 1. But since the probabilities come from a softmax over finite logits, they can’t actually reach \(1.0\) - that would take an infinitely large logit. So even a flawless model on a perfectly deterministic task has a tiny positive floor it can only ever approach, never touch.

  2. Training is inherently stochastic. Each step’s gradient comes from a different random batch, which is different every epoch, and each batch pulls the weights in the direction that’s best for it specifically. It’s likely that essentially the same weights are good for this task regardless of batch (that’s what generalization is), but you still expect a little bit of push and pull based on the randomness of the batches.

  3. The learning rate is still nonzero. Something has to move in some direction every time. Even with small steps, you have to move a little bit around the exact perfect solution just by virtue of being forced to take some kind of step at all. A learning rate decayed all the way to near zero would shrink those updates to zero, but ours here stays small but positive.

  4. Dropout introduces randomness. During training, dropout randomly zeroes different activations on every forward pass. Even if the current forward pass’s weights were literally perfect, next time, the set of weights allowed to contribute would be slightly different. So really, the network computing the loss is a slightly different network each batch. It helps no specific configuration of weights matter too much, but it also means the training loss can’t be perfectly minimal or perfectly smooth.

None of this matters for whether the model works: the loss ending up at around \(0.00x\) means that the model is putting essentially all its probability on the right character every time. But that little bit of randomness exists due to many of the exact mechanics we’ve been laboring over this whole time, so it’s worth a moment. And this little bit of randomness is a good thing! Yes, on this task, you could find a very exact mapping with a lookup table. But most tasks are not so exactly deterministic, and all of this randomness helps protect against overfitting and forces the model to learn something more general. For almost every use case, that’s exactly what’s desired.

Now, of course, a model can be perfectly performing on the training data, and still not have learned any general mapping. So the thing to do next is to actually give this model examples it has never seen before, and see if it still performs. We’ll dive deep into all of that next!

9. Using the Model

Now, we have a trained model. The next thing you of course want to do is use it - give it a date, and see if it actually knows what to do.

And beyond that, you might want to know how well it does its job. Not just what one answer looks like - but measuring accuracy on some evaluation set, checking what it does on edge cases, or even on pieces of data that you haven’t exactly shown it before.

9.1 Generating One Token at a Time

The first thing we need to do, then, to see how the model performs, is to be able to run it! And that means use it to generate one token at a time. Back in Section 5 we described generation as a loop, where we start with <bos> token, run it through the decoder, read off the last row, pick a token, append it, feed the longer sequence back in, and repeat until <eos>.

That’s exactly the loop we now need. And we’ll write it into code so that we can actually call it, and then see what it makes for a test sequence.

def generate(model, src_str, max_len=12):
    model.eval()                                           # turns dropout OFF
    src = torch.tensor([encode_src(src_str)], device=device)   # tokenize it with our encode_src
    trg = torch.tensor([[bos_idx]], device=device)             # start with just <bos>

    with torch.no_grad():                                  # no gradients needed at inference
        for _ in range(max_len):                           # generate until max_len is reached
            logits    = model(src, trg)                    # get the logits from the model. size (1, current_len, vocab_size)
            next_id   = logits[:, -1, :].argmax(dim=-1)    # take ONLY the last row only. Greedily pick next token
            trg       = torch.cat([trg, next_id.unsqueeze(0)], dim=1)   # append to the running target sequence
            if next_id.item() == eos_idx:                  # stop if the model emits <eos>
                break

    out_ids = trg[0].tolist()[1:]                          # drop the leading <bos>
    if eos_idx in out_ids:
        out_ids = out_ids[:out_ids.index(eos_idx)]         # cut at <eos>
    return "".join(id_to_char[i] for i in out_ids)         # turn tokens back into characters

So, what exactly is going on here?

  1. First, model.eval() to turn dropout off.
  2. Seed trg with [[bos_idx]] - always signaling the start of generation.
  3. Run it through the model, and get logits[:, -1, :] - just the last row.
  4. Use .argmax for greedy decoding. We could sample here, but the task isn’t really suited to that.
  5. Append the chosen token with torch.cat([trg, next_id ...]) so the next time the loop comes around, trg now includes it.
  6. Break if we produce an end token. if next_id == eos_idx: break is when the model tells us it’s done.
  7. Strip the <bos>, cut at <eos>, and map the IDs back to characters.

One honest inefficiency with writing it like this: calling model(src, trg) runs the whole model every step, so the encoder re-processes the same source each time, even though (as we said in Section 4.5), it never actually changes. A real implementation would encode once and reuse enc_out. We keep it this way so generate uses the exact same forward we built. (The decoder has the same kind of waste, as it re-processes its own earlier tokens every step, even though they never change. But fixing that requires the more complex KV cache that we touched on in Appendix 4.)

We can use this function to see what our model actually generates:

test_date = "march 03, 2019"
generate(model, test_date)
'2019-03-03'

It works!

9.2 Checking Generalizability

\[ \text{"It works!"} \]

… on one example. Testing your model and seeing the output you want is encouraging, of course, but a working model should work on many, many examples. More than you can just try by changing what that test_date string is manually and re-generating over and over again. That’s why when measuring model performance, we want to be able to quantify how well the model does on a real set of examples it’s never seen before. We want to be sure it didn’t just memorize the specific dates we gave it, but actually learned a pattern.

So let’s test this model on some hold-out data. We’ll freshly generate new examples that are guaranteed not to be in the training set, and then see how often we get those right.

For this task, we’ll define “correct” with exact-match accuracy. In our case, the entire ISO string has to be correct, character for character. A date that’s off by a number is wrong, not “80% right”, so this is a good metric to use.

Let’s see how our model does on this hold-out set:

# a held-out test set: fresh random examples NOT in the training data
# since we made the training set randomly, we just guess and check here
train_set = set(dataset)
random.seed(999)                          # a different seed from training
test = []
while len(test) < 2000:
    ex = make_example()
    if ex not in train_set:               # guarantee these are unseen
        test.append(ex)

# exact-match accuracy: the whole output string must be right
correct = sum(generate(model, human) == iso for human, iso in test)
print(f"exact-match accuracy on {len(test)} held-out dates: {correct/len(test):.1%}")
exact-match accuracy on 2000 held-out dates: 100.0%
for human, iso in test[:8]:
    pred = generate(model, human)
    print(f"  {'✓' if pred == iso else '✗'}  {human:28s} -> {pred}   (truth {iso})")
  ✓  thursday, may 14, 1964       -> 1964-05-14   (truth 1964-05-14)
  ✓  september 25, 2036           -> 2036-09-25   (truth 2036-09-25)
  ✓  30/01/2007                   -> 2007-01-30   (truth 2007-01-30)
  ✓  25/06/1967                   -> 1967-06-25   (truth 1967-06-25)
  ✓  september 25, 1983           -> 1983-09-25   (truth 1983-09-25)
  ✓  15/07/1959                   -> 1959-07-15   (truth 1959-07-15)
  ✓  jul 10 1995                  -> 1995-07-10   (truth 1995-07-10)
  ✓  30 may 1998                  -> 1998-05-30   (truth 1998-05-30)

It really did learn the task. This in itself is kind of obvious. “Memorization” is often thought of as a “low hanging fruit” during training, but especially for a task like this - even with only 50,000 examples - just learning how to do the translation is probably easier than memorizing all of them.

Modern modeling practices also have a lot to protect from this - and more than we’ve touched on here. Between batching, dropout, and optimizer regularization (which we didn’t tackle), there’s a lot to protect from true complete memorization. That being said, it wouldn’t be uncommon to see better performance on the training data than the evaluation - the model is fundamentally only trying to fit the training data, after all. That’s just another thing that is a judgement call when it comes to model evaluation.

And speaking of judgement calls - the whole decision of what metric to use for evaluation is one that of course, say it with me, depends on the task. Maybe for some tasks, you can check “accuracy” with a token-per-token yes/no. Maybe you have to scan for key words, maybe you need a real person to just read every answer. Maybe you have an external rubric. Evaluating a model on how good it is, is really an entire field - sometimes, often, more than half the work of actually building one that’s ready for use.

So while we got to use a very simple metric here for a very simple task, and prove that the model did well with it, it won’t often be so easy.

9.3 When it’s Not so Easy

Sometimes, finding the mismatch between what your model generates and what you wanted it to requires a lot of digging and effort. And as I said, even deciding in what ways it’s not doing what you want can be a whole challenge.

But sometimes, there are a few common things that can go wrong, which aren’t super strongly tied to rigorous evaluations, and which have obvious signals for what you might want to tweak first. We’ll cover an extremely non-exhaustive list here.

The loss sits flat at its starting value. (Around \(-\ln(1/\text{vocab\_size})\) - random guessing) and won’t budge.

  • Likely culprit: It’s probably a bug, not a tuning problem. It means nothing is learning at all. Usually, this means something like: the learning rate is zero, labels are misaligned (a teacher-forcing shift off by one), a mask applied wrong, or gradients that aren’t actually flowing.
  • To fix: Double check all of the wiring. Try to look at gradients and be sure they can move through the model.

The loss blows up. Either it shoots upward, oscillates violently, or eventually goes to NaN/inf.

  • Likely culprit: The learning rate is too high -or- gradients are somehow exploding.
  • To fix: Try to lower the learning rate (often by 10×). Add gradient clipping.

The loss falls, but painfully slowly. Or it falls just a little, then flattens out while the model is still bad.

  • Likely culprit: The learning rate is too low -or- the model might be too small to fit the task at all -or- gradients are vanishing.
  • To fix: First, raise the learning rate as it’s a cheap fix. If the training loss simply won’t get low no matter what, the model is probably too small, so try to go bigger (more layers, wider \(d_\text{model}\)). Residuals and normalization are what mostly prevent vanishing gradients, so they’re fairly unlikely.

The training loss gets low, but held-out performance is poor. There’s a big gap between the same evaluation metrics on train vs eval data.

  • Likely culprit: This means overfitting. Either the model is too big for the data -or- there’s too little data -or- too little regularization.
  • To fix: Try more data (or augmentation), more regularization (stronger dropout, weight decay), or a smaller model.

One thing that can help diagnose the middle two is to print the gradient norm for each step during training. This gives you the direct information about if your gradients are exploding or collapsing toward zero, which can help eliminate those as issues.

9.4 How Far Does It Generalize?

Showing that your model generalizes well to held-out data that matches the training data distribution is a great sign - it’s literally the task your model was trained on. But real data, real use cases, are often not so clean cut. A training dataset is almost never an exact distributional match to what you will ultimately use the model on. You of course try to get as close as you can when designing your dataset, but very rarely will it ever really happen.

So the natural next question is, how far can I push what it’s learned? How “general” is it? Beyond new instances drawn from the exact same data generator, can the model figure out what to do with something slightly different? Can we reasonably expect it to?

9.4.1 Interpolation vs Extrapolation

The usual language for this is interpolation versus extrapolation. You interpolate when you fill in gaps inside the region your data covers, and extrapolate when you move outside it. For a long time in machine learning, the classic belief was that neural networks were mostly strict * interpolators*. They do very well inside the space they know, and very poorly as soon as they’re outside of it. It’s a useful intuition, but turns out to be not quite true.

For one thing, deep networks have been shown to generalize far better than classical learning theory predicted they had any right to, enough that “why do these models generalize at all?” became a well-studied research topic in and of itself. For another, the clean geometric picture of “inside the data” has been shown to not really be how real data works. Indeed, it’s been shown, notably by Balestriero, Pesenti, and LeCun (2021), that in high dimensions, essentially every new point lands outside the convex hull of the training set. By that strict definition, a model is always extrapolating, and “interpolation” isn’t really on the table at all.

That said, a lot of current research on this topic happens on frontier-scale models, which are large and capable enough to do things our little date-translator simply won’t, like in-context learning or the apparent jumps in capability that show up as models scale. We have a small, easy problem with a single clear right answer. So rather than worry about the exact line between interpolation and extrapolation, or trying to figure out exactly how far a test example sits from the training data in some mathematical sense, we’ll take the broader, more practical view and just ask what happens when we push the model “out of distribution”.

“Out of distribution” just means, anything that the model hasn’t seen anything like before. It’s often brought up when people talk about the model failing catastrophically - like starting to produce total garbage because it has no idea what it’s being asked anymore. But data can also be just slightly out of distribution in different ways. The whole idea is just that we’re going to choose a few examples to put through our model that we know it’s never seen before, and just see what happens! We’re trying to find some practical bounds of our model’s capabilities, empirically. Looking ahead, we’ll see in Section 10 if we can get any hint into why it behaves that way, in whatever way it does, but that will come later.

And I say “in whatever way it does”, because trying your model on things you shouldn’t really expect it to know is always a mixed bag. How well it performs on data that does match what you gave it can easily become a strict “goodness” metric. But you can never really expect a model to do well on a task you’ve never shown it. It really is just a “try and see”.

So that’s what we’ll do. We’ll try to pick harder and harder cases for our model, hypothesize about what we might expect it to do, and then just try it and see what happens. Where we can try to say something more about the behavior we see, we will.

9.4.2 Case 1: A Year It Hasn’t Seen

We’ll start with the most obvious thing to try, and a task that seems like it should trivially extend. We’ll give the model a year that wasn’t in its training range. Our generator sampled dates from 1950 to 2049, so what happens if we ask it about 1850, or 2088, or 3017?

Before we run anything, let’s stop for a second and think about what should happen. The first thing to note, is that the year is always a copy from the input. Of the three things the model does to a date, the year is mechanically the easiest. It must learn to turn “march” into “03”. It must learn what ordering to expect for day/month/year to re-arrange. But in all of the data we generated, the year itself was always just plainly there. The model just needs to recognize what part is the year, and copy it, the same four digits in the same order, somewhere else. Notably, additionally, the model works one character at a time. So it never needs nor has inherent knowledge about what number the year is. It doesn’t think “1950” is a “smaller number” than “1960” - it should just know that when it sees those four numbers in that order (or maybe, any four numbers), to put them, in that same order, in a certain spot of the output.

Thinking about it that way, a year like 1850 isn’t novel because of its value - every one of 1, 8, 5, 0 is a character the model has copied thousands of times. The real question is whether anything about its ordering is new. To see that, we look one level down, at where each digit sits.

Here is every year our model saw, broken into its four character-slots:

Slot Digits it ever saw (1950–2049) Why
Thousands 1, 2 every year is 19xx or 20xx
Hundreds 0, 9 19xx → 9, 20xx → 0
Tens all of 0–9 (but 5–9 only ever beside a 9, and 0–4 only ever beside a 0)
Units all of 0–9 every possible ending shows up

So the model is very used to seeing all digits 0-9 in the last two digits of a year. But the first two digits have only ever been in a very narrow range. So we’ll want to see what the model does outside of that range.

So we’ll pick a bunch of unseen years, and see if it breaks as the task gets harder, and more out of distribution. The 3 specific test cases below will change just the year, but we’ll hold the rest of the date fixed to test all things cleanly. We’ll choose 2 different formats:

  • 15 june YYYY, and
  • 15/06/YYYY

as its success might depend on how the information is delivered. We hold the day and month fixed (15 june, both squarely in-distribution) and use a named-month format so there’s no day/month ambiguity to muddy things. The only thing changing between probes is the year.

We’ll pick a bunch of unseen years, then, and see if it breaks as it gets harder, more out of distribution. We’ll try:

1) A year whose digits all land in familiar slots. Out of our specific range, but with numbers the model has all seen in that slot before.

Test cases: 1912 and 2088.

Every digit sits in a position that’s seen it; only the exact value (and the particular hundreds/tens pairing) is new. If the model really learned “copy the year block,” this should just work with no issues. It barely counts as leaving the distribution at all. Let’s see how it does!

test_probes = [
    ("15 june 1912", "1912-06-15"),  # unseen value, every digit in a familiar slot
    ("15/06/1912", "1912-06-15"),
    ("15 june 2088", "2088-06-15"),  # same idea, the other century
    ("15/06/2088", "2088-06-15")
]

for human, expected in test_probes:
    got = generate(model, human)
    flag = "ok" if got == expected else "XX"
    print(f"[{flag}] {human:>16}  ->  {got:<12}  (wanted {expected})")
[XX]     15 june 1912  ->  1992-06-15    (wanted 1912-06-15)
[XX]       15/06/1912  ->  1972-06-15    (wanted 1912-06-15)
[ok]     15 june 2088  ->  2088-06-15    (wanted 2088-06-15)
[ok]       15/06/2088  ->  2088-06-15    (wanted 2088-06-15)

So, this is interesting. We do ok for 2088 but not for 1912. What’s different?

Well, back to the slot table: in a 19xx year (hundreds = 9), the tens digit was always 5-9 in training. A 1 in the tens slot beside a 9 is a pairing the model never saw - and here it quietly “corrected” it to a 9, a tens value it has seen in the 1900s.

But it’s the exact same issue with 2088, and the model has no issue. For whatever reason, the model has stronger priors about what 19xx years are allowed than what 20xx years are (you could try out a few more dates, and find this is generally true).

One more thing to notice, is that in all cases, the last digit of the year copies faithfully. As we saw, this is the one slot that was always allowed to be any digit, so it makes sense that it would copy even when other parts of the year don’t.

2) A year with digits in slots they’ve never been in. Next, we’ll put digits in places they’ve never been. This asks something real: Did the model learn a clean, position-blind “copy these digits,” or did it just learn what years tend to look like?

Test cases: 1850 and 3017

We should expect a well generalized model to do this, but can’t be sure. We already saw it struggle with the last example, and this one is harder. We might expect 3017 to do worse than 1850 because the start digit is different, so the model’s immediate understanding is worse. But maybe, it’ll default back to the “copy-exact” behavior we’d hoped for when it doesn’t have all of its assumptions of a “19xx” year behind it.

test_probes = [
    ("15 june 1850", "1850-06-15"),  # unseen value, easier case (year still starts 1)
    ("15/06/1850", "1850-06-15"),
    ("15 june 3017", "3017-06-15"),  # same idea, harder (year starts 3)
    ("15/06/3017", "3017-06-15")
]

for human, expected in test_probes:
    got = generate(model, human)
    flag = "ok" if got == expected else "XX"
    print(f"[{flag}] {human:>16}  ->  {got:<12}  (wanted {expected})")
[XX]     15 june 1850  ->  1950-06-15    (wanted 1850-06-15)
[XX]       15/06/1850  ->  1950-06-15    (wanted 1850-06-15)
[XX]     15 june 3017  ->  2017-06-15    (wanted 3017-06-15)
[XX]       15/06/3017  ->  2017-06-15    (wanted 3017-06-15)

So years this far outside the range have fully broken it. The model changed 1850 → 1950 (the novel hundreds digit 8 snapped to 9), and 3017 → 2017 (the novel thousands digit 3 snapped to 2).

The model did not learn a position-blind “copy these digits.” It learned that the thousands slot is a 1 or 2 and the hundreds slot is a 0 or 9, and when you hand it a digit that violates that, it overrides the copy and substitutes the nearest value it has seen. The fact that both 3017 and 1850 fail completely suggests the thousands and hundreds priors are each absolute.

50 and 17 did copy though - so the model is consistent. It has no trouble copying the final 2 digits correctly if they’re what it thinks can follow the first 2. Once it “corrects” the first 2, those can copy as-is without issue.

3) A year that’s the wrong length - Years of course aren’t all 4 digits, but the model has never seen that. We’ll try what we know to be a valid year, sometime before 1000. This is a very different task - it doesn’t just go outside of values the model expects, but outside of the structure it’s come to expect as a pattern.

Test cases: 841 and 57

We can expect that this probably won’t work. The model even needs to decide on the behavior for this. Does the year in ISO format become “0841” or stay “841”? It likely won’t interpret “57” as the last 2 digits of the year, as a human might. Even then, the task could be interpreted ambiguously. We’ll mean here “the year 57”, but even a human might assume you meant, for instance, “1957” and are just referring to it as ’57.

test_probes = [
    ("15 june 841", "0841-06-15"),  # three digit year
    ("15/06/841", "0841-06-15"),
    ("15 june 57", "0057-06-15"),  # same idea, harder (2 digit year)
    ("15/06/57", "0057-06-15")
]

for human, expected in test_probes:
    got = generate(model, human)
    flag = "ok" if got == expected else "XX"
    print(f"[{flag}] {human:>16}  ->  {got:<12}  (wanted {expected})")
[XX]      15 june 841  ->  1981-06-15    (wanted 0841-06-15)
[XX]        15/06/841  ->  1981-06-15    (wanted 0841-06-15)
[XX]       15 june 57  ->  2057-06-15    (wanted 0057-06-15)
[XX]         15/06/57  ->  1977-06-15    (wanted 0057-06-15)

As expected, it simply can’t represent a year that isn’t four digits, so it forces one. We wanted “00”, but of course learned in the last example that the “19” or “20” starts are absolute - so of course we instead get a 19xx or 20xx year.

But notice that the numbers in our short years aren’t necessarily reliably appearing, either. In the last example, 57 becomes 1977 - not 1957 - and 841 becomes 1981. It seems, again, that the ones place - the only slot that got to see every number - is the only one that reliably copies into the correct place.

In both cases though, we’re off the edge of the structure entirely, and just grabbing for the nearest valid-looking year.

Put the three probes together and a precise picture emerges - one that maps almost exactly onto the slot table we built. The model learned to copy the year, but with a per-slot prior whose strength tracks how constrained that slot was in training:

  • The units slot saw every digit freely, so it copies anything, no questions asked.
  • The tens slot saw every digit but only in certain pairings, so it mostly copies but will occasionally “fix” a bad pairing (our 1912 and 2088 cases), though inconsistently.
  • The hundreds and thousands slots only ever saw two digits each, so the model treats that as law, snapping any novel digit back to a seen one every time.

So “does it generalize to an unseen year?” Not really. Or rather, only as far as each digit-slot’s training coverage allows. If the data had full coverage on a slot, it generalizes perfectly. If the data pinned a slot down to a small set, it treats that constraint as a rule and overrides the evidence in front of it to obey.

9.4.3 Case 2: A Convention It Was Never Told

In all of the Case 1 instances, we focused on changing the values the model had to handle. This, of course, does change its understanding, but we still gave it those new values within the same structure it came to expect. We can try instead, to probe structures it hasn’t seen before, to see if it learns, not just generally what to do with numbers, but some semblance of meaning.

When we made the data, we chose a few rigid formats. Those were:

INPUT FORMAT EXAMPLE
%d %B %Y 03 march 2019
%B %d, %Y march 03, 2019
%d %b %Y 03 mar 2019
%b %d %Y mar 03 2019
%A, %B %d, %Y sunday, march 03, 2019
%d/%m/%Y 03/03/2019 (always day/month/year)

For instance, dates that were given in the “/” convention assumed %d/%m/%Y - day, then month, then year. We chose one so that “03/04/1994” was clearly unambiguous - the model only ever saw that as day first. But it’s also very common for that same date “03/04/1994” to be month first. So we can push the boundaries of the conventions we had it learn, to see if some of them are more general.

1) Day and month swapped. The first thing we can do is to try that exact %d/%m/%Y with swaps for day and month.

Test cases: - 07/04/1994 and 07/15/1994

In the first case, the model will certainly give us back April 7, 1994 - 1994-04-07. It cannot possibly know that we might have meant “July 4, 1994”, and indeed no one could without knowing what convention we were using. The model, not knowing two conventions even exist, will surely give back the interpretation that’s well inside its distribution - that the data was DD/MM/YYYY.

The second case though, is more interesting. There is no 15th month. So it can’t give back an ISO date that actually exists if it follows its usual conventions. But we also can’t really reasonably expect it to somehow now know that MM/DD/YYYY is also a valid format. We’ll see what this produces, but it’s likely that, similar to above, if it’s a good copier of pieces into the right place (so long as it thinks the numbers can go in that slot), it might just give back 1994-15-07, as it probably makes more sense to assume the rule still follows, and just that 15 is valid in the second slot, than that the convention had flipped.

test_probes = [
    ("07/04/1994", "1994-04-07"),  # ambiguous DD/MM/YYYY format
    ("07/15/1994", "1994-07-15"),  # no 15th month, must be MM/DD/YYYY
]

for human, expected in test_probes:
    got = generate(model, human)
    flag = "ok" if got == expected else "XX"
    print(f"[{flag}] {human:>16}  ->  {got:<12}  (wanted {expected})")
[ok]       07/04/1994  ->  1994-04-07    (wanted 1994-04-07)
[XX]       07/15/1994  ->  1994-15-07    (wanted 1994-07-15)

Both exactly as predicted. 07/04/1994 → 1994-04-07: the model reads day-first, gives April 7th, and has no way to know we might have meant July 4th. This is it doing well on the task, exactly within its wheelhouse.

And 07/15/1994 → 1994-15-07 has the failure mode we saw coming. The model has no idea ‘15’ isn’t a valid month, so it follows the ordering rigidly and puts ‘15’ in the month slot. It’s not “choosing a convention”, it’s just following the rule, since that’s all it knows.

One detail worth flagging: here the model faithfully copied the out-of-range 15 into the month slot, rather than snapping it to a valid month the way it snapped 8 → 9 in the year’s hundreds slot. But that’s consistent with what Case 1 taught us, because the month slot saw all twelve values 01-12 in training - both a “1” in the tens slot and a “5” in the ones slot, so it’s likely actually just ok with this.

2) An entirely unseen format. We can also try to give the model some formats that weren’t in the data at all. Not ones that we know are different but look identical to what it’s seen - ones that are genuinely different from the set of formats we gave.

Test cases: 2014, feb 09, and 19 f 2014 and 9 f 2014

The first case is the same information it’s seen before, in a different order. We’ve given it “shorthand month, day, year” before, so this just flips the year to be first. If it’s robust in recognizing snippets of patterns it’s seen, this would do okay.

In the second 2 cases, we’re giving genuinely never before seen pieces. In 19 f 2014 we drop the “feb” for just “f”. February is the only month of the year that begins with an “f”, and 19 has only ever been a day, so maybe this will work. In the 9 f 2014 case, we go a step further, also dropping the standard “0” formatting from the day. This is quite a different case from what the model has seen, though still a human would quickly figure out what this means. The model will likely get tripped up.

test_probes = [
    ("2014, feb 09", "2014-02-09"),  # swapped ordering
    ("19 f 2014", "2014-02-19"),  # "f" as only month information
    ("9 f 2014", "2014-02-09"),  # month and day both different to parse
]

for human, expected in test_probes:
    got = generate(model, human)
    flag = "ok" if got == expected else "XX"
    print(f"[{flag}] {human:>16}  ->  {got:<12}  (wanted {expected})")
[XX]     2014, feb 09  ->  2009-02-24    (wanted 2014-02-09)
[XX]        19 f 2014  ->  2014-07-19    (wanted 2014-02-19)
[XX]         9 f 2014  ->  2044-07-20    (wanted 2014-02-09)

So, interesting.

  • 2014, feb 09 → 2009-02-24 is the interesting case. If the model picked up general patterns, we expect this one to be easy: every token snippet in it (feb, 09, 2014) is something the model has seen hundreds of times, just never with the year first. Instead, it produced something with a lot of our original information garbled. 09 from the end of the sequence, got stuck on the year. Our years normally end our inputs, so it likely sees digits at the end and thinks those are the year. The “2” and “4” from the year (first and last digits) make it to the day, again, maybe because digits at the start are expected to determine the day. The ‘feb’ copies ok, but that’s in the place the model expects it to be. So this is the most reliable piece of evidence that the model has not learned portable facts like “feb is a month” or “a four-digit number is a year” that it can recognize wherever they appear - it’s more likely learned what positions of the input correspond to positions of the output.
  • 19 f 2014 → 2014-07-19 gets close. It nails the year and the day, but gives 07 - july - for the month. It’s never seen f before, so is probably just guessing.
  • 9 f 2014 → 2044-07-20. This is surprising. You maybe wouldn’t expect this to be too different from the previous case, but now even the year has an issue, despite being within range and in its correct spot. The day and month don’t seem based on much. We’ve likely pushed it so far out of it’s comfort range here, it just fills in something sensible, not strongly based on the input.

So did the model learn any generalizable meaning? Seems like the answer is mostly no. It learned the exact conventions and formats it was shown (day-before-month, these six layouts, these spellings of the months) as surface patterns, not as facts about what a day or a month or a year actually is. There do seem to be some pretty strong positional assumptions that drive a lot of how the inputs are processed. And of course, there are times where the patterns it has learned are reusable enough that we get some surprising results, but that’s not the same thing as learning “meaning”.

📌 Note: (In general, what we’ve seen here are some small, clean instances of what’s called shortcut learning (Geirhos et al., 2020). Models latch onto whatever rule happens to solve training, with no notion that you meant something more general.)

So, what went wrong?

Well, nothing. We saw 100% accuracy on our hold-out set - the model is exceptionally well trained to the problem. You might call this “overfitting”, but it’s not, because that 100% accuracy is on hold-out data - so the model generalizes to data it’s never seen, but is like what it’s seen, incredibly well.

So then, what do we call it, when a model doesn’t perform as well as we want on cases that are different in kind from what we trained it on? We don’t call it anything - you shouldn’t expect that! We call it poorly chosen training data. Because if you knew you needed your model to perform on certain cases, you should’ve trained on them.

In a lot of ways, generalizability is more of a “nice to have” than something you can expect. You don’t get it for free, but you’re not entirely powerless, and it does seem to show up more readily in some setups than others.

One thing you can do to encourage generalizability is the same way you prevent overfitting: regularization. These techniques stop a model from committing too hard to the specifics of its training data, which also keeps its learned rules a little softer and more flexible.

And our own result hints at a different one, with a concrete experiment you could run. Recall that our model was essentially done learning by around epoch 3 - the last several epochs mostly just drove the training loss lower. A model trained for fewer epochs would carry softer priors, so the “the hundreds digit is a 0 or a 9” rule would likely be less rigidly burned in. You could try it yourself: retrain with far fewer epochs, rerun these probes, and see whether the less-fit model is the more general one. The tradeoff, though, is fit for generality. A model stopped earlier in training might have more general understanding still, but it’s also likely not fit as well. That’s why optimizing for the cases you really want is the better solution - you don’t risk underfitting in the name of getting a model that’s well generalized when that can mean it only learned weakly how to solve problem.

But even then, temper expectations. Softer priors can only help the Case 1 probes where the model had all the necessary information and was just applying a rule too hard. Stopping training early likely won’t move the needle on the Case 2 tests, because there the information is coming in in a way it’s never seen before - essentially incomplete. For those, the real lever would be what we already discussed - if you need a behavior, put it in the data.

9.4.4 Takeaways

We could keep probing specific cases for a long time, but we already made the key point: there is no simple, finite checklist that tells you how a model handles everything outside its training. The space of “things you didn’t train on” is infinite - you can’t know how it does on literally everything out of distribution.

What you should do - and what we tried to do here - is reason first. Take a hard look at your data, and ask if it really covered everything that you need the model to be able to handle. Try to understand the pattern the model could possibly have learned from the data. Assume it picked the one that was a shortcut - does that apply to the cases you need it to?

But of course “learning the pattern your model learned” is often a really hard thing to put your finger on. In this example, we had 6 distinct date formats, and we still had to do some digging to try to understand what patterns the model was really applying.

For models at larger scale, the “patterns” are very much not clear. But also, larger models have been shown to, in general, generalize better. So let’s also take a quick step back to talk about how what we’ve seen here applies to the enormous, general purpose LLMs that most of the modern field revolves around.

9.4.5 Generalizability for Large Models

As it turns out, large language models are often strikingly robust to inputs they were never explicitly trained on. Part of why is simply coverage. If you train on a large enough slice of everything people write, then a great deal of what you might think was out of distribution was actually probably somewhere in the data all along.

Sometimes though, it is different in kind. For instance, large models exhibit in-context learning (Brown et al., 2020), where they can pick up a brand new pattern from a handful of examples given in the prompt itself. Our model obviously cannot do this. But a large model, if shown two or three mm/dd examples and what to do with them in its context, can simply adapt on the spot. It’s a real, qualitative gap between the kind of ultra-small, single-purpose model we built here and what runs at the frontier.

That being said, scale does not simply dissolve the problem of out of distribution behavior and generalizability gaps. Frontier models have still been shown to latch onto shortcuts, lean on spurious correlations, and stumble on tasks requiring too much extrapolation. And whether their most impressive generalization actually reflects something like real understanding, or just incredibly broad interpolation, is honestly still an open and actively argued question.

So the core idea still stands - a model can only extend what was present in its training data. It’s just that “extending what was present” can be staggering for large models. The “patterns” inside large-scale training data are much, much more than “the ordering of the words”, and capable models can recombine the structures they learn into things that appear nowhere in their data. The ability to generalize, then, isn’t hard-capped by novelty, but it is on information. A model can’t, for instance, supply a specific fact that was never derivable from anything it was given. It won’t know your name if you don’t tell it. That is the part that, definitively, no amount of scale or cleverness will solve if the pieces aren’t there.

So how far can you expect a model to extend?

  • For information you know it never saw and could never derive from what it did see, it’s a flat no. The information simply isn’t there to find, so you can’t expect this.

  • For information you know was squarely within what it saw, it should be a reliable yes. Given a model well-trained, shown that pattern and information often, this is the case you trained for, and exactly where you expect performance.

  • But for everything in between, which is most things, with some pieces it has seen, but asked to come together in a way that was never in training - it’s a big maybe. The whole middle is always a question mark.

Bigger models tend to reach further into the “maybe” space, and smaller ones less far. So capacity genuinely matters here, but you can never quite know by how much until you look. The wide uncertain middle is answered by probing, the way we just did, not by working it out in advance.

9.5 Benchmarking

Of course, even though evaluating a modern model is not as clean as evaluating something like our toy model, it’s useful to establish some generalized evaluations to compare performance. Yes, different models will be good at different things. Yes, how far you can push them out of distribution, and in what ways, is very hard to quantify. And yes, evaluations are best done on the specific things you care about. But if I’m choosing a model to use, how do I know to pick Qwen or Mistral at 8B parameters? If I train a model, how do I know how it compares to others?

So, the standard tool for answering “how do models compare” is the benchmark: a fixed, public test set (MMLU for broad knowledge, GSM8K for grade-school math, HumanEval for coding, and hundreds more) that produces one comparable number per model. Benchmarks are useful and necessary. You need some common yardstick to track progress and compare model performance, and standardized datasets are about the best you can do for that. However, on top of the general difficulty of evaluation we’ve already seen, benchmarks have their own specific issues that make a single number a far weaker signal than it looks.

Some issues with benchmarking:

  • Contamination. Benchmarks are public, so many of their questions, or light paraphrases of those questions, routinely leak into training data. Sometimes, models score brilliantly by having effectively memorized the test rather than being able to do the task. This is pervasive and very hard to detect.
  • Goodhart’s law. “When a measure becomes a target, it ceases to be a good measure.” Benchmarks are often easy to optimize for specifically, and scores on them can climb without the underlying ability climbing with them. Models can be very good at certain benchmarks specifically, and not good in general.
  • Narrowness. A benchmark measures one skill on one distribution. Topping a math benchmark says nothing about whether a model writes well, follows instructions, stays safe, or helps with how you want to actually use it. Compressing a whole model down to one number can throw away everything that doesn’t fit that one test. So that specific performance number can be useful when the benchmark is truly a direct analog to your use case - which it very often won’t be.

This is exactly why it can be a problem to read a common headline like “tiny new 2B model beats the giant frontier model on benchmark X!” and interpret it as “tiny new 2B model must be amazing”. This almost never translates to a general “the small model is better”. It’s usually “someone optimized a small model for benchmark X,” often on data close to the test, at the cost of being worse at nearly everything the benchmark doesn’t measure.

So is there something better? Yes, but it’s exactly what we discussed, evaluating on data the model could not have seen and measuring the thing you actually care about. There are examples of this more nuanced evaluation on a larger scale, like the use of human preference comparisons, where people judge real outputs head-to-head (Chatbot Arena is the well-known public version, and much harder to game than a static test), and increasingly, a strong model used as a judge (scalable, but with biases of its own). For a holistic view across many dimensions at once rather than a single score, HELM is the canonical effort for a well-rounded benchmark.

So benchmarks are not useless. And the field is actively working to make them better. They’re a necessary common language, and good ones to help move the field forward (like MMLU and its relatives). It’s just that it’s often better used as more of a “ground floor” than a verdict. Models should at least perform well on benchmarks, usually on many of them, if they’re generally good models. But doing better on one or two, especially of the easier ones to perform well on, doesn’t tell you all that much.

10. Inspecting the Model

Now, whether your model performs great or performs poorly, there are a lot of reasons you might want to look more closely at what it’s doing when it runs a forward pass. Not just what it outputs - literally, what goes on in there, so you can pick apart how the model is actually solving the task you’ve given it.

This is a huge field in AI research - often referred to as “interpretability”, and so there is a ton that people have done to test exactly what the model comes to internally represent about the problem.

Here, we’ll just skim the surface. We’ll mostly look at attention maps and see if we can say anything about what the model learned to do through them. And for our model specifically, that’s where there’s the most to learn, as a vocabulary containing only characters won’t have the same interesting relationships between embedded words that we saw back in the first few sections anyway.

10.1 Getting and Using Attention Weights

Our MultiHeadAttention returns the attention output, but it computes the attention weights along the way. Remember, that these are an intermediate product of attention - the softmaxed (query × key) grid that says how much each query position attended to each key position. These are useful to look at exactly because they don’t yet incorporate the values from those keys. Instead, they just give us a normalized view of how much every token looks at every other token.

10.1.1 Saving the Attention Weights

To inspect them, we’ll want to actually get the values out. Fortunately, we actually already quietly added a line to stash them back when we defined MultiHeadAttention. In the MultiHeadAttention.forward, definition, we added:

attention = torch.softmax(attention_scores, dim=-1)
self.attention_weights = attention.detach()   # <- save a copy for inspection

This just creates a version of them we can pull out during a forward pass. .detach() keeps them as a plain copy, so we don’t get the whole autograd graph with them.

This means that during any forward pass we can just save them. We’ll write a function to do just that - give us back the attention weights during a forward pass:

def get_attention(model, human, iso, kind="cross"):
    """
    Run one teacher-forced pass and pull out the RAW attention weights for one of
    the model's three attention operations.
    """
    model.eval()
    src = torch.tensor([encode_src(human)], device=device)
    trg = torch.tensor([encode_trg(iso)],  device=device)
    with torch.no_grad():
        model(src, trg[:, :-1])            # one pass fills in attention_weights everywhere

    if kind == "encoder":
        mods = [blk.attention       for blk in model.encoder.layers]
    elif kind == "decoder":
        mods = [blk.attention       for blk in model.decoder.layers]
    elif kind == "cross":
        mods = [blk.cross_attention for blk in model.decoder.layers]
    else:
        raise ValueError(f"unknown kind {kind!r} (use 'encoder', 'decoder', or 'cross')")

    # each module's weights are (1, heads, q_len, k_len); stack over layers, drop the batch dim
    A = torch.stack([m.attention_weights[0] for m in mods])   # (layers, heads, q_len, k_len)
    return A.cpu().numpy()

What this does, step by step:

  1. Puts the model in eval mode (dropout off), so what we inspect is its deterministic, deployed behavior.
  2. Runs a single teacher-forced forward pass - we hand it the whole known target so the entire output is produced at once. We don’t care about the output itself; we run the pass purely so that every attention module stashes its attention_weights. One pass fills in all of them - encoder, decoder, and cross.
  3. Then, we pick out the modules for where we actually want to get the attention from ("encoder", "decoder" self-attention, or "cross"), grabs each layer’s saved weights, and stacks them.
  4. It returns the whole thing as one array of shape (num_layers, num_heads, query_len, key_len) - which contains per layer, per head, the attention weight matrix.

When we run this function, we get back, for our chosen attention location, the query_len × key_len matrix, for every attention head, and every layer. Within this matrix, entry (i, j) is how much query position i attended to key position j. Because of the softmax, each row sums to 1 - it’s a probability distribution for how much that query - the thing that’s looking - needs to look at every key - the thing that has information to give - to find the answer. What “query” and “key” are referring to depends on the “kind” - for cross-attention, queries are output positions and keys are input positions; for the self-attentions, it’s one sequence attending to itself. When we stack all of the layers and heads together, we have the model’s entire attention story, for this one example, in a single object.

We pull all of it out of this function, but will ultimately end up slicing this into different portions depending on what we want to look at. Throughout this section, we’ll investigate what all of the different types of attention can tell us - we’ll look at the cross-attention, self in the encoder, self in the decoder, and at how they all differ with layer and head. Of course, for every part of this overall attention output that we take, we’ll explain exactly why we’re looking at that piece, and what we can tell from it.

10.1.2 Making Attention Maps

When we look at attention, we’ll look at maps. All this means is that we’ll plot the matrix itself as a heatmap, so that we can easily see the larger weights as brighter spots.

But this also means that we need to decide on a single query_len × key_len matrix to look at. In practice, this doesn’t need to mean choosing one layer and one head - we can average over heads in a layer, or even over layers, if we want to still get a single matrix out, but have it represent more than a single matrix within the model’s internals.

We will leave that choice agnostic, as throughout this section we’ll try many versions of ultimately making the maps. For now, let’s just write our simple functions that will help us visualize the matrices as maps once we have them:

def attention_labels(human, iso, kind):
    """Row (query) and column (key) tick labels for each attention kind."""
    if kind == "cross":
        # decoder output attends to encoder input: rows are what we produced, cols are what we read
        return list(iso) + ['<eos>'], list(human)
    if kind == "decoder":
        # decoder attends to itself: both axes are the decoder's own tokens
        toks = ['<bos>'] + list(iso)
        return toks, toks
    if kind == "encoder":
        # encoder attends to itself: both axes are the input characters
        return list(human), list(human)

def plot_attention(M, row_labels, col_labels, ax=None, title="",
                   ylabel="query (attending from)", xlabel="key (attended to)"):
    if ax is None:
        # size the figure to the matrix so the cells stay legible
        _, ax = plt.subplots(figsize=(max(4, len(col_labels) * 0.45),
                                      max(3, len(row_labels) * 0.40)))
    # vmin/vmax pin the color scale to [0, 1] so brightness is comparable ACROSS maps
    im = ax.imshow(M, aspect='auto', cmap='viridis', vmin=0, vmax=1)
    ax.set_xticks(range(len(col_labels))); ax.set_xticklabels(col_labels)   # label each column (key)
    ax.set_yticks(range(len(row_labels))); ax.set_yticklabels(row_labels)   # label each row (query)
    ax.set_xlabel(xlabel); ax.set_ylabel(ylabel); ax.set_title(title)
    plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)                       # weight → color key
    return ax

These two small helpers turn any 2-D slice we choose into a readable picture:

  • attention_labels defines the tick labels, to help us read directly which actual tokens are attending to each other. It depends on the “kind” of attention because the different attentions operate on different sequences (input vs output).
  • plot_attention does the visualization part. This just uses the simple matplotlib imshow, which renders the matrix as a heatmap. The one important thing for this visualization is the vmin=0, vmax=1. We pin the color scale to the full [0, 1] range, to make sure that a bright cell means the same thing in every map we make. This will be very helpful in later sections when we’re trying to directly compare maps.

So, now we’re all set up, and we’ll re-use the code we wrote here throughout the rest of Section 10.

10.2 Averaged Cross-Attention Maps

For our first stop, we’re going to look at cross attention, since it’s in a lot of ways the most intuitive type to interpret. Cross attention shows us how the decoder’s output (as query) is attending to the encoder’s input (as the keys). So, this is a direct alignment between the two sequences we care about, the human date and the ISO date it becomes.

We’re also going to do some averaging here - we’ll take just the last layer, and average over all of the heads. This is the best place to start, because the last layer is our best probe for what the model is about to output as its final answer, and averaging over the heads tells us, in general, what that layer is caring about.

10.2.1 A Success Case

Let’s start by just looking at a map and seeing what we can tell. One of the best places to start is a case where the model was successful - where the output it produces is what we want, because then we know that the attention there should be a good signal - it does what we want.

Let’s choose a date the model performed well on - pass it through to get the attention out, then take the last layer and average all of the heads:

human = "15 june 2017"
iso   = generate(model, human)
A     = get_attention(model, human, iso, "cross")   # (layers, heads, out_len, in_len)

M = A[-1].mean(0)                                   # <- the choice: last layer, averaged over heads
rows, cols = attention_labels(human, iso, "cross")
plot_attention(M, rows, cols,
               title=f"{human!r} → {iso!r}  (cross · last layer · head-avg)",
               ylabel="output (produced)", xlabel="input (read)")
plt.tight_layout(); plt.show()

So, what does a clean map tell us?

  • First, the day is a clean copy. The brightest cells in the map are between the input and output’s day numbers, so at least in a format it expects, it seems to just read the day numbers it needs from the input pretty reliably.
  • The month is better informed with more letters. The attention to the “0” and “6” from the output is generally more spread, but out over the letters of june. Interestingly, the “j” seems to motivate the “0” more strongly than it does the “6”, with the rest of “une” motivating “6” but not “0”. This would make sense - all “j” months start with a 0 in our ISO dates. But overall, all of the letters of “june” get some attention from the “0” and “6” for the month.
  • The year’s end numbers matter more than their starts. We already saw that this is likely how the model behaves from Section 9 - even though the output starts with the year, the “2” in the year output isn’t strongly motivated by the “2” in the input. And the “2” in the output, if anything, attends more to the input’s 0 (though very weakly) than it’s own 2. This already aligns with the failure modes we saw - where the year would start with a “19” or “20”, always.

It’s additionally telling when the attention is diffuse. Yes, the brighter spot for producing “0” in the output is the “0” from the year in the input - but it’s not overwhelmingly strong. In general, anytime the attention for something is diffuse, small differences can just be down to noise. Diffuse attention means the model doesn’t particularly have a strong signal to read off - but remember, thanks to the softmax, it must always put total 1 probability across the row, so you’ll always see attention somewhere.

Sometimes, that diffuse attention makes sense and is what you would expect. In this case, we’d say the attention is distributed, not really diffuse. It’s looking at different things, but has a clear picture of what it needs to look at. This is the case for the month - in this input format, there isn’t a clear single token to read from, so the attention is spread over the letters of the input month. But the “0” in the hundreds place of the year for the output is diffuse rather than attending strongly to the obvious “0” analog from the input - that tells you already that it’s not really considering that number from the input year when deciding what to go with.

10.2.2 What Values Matter

So, now let’s look at a failure case, where the model fails to get the values right, even though the information is readily available and in the exact format the model was trained to expect it in. Back in Section 9, we found that the model mangles unfamiliar years in a consistent way. When it sees 1850, it returns 1950, “correcting” an unseen hundreds digit to one it knows. So, how is that different from when it’s just given 1950?

These maps will hopefully tell us a bit more about what goes on when that happens:

humans = ["01 may 1850", "01 may 1950"]               # <- our two year cases
fig, axes = plt.subplots(1, len(humans), figsize=(11, 4.5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)                    # the model's OWN output
    A   = get_attention(model, human, iso, "cross") # (layers, heads, out_len, in_len)

    M = A[-1].mean(0)                               # same choice: last layer, head-avg
    rows, cols = attention_labels(human, iso, "cross")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (produced)", xlabel="input (read)")

plt.tight_layout(); plt.show()

So, these aren’t at all different, really! This confirms actually the hypothesis we had back in Section 9.4. Both the 19 and 18 in the output aren’t strongly motivated by their input analogs or read off from anywhere. The 9 in the output attends a teeny bit more to 9 when it’s in the input, but it’s very weak. Those first 2 digits are strong priors that the model has, and it’ll output them regardless of what the input is. Here, we see that ignoring of those digits from the input happening.

While we don’t know exactly where the “9” that overrides the “8” comes from, what we do know is that a digit the model reads straight from the input lands on its twin as a single bright cell, while a digit it overrides has nowhere to point, and smears across the row. In 1850 → 1950, the 5 and 0 lock onto the input’s 5 and 0 basically exactly as much as they do for 1950. The 9 that the model substitutes is diffuse. There’s no 9 in the input to read, because the model isn’t reading that digit at all. It’s imposing it, straight from its existing expectations. And actually, it’s doing it in both cases. The “9” never comes from the input year, it’s always just an assumption the model is making.

10.2.3 What Formatting Matters

Remember back in Section 9.4, when 2088 worked and 1912 didn’t, I said that if you tried it for a few more 20xx dates, you’d see it generally works. But we only tried two formats there. Let’s see how robust these attention patterns are for the same date in different formats.

humans = ["may 01 2088", "saturday, may 01 2088"]               # <- our two year cases
fig, axes = plt.subplots(1, len(humans), figsize=(11, 4.5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)                    # the model's OWN output
    A   = get_attention(model, human, iso, "cross") # (layers, heads, out_len, in_len)

    M = A[-1].mean(0)                               # same choice: last layer, head-avg
    rows, cols = attention_labels(human, iso, "cross")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (produced)", xlabel="input (read)")

plt.tight_layout(); plt.show()

So, this particular date is robust to formatting, too. A few things to note comparing these to other attention maps, though.

We can see that the model is doing better than just position-for-position copying. It’s robust to the day and month being flipped - it still puts the attention in the correct, and relatively the same, places when we flip the ordering.

In the spelled-out day of the week case, the model (appropriately) almost entirely ignores the “saturday”. This is also good - that information actually doesn’t motivate the ISO format at all, and the model knows not to spend much attention on it. Besides that, we see a lot of similar patterns in the attention. Though, not even the day of the week being there means the attention is a little more diffuse - we lose some of those bright yellow cells.

10.2.4 What Order Matters

Another case we encountered - when we instead gave the model the year first - it had no clue what to do with that. We weren’t totally sure what caused that, and we’ve seen it be robust to other position swaps, so can we figure out here what happened?

humans = ["2014 feb 09", "09 feb 2014"]               # <- our two year cases
fig, axes = plt.subplots(1, len(humans), figsize=(11, 4.5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)                    # the model's OWN output
    A   = get_attention(model, human, iso, "cross") # (layers, heads, out_len, in_len)

    M = A[-1].mean(0)                               # same choice: last layer, head-avg
    rows, cols = attention_labels(human, iso, "cross")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (produced)", xlabel="input (read)")

plt.tight_layout(); plt.show()

There’s something really interesting going on here, which is that the shapes of the two maps are very similar. It doesn’t matter if the year was first - the model gives the same attention pattern as if it was the day that came first. But in the previous example, we saw plenty of flexibility to swapping day and month.

But, we have some rigid formatting patterns across all of our examples, that it probably learned just as it learned our rigid slot-digit relationships. In our training data, every time an input starts with a digit, those digits are the day - that allows that to be a fixed assumption the model makes. It’s also true of the year, in our training data, no matter the format, that was always last. So it’s likely not that an edit to this specific format causes a problem - the model probably just sees a digit come in the first slot and says “ok, that’s the day”. And indeed, during training, that was a rule.

10.2.5 What Tokens Matter

One case that we saw back in Section 9.4 was 19 f 2014. The model got the day and year right, but couldn’t figure out the month, even though “f” is a unique letter to “february”. Instead, it gave us 07.

Let’s look at the attention maps for both 19 f 2014 and 19 feb 2014 and see how they’re different. Do we see stronger attention to e and b than we might expect for how tightly f should link to 02?

humans = ["19 f 2014", "19 feb 2014"]               # the examples
fig, axes = plt.subplots(1, len(humans), figsize=(11, 4.5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)                    # the model's OWN output
    A   = get_attention(model, human, iso, "cross") # (layers, heads, out_len, in_len)

    M = A[-1].mean(0)                               # same choice: last layer, head-avg
    rows, cols = attention_labels(human, iso, "cross")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (produced)", xlabel="input (read)")

plt.tight_layout(); plt.show()

So, these particular attention maps aren’t that telling. The model attends pretty strongly to f for the output month in both cases. But, when it’s just f it becomes 07, and when it’s feb, it seems that it uses the f to get 0 and the e to get 2. This is reasonable, still. “February” being the only month to start with f doesn’t mean f must shoehorn it - that also means the presence of an f shoehorns 0. Then, maybe the model got used to an e in the second slot informing 02 after an “f”. It’s hard to guess at what rule the model is making - even a shortcut rule you could think of isn’t necessarily the one it’ll take.

Let’s look at one more similar example, “july” and “june”, as they have a similar issue - with some letters matching, what does the model use?

humans = ["19 july 2014", "19 june 2014"]               # the examples
fig, axes = plt.subplots(1, len(humans), figsize=(11, 4.5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)                    # the model's OWN output
    A   = get_attention(model, human, iso, "cross") # (layers, heads, out_len, in_len)

    M = A[-1].mean(0)                               # same choice: last layer, head-avg
    rows, cols = attention_labels(human, iso, "cross")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (produced)", xlabel="input (read)")

plt.tight_layout(); plt.show()

So, we see here maybe more what we would expect, with strong attention to the l and n to decide the month number. And in both, the 0 looks again more closely at that j. So maybe, if anything, we see a similar sort of rule forming here, where the model reliably gets several month letters, and has learned to mostly pick the first digit - “0” or “1” - with the first letter, and the second digit with the rest.

So, we’ve looked at a lot of maps here. And it’s clear that sometimes, the model is reading the answer from the input, and other times, it’s diffuse. Whether or not this makes sense really just depends on case. Sometimes, it’s pooling a signal spread across several tokens (the letters of a written-out month). But other times, it’s clearly stopped consulting the input altogether and fills the slot from a learned prior (like with year digits it overrides).

But you can see, hopefully, how looking at the attention can tell you something different from what just the results do. For example, these maps reflect the model’s positional assumptions as much as the input’s content, which we saw in the case where we put the year first. But there’s also the other case, where, for instance, we were able to see how the exact letters are being considered. Other times, the maps can’t tell us much. They can’t tell us why 1912 is a problem and 2088 is ok.

That isn’t a failure of our reading or of the tool of attention maps - just a real limitation. A cross-attention map shows you where the model looks, but it can’t tell you what it does with what it finds. That can be very useful, but it’s certainly not the entire story.

10.3 Attention Per Head and Layer

For all the maps we just saw, we took the attention of the last layer and averaged over heads. This is a good general metric, but averaging wipes out differences in what the heads look at - and the whole point of heads is that they should learn to specialize. Additionally, while the last layer is easiest to interpret in that it’s the final set of weights before the output, it’s of course not the only one that matters.

So let’s try to poke a little deeper - seeing how attention differs across heads and layers - to see if we can figure out any more nuanced structure.

10.3.1 Attention Across Heads

First, instead of averaging over all of the heads in a layer, let’s just look at all of them. Fortunately, our model only has 8 heads, which isn’t so bad to just inspect individually.

We’ll still take the last layer here, but look now at every cross attention head for a working example within that layer.

human = "15 june 2017"                               # one specimen, now at full resolution
iso   = generate(model, human)
A     = get_attention(model, human, iso, "cross")    # (layers, heads, out_len, in_len)

layer = -1                                           # the choice: open up ONE layer...
rows, cols = attention_labels(human, iso, "cross")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))      # ...and show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (cross · last layer · each head)")
plt.tight_layout(); plt.show()

The first thing that jumps out is that the averaged map dilutes a lot of the sharpness that’s actually there. Every individual head is nearly binary - with cells either near 1.0 - put all your attention here - or otherwise 0 - don’t look at this at all. So all of the soft, diffuse signal we saw in the previous section wasn’t about heads being wishy-washy - it was precisely because the different heads are telling the model to look at different things.

Which is great, because that’s exactly what multi-head attention is supposed to provide. The heads should be able to specialize to their own part of the problem. We can see that division of labor here when we look more closely:

  • All of the heads do day copying. All except 0 have bright attention between 1 and 1 and 5 and 5 - all maps have learned that rule.
  • All heads participate in choosing the month, but different letters are in focus for different heads. All heads have at least one bright spot between one of the month letters and the first or second month digit, so it’s more of an aggregate over them that’s likely deciding month.
  • Heads 0, 1, and 2 are the ones that give the most attention to the whole year, though similar to the day, the last 2 digits of the year show up in almost all maps. The first two digits are messier. Heads 5, 6, and 7 do pick out the 2 correctly, and heads 0, 1, and 2 put the output 0 on the input 0. But those same three heads put the 2 on that input 0 as well, so they’re treating the first two digits as one block rather than copying them one by one. No single head cleanly copies both. Heads 3 and 4 attend those first 2 digits back to the month and day instead.

So our reliable pure copies show up in all heads - the second digit of the day in all, and the last digit of the year and first digit of the day in all but 1. These are the exact digits that we saw get faithfully copied again and again, and they are attended to by nearly all maps. This, of course, also shows up in our averaged maps clearly - there was always a reliable bright spot on those two digits, and we see here why - because every attention head knows those are important.

But this is also something worth pausing on, because just from this use case, it might seem like having multiple heads can hurt you. The digits that we saw the model get consistently correct are bright in every single map, and the digits that the model was shakier on are still sometimes bright - just only in one or two. Does that mean the model should have learned to make all of the heads the same?

The answer is no, of course. But let’s be precise about why not. Remember how multi-head attention actually comes back together. We averaged because that’s convenient to look at, but the model does not average its heads. Each head computes its own output, and those outputs are concatenated - laid side by side, and kept separate - and then mixed by the output projection.

So a digit that’s bright in only one head isn’t “weakly attended to” from the model’s point of view - if one head carries a clean signal into the concatenation, the output projection is free to lean on it entirely. This would be true even for the digits that are bright in every single head - the output projection is free to discount it entirely.

The problem with, for instance, the first 2 digits of the year here isn’t that they aren’t bright in enough heads, it’s more likely that they’re too bright in too many heads. The output projection could learn which head to trust to pull that digit from, but with a less clear signal, that’s harder to do. The attention to that digit is neither decisive on a broad scale (like the last digit of the year was) nor decisive within specialized heads. The signal is spread out, so picking it out is harder. And more so, it’s likely that, as we saw with those digits snapping to the 20 and 19 the model knows, it’s probably not really deciding them from the attention at all. Attention says “hey, you need to look at this thing”, but just because the model looked, doesn’t mean it does anything with it.

It’s tempting to go further and start picking apart what happens downstream, in the value mixing, the output projection, and the feed-forward layers, but those are in general much harder to look at, so we’ll skip them. And we already know what happens from the behavior we saw in Section 9: hand the model any year and it still produces a 19xx or 20xx, snapping the leading digits to what it expects no matter what the input said. So we know those digits aren’t being set by the input at all, and instead by the assumptions the model makes. The messy, conflicting head-attention here is consistent with that, and finding where exactly that information gets missed is a deep dive for a different kind of notebook!

But one more thing we can do, just to sanity check some of our claims we made from the single example above, is just look at the set of maps for a different date, in a different format. Are the heads still specialized in the same way?

human = "12/11/2017"                               # a different date choice
iso   = generate(model, human)
A     = get_attention(model, human, iso, "cross")    # (layers, heads, out_len, in_len)

layer = -1                                           # the choice: open up ONE layer...
rows, cols = attention_labels(human, iso, "cross")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))      # ...and show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (cross · last layer · each head)")
plt.tight_layout(); plt.show()

There are some consistent things these heads are doing. First off, the day digits and the final digit in the year are strongly attended in all but 0, same as in the last example. That’s expected. But notice where the specialization is happening. Heads 2, 3, 4, 5 and 7, are where the month is most straightforwardly attended, which is similar to where “june” had some of the strongest signal on “j” and “u”. So those heads probably are the ones most responsible for the month, it’s just that when the month is spelled out, that shows up in more pieces than it does for the clean format we have here.

But… you also might look at this and say: “Sure, I see some specializing, but if a lot of what is done well just shows up as the same sort of map, do we even need eight heads?” And that in general is a very fair question - it’s actually been shown that trained models often do carry more heads than they strictly need. But - you can’t really conclude it from maps like this alone. A redundant (or, even a flat) head isn’t necessarily not doing anything. Near-redundant and near-uniform attention both still produce an output, and a head can carry its real contribution through its values and the output projection rather than through the specific pattern we can see. On top of that, as we touched on here, different inputs may activate different heads more or less. Some heads can be specialists for a specific formats or values, and just don’t have much nuanced to say about some specific case.

So asking “do we need this head” is an important question - and too much attention head redundancy or consistently flat maps could be an early sign. But it’s not a diagnosis. To really determine if the head is useless, we’d need to try to remove it and see if the model misses it - a technique called head pruning. That’s again, way out of scope for this notebook, so we’ll just say that for those heads, we can’t really tell what it is exactly they provide from maps alone.

10.3.2 Attention Across Layers

The next thing we’ll do is look at how attention differs across layers. The intuition for different heads is that they’re meant to specialize across different features. The intuition behind layers is that they tend to sharpen features - earlier layers will look more broadly, and later layers will be more focused. Let’s see if that’s true.

We’ll return to the average over the heads, to try to get that aggregate sense again for what attention is doing. So we’ll just see what the average attention looked like at each layer depth.

human = "15 june 2017"                                  # the same specimen as 10.3.1
iso   = generate(model, human)
A     = get_attention(model, human, iso, "cross")      # (layers, heads, out_len, in_len)
rows, cols = attention_labels(human, iso, "cross")

fig, axes = plt.subplots(1, A.shape[0], figsize=(5.5 * A.shape[0], 4.5))
for l, ax in zip(range(A.shape[0]), axes):
    plot_attention(A[l].mean(0), rows, cols, ax=ax, title=f"layer {l}")
fig.suptitle(f"{human!r} → {iso!r}  (cross · head-avg · each layer)")
plt.tight_layout(); plt.show()

So, the intuition holds for layer 0, but layer 1 is actually already pretty sharp. As we get further in depth, overall the structure tightens. Layer 0 is a pretty broad, low-contrast wash, with a bit of focus on the year. The model at this point is gathering information widely, not committing to anything yet.

But by layer 1, we see the model start to hone in on the days, a little bit across the month, and the last 2 digits of the year. By layer 2, the second digit of the day has fully sharpened, and more attention has collected around key letters of the month (the “n”) and year. As we saw splitting into heads, at least for the month that’s probably more about a real separation of signal, not it being indecisive.

It should also be more obvious now why we chose the last layer, for the most part, to investigate attention maps. In the final layer, the model is more decisive about where to put its focus, as those attention weights are more directly influencing the output. In earlier layers, attention is so spread because attention heads are looking at many different things, gathering more information, and it can be hard to determine where the key attention points will end up.

But, just while we’re here, let’s confirm that with one more set of plots. Let’s look at all of the heads in the first layer instead of the last.

human = "15/06/2017"                                 # a different date choice
iso   = generate(model, human)
A     = get_attention(model, human, iso, "cross")    # (layers, heads, out_len, in_len)

layer = 0                                            # the choice: open up ONE layer...
rows, cols = attention_labels(human, iso, "cross")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))      # ...and show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (cross · first layer · each head)")
plt.tight_layout(); plt.show()

You might’ve expected these to look diffuse, but even early on, the heads are learning to put their attention in specific places. It’s just that they’re grabbing for more things across heads. Look at the 5 in the day. Across heads, it attends to: the 5 in the day (itself) (heads 0 and 7), the 0 in the month (head 1 and 5), the 1 in the day (heads 2 and 3), the 6 in the month (head 4), the / after the day (head 5), and the 7 in the year (head 6) - a far cry from the decisive 5 \(→\) 5 we see in the last layer.

But is that wrong? No! Earlier layers in the model are still going to be transformed by later layers, so their focus is more on prep work. This is clear in the structure they do have. Head 5 spreads attention all over the place. Consistent verticals, like heads 2, and 6 have, seem to be mostly about attending the digit in that slot to everything, making sure all of the output knows about it. Early structure prefers this type of attention over clean copies, because before the model can copy the right digit, it has to work out which part of the input is the day, the month, the year - and that’s a matter of relating positions to each other. This goes back to something that we’ve talked about many times in this notebook - the decoder itself still has a job of understanding to do, and we can see it here. Attention like this is the model understanding how the input structure needs to relate to its output, which is a key prerequisite to making decisions that depend on that structure.

So, all layers of attention are doing something important. It’s just that earlier layers can be harder to interpret, because they’re informing later layers of the model more so than the output from the model. It’s harder for us to say exactly how this structure will return a result, but we can trust that it’s important for the model, because the model has decidedly honed in on those features.

10.4 Self-Attention Maps

When we read the cross-attention maps, some things came through crisply - the day, the month. But other parts were stubbornly diffuse, right where you’d least expect it: the leading digits of the year, which the model isn’t sourcing from the input even though an obvious source is sitting right there. That left a loose end. The model isn’t guessing randomly - hand it the same date and it produces the same output every time, so something is deciding those digits. If it isn’t the input, where is it?

The only other thing the decoder has to look at is its own output so far - the characters it has already produced. That’s exactly what the decoder’s self-attention sees. So this is the natural place to chase the loose end down: if the hundreds digit isn’t read from the input, maybe it’s read from the thousands digit the model just wrote. Let’s look.

(There are two self-attentions - one in the encoder, one in the decoder - so we’ll look at both. The encoder’s is the quieter of the two; the decoder’s is where our loose end might get tied up.)

10.4.1 Encoder Self-Attention Maps

Let’s start by looking at the encoder’s self-attention, because it comes first in the model’s flow. Before the decoder generates anything, the encoder passes the input through its own attention, trying to build a rich representation of the input. Its job is to let each character of the date gather context from the rest.

This means, unlike cross-attention where we expect to see where the output is coming from, we expect to just be able to tell what tokens are contextualized by other ones. The encoder should be where the input gets organized. So, for instance, where in cross attention we saw that the output could read the month off just a couple of the letters of june - we’d expect that the encoder does some work to gather the letters of “june” into some coherent signal - maybe we’ll see attention across those letters to each other. That being said, expect these maps to be quieter than cross-attention, because the job the encoder has to do is less “clean”, and encoding naturally means spreading across more of the input.

We’re going to go back to looking at averages first, as interpreting a single map is always cleanest. And, we’ll go back to the last layer, where there’s theoretically the clearest signal to see.

# encoder self - the full square (no mask), + the month-pooling check
human = "15 june 2017"
iso   = generate(model, human)
A     = get_attention(model, human, iso, "encoder")   # (layers, heads, in_len, in_len)

M = A[-1].mean(0)
rows, cols = attention_labels(human, iso, "encoder")
plot_attention(M, rows, cols,
               title=f"{human!r}  (encoder self · last layer · head-avg)",
               ylabel="input (attending from)", xlabel="input (attended to)")
plt.tight_layout(); plt.show()

So, the encoder’s self-attention is a pretty quiet map, just as we thought it might be. But this map is very flat - we really can’t pick out anything distinct at all besides a little on the day, a little on the year, and some spread for “june”.

But, we saw in the last section how an unstructured map like this might just mean head-averaging killed our signal and not that it’s not there, so let’s take a look at the heads:

human = "15 june 2017"
iso   = generate(model, human)
A     = get_attention(model, human, iso, "encoder")    # (layers, heads, out_len, in_len)

layer = -1                                             # last layer again
rows, cols = attention_labels(human, iso, "encoder")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))        # now show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (encoder self · last layer · each head)")
plt.tight_layout(); plt.show()

So we do see some structure here. Still not as strong and crisp as the cross-attention, but these maps are clearly serving a purpose.

  • There are a lot of vertical stripes here, which work to pool together nearby information. Heads 6 and 7 aggregate more so over the month, where heads 3, 5, and 7 seem to pull together year information.
  • Some do lock digits onto themselves or to their neighbors, but different ones handle different digits. So the same division of labor we saw in the cross-attention grid shows up on the input side too.

So, the encoder is organizing its input, it’s just not as obvious. It does some work to pool together some information, and anchor some digits or segments. The average blurs what the specialists are doing, but they’re there.

10.4.2 Decoder Self-Attention

Now, the decoder. The decoder’s attention is output-to-output, so here we’ll see how each new token the model generates attends to the tokens it has already generated. Besides the input, which we looked at in cross-attention, this is where the model gets the other part of its decision-making from.

This is where that loose end about the years might finally get tied. We saw that the year’s leading digits don’t come from the input, because cross-attention went diffuse where an obvious source was sitting. So if they come from anywhere the model can see, it’s from the output it has already produced. We’ll hopefully see in these maps, if we can figure out where the year “correction” might stem from.

A few things to note before we look at this. First, these maps will be triangular, because the causal masking doesn’t allow looking ahead. Second is about how we have to read these, because it’s different from the ones we’ve read so far. For the decoder, each row’s attention is the computation that produces the next token. So to ask where a digit in the output (attended to) comes from, you look at the previous digit’s row. Essentially, say I want to understand what motivated the final digit in the year 2017. I’d look at what the 7 “attended to” on the x-axis, and follow it up to the row that corresponds to 1 from the “attending from”, because each new token is generated from the attentions of the previous.

Finally, we should expect this to carry more visible structure than the encoder. Since in the decoder, the final layer is back to being right before the output, this is closer to an actual decision.

# decoder self - the triangle; a match case beside an override case
humans = ["15 june 2017", "01 may 1850"]               # <- your match + override strings
fig, axes = plt.subplots(1, len(humans), figsize=(11, 5))

for human, ax in zip(humans, axes):
    iso = generate(model, human)
    A   = get_attention(model, human, iso, "decoder")  # (layers, heads, out_len, out_len)
    M   = A[-1].mean(0)
    rows, cols = attention_labels(human, iso, "decoder")
    plot_attention(M, rows, cols, ax=ax,
                   title=f"{human!r} → {iso!r}",
                   ylabel="output (attending from)", xlabel="output (attended to)")
plt.tight_layout(); plt.show()

The decoder’s attention is dominated by, if anything, just a generic autoregressive pattern. Tokens mostly lean on a few tokens right before it. That’s a sensible thing for a generator to do (keep track of where you are), but it isn’t specific to any of the tokens themselves. Some rows are just generally, uninterestingly, spread, which doesn’t tell us very much.

Here, we also put a map where we successfully copied the year, and see that the map has nearly the same shape as when we didn’t successfully copy the year. Of course though, by now we know not to trust the averaged maps as an absolute verdict, so let’s go the step further and look at the heads themselves:

human = "15 june 2017"
iso   = generate(model, human)
A     = get_attention(model, human, iso, "decoder")    # (layers, heads, out_len, in_len)

layer = -1                                             # last layer again
rows, cols = attention_labels(human, iso, "decoder")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))        # now show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (decoder · last layer · each head)")
plt.tight_layout(); plt.show()

# decoder self - the triangle; a match case beside an override case
human = "15 june 1850"
iso   = generate(model, human)
A     = get_attention(model, human, iso, "decoder")    # (layers, heads, out_len, in_len)

layer = -1                                             # last layer again
rows, cols = attention_labels(human, iso, "decoder")
fig, axes = plt.subplots(2, 4, figsize=(16, 7))        # now show all 8 heads of it
for h, ax in enumerate(axes.flat):
    plot_attention(A[layer, h], rows, cols, ax=ax, title=f"head {h}")
fig.suptitle(f"{human!r} → {iso!r}  (decoder · last layer · each head)")
plt.tight_layout(); plt.show()

So, it’s a lot to look at, but once again, a specific hypothesis is best confirmed by looking at all of the heads, because the average can wash them out, and we see some real structure from this view. I’ll just point out a few things.

The biggest difference is how head 3 acts on these two cases. In the case of the year it gets right - 15 june 2017 - this head is diagonal over the year digits - that pure autoregressive “look at the previous digit for the next”. But for the 15 june 1850 case, we see a vertical line over the <bos> or the 1 instead. Heads 2, 4, and 6 seem to do a better job attending to the previous token, the autoregressive behavior, than the 1850 case, which you might expect is doing more guessing. Most other heads are largely the same - and we do have the same format and day/month combination here.

So, what can we tell from what we see? Well, we can see that attention is different when the model is inventing rather than copying, but we still don’t get to see where the invented digit is coming from. Head 3 “messing up” and attending to <bos> isn’t a “source” of information so much as it’s a fallback. And heads 2, 4, and 6 all more cleanly attending to the token behind them might just mean the model has more license to “make it up”, based on what it does with this information downstream. We really can’t say for sure what actually happens to this information. All we can really tell is that between cross-attention, which never locked onto the input year, and self-attention, which pretty well follows the autoregressive pattern in both cases, though for different heads, - the model doesn’t clearly just “get” the information from somewhere. The 9 comes from somewhere else that attention maps can’t tell us.

10.5 Attention Maps Come to Life

Every map we’ve looked at so far has been a snapshot from the finished model. This is useful to figure out what the model ended up with, but of course, a model’s attention learns and adapts during training. Elements that are obvious in the final map, like the clean digit copies, the specialized heads, the diffuse year, are all features that form during training.

So one more thing we can do, is actually watch it form. If we look at how attention takes shape over time, we can see it form, in real time, what eventually become the rigid rules we saw in the final maps.

10.5.1 Attention Forming Over Training

When we were training, I saved off frequent checkpoints and said we’d use them later. This is what we were saving them for. Looking at how attention forms over time means we need to be able to look at what attention looked like during previous forward passes, which are lost if we don’t save the checkpoints. Here, we can just re-run our calculation through those checkpoints to see how attention evolved.

We’ll just look here at cross-attention, as that’s the most telling, and again the average over heads and last layer so we just have one object to look at.

📌 Note: Even if you reloaded my model checkpoint earlier, that does not have all of the checkpoints needed to create this visualization. So don’t re-run this cell unless you’ve also re-run this whole notebook so far, with a newly trained model and your own checkpoints folder.

human = "15 june 2017"
iso   = "2017-06-15"                     # FIXED target, teacher-forced at EVERY frame
rows, cols = attention_labels(human, iso, "cross")   # computed once; identical every frame

CKPT_DIR = "checkpoints"
manifest = json.load(open(f"{CKPT_DIR}/manifest.json"))
frame_paths = []
for step, path in manifest:
    model.load_state_dict(torch.load(path, map_location=device)); model.eval()
    M = get_attention(model, human, iso, "cross")[-1].mean(0)
    fig, ax = plt.subplots(figsize=(6, 5))
    plot_attention(M, rows, cols, ax=ax, title=f"step {step:>5d}    {human!r} → {iso!r}")
    fp = f"{CKPT_DIR}/frame_{step:06d}.png"
    fig.savefig(fp, dpi=110)             # fixed figsize+dpi, NO bbox_inches='tight'
    plt.close(fig)
    frame_paths.append(fp)

frames = [imageio.imread(fp) for fp in frame_paths]
imageio.mimsave("attention_forming.mp4", frames + [frames[-1]] * 8, fps=2)
WARNING:imageio_ffmpeg:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (660, 550) to (672, 560) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
data = base64.b64encode(open("attention_forming.mp4", "rb").read()).decode()
HTML(f'<video src="data:video/mp4;base64,{data}" controls loop autoplay muted width="600"></video>')

So you’ll notice that it actually hones in really fast. We have many more checkpoints saved earlier in the model here, so while we do see the first several steps represent still the largely untrained model, the attention signal forms very quickly. It zeroes in on day-to-day, month-to-month, and year-to-year in just a few hundred steps. The day is the quickest signal to be firmly decided, with the month and year continuing to change their shapes slightly until more towards the end. Though of course, a good amount of that is due to natural randomness, and some of the effect is washed out from the averaging.

But one question this does raise, is why does this seem to settle so fast? And it’s really because, in this case, the problem really is that easy. “Copy the digit” is about the simplest thing you can ask a transformer to do, so it makes a lot of sense that the model has essentially decided what it needs to look at by the end of the first epoch. A lot of what we see after that really is just some randomness, not the model changing its mind on what to look at.

10.5.2 Entropy Curves

The video makes the sharpening obvious to the eye, but “obvious to the eye” isn’t something you can quantify. Fortunately, this effect is something we can quantify fairly easily.

Remember what a single row of an attention map really is: a probability distribution. Every query spreads a total of 1.0 across the keys it could look at. A sharp row dumps almost all of that 1.0 onto one or two keys, so the model is sure where to look. A diffuse row spreads it thinly across many, with no strong preference. So “how sharp is this attention” is really “how concentrated is this distribution,” and that’s a question with a standard answer.

That answer is entropy. For a distribution with probabilities \(p_1, p_2, \dots\) over the keys, \[ H = -\sum_i p_i \log p_i \]

If all the weight lands on a single key (\(p=1\) there, \(0\) elsewhere), the sum is \(0\), so zero entropy means perfectly concentrated. If the weight is spread perfectly evenly across \(n\) keys (\(p = 1/n\) each), the entropy hits its maximum, \(\log n\). This gives one number per row, so to get a single number for a whole map, we average the entropy over all the query rows.

We’ll also plot two things here. First, the entropy of the head-averaged map, which is useful, but we know the blend is not the sharpest signal. So we’ll also plot the entropy of the single sharpest head (the most concentrated specialist at that moment). The average says how focused the overall picture looks, and the second says how focused the model’s most specialized individual head has gotten.

human, iso = "15 june 2017", "2017-06-15"
manifest = json.load(open(f"{CKPT_DIR}/manifest.json"))

def row_entropy(M):                       # M: (q,k), each row sums to 1
    p = np.clip(M, 1e-9, 1)
    return (-(p * np.log(p)).sum(1)).mean()   # mean entropy over query rows (nats)

steps, avg_H, best_H = [], [], []
for step, path in manifest:
    model.load_state_dict(torch.load(path, map_location=device)); model.eval()
    A = get_attention(model, human, iso, "cross")
    steps.append(step)
    avg_H.append(row_entropy(A[-1].mean(0)))
    best_H.append(min(row_entropy(A[-1, h]) for h in range(A.shape[1])))

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(steps, avg_H,  'o-', label="head-average")
ax.plot(steps, best_H, 'o-', label="sharpest head")
ax.axhline(np.log(len(human)), ls='--', c='gray', lw=1, label="uniform (diffuse)")
ax.set_xscale('symlog')                       # so step 0 shows on a log-ish axis
ax.set_xlim([0,10000])
ax.set_xlabel("training step"); ax.set_ylabel("attention entropy (nats)")
ax.set_title("Cross-attention sharpening")
ax.legend(); plt.tight_layout(); plt.show()

So a few things. First, the drop is a cliff, just like we saw. After around 200-300 steps, it starts to plummet. By 1000 steps, not much more additional sharpening happens overall, though the sharpest head does continue to get sharper. With 50,000 examples at batch size 16, one epoch is 50000 / 16 = 3,125 steps - so even by 1/3 into the first epoch, the model has decided most of what information it needs to get, from where.

The second thing is that gap between these two lines is exactly what we already saw with head averaging - the sharpest single head plunges far lower than the head-average, because the average blends different specialists, but the heads are specializing strongly.

The overall speed here acts as a kind of a difficulty gauge. Our toy locks in almost instantly because its task is trivial. A “real” model would train for hundreds of thousands of steps precisely because its problem is hard, and this entropy would take much longer to come down rather than crashing like we see here.

10.6 The Input Embeddings

Pivoting away from attention, there’s one more easy thing worth looking at: the embeddings that come out of the input. Back in Section 1.4 we saw that real word embeddings like GloVe carry a surprising amount of structure, where similar words land near each other, and you can even do math over the geometries.

That comes from learned embeddings, and our model learned its own embeddings for our tiny vocabulary. So the natural question is: did they pick up any structure of their own? While probably not as rich as words that get contextualized by a whole text corpus, we still might expect to see something here. The digits especially have an obvious candidate as they’re numbers. So let’s look at them and see if anything interesting came out.

10.6.1 Principal Component Analysis

Each embedding is a 128-dimensional vector, which is impossible to view directly. So we’ll use PCA (principal component analysis), which finds the two directions along which the vectors vary the most and projects everything onto them. PCA is useful, but it’s a very compressed view of what’s actually there. So while this can be a good way to view high-dimensional data in lower dimensions, we definitely lose a lot of nuance.

In fact, we’ll show on the axes how much of the vectors’ total spread each direction captures, so we can see just how much of the variance we lose by doing this.

# ---------------------- get the embeddings out -------------------------------
embeddings = sorted((m for m in model.modules() if isinstance(m, nn.Embedding)),
                    key=lambda m: m.num_embeddings)
W_out  = embeddings[0].weight.detach().cpu().numpy()   # (out_vocab, d_model)
# -----------------------------------------------------------------------------

show   = list("0123456789-")                           # content tokens only
idx    = [i for i, t in id_to_char.items() if t in show]
labels = [id_to_char[i] for i in idx]
W      = W_out[idx]

# PCA by SVD: center, decompose, project onto the top 2 directions
Xc = W - W.mean(0, keepdims=True)
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
coords = Xc @ Vt[:2].T
evr = (S**2 / (S**2).sum())[:2]                         # how much the 2 axes explain

fig, ax = plt.subplots(figsize=(6, 5))
ax.scatter(coords[:, 0], coords[:, 1], s=20)
for (x, y), t in zip(coords, labels):
    ax.annotate(t, (x+.25, y-.25), fontsize=13, ha='center', va='center')
ax.set_xlabel(f"PC1 ({evr[0]:.0%} var)")
ax.set_ylabel(f"PC2 ({evr[1]:.0%} var)")
ax.set_title("Input embeddings: geometry of the digit tokens")
plt.tight_layout(); plt.show()

So… we can’t tell that much here. The one unmistakable feature is the dash, which sits off on its own, far from every digit. That makes sense, as its function in the input and output is fundamentally different, so the model gives it a correspondingly different vector. Among the digits themselves, it’s mostly a scatter. If you squint, there may be a faint drift from smaller digits on the top toward larger ones on the bottom, but that’s very shaky.

But also notice, the two axes together hold only about a third (13% + 17%) of the variance. So 70% of how these tokens differ is in directions this plot can’t show, and two points that look close here might be far apart in the full space. So really, this is just a rough likeness, and nothing super clean jumps out.

10.6.2 Cosine Similarities

So, PCA wasn’t all that useful here. Or at least, a 2D projection that throws away most of the variance isn’t where we want to rest a conclusion. So let’s use a second lens, one that does use the full vectors, in cosine similarity between every pair of embeddings.

We’ve already talked about cosine similarity elsewhere in this notebook, but remember that it tells us how aligned each token’s vector is with every other’s, with 1 being completely aligned (identical direction), and 0 being totally unrelated (perpendicular directions), to negative -1 being completely opposed (opposite directions).

Wn = W / np.linalg.norm(W, axis=1, keepdims=True)
Scos = Wn @ Wn.T

fig, ax = plt.subplots(figsize=(5.5, 4.5))
im = ax.imshow(Scos, cmap='viridis')
ax.set_xticks(range(len(labels))); ax.set_xticklabels(labels)
ax.set_yticks(range(len(labels))); ax.set_yticklabels(labels)
ax.set_title("cosine similarity between digit embeddings")
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
plt.tight_layout(); plt.show()

So truly, apart from the bright diagonal (where every token is identical to itself and thus perfectly aligned), the grid is mostly dark. Sometimes digits get a tiny bit more signal with their neighbor, but not often, and not enough to say it’s anything meaningful. This means that mostly, the digits are orthogonal to one another - their vectors point in distinct, unrelated directions, with none of the adjacency you’d see if the model had laid them out as a number line.

So, the model did not learn that its digits are numbers. It learned them as ten distinct, essentially arbitrary symbols. Thinking back to Section 9, this is exactly what we should have expected. The model works one character at a time, and does a lot to just copy digits into the right slots. There’s no real reason to learn them as a magnitude scale, so it didn’t.

It’s a sharp contrast with the GloVe embeddings from Section 1.4. Those were rich with structure because their task - predicting words in context - requires capturing meaning. Ours are nearly structureless because our task mostly requires telling symbols apart, which actually, orthogonal vectors do very well. (Though, it’s worth pointing out that 128-dimensional vectors will start essentially orthogonal, so this does not need to be learned by the model so much as exploited). Either way, it seems the embeddings, like everything else we’ve looked at, just mirror what the model needed to know.

10.7 Beyond Attention Maps

Similar to Section 9, where we could’ve looked at thousands of outputs, you can keep going forever with attention maps. We’ve looked at them every which way, and even peeked at the embeddings. So while we can keep going, I think we’ve proved how and when those things are useful, and when they’re not. They can tell us many things, but are also not the full story.

So you might wonder, how do we get the full story? But that answer is a very, very long list of things involving editing the model, picking it apart thoroughly, and very complex visualizations. So we’ll just mention a few more things you could do, but then wrap up looking at the model’s internals for any more answers.

Most of these things try to get at exactly the wall we kept hitting in this Section - attention helps us understand what the model decided to look at, but not what it decided to do with that information. A lot of that exists in the output projections and the feedforward networks, which are a lot harder to investigate. But, if you wanted to, here are some of the ways you’d do that:

  1. The logit lens. The model’s final step turns a position’s vector into a next token by running it through the output layer. The logit lens does that early, taking the half-finished vector from an intermediate layer and pushing it through that same output layer, to see what the model would say if it only had part of the information. If you do it at every layer, you can watch the guess sharpen from nonsense into the final answer.
  2. Ablations and activation patching. Rather than just looking at what the model did and inferring, these methods actively intervene. You prove that a part of the model matters by changing it, and seeing what happens. During ablations, you delete a component to see if the model needed it. In activation patching, you swap a component between a working and a broken run to see if the output moves. This lets you see what the model actually uses to come to its decisions.
  3. Probing. You can freeze the model, grab its internal activation at some layer, and train a small classifier on top of that to predict a property. Essentially, see if with some internal representation, you’d be able to find some information you care about, or think the model ought to care about. If the classifier succeeds, that information is linearly present in the representation. This is cheap to run and popular, with of course the caveat that present doesn’t mean used, so the probe can read off something the model itself never acts on.
  4. Sparse autoencoders (SAEs). A single neuron usually fires for many unrelated things (called being “polysemantic”), because the model crams far more concepts than it has neurons by spreading them across combinations of neurons (called “superposition”). An SAE is a second, small network trained to take one of the model’s activation vectors and rewrite it as a short list of active features pulled from a much larger learned “dictionary” of directions, forced to use only a few at a time. When it works, each dictionary feature means one clean thing, so you can finally read an activation as “these specific features fired.” This is the current frontier, and where a lot of today’s excitement lives, so a lot of active work is being done here.
  5. Tracing whole circuits. Less a separate tool than the goal the others serve. Once you can examine components one at a time, you can try to find the handful that together carry one specific behavior, work out each one’s job, and show how they wire into a complete mechanism - a “circuit.” It’s the closest you can get to reverse-engineering a piece of a model. (The famous work on this is via “induction heads,” the mechanism behind in-context copying, though it’s quite involved and difficult to describe in just a few sentences.)

And it goes much deeper. If you want a sense of the frontier, the work on sparse autoencoders and “monosemanticity” (Elhage et al., 2022; Bricken et al., 2023) is where a lot of the current excitement lives, and the effort to trace full circuits (Olah et al. , 2020; Olsson et al., 2022) shows how far the reverse-engineering can go. These are massive, active areas of research. On a model as small as ours, you can be fairly thorough without many tools at all, but on a frontier model, interpretability is very much a live problem. And it’s quite an important one, since much of the case for trusting these systems rests on being able to see inside them. There’s easily a whole other notebook’s worth of content here, so we’ll leave it at that!

11. Transformers in the Wild

If you’ve somehow made it all the way to the end of this tutorial, then you (hopefully!) have a strong grasp on all of the fundamentals of the transformer model - from the math that drives it, to how to build one in code, to how to prod at it once it’s trained. And though the transformer we built here is small, it is the same exact underlying fundamentals that even the largest, most capable models today use.

Of course, outside of the fundamentals, there is a lot that modern, large LLMs do a little bit differently. And the field is constantly evolving to continue to develop new methods. So while the underlying architecture still stands, many of the individual pieces we’ve discussed have been updated to a more efficient or better-behaved version of itself. And the exact ways in which these models are now used, and the cases to which they’ve been adapted, has also evolved as the problems the field tries to solve evolve.

So while we absolutely cannot exhaust a decade of research in this section, I’ll try to give a little bit of a map of the landscape of current, modern models you may see out “in the wild”.

11.1 Component Swaps

One of the most obvious things that’s happened in the last decade is that many of the components of the original transformer have been swapped out for “better” versions of themselves. And almost every one of those swaps is driven by the pressure of massive scale, so in particular, every component has mostly been re-engineered to be faster, lighter on memory, more stable to train, and able to handle far longer inputs.

Now that we have a full understanding of the core components, it’s hopefully a little bit easier to understand why you’d swap these things. I’ll give a brief intro, but mostly here just enumerate what, for each of the things that we built, the more modern common alternatives are, and what they buy us.

11.1.1 Positional Encodings

Our implementation: We used fixed sinusoids, added to the embeddings at the input. This works, but some of the most useful structure of those position vectors gets muddied in the addition. The model has to learn to exploit how this encodes relative position, the absolute positions are too easy to learn on instead, and it can’t extrapolate to sequences longer than those seen in training.

Key alternatives to know about:

  • RoPE (rotary position embeddings). This takes the rotational property that should have been useful about sinusoidal encodings, and exploits it better by moving it to inside attention. Rather than adding a position vector, you rotate the query and key vectors themselves by a position-dependent angle. The query-key dot product then depends only on the relative offset between the two tokens. Absolute position cancels by construction, instead of being something the model must learn to ignore. This also makes base-frequency length-extension methods possible. (See Appendix 2 for some more details if you haven’t - we extensively work through some of the main properties of sinusoidal encodings, and touch on RoPE, there.)
  • ALiBi (attention with linear biases). This method is even simpler. It skips position embeddings entirely and just adds a penalty to each attention score that grows with the distance between the two tokens. Its appeal is that it carries no learned position parameters at all, and it was built specifically to extrapolate well. You can train on short sequences, and run on much longer ones, out of the box.

The state of the field: RoPE is the default in most modern LLMs, with ALiBi as a less-common alternative. The reason is roughly what you’d guess from the two descriptions - ALiBi’s linear penalty hard-codes a “closer is always better” bias that it never gets to unlearn, which is fragile, especially for the long-range cases (retrieval, a fact buried early in a long document) that models increasingly have to handle. RoPE assumes none of that, which is a big part of why it won out. The active frontier here is all about context length: how to take a model trained at one length and stretch it to far longer ones. A lot of the active research tries to rescale RoPE’s frequencies or the positions they act on (position interpolation, YaRN, and relatives).

11.1.2 Attention

Our implementation: We built “full” multi-head scaled dot-product attention. This version includes every token attending to every other one (every earlier one, under the causal mask), and each head with its own queries, keys, and values. It’s the heart of the model, but has two major costs that become the defining problem at scale. First, the attention matrix is sequence-length by sequence-length, so both compute and memory grow with the square of the input length - double the context, quadruple the cost. Second, when generating autoregressively, a naive implementation must re-run the entire past through attention at every single new token, costing tons of time, again, especially for longer sequences.

Key alternatives to know about: Since it’s the most central part of the transformer, a lot of work has been done here. Major upgrades come from two very distinct sides - 1) trying to compute the exact same result, just faster and/or with less memory, and 2) changing what attention actually computes, to trade a little quality for a lot of efficiency.

1) Same result, just faster or lighter:

  • KV caching. If you store the keys and values you already computed for past tokens in the sequence instead of recomputing them each time, then each new token can be produced much, much faster. The output is exactly identical, you just store a bit of the information instead of recalculating it every time. It results in massive time speed-ups, and is what makes autoregressive generation practical at all. So this is actually so standard that you’ll basically never see generation without it in the backend, unless it’s a very small model anyway or quick demo. (We dive deep into why this is possible and relatively easy to do in Appendix 4, which also sets up the notebook I have on KV Caching. Read both for more details!)
  • FlashAttention. This is a way of computing exactly the same attention we already know, but with the work reorganized so the full n×n matrix is never written to slow memory. Essentially, the computation is broken up into pieces that can be done in faster memory. You end up with the exact same numbers out, but just much less memory traffic. By avoiding storing the full matrix, memory grows linearly rather than quadratically, and skipping all that memory traffic makes it much faster.

2) Changes to attention itself (and therefore a real tradeoff):

  • Multi-query and grouped-query attention. The KV cache makes generation much faster, but the cache rapidly becomes huge and memory intensive, in part because every head keeps its own keys and values. MQA goes to the far extreme to reduce that size - it makes all query heads share a single key/value head, shrinking the necessary size of the cache by a factor of the head count. Huge memory saving, but it costs quality, so GQA splits the difference by keeping a handful of key/value groups instead of just one. That way, you get most of the memory savings and most of the quality. Note, this is a trained-in architectural choice, so not a totally free optimization you can do at inference-time (though, can require only minimal extra training to apply). GQA is where most large models have landed.
  • Multi-head Latent Attention (MLA): Introduced with DeepSeek-V2, it’s a way to shrink the KV cache. Where GQA saves memory by having heads share K/V, MLA instead compresses each token’s keys and values into a single small latent vector. Only that latent is cached, and it gets projected back up into full per-head K and V when it’s time to attend. In effect, it’s a learned low-rank bottleneck on what the cache has to remember. Rebuilding K/V costs extra compute at every step, though, so you’re trading memory for compute. It also doesn’t play nicely with RoPE out of the box, so MLA carries a small separate key just for position information. Like GQA, it’s a trained-in architectural choice.
  • Sliding-window / sparse attention. Here, you simply drop the “every token sees every token” rule and let each token attend only to a local window, or a sparse pattern plus a few global tokens. It makes the computation an approximation rather than the full thing, but it does pull the quadratic cost down toward linear. Information still travels across the whole sequence because each layer’s window compounds with depth, so it’s not quite as bad as it first seems. Though the bet is still that most of what a token needs is nearby.

The state of the field: These are mostly complementary rather than competing / “choose one”, so a modern model typically runs all of the “exact” ones together (KV caching and FlashAttention), with GQA also being very popular (even though it does lose some quality). MLA is a newer alternative that some frontier open models (notably DeepSeek’s) have adopted. Sliding-window and sparse variants show up more selectively, often where very long context is much more of the point, since their efficient growth with sequence size is such an advantage. The active question is whether attention’s quadratic cost can be escaped without surrendering quality. Faster attention implementations that actually change the computation are rarely chosen to replace full attention outright. The most serious alternatives right now actually scrap attention entirely rather than make it faster. State-space models like Mamba, which process sequences in linear time and are strong on long inputs, are a strong contender, though whether they fully match attention-based implementations across the board is still being argued. So attention’s computation cost remains the central engineering problem of the modern transformer, with much active work still being done.

11.1.3 Normalization

Our implementation: We used LayerNorm, in the pre-norm position. For each token’s vector it subtracts the mean and divides by the standard deviation (across the d_model features), then applies a learned scale and shift. The things to note there are 1) it runs at every block (twice per layer, before attention and before the feed-forward), and 2) it’s really doing two jobs at once, both re-centering (subtracting the mean) and re-scaling (dividing by the standard deviation). There’s really nothing wrong with it. But, it turns out the re-centering pulls much less weight than the re-scaling, while still costing a full mean-and-subtract every single time it runs.

Key alternatives to know about:

  • RMSNorm (root-mean-square normalization). It drops the mean-subtraction entirely: it normalizes only by the root-mean-square of the vector. It usually drops the learned shift too, so just keeps the re-scaling. The reasoning is exactly what we just mentioned about the LayerNorm - the stabilizing benefit is coming almost entirely from controlling the scale, not from re-centering, so you can just skip that part with no measurable loss in quality. It’s a small change, but it’s run at every block of a very deep network, so it adds up.

The state of the field: RMSNorm is the default in most modern large models now. But more than just the method, there are two more things worth knowing.

First, on placement. The original paper put the norm after each sublayer’s residual addition (“post-norm”), whereas we used “pre-norm” (normalizing before the sublayer). We did this because it changes nothing about the method itself, so we didn’t have to learn anything new. But it has been shown to make very deep stacks far more stable to train. So we went ahead and did the modern version there already rather than staying faithful to the paper. But just pointing out that post-to-pre norm is something that was investigated, and then settled on in the field.

Second thing, is that normalization is actually one of the fiddlier levers in training enormous models. Beyond just the RMSNorm swap, which is nearly universal, there is a lot of work being done on where to add normalization. Some teams have added normalization in new spots as a way to keep training stable at scale - like normalizing the queries and keys before attention (QK-norm). Exactly where norms can benefit is still something people tune.

11.1.4 The Feed-Forward Network

Our implementation: We used a position-wise feed-forward network: two linear layers with a ReLU in between, applied independently to every position (with an inner dimension four times d_model). We talked extensively in Appendix 3 about why models need a non-linearity like ReLU, but ReLU itself is a rather blunt one. It hard-zeros every negative value, so has a hard corner at zero, it discards everything on the negative side, and a unit stuck in the negative region gets no gradient at all, so it can effectively “die.” It still works, but is also improvable.

Key alternatives to know about:

  • GeLU (Gaussian error linear unit). This is a smooth drop-in for ReLU - instead of a hard cutoff at zero, it multiplies each input by a smooth gate (the Gaussian CDF) that’s near 0 for very negative inputs and near 1 for clearly positive ones. This just means negatives are suppressed gently rather than clipped off. It gives cleaner gradients and a modest quality bump. It’s the activation in BERT and the GPT-2/3 line.
  • SwiGLU and other gated units. This involves more than simply switching the activation function. It also changes the block’s shape. Instead of one projection → activation → projection, you run two parallel input projections: one passes through an activation and then gates (elementwise-multiplies) the other, before the final projection. That gating gives the layer input-dependent, multiplicative control over what flows through, and it trains better per parameter. (Because it adds a third weight matrix, the inner dimension is usually shrunk to about two-thirds to keep the parameter count comparable.) It’s the default feed-forward in most modern LLMs, like LLaMA and PaLM.
  • Mixture-of-Experts (MoE). This is an even bigger departure. You replace the single feed-forward network with many parallel ones (“experts”), plus a small router that, for each token, picks just a few to actually run (say 2 of 8). Each token only flows through those few, so the compute per token stays roughly fixed while the total parameter count can grow enormously, since now how big the model is, is very different from how much it costs to run. It has downsides though, like how you still have to store every expert (so, large memory), and because the router has to be trained and kept load-balanced, it adds a real training instability. Mixtral is the best-known open example.

The state of the field: Gated feed-forwards of the SwiGLU family are the standard middle of a modern transformer block now, with plain ReLU mostly historical. MoE is increasingly common at the frontier. One of the main reasons why is, as we’ve discussed, the feed-forward network is actually where the majority of a transformer’s parameters are. So when you want to grow a model’s capacity, the feed-forward is where the capacity already is. When you want to sparsify your model, it’s also where it makes the most sense to. MoE helps to do that. However, its surrounding engineering - handling the routing, load-balancing, and the cost of holding every expert in memory - is complicated and hard. So it’s powerful, but far from free.

11.1.5 Tokenization

Our implementation: In our date formatting model, we tokenized at the character level deliberately, so every token stayed legible while we learned. The cost is that sequences get long (every character is its own token, which is expensive given attention’s quadratic scaling) and the model has to rebuild every recurring pattern from loose characters each time. But back in Section 1, we talked about the way that tokenization is more commonly done, using BPE, in depth.

Key alternatives to know about. Most of the modern alternatives and extensions: byte-level BPE, WordPiece, Unigram/SentencePiece, as well as their quirks, we covered in Appendix 1. So I will point you there to learn about the modern alternatives. But summarize just briefly here. Essentially every modern model will use subword tokenization of some sort. Recent frontier models have converged specifically on byte-level BPE with large (100k+) vocabularies.

Character-level, like we did above, is the kind of choice you’d really only make if, for some specific reason, you have a very narrow task or very specific set of tokens (like we did!)

The state of the field: As we said, byte-level BPE is the most commonly used tokenization scheme. But one more thing being looked into is whether to drop the tokenizer entirely, with byte or character-level models that learn straight from raw bytes with no merge step at all (ByT5, MEGABYTE). They erase tokenization’s quirks, but reintroduce exactly the long-sequence problem our character-level scheme had, which then causes the exact attention efficiency issues we got into in 11.1.2. So for now, the tokenizer stays standard.

11.2 Encoder, Decoder, or Encoder-Decoder

Back in Section 4.1, we discussed briefly how the full transformer - an encoder-decoder - in its complete form is not actually the only transformer architecture you’ll see. There’s also encoder-only models for understanding, and decoder-only models for generation. And while the encoder-decoder is still used for turning one sequence into another, decoder models are actually the most common type, and what almost all modern frontier models are. But all still have use cases, and you’ll see each of the three show up as modern models.

So now that we’ve completed our understanding of the full encoder-decoder architecture, let’s revisit the three types to talk about when you’ll still see them, what modern examples are, and how the three differ in how they’re trained.

11.2.1 Decoder Only Models

Decoder-only models, of course, contain only the decoder part of the architecture, with one additional edit. There’s no cross-attention, because there’s no encoder output to cross-attend to. They’re a single stack of masked (causal) self-attention that only ever looks backward, producing one token at a time.

At training time, you simply give full sequences from your data, shifted one like we did with teacher forcing. It’s just next token prediction over your enormous corpus. At generation time, they produce one token at a time, starting typically from a prompt that sets up what to generate next. The prompt often uses special tokens for tagging (similar to the <bos>, <eos> tokens) to help separate what was “input” vs “output” (though it’s all processed the same way).

decoder-only

This is what most models you’ll hear about are. The GPT line, LLaMA, Mistral, Qwen, Gemma, and essentially every frontier chat and generation model.

Today, they’re the most common because that single objective (predict the next token) can actually cover an astonishing range if you just define your task right. Many tasks can be reframed as generation, even ones that sound like they should require understanding: \[ \text{"[REVIEW]. The sentiment here is \_\_\_"}\]

as we noted in 4.1.

Of course, this works because the decoder stack can also understand, just only based on things behind it. But the same understanding benefits that you get from attention and the feedforward network are of course still there.

It’s also the simplest of the three, the one that scales most cleanly, and in-context learning falls out of it for free.

11.2.2 Encoder Only Models

Encoder-only models are then, the encoder stack alone. The blocks have the full bidirectional self-attention, and they aim to produce rich representations rather than generated tokens.

At training time, you add a prediction head whose job is to predict what word was masked out from your input data. Then, when you want to actually use the encoder, you just throw that out and the rest remains the same.

encoder-only

The original canonical example of an encoder-only is BERT. But although that’s an older model, the family is very much alive in modern times as well. The encoder-only architecture is the backbone of the embedding and retrieval models behind semantic search and RAG (Sentence-BERT and its descendants), and it’s still being actively modernized (ModernBERT, 2024).

The field is still alive, because you sometimes do genuinely just want to represent text rather than produce it. Classification, retrieval, reranking, entity tagging, are all tasks where a bidirectional encoder that can look over the whole sequence at once gives richer representations, and can be far cheaper to run.

Typically, what comes out of an encoder is an embedding you want to use for something. It’s very common to attach a small task-specific head to the output, and train that to do a specific task (like score sentiment). Because you can often do this trusting that the encoder’s embeddings are already good, this means you can freeze the encoder portion and just train the head. So for tasks like that, an established encoder + head is often much, much cheaper to train as well.

11.2.3 Encoder-Decoder Models

Encoder-decoder models are the full version that we looked into. They’re by far the most complicated since they have both an encoder and decoder. The encoder reads the entire input bidirectionally, and a decoder generates the output one token at a time while attending back to the encoder through cross-attention.

As we already saw, they’re trained with the encoder turned on and the decoder doing the task with teacher forcing. At generation time, the encoder path still runs, passing its representation into the cross-attention of the decoder, which generates one token at a time.

encoder-decoder-basic

The canonical modern example is T5, which frames every task as text-to-text. But a model like Whisper, which is a speech-to-text model, is an enormously popular one too. It takes audio in through the encoder, text out through the decoder. We’ll discuss different modalities like this a bit more in depth in just a little bit.

Encoder-decoders persist even though decoder-only models now translate and summarize perfectly well by prompting, because sometimes having the entire input available up front just really is a major advantage. This is especially true of course when the output is a genuinely different sequence - those cases where we needed “Input” and “Input2” to be distinct segments. The clean separation of tasks means the encoder can process the source with no causal constraint while the decoder focuses on generating. That means it’s still the natural fit for tasks like translation, speech, and much multimodal work where “read this completely, then produce that” is still the best way to approach the problem.

11.3 Different Training Methods

In this tutorial, we saw actually a quite narrow view of training. We had a very specific task, input and output data that we could generate and match exactly, and so we were able to put that through a standard training pipeline and easily grade behavior.

But “training” can take a lot of different shapes. As we just saw in 11.2, the pretraining objective (the thing you train to produce) is tied to the architecture family. Decoder-only models learn on next-token prediction, encoder-only models learn to fill in masked tokens, and encoder-decoders learn sequence-to-sequence. And even within those families, the space of training techniques that are done is very vast.

So while the core loop of training never changes: find a way to measure how well the model is doing, then nudge its parameters toward improving that measure - the exact way you do that, what you measure and how you improve, actually has a lot of wiggle room. We’ll talk about what some of those different choices are, as well as some of the additional considerations of modern training here.

11.3.1 Multi-Stage Training

The first thing to be aware of is that modern models, for a variety of tasks, are very often trained in a multi-stage sort of way. This means training the model distinct phases, where each is able to use different data, a different objective, or even to train a different part of the model.

The logic and reasoning behind doing multi-stage training is always the same. It’s cheaper and more effective to teach something broad and general, or easy and concrete, first, then specialize, than to do everything at once from scratch. And this is a general idea, well beyond language models. The core versions of this that show up in many places are:

1) Starting with a “pre-training” phase. Especially when teaching a model from scratch - fully randomized weights - a lot of what the model needs to learn doesn’t require high-quality, or even task-specific data. For example, learning the basic rules of grammar and sentence construction doesn’t need well-written novels and textbooks - you could do it from (presumably readable) Reddit comments. But this idea applies to tons of cases - you can start your training on a large, general dataset to build broad capability, then fine-tune on a smaller, task-specific one.

This is the backbone of modern transfer learning across all of deep learning. And it’s not just about “saving the good stuff”, even though that is a huge advantage (high-quality, task-specific data is often very scarce compared to broad, large datasets). It’s also been shown to make the model a better generalizer when it finally trains on the specific task.

The deeper reason is about representations: a pretrained model has built a general-purpose internal model of its domain. For language, that’s grammar, structure, meaning, broad world knowledge - all things that transfer to almost any task. Fine-tuning, then, doesn’t need to and shouldn’t rebuild them. It just learns to use them for the job at hand, so the scarce task-specific data is spent entirely on the task rather than relearning the basics. This has become so central that the whole modern field is largely organized around it: train one enormous, general “foundation model” once, at great expense, then adapt it to countless downstream uses (Bommasani et al., 2021).

2) Freezing certain parts of the model and training others. There are also often cases where you want to train the model in parts - holding some portions frozen while updating others - rather than the whole thing at once. The advantage is that training this way is often cheaper, but also that it protects what the model already learned from being trampled. A few common cases and ways you’ll see this done:

  • Hold a pretrained backbone frozen, and train only a small new head on top for a specific task. This is exactly what we described for how most encoder-only models are used. Assuming the embedding out of the frozen part is good, you can take good advantage of those without tweaking them at all.
  • Unfreeze just the last layers, and continue to train those. It’s been shown that, in particular, the earlier layers of a model tend to be where most broad, general knowledge is learned, and later layers tend to be where more of the task-specific subtleties are learned (see Yosinski et al. (2014), and notice, this predates transformers - it’s a much more general property). So one common technique is to fine tune a model by freezing everything but the last few layers, and allowing those to adjust to the new data and task.
  • Training portions of a complex architecture on their own. The idea here is a bit like holding the backbone frozen, but instead of taking an established backbone, you inspect which parts of your model could be trained on their own. A classic example in modern architectures are latent diffusion models (like StableDiffusion), which first train a variational autoencoder (VAE) to learn how to compress and then decompress an image. Since this task is highly well-constrained, and does a very specific thing (taking in an image, and giving out the same image), it can be trained on its own, as a separate task, before becoming part of the larger architecture.

3) Training using something, then throwing it out. We’ve already talked about a specific instance of this in encoder-only training, where a prediction head is used during training time and thrown out later. But this is a general pattern that can be used for training many models. You use a piece of the architecture just to create a useful training signal, and throw it away once training is done. What you keep is the representation it forced the rest of the model to learn. It often looks like:

  • A task head used only in training. The clearest case is the masked-language-model head on an encoder-only model. It’s used to define the fill-in-the-blank task, then discarded. The point of the head was just to define something to measure, but the goal was to keep the encoder’s representations, not the head that trained them.
  • An autoencoder where you keep only the encoder. This is a common pattern to train a feature extractor. An autoencoder is a network that compresses an input, then reconstructs it (input in, same input out). If you train this well, then throw away the part that reconstructs the image, then the first part is just a high quality feature extractor. The reconstruction task was only ever a way to force the encoder to learn a good compression. (This is a different use of an autoencoder than the latent-diffusion VAE we just discussed, which is trained separately and then kept - its encoder turns training images into latents during training, and its decoder renders the final image at generation. What we’re talking about here is throwing one half away entirely.)

The Modern Case

One of the fullest expressions of the idea of multi-stage training actually comes from the pipeline behind a modern chat assistant, which stacks several of these moves into one process. These models are built in three distinct stages, with a changing objective, and even changing training procedures in the different stages. And remember, the architecture to have in mind here is the “simple” decoder-only model.

The stages are:

Stage 1: Pretraining. The model is trained on next-token prediction over an enormous, diverse corpus, created from much of the web, books, and code. This is the task we’ve already discussed in detail, with the input and a teacher-forced shift of the input making up the data. The model’s goal is just to learn text at colossal scale. The crucial thing here is that this training is self-supervised - the “label” for each position is simply the next token, that literally comes from all of the text you scraped. So you’re just learning from massive data, no “truth” labeling needed, which is what lets it scale to trillions of tokens.

  • What comes out is a base model: an extraordinary text-completer, with an abundance of knowledge baked in.
  • If you input text, you’ll get out completion. Finishing of your sentence. It can’t yet, especially reliably, answer questions, because all it has learned to do is continue text, not answer.

Though it’s not a full-fledged chat assistant, a model coming out of this stage is still very capable. This is where the majority of the model’s world knowledge and language ability is learned. GPT-3 is the canonical demonstration that at enough scale, this stage alone produces startling few-shot ability.

Stage 2: Instruction Tuning. This is where the “following instruction” behavior is learned. The base model is fine-tuned on curated examples of (instruction, good response) pairs. This is ordinary supervised learning, so the same type of training task we did on our toy model, just on instruction-following data instead of dates. It mostly doesn’t add new knowledge, but it does teach new behavior: given a request, produce a helpful response rather than ramble onward the way a base model would.

  • What comes out is an SFT model (or sometimes called IFT): a model that can be chatted with and asked questions.
  • If you input text, you get a response, the assumption after this stage is that what you say is meant to be answered, not continued.

Though it’s not yet tuned for the quality of its answers, a model coming out of this stage is already a genuine assistant. This is where the ability to follow instructions - including ones it was never explicitly trained on - is learned. FLAN is the canonical demonstration of exactly that: instruction-tuning on a collection of tasks produces strong zero-shot performance on entirely new ones.

Stage 3: Preference Learning. The last step shapes the qualities that are hard to pin down as a supervised target. Things like helpfulness, tone, honesty, and harmlessness are all features you want the model to have, but they’re hard to put your finger on and rare to find cleanly labeled in data. You can’t easily write the one “correct” response that perfectly encapsulates helpfulness, but you can reliably say “this response is more helpful than that one.” So the classic recipe for this stage, RLHF, does just that. You collect human comparisons between responses, train a reward model to predict those preferences, then use reinforcement learning to push the model toward higher-reward outputs.

  • What comes out is an aligned model: This is the finished chat assistant, which is now not only capable and responsive, but shaped to be helpful, honest, and pleasant to interact with.
  • If you input text, you still get a response - but now one pushed toward what a person would actually prefer. The change from the last stage is in the response’s quality and character, not in what the model fundamentally does.

This is a genuinely different paradigm from everything above - the model stops being trained by “here’s the right answer” and starts being trained by “here’s which answer people preferred”. And the full machinery to train via reinforcement learning is pretty different from supervised and self-supervised training.

If you want all of the details, I have a notebook all about classic reinforcement learning here: RL for LLMs!

(Or, if you prefer other reading, a popular modern simplification, DPO, skips the separate reward model and optimizes for the preferences directly. InstructGPT is the canonical full RLHF pipeline.)

All modern models are trained in some mix of these stages. And with the typical pipeline of broad ability first, then instruction-following, then final alignment to preference. But treat that order as a common backbone, not a strict law. Both the sequence and the weight of each of these stages can actually vary quite a lot, and stages can get repeated, interleaved, run in multiple rounds, or occasionally skipped.

More importantly, how much a model actually learns in each stage has recently undergone pretty major shifts. In a standard chat model, the final RL stage is mostly used for polish - aligning tone and helpfulness on top of abilities that pretraining already established. But in, for example, reasoning models, that stage does far heavier lifting. RL algorithms like GRPO, with rewards that can verify an answer (did the math check out? did the code pass its tests?), have been able to teach the model a large portion of its actual reasoning capability, not just how it answers, but what it answers. DeepSeek-R1-Zero even showed strong reasoning emerging from large-scale RL run directly on a base model, with no supervised fine-tuning first.

So the three stages are the right mental model to start from, but where exactly most of the real learning happens is very much flexible, and the field is actively experimenting with how far each stage can be pushed.

11.3.2 It’s (Mostly) About the Data

There’s something beyond training methodology that’s often overlooked, and it’s about the data quality. When you train, you start from enormous amounts of messy, uneven, often outright junk data scraped from the internet, and a huge share of the real work is in cleaning it and deciding what goes in. In fact, now that the architecture has largely settled - nearly universally some version of the transformer we built here is the go-to architecture, data has become the single biggest lever separating a great model from a mediocre one.

I’ll keep this section brief, but a few things are worth knowing about how that plays out:

1) Quality beats raw quantity. More data is not automatically better, with low-quality text actually often degrading a model. So much of the work at the pretraining stage is really filtering. Removing junk, and especially deduplicating, as repeated data wastes compute and drives memorization (Lee et al., 2021), reliably improves the result. Pushed to the extreme, carefully curated high-quality data can let a much smaller model punch far above its weight - “Textbooks Are All You Need” showed surprisingly strong small models when trained on small, clean, textbook-like data.

2) Composition matters a lot. It’s not only how much data but what mix of it. How much web versus code versus math versus books versus other languages changes what the model is good at. Those proportions are often carefully tuned.

3) Synthetic data and distillation are rising fast. Increasingly, training data isn’t scraped at all, but is instead generated by prompting a capable model to produce examples, or by training a smaller model on a larger one’s outputs (distillation). This can be very powerful where good human data is scarce, like step-by-step reasoning traces. But it carries a real risk - lean too heavily on model-generated data and quality can degrade across successive generations, a failure mode called model collapse (Shumailov et al.).

Unfortunately, the good data might be running out. Frontier models are now large enough to be bumping against the limits of high-quality human text on the internet - the so-called “data wall” (Villalobos et al.). That pressure is much of why synthetic data, multimodal data, and squeezing more signal from less are such active frontiers.

So while data work is unglamorous, it’s where an enormous share of real frontier progress now comes from. It’s also increasingly the part labs want to keep secret - two models with near-identical architectures can differ wildly based purely on what they were fed, so the data is truly one of the most important components of training.

11.3.3 Training at Scale

We’ve covered pretty extensively here what models to train, on what data, and even how, conceptually. But there’s a major reality that we haven’t touched on much which is about how, physically you pull it off. A frontier model is far, far too large to train on a single GPU. In fact, most “small” (few billion parameter) models are, too. But the bigger the model gets, the more challenging dealing with all of that size, and all of that compute, gets. Most large, or even medium, models need to run across many computers at once, for weeks or months on end. So training at scale is a difficult engineering problem in its own right. Knowing some of the most major considerations that need to happen for large scale training can be helpful context, so we’ll touch the main ideas here quickly.

The core idea for making training feasible is that you need to intelligently split the work by parallelizing the problem. There are three main ways to spread training across many devices, usually combined:

  • Data parallelism - put a full copy of the model on each device, split the batch between them, and average the gradients. It multiplies throughput, but every device still holds the whole model, so it doesn’t help when the model itself is too big.
  • Tensor (model) parallelism - split the individual layers across devices, so each holds and computes a slice of each weight matrix. This is necessary when a single layer won’t fit on one chip.
  • Pipeline parallelism - you can also split the stack of layers into stages, one group of layers per device, and feed micro-batches through like an assembly line. This helps when the whole depth of the model won’t fit.

But even split up, memory is often still very tight, so several additional tricks can help keep it in check. Some of the most common ones include:

  • Sharding - with plain data parallelism, every device redundantly stores a full copy of the parameters, gradients, and optimizer state (Adam’s running averages alone are about twice the size of the model). Sharding methods partition all of that across devices so no single one holds the whole thing.
  • Mixed-precision training (storing most things in 16-bit) and gradient accumulation (simulating a big batch in several small steps) are both very common techniques, which we already discussed briefly in Section 7.
  • Gradient checkpointing is a similar idea, where instead of storing every activation for the backward pass, you recompute some on the fly, trading extra compute for memory.

There’s also, of course, the whole hardware side of things. All this parallelism requires constantly shuffling enormous tensors between chips, so not only how big the chips are, but how they interconnect, becomes as much of a bottleneck as the chips themselves. Frontier training runs on clusters of thousands of specialized accelerators (GPUs or TPUs), wired together with very high-bandwidth links, for extended stretches of time.

That’s all I’ll say, and honestly, all I can. First, as I’m no expert here. But also, because engineering around large-scale training is a deep and specialized engineering discipline of its own, and much of the knowledge that is necessary to train frontier models lives in those labs, as they’re the only ones with the scale to do it! So I’ll point you in the direction of some canonical references for deeper reading, but otherwise leave it at that. (See Megatron-LM, GPipe, ZeRO)

11.3.4 Parameter Efficient Fine Tuning

Ok, one final thing to touch on when it comes to training. We talked already about some methods for avoiding retraining an entire, large model, like freezing sections, or fine-tuning on top of an existing, well-performing model. But there’s an additional trick that makes doing additional fine-tuning very time and memory efficient, and it’s called parameter efficient fine-tuning.

The idea is this: if you start with a model that has plenty of existing knowledge for a task, but you just want to tweak it, doing additional full fine tuning is kind of a waste. It costs enormous memory and compute, and you risk messing up good behaviors the model already did have, especially if you’re fitting an enormous model to a relatively small problem.

Parameter-efficient fine-tuning (PEFT) methods get around this by freezing the pretrained model entirely, and training only a tiny set of new parameters bolted onto it. The best-known method is LoRA (and its cousin, DoRA). Instead of changing a big weight matrix W directly, LoRA leaves it frozen and learns a small low-rank update beside it - two skinny matrices whose product has the same shape as W but far fewer parameters. The intuition is that the change to the model that a given task requires actually just lives in a low-dimensional subspace, so you don’t need to touch the full matrix - a thin low-rank correction captures it. In practice, this can mean training well under 1% of the model’s parameters. Extensions like QLoRA push it further still, quantizing the frozen base to 4-bit so even very large models can be fine-tuned on a single GPU.

The payoff is huge, and the technique allows you to fine-tune a huge model on modest hardware, since you only ever compute gradients and optimizer state for that tiny slice. And because the base is untouched, you can sidestep the catastrophic forgetting that full fine-tuning risks.

If you want to know more, see my LoRA Tutorial!

11.4 Other Data Modalities

Throughout this whole notebook, we’ve mostly been talking about feeding our transformer text data. And indeed, most modern transformer-based models you interact with are largely text-based. But there’s nothing really special about text. Looking back to Section 1, we had to do a ton of work to even get text to look like something the model could process: numbers. And the architecture itself doesn’t care at all where the numbers come from. Really, the only thing that matters is that the numbers ultimately come to represent some sort of a sequence so that attention can exploit that structure. Where you get that sequence from… well that can be a ton of different places, actually.

So, if you haven’t fully appreciated it already, do now: a transformer is a general-purpose engine for processing sequences of vectors. As long as your data is, or can be interpreted as, a sequence. A transformer will work. That’s really why they are so ubiquitous. Yes, they’re amazing general purpose learners. But also, sequences are everywhere. And for so many other data types, it’s barely even a stretch to turn them into the kind of input data the transformer wants. A few key examples:

  • Images. The Vision Transformer (ViT) chops an image into a grid of small patches, linearly projects each patch into a vector, and feeds that sequence into a standard transformer, with patches as “tokens.”
  • Audio. We talked briefly about Whisper already, which turns speech into a sequence of audio-frame vectors and runs them through a transformer encoder-decoder.
  • Others. Beyond what’s easily perceptible to humans entirely, there are tons of other cases. Protein sequences, DNA, time series, even sequences of game states - anything you can lay out as a sequence of vectors is fair game.

And of course, many models can now effortlessly deal with combinations of modalities. Image-patch tokens and text tokens can go into the same model (the basis of modern multimodal and vision-language systems), or image and text encoders can be trained to share an embedding space, as in CLIP. The list goes on, because the space of different modalities that the model can accept isn’t really bounded by much - it’s as big as the space of problems you could think to solve, and as architectures you could think to put together.

So the key takeaway here isn’t that certain modalities need certain architectures. It’s that almost all modalities are compatible with the transformer. Yes, with their own nuances of how to process the input (like we needed with text), and always their own considerations for the specific task you’re solving, but not simply because the data comes from somewhere different.

And that’s true of this whole section. And a takeaway of the whole notebook! Even though we’ve spent a lot of time talking about different ways to train and use these models, how to pick them apart, and know what they do, the architecture we built here is the reusable idea, and remains, at the core. Everything else builds on, and works with, those central components. But the backbone - the basic transformer - is still all you need :).

Appendices

The appendices here are “optional reading” - meant to give more clarity or detail to something we already touched on in the main tutorial. I suggest referencing them alongside the main tutorial’s content when you read them!

Appendix 0: Dimensionality Reference Guide

One of the most important things for keeping straight in your mind everything that’s happening in a model is what dimensions things are as they are getting transformed. Here, I’ll lay it all out simply as a place to re-reference as you work through this notebook.

Key dimensions:

  • \(\bf{n_\text{seq}}\): the sequence length going into the model.
  • \(\bf{d_\text{model}}\): the dimensionality of the model
  • \(\bf{n_\text{embeds}}\): the size of the word embeddings (\(\bf = {d_\text{model}}\)), see note below

1. Word Embeddings

Word embeddings create, for each value in the starting sequence, \(n_\text{seq}\), a vector of size \(n_\text{embeds}\) that makes up our input into the model \(X\):

\[ X = n_\text{seq} \times n_\text{embeds} \]

2. Positional Encodings

Positional encodings are added element-wise to the word embeddings, so they must be the same shape: \[ PE = n_\text{seq} \times n_\text{embeds} \]

3. Attention

Attention starts by making \(Q, K, V\) matrices out of \(W^Q, W^K, W^V\) weight matrices.

\(W^Q, W^K, W^V\) must all have the same dimensionality:

\[ W^Q, W^K, W^V = n_\text{embeds} \times d_\text{model} \]

so then, \(Q, K, V\) end up with size: \[ Q, K, V = n_\text{seq} \times d_\text{model} \]

Attention is calculated by doing \(Q K^T\), \[ A = n_\text{seq} \times n_\text{seq} \] Scaled and softmax’ed in scalar operations which don’t change the shape, then multiplied by \(V\), resulting in: \[ O = n_\text{seq} \times d_\text{model} \]

This \(O\) becomes the input for the next layer.

📌 Note: Notice, that this means that, for attention specifically, \(d_\text{model}\) does not need to be a fixed value/property. Here, we see the \(n_{embeds}\) dimension drop out and get transformed to \(d_\text{model}\). The math for attention would work out all the same if the next layer had a different \(d_\text{model}\) size.

In practice though, we do fix \(d_\text{model}\) for every layer in the entire model because we use residual connections, which do need it to be the same for every layer. In fact, these need the input and output to an attention layer to be exactly the same size. So it must be enforced in a transformer that \(\bf{n_{embeds} = d_\text{model}}\) as well, and that \(d_\text{model}\) is a fixed property for all layers in the model. If the desired embedding model has \(n_{embeds} \neq d_\text{model}\), a simple linear projection layer takes care of this.

I will continue to label them separately for clarity of their meaning, but know that this is true, general property.

4. Multi-Head Attention

This is also why multi0head attention works so smoothly. In multi-head attention, we additionally have:

  • \(\bf{\text{h}}\): the number of heads
  • \(\bf{d_k}\): the dimensionality of heads (\(d_\text{model} / \text{h}\)), by definition

Similar to self attention, we start by making \(Q, K, V\) matrices out of \(W^Q, W^K, W^V\) weight matrices. Except now, we have \(i = 1 ... \text{h}\) heads of size \(d_k\) instead.

\(W^{Q_i}, W^{K_i}, W^{V_i}\) must all have the same dimensionality:

\[ W^{Q_i}, W^{K_i}, W^{V_i} = n_\text{embeds} \times d_\text{k} \]

so then, \(Q_i, K_i, V_i\) end up with size: \[ Q_i, K_i, V_i = n_\text{seq} \times d_\text{k} \]

Attention is calculated by doing \(Q_i K_i^T\), \[ A_i = n_\text{seq} \times n_\text{seq} \]

This is scaled and softmax’ed in scalar operations which don’t change the shape, then multiplied by \(V_i\), for each of the heads, resulting in: \[ O_i = n_\text{seq} \times d_\text{k} \]

The \(\text{h}\) \(O_i\)’s are combined with a concatenation: \[ O' = \begin{bmatrix} O_1 & O_2 & ... & O_\text{h} \end{bmatrix} \\ O' = n_\text{seq} \times d_\text{model} \]

Then, \(W^O\) recontextualizes these relative to each other, \(W^O = d_\text{model} \times d_\text{model}\) \[ O = O' W^O \\ O = n_\text{seq} \times d_\text{model} \]

Same result in terms of dimensionality! Even though we broke into distinct heads.

5. The Feed-Forward Network

The FFN is applied to each position independently. It expands to a larger inner dimension and then contracts back. We add:

  • \(\bf{d_\text{ff}}\): the inner (hidden) dimension of the FFN, typically \(4 \times d_\text{model}\).

With \(W_1 = d_\text{model} \times d_\text{ff}\) and \(W_2 = d_\text{ff} \times d_\text{model}\), the input \(O = n_\text{seq} \times d_\text{model}\) is widened, passed through a nonlinearity (which doesn’t change shape), then narrowed back: \[ H = O\,W_1 = n_\text{seq} \times d_\text{ff} \qquad\rightarrow\qquad O\,W_1 W_2 = n_\text{seq} \times d_\text{model} \] So the FFN takes \(n_\text{seq} \times d_\text{model}\) right back to \(n_\text{seq} \times d_\text{model}\) - same in, same out - just passing through a wider \(d_\text{ff}\) in the middle.

📌 Note: LayerNorm and the residual additions don’t change shape either - everything stays \(n_\text{seq} \times d_\text{model}\). That’s the whole point: because every sublayer returns \(n_\text{seq} \times d_\text{model}\), a full encoder (or decoder) block takes \(n_\text{seq} \times d_\text{model}\) in and hands \(n_\text{seq} \times d_\text{model}\) out - which is exactly what lets us stack \(N\) of them.

6. Cross-Attention

Cross-attention (in the decoder) is the same process as multi-head self attention, but the two sides’ lengths differ. The query comes from the decoder’s sequence (length \(n_\text{out}\)), while the keys and values come from the encoder output (length \(n_\text{seq}\)): \[ Q = n_\text{out} \times d_\text{model} \qquad K, V = n_\text{seq} \times d_\text{model} \] So the attention matrix is rectangular, not square: \[ A = Q K^T = n_\text{out} \times n_\text{seq} \] (each decoder position attending over every encoder position). After softmax and \(\times V\): \[ O = n_\text{out} \times d_\text{model} \] So this attention’s score matrix isn’t square - it aligns a length-\(n_\text{out}\) sequence against a length-\(n_\text{seq}\) one.

(Masked self-attention, by contrast, keeps the square shape - the mask just zeros the upper triangle before the softmax, changing the values, not the dimensions, so we won’t re-cover it here, it’s the same.)

7. The Output (Prediction) Layer

Finally, the decoder’s output is projected from \(d_\text{model}\) out to the full vocabulary, with \(W_\text{out} = d_\text{model} \times n_\text{vocab}\): \[ \text{logits} = X_\text{out}\,W_\text{out} = n_\text{out} \times n_\text{vocab} \] A softmax over the vocabulary dimension turns each row into a probability distribution over the \(n_\text{vocab}\) possible next tokens. (At generation time we usually only need the last row, \(1 \times n_\text{vocab}\))

So, dimensionally, the whole forward pass is \(n_\text{seq}\) (or \(n_\text{out}\)) \(\times\, d_\text{model}\) everywhere inside, widening to \(d_\text{ff}\) within each FFN, and finally fanning out to \(n_\text{vocab}\) at the very end.

Take me back to the main tutorial

Appendix 1: Bytes, Unicode, and Byte-Level BPE

In Section 1.2 we ran BPE by merging characters. And in that section, we saw that we needed a special <unk> token to accommodate anything that wasn’t a known letter from our vocabulary. You can imagine, real text scraped from the internet would have the opportunity to have tons of things represented by an <unk>, especially if we didn’t have an ideal, complete, representative corpus to begin with.

But as we’ve also discussed, there’s nothing really special about letters other than that they’re the symbols of which words are comprised. But are letters really the best symbols you can use to represent words? Well, just words, maybe, but real life text is full of all sorts of non-word things.

Real tokenizers choose a different symbol to represent words and other text - bytes. It’s the reason they can handle emojis, Chinese characters, code snippets, or any random string of pure garbage, all without ever needing to actually rely on that <unk>.

What is a byte, and what is Unicode?

Everything on a computer is ultimately stored as numbers, and text is no exception. The system for deciding which numbers represent which characters is called Unicode - a giant, agreed-upon table that assigns a unique number to essentially every character in every writing system humans use: the English alphabet, every other alphabet, mathematical symbols, emojis. Unicode just assigns an ID per character - say, 界 is number 30028, 🙂 is number 128578, and so on - for ~150,000 characters and counting.

It’s sort of like starting with a 150,000-character alphabet. Which is useful, but very extensive and large numbers are difficult to store. That’s where UTF-8 comes in. It’s the standard way of writing Unicode characters as sequences of bytes. A single byte is a single value from 0 to 255 that takes up the smallest unit of memory on a computer. Since 0 to 255 isn’t enough to hold all of the different Unicode options, UTF-8 uses a variable number of bytes per character: the common ones (ordinary English letters) take a single byte, while rarer characters take two, three, or four. Let’s see how a few words actually get encoded:

for s in ["hi", "café", "世界", "🙂"]:
    b = s.encode("utf-8")
    print(f"{s!r:8} -> {len(b)} bytes -> {list(b)}")
'hi'     -> 2 bytes -> [104, 105]
'café'   -> 5 bytes -> [99, 97, 102, 195, 169]
'世界'     -> 6 bytes -> [228, 184, 150, 231, 149, 140]
'🙂'      -> 4 bytes -> [240, 159, 153, 130]

So “hi” is 2 bytes, 2 letters, while an emoji takes 4 bytes for the single character.

But in this way, we’ve actually reduced our “alphabet” to 256. The unit of the “letter” or “character” is now no longer the smallest unit - some will take more than 1 byte to represent, but everything can be represented by the set of numbers between 0 and 255. Everything.

And every possible new thing, not even in our dictionary, can be represented by a series of bytes. With how we did things in Section 1.2, having the vocabulary contain all of the individual letters means some non-real word like “blacapeptopotamus” could still be represented, at worst by all of its letters. With bytes, everything that can be typed into a computer can be represented by bytes - the <unk> is all but completely unnecessary.

How This Changes the BPE Algorithm

The nice thing is that it doesn’t! Everything about how we worked through BPE works exactly the same. You count adjacent pairs, glue the most frequent, repeat. It’s just that the starting alphabet is 256 byte-values instead of all of the letters. Frequent byte-sequences get merged into single tokens exactly as frequent character-sequences did - so naturally multiple bytes in a row that make up a single letter probably get merged early on. Common English words still likely build up as whole tokens (their byte sequences are everywhere in the corpus). And other common byte sequences, like a very common emoji, might get merged into a single token too. The algorithm is completely indifferent to what a byte represents, it works just the same.

For that reason - I’ll skip reworking the example here over bytes. It’s not really intuitive to watch bytes connect, or build up a vocabulary and really it all does come together the same way. The intuition is still best developed using an example like we did in Section 1.2.

Some Consequences

Two final little things worth noting on the BPE discussion (one specific to bytes and one not), because some real model behaviors can be explained by tokenizing this way.

Consequence 1 is general to tokenization. Tokens are usually bigger than letters. So a word like “strawberry” won’t be treated as ten letters - BPE will merge it into a couple of chunky tokens, maybe straw + berry. Each of those is a single ID pointing to a single learned vector. The token for berry relates inherently not at all to the tokens for b, e, r, or y. So the famous case, asking “how many r’s in strawberry?”, is something models will be bad at, for a reason coming all the way from how they understand words at all. Being able to do this task must come from specifically memorizing that berry contains two r’s, and straw contains one. This can be held in the embedding of the tokens “straw” and “berry” themselves, but it’s NOT going to come from knowing to divide either component back down into its letters. Models are famously so bad at spelling and letter-counting because all access to that level of word construction is really gone unless it gets baked back into the embedding during training.

Consequence 2 does come from specifically using bytes, and it’s that doing BPE on bytes can mean you end up with tokens that contain sub-characters. Take ‘🙂’ -> [240, 159, 153, 130]. BPE could merge just the first two of those bytes into a unit, and leave the other two. A token can end up as a fragment of a single character - a couple of bytes that don’t complete any single letter on their own. Literally, half an emoji. And this can complicate things at decode time, because the model can now also theoretically output half of an emoji… which is invalid UTF-8 that can’t be turned back into anything at all (UTF-8 is constructed smartly enough that two bytes of what was a 4-byte string simply map to nothing at all). In practice, this really won’t happen, the model won’t learn to print out the first half of 🙂 without the second half. But if it gets cut off for some reason, reaches an output limit before it could finish, we can end up with a � after decoding - the universal symbol for “I don’t know what those bytes are”.

So, the mismatch between tokens and characters can run in both directions. Sometimes a token can encompass many letters and lose sight of what those letters were, and sometimes a single character gets split in half. Either way, the clean “characters” we take for granted are not the units the model actually operates on. Fortunately, both lead to just small quirks, not critical failures.

Alternates to BPE

In your study of modern architectures, you may find that tokenizers use some common alternates to BPE. They all share the same goal of building a vocabulary of common subword pieces, but they differ in the exact rules for how to choose which pieces to form. We won’t cover these in depth, but here are some of the common ones you might see, and what’s different.

WordPiece (used by BERT). This is a common alternate that’s very close to BPE. It also merges pairs, but instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training text. In practice that means scoring each candidate pair by roughly: \[ \frac{\text{count}(ab)}{\text{count}(a)\,\text{count}(b)} \] and favoring pairs that co-occur more than their individual frequencies would predict rather than pairs that are merely frequent. It also tokenizes new words a bit differently. Instead of replaying merges in order (as we did), it greedily takes the longest vocabulary piece that matches at each step (which we actually discussed as a potential way to tokenize in the main tutorial). WordPiece also marks pieces that continue a word with a ## prefix, so “playing” becomes play + ##ing.

Unigram / SentencePiece (used by T5, mBART, and many multilingual models). This employs a truly different algorithm, in that rather than building up by merging small pieces, it starts with a large candidate vocabulary and prunes down by repeatedly removing the tokens that “don’t matter” in a probabilistic model of the text - basically asking, “if I removed this piece, could I still come up with essentially the entire corpus anyway?”. It’s also probabilistic at tokenization time - a word can have several valid segmentations, each with a probability. In normal use, it just takes the single most likely one (so it’s used deterministically), but it can sample among the alternatives, for instance, to add variety during training. (One note: “SentencePiece” is technically the library that usually runs this, but it can also run BPE.)

So how does a team building a real model pick one? Why would someone use BPE vs WordPiece? It really is less about quality, as most of these schemes have been shown to perform pretty similarly, and more about some practical implications.

  • The biggest factor is language coverage. Approaches like SentencePiece run on raw text with no assumptions about spaces, which matters enormously for languages like Chinese, Japanese, and Thai that don’t separate words with spaces, so multilingual models tend to favor it.
  • Another factor is robustness. Byte-level fallback (like we have with byte-level BPE) guarantees anything can always be encoded and represented, with truly no unknowns.
  • Finally, there’s of course the very practical consideration of ecosystem. The tokenizer is a fundamental choice to the model that can’t be changed, so teams often just inherit whatever tooling they already have and whatever the models they build on have already used.

But really, the above considerations are fairly small, and tweaks can help accommodate them, so truly all of the above choices for algorithm here will work well. In modern times, the choice is converging, and most recent frontier models have landed on using byte-level BPE with large (100k+) vocabularies.

Appendix 2: The Rotational Property of Sinusoidal Encodings

As we saw in the positional encoding section, the cosine similarity between all consecutive positional encoding vectors is exactly the same value: approximately 0.9691 (for \(d_\text{model}=100\)). The similarity between positions 1↔︎2 is identical to 200↔︎201.

This isn’t a coincidence, and it hints at a deep structural property of sinusoidal encodings. Let’s work through why this happens mathematically, and then uncover the more fundamental property underneath - the one the transformer actually exploits.

Doing the Math

Recall that a positional encoding vector for position \(pos\) has \(d_\text{model}\) entries, arranged in sin/cos pairs. For frequency index \(j\) (where \(j = 0, 1, ..., \frac{d_\text{model}}{2} - 1\)):

\[\text{Column } 2j: \quad \sin\left(\frac{pos}{\omega_j}\right), \qquad \text{Column } 2j+1: \quad \cos\left(\frac{pos}{\omega_j}\right)\]

where \(\omega_j = 10000^{2j/d_\text{model}}\) sets the scale of that frequency-specific band (whose wavelength is \(2\pi\omega_j\)).

So we can think of each PE vector as \(d_\text{model}/2\) pairs of (sin, cos) at different frequencies:

\[PE(pos) = \left[\sin\!\left(\frac{pos}{\omega_0}\right),\ \cos\!\left(\frac{pos}{\omega_0}\right),\ \sin\!\left(\frac{pos}{\omega_1}\right),\ \cos\!\left(\frac{pos}{\omega_1}\right),\ \ldots\right]\]

Step 1: The Norm is Constant

The squared norm of any PE vector is:

\[\|PE(pos)\|^2 = \sum_{j=0}^{d/2-1} \left[\sin^2\!\left(\frac{pos}{\omega_j}\right) + \cos^2\!\left(\frac{pos}{\omega_j}\right)\right]\]

By the Pythagorean identity (\(\sin^2\theta + \cos^2\theta = 1\)), every term in the sum equals 1:

\[\|PE(pos)\|^2 = \sum_{j=0}^{d/2-1} 1 = \frac{d_\text{model}}{2}\]

Therefore:

\[\|PE(pos)\| = \sqrt{\frac{d_\text{model}}{2}} \quad \text{for all } pos\]

Every positional encoding vector has exactly the same length, regardless of position. This already tells us something: positions are distinguished purely by direction, not magnitude. They all live on the surface of a hypersphere of radius \(\sqrt{d_\text{model}/2}\).

Step 2: The Dot Product Depends Only on the Offset

Now, consider the dot product between two PE vectors at positions \(pos\) and \(pos + k\) (for any offset \(k\)):

\[PE(pos) \cdot PE(pos+k) = \sum_{j=0}^{d/2-1} \left[\sin\!\left(\frac{pos}{\omega_j}\right)\sin\!\left(\frac{pos+k}{\omega_j}\right) + \cos\!\left(\frac{pos}{\omega_j}\right)\cos\!\left(\frac{pos+k}{\omega_j}\right)\right]\]

The cosine difference identity states:

\[\cos(\alpha - \beta) = \cos\alpha\cos\beta + \sin\alpha\sin\beta\]

So if \(\alpha = \frac{pos}{\omega_j}\) and \(\beta = \frac{pos+k}{\omega_j}\), each pair collapses:

\[\sin\!\left(\frac{pos}{\omega_j}\right)\sin\!\left(\frac{pos+k}{\omega_j}\right) + \cos\!\left(\frac{pos}{\omega_j}\right)\cos\!\left(\frac{pos+k}{\omega_j}\right) = \cos\!\left(\frac{pos}{\omega_j} - \frac{pos+k}{\omega_j}\right) = \cos\!\left(\frac{-k}{\omega_j}\right)\]

Since \(\cos\) is an even function (\(\cos(-x) = \cos(x)\)):

\[PE(pos) \cdot PE(pos+k) = \sum_{j=0}^{d/2-1} \cos\!\left(\frac{k}{\omega_j}\right)\]

The absolute position \(pos\) has completely cancelled out. The dot product depends only on the offset \(k\).

Step 3: Therefore, Cosine Similarity is Constant at Fixed Offset

Cosine similarity is defined as:

\[\text{CosSim}(PE(pos),\ PE(pos+k)) = \frac{PE(pos) \cdot PE(pos+k)}{\|PE(pos)\| \cdot \|PE(pos+k)\|}\]

We’ve shown:

  • The numerator = \(\sum_{j} \cos(k/\omega_j)\), which depends only on \(k\)
  • The denominator = \(\sqrt{d/2} \cdot \sqrt{d/2} = d/2\), which is the same for all positions

So:

\[\boxed{\text{CosSim}(PE(pos),\ PE(pos+k)) = \frac{1}{d_\text{model}/2}\sum_{j=0}^{d_\text{model}/2-1} \cos\!\left(\frac{k}{\omega_j}\right) \quad \text{for all } pos}\]

This is a constant that depends only on the offset \(k\) and the model hyperparameters (\(d_\text{model}\) and the base 10000).

Verifying the Number

For our case (\(d_\text{model} = 100\), \(k = 1\)):

\[\text{CosSim} = \frac{1}{50}\sum_{j=0}^{49} \cos\!\left(\frac{1}{10000^{2j/100}}\right) = 0.969108...\]

We can verify this in code:

d_model = 100
k = 1
def cosine_sim(d_model, k):
    return sum(np.cos(k / (10000 ** (2*j / d_model))) for j in range(d_model // 2)) / (d_model / 2)

print(f"Predicted: {cosine_sim(d_model, k)}")
Predicted: 0.9691077369162043

This matches the flat line in our cosine similarity plot exactly. Of course, the usefulness of this is much more than just the fact that two neighboring positions have the exact same offset. The usefulness is that any two positions separated by an offset k have the same value as any other two positions separated by that same k.

But Why? The Rotation Property

Let’s talk about why this really happens. We’ve proven that cosine similarity is constant at fixed offset. But cosine similarity is a scalar summary - it collapses the relationship between two 100-dimensional vectors down to a single number. And the model doesn’t actually compute cosine similarities between PE vectors - it operates on the full vectors through learned linear projections. So cosine similarities being the same is not a useful property in and of itself. Rather, it’s a visible symptom of something deeper: shifting position by a fixed offset \(k\) is a rotation in embedding space.

In their paper, Vaswani et al. state a much more generalized property of sinusoidal positional encodings:

“We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset \(k\), \(PE_{pos+k}\) can be represented as a linear function of \(PE_{pos}\).”

If that doesn’t make immediate intuitive sense to you - it didn’t to me, either. So let’s dig into what exactly they meant, and why that leads to the downstream property we saw.

What This Means

Consider a single frequency band \(j\), which contributes a (sin, cos) pair to the PE vector. When we move from position \(pos\) to position \(pos + k\), these two entries transform as:

\[\sin\!\left(\frac{pos}{\omega_j}\right) \longrightarrow \sin\!\left(\frac{pos+k}{\omega_j}\right)\] \[\cos\!\left(\frac{pos}{\omega_j}\right) \longrightarrow \cos\!\left(\frac{pos+k}{\omega_j}\right)\]

Applying the trigonometric addition formulas:

\[\sin\!\left(\frac{pos+k}{\omega_j}\right) = \sin\!\left(\frac{pos}{\omega_j}\right)\cos\!\left(\frac{k}{\omega_j}\right) + \cos\!\left(\frac{pos}{\omega_j}\right)\sin\!\left(\frac{k}{\omega_j}\right)\]

\[\cos\!\left(\frac{pos+k}{\omega_j}\right) = \cos\!\left(\frac{pos}{\omega_j}\right)\cos\!\left(\frac{k}{\omega_j}\right) - \sin\!\left(\frac{pos}{\omega_j}\right)\sin\!\left(\frac{k}{\omega_j}\right)\]

This is a matrix equation:

\[\begin{bmatrix} \sin\!\left(\frac{pos+k}{\omega_j}\right) \\[4pt] \cos\!\left(\frac{pos+k}{\omega_j}\right) \end{bmatrix} = \begin{bmatrix} \cos(k/\omega_j) & \sin(k/\omega_j) \\ -\sin(k/\omega_j) & \cos(k/\omega_j) \end{bmatrix} \begin{bmatrix} \sin\!\left(\frac{pos}{\omega_j}\right) \\[4pt] \cos\!\left(\frac{pos}{\omega_j}\right) \end{bmatrix}\]

That transformation matrix is a 2D rotation matrix \(R_j(k)\), rotating by an angle of \(k/\omega_j\) radians. A rotation matrix is a transformation that changes the direction a vector points without changing its length. This is consistent with what we showed earlier, that all positional encoding vectors have the same norm. They’re distinguished purely by direction, and moving between positions is just a rotation. Since each frequency band transforms independently, the full PE vector transforms as:

\[PE(pos + k) = M_k \cdot PE(pos)\]

where \(M_k\) is a block-diagonal matrix:

\[M_k = \begin{bmatrix} R_0(k) & & & \\ & R_1(k) & & \\ & & \ddots & \\ & & & R_{d/2-1}(k) \end{bmatrix}\]

Each 2×2 block rotates its frequency band by an angle that depends on \(k\) and that band’s wavelength (\(2\pi\omega_j\)), but is completely independent of \(pos\). Shifting by \(k\) positions is a fixed linear transformation, regardless of where you start.

So cosine similarity is constant because rotations preserve geometry. A rotation doesn’t stretch, shrink, or distort vectors. It preserves all norms and all angles. So the angle between \(PE(pos)\) and \(PE(pos+k)\) is determined entirely by the rotation \(M_k\) that connects them, which depends only on \(k\). Cosine similarity measures the cosine of that angle, hence it’s constant. That is an outcome of how rotational transformations operate.

Why the Rotation Property Matters for Transformers

This rotational property is really what Vaswani et al. were referring to. Here’s why it’s so important. The attention mechanism computes:

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]

where \(Q = XW_Q\) and \(K = XW_K\) are linear projections of the input. If the input \(X\) includes positional encodings, then the attention score between positions \(pos_a\) and \(pos_b\) involves, among other things, the interaction between \(PE(pos_a)\) and \(PE(pos_b)\) through these learned weight matrices.

Suppose the model needs to learn “attend to the word 3 positions back.” That means it needs the attention score to be high when \(pos_a - pos_b = 3\). Since:

\[PE(pos_a) = M_3 \cdot PE(pos_b)\]

The relationship between the two PE vectors is captured entirely by the fixed matrix \(M_3\). The model’s learned projections \(W_Q\) and \(W_K\) can compose with \(M_3\) through ordinary matrix multiplication, so the same \(W_Q\) and \(W_K\) that detect “3 positions back” at position 10 will automatically detect “3 positions back” at position 200. The model learns this pattern once and then, (in theory), it could generalize everywhere.

If the positional encoding didn’t have this rotation structure and moving by \(k\) positions involved a different transformation depending on where you started, then the model’s fixed \(W_Q\) and \(W_K\) couldn’t capture relative position as a general concept. It would need to learn separate attention patterns for every absolute position, which scales terribly with sequence length.

Ok, one more thing though. Because the keen eye might have noticed - we DO have fixed cosine similarities between fixed positions k, great. And typically distinct cosine similarities for any k. And we have rotational transformations as a general property.

But we do not have monotonically decreasing cosine similarities for increasing k, and we made a statement earlier that we want close positions to have similar embeddings as one another. So, does that matter?

ks = np.arange(1,101,1)
def cosine_sim(d_model, k):
    return sum(np.cos(k / (10000 ** (2*j / d_model))) for j in range(d_model // 2)) / (d_model / 2)

cossims = [cosine_sim(d_model, k) for k in ks]

plt.scatter(ks, cossims); plt.show()

Short answer, not totally, but a little. We know that cosine similarity is a lossy scalar summary of a rich high-dimensional relationship - so the number itself isn’t important.

BUT… it is a useful property that nearby positional encodings are similar, because it can help the model learn relationships more easily. This is actually up to that choice of frequency - the “10000” that we sort of shrugged off before.

In truth, sinusoidal positional encodings have been largely replaced in modern transformers by methods like Rotary Position Embeddings (RoPE), which apply the same frequency-based rotation idea inside the attention mechanism rather than as an additive encoding. Because, as it turns out, while the rotational properties of sinusoidal encodings are present, the model can have issues fully exploiting them. But before we get into every single little detail here about positional encodings, I’ll instead save the full story - including why sinusoidal PE fell out of favor, how RoPE works, and why the base frequency turns out to be one of the most actively studied hyperparameters in modern LLMs - for a future, dedicated positional encodings notebook. For now, the sinusoidal version gives us exactly the right intuitions to understand the rest of the transformer architecture. And just know - it was still a good, calculated choice!

Take me back to the main tutorial

Appendix 3: Why Neural Networks Need Non-Linearity

As we worked through what the encoder produced, we saw a very key observation about the architecture come into play. That while attention is the mechanism that allows for complex information retrieval and word relationships, an MLP is still necessary for learning, because attention is fundamentally a layer of linear transformations, and non-linearity is a requirement for learning complex relationships.

In fact, this is one of the most consequential insights in all of modern machine learning, so worth digging deep into. If this isn’t your first introduction to neural networks, this concept is probably familiar, but we’ll go over again here, in detail, why non-linearity is such an essential feature.

What “Linear” Actually Means

First we have to be precise, because non-linearity being so important rests on what a linear function can and cannot give us. A function \(f\) is linear if it satisfies one property:

\[ f(a\mathbf{x} + b\mathbf{y}) = a\,f(\mathbf{x}) + b\,f(\mathbf{y}) \]

for inputs \(\mathbf{x}, \mathbf{y}\) and scalars \(a, b\).

There are two valuable properties in this definition.

  1. First, this means that the inputs are additive - any way that the inputs can be broken up into an addition of parts, the function can be applied independently to those parts, and added later, to get the same result.

  2. Second, it means that the function scales linearly. If you scale the input, the output scales by the exact same amount - putting “20” through the function is exactly equivalent to putting “5” through the function, then multiplying the result by 4.

📌 Note: You might wonder if these two properties are redundant. Over strictly rational numbers \((\mathbb{Q})\), additivity actually forces scaling to work automatically. And physically speaking, digital computers only use rational numbers (finite floating-point fractions). However, because we design neural networks under the theoretical, continuous framework of real numbers \((\mathbb{R})\) so that calculus and gradients work, we must explicitly include both properties in our definition, as over real numbers, they are completely distinct.

Simply put, a linear function preserves addition (and therefore subtraction) and scalar multiplication (and therefore division).

The Linearity of Matrix Multiplies

A matrix multiply \(f(\mathbf{x}) = \mathbf{x}W\) is the canonical linear function, as “linear algebra” as a field deals with this class of transformations. Any matrix multiply (as we’ve seen work out many times in this tutorial) is just a combination of products - multiplies between row and column values, summed. Since addition and multiplications themselves of course satisfy the property of “preserving addition and multiplication” - any matrix multiply with a matrix constructed of numbers is a linear function.

(Strictly speaking, adding a bias, \(\mathbf{x}W + \mathbf{b}\), to a matrix multiply, as is typical in neural networks, makes it affine - shifted on the origin - rather than linear, but the same conclusions follow, so we’ll use “linear” loosely to include it, as everyone does.)

The core insight that will lead to nonlinearity being necessary is this: a composition of linear functions is itself a linear function. Or basically, it doesn’t matter how many matrix multiplies you do in a row - since a matrix multiply is a linear combination, as many as you stack will always also be a linear combination.

Consider two linear layers:

\[ \mathbf{f(x)} = \mathbf{x}W_1, \qquad \mathbf{g(x)} = \mathbf{x}W_2 \]

If we stack them, doing them one after another, we get:

\[ \mathbf{z} = \mathbf{g ( f(x) )} = (\mathbf{x}W_1)W_2 \]

But remember the property above. Since f(x) and g(x) are both linear functions, and matrix multiplication is associative, we have:

\[ (\mathbf{x}W_1)W_2 = \mathbf{x}(W_1 W_2) \]

But \(W_1 W_2\) is itself just a matrix. If we call it \(W' = W_1 W_2\), then we can simply write:

\[ \mathbf{z} = \mathbf{x}W' \]

So two stacked layers is exactly equal to one with some other weight matrix \(W'\). And this doesn’t stop at two. Stack a hundred linear layers, or a thousand, and the same argument collapses every one of them:

\[ \mathbf{x}W_1 W_2 W_3 \cdots W_{1000} = \mathbf{x}W', \qquad W' = W_1 W_2 \cdots W_{1000} \]

What this means is that two, three, four, thousands of stacked linear layers are exactly equivalent to a single linear layer (with different numbers, of course, but always still expressible as just a single matrix). Since linear functions don’t care how many ways you decompose the input as a series of multiplies and adds, you get the exact same answer doing the math as 100 operations as you would just doing one.

Now, linear combinations can of course learn something. A vector, for instance, is interpretable as a combination of unit vectors, and how to combine those unit vectors is a learnable combination. That’s why linear layers are still useful in some layers of the network. But the other issue here is just that: that a linear function can only learn linear relationships.

The important geometrical interpretation is that linear functions can only separate their space with straight lines, or in higher dimensions, flat planes. Its decision boundaries are hyperplanes, and nothing it does can bend them. Geometrically, any linear mapping can rotate, scale, shear, and reflect space, but only uniformly - moving all of the points by the same amount, at once. This means that any decision boundary that can’t be divided by a hyperplane, cannot be learned by a purely linear function. But as we just saw, no matter how many layers of linear functions we stack, we can never get out of that trap.

The XOR Example

An incredibly simple example where we can observe the linear function being unable to learn is the simple XOR (“exclusive or”) function. Consider four points and labels we want a model to produce for each of them:

\[ (0,0) \to 0, \qquad (1,1) \to 0, \qquad (0,1) \to 1, \qquad (1,0) \to 1 \]

The output is 1 when exactly one input is 1, and 0 otherwise. Plot the four points: the two 0-labeled points sit on one diagonal (bottom-left and top-right), and the two 1-labeled points on the other (top-left and bottom-right).

Now try to separate them the only way a linear classifier can: with a single straight line, 1’s on one side, 0’s on the other. In the demo below, we’ll plot the points, and you can freely tweak all parameters that a linear function has available to fit the data.

The below code makes the demo - unhide if you want to see it

# @title
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact, FloatSlider

# 1. Define the XOR dataset
X = np.array([[0,0], [1,1], [0,1], [1,0]])
y = np.array([0, 0, 1, 1])

def xor_challenge(w1, w2, bias):
    # Set up the plot canvas
    fig, ax = plt.subplots(figsize=(6, 6))
    ax.set_xlim(-0.5, 1.5)
    ax.set_ylim(-0.5, 1.5)
    ax.grid(True, linestyle='--', alpha=0.5)

    # Generate the decision line: w1*x + w2*y + b = 0 -> y = -(w1*x + b) / w2
    x_vals = np.linspace(-0.5, 1.5, 100)
    if abs(w2) > 1e-3:
        y_vals = -(w1 * x_vals + bias) / w2
        ax.plot(x_vals, y_vals, color='purple', linewidth=2.5, label='Decision Boundary')

        # Color the spaces to show the model's predictions (Positive vs Negative space)
        # We check the sign of the center point to determine shading direction
        if w2 > 0:
            ax.fill_between(x_vals, y_vals, 2, color='blue', alpha=0.05)
            ax.fill_between(x_vals, y_vals, -2, color='red', alpha=0.05)
        else:
            ax.fill_between(x_vals, y_vals, 2, color='red', alpha=0.05)
            ax.fill_between(x_vals, y_vals, -2, color='blue', alpha=0.05)

    elif abs(w1) > 1e-3:
        # Handle perfectly vertical line scenario
        ax.axvline(-bias / w1, color='purple', linewidth=2.5, label='Decision Boundary')

    # Calculate live classification accuracy
    # Model predicts 1 if w1*x1 + w2*x2 + b > 0, else 0
    raw_outputs = X[:, 0] * w1 + X[:, 1] * w2 + bias
    predictions = (raw_outputs > 0).astype(int)
    correct_mask = (predictions == y)
    accuracy = np.mean(correct_mask) * 100

    # Plot the XOR points (Green if model gets it right, Orange/Red if it fails)
    for i in range(4):
        marker = '^' if y[i] == 1 else 'o'
        edge_color = 'green' if correct_mask[i] else 'crimson'
        linewidth = 4 if not correct_mask[i] else 2

        # Plot point colored by its true class, outlined by its correctness
        ax.scatter(X[i, 0], X[i, 1],
                   color='blue' if y[i] == 1 else 'red',
                   s=250, marker=marker, zorder=3,
                   edgecolors=edge_color, linewidths=linewidth)

        ax.text(X[i, 0] + 0.05, X[i, 1] + 0.05, f"Label: {y[i]}", fontsize=10, fontweight='bold')

    # Status box
    status_text = f"Accuracy: {accuracy:.0f}%\n"

    ax.text(-0.4, -0.4, status_text,
            bbox=dict(facecolor='white', alpha=0.9, boxstyle='round,pad=0.5'), fontsize=11)

    ax.set_title("XOR Linear Classifier", fontsize=12, fontweight='bold')
    plt.show()
# 2. Build the interactive sliders and launch the challenge
interact(
    xor_challenge,
    w1=FloatSlider(value=1.0, min=-3.0, max=3.0, step=0.1, description='W1'),
    w2=FloatSlider(value=-1.0, min=-3.0, max=3.0, step=0.1, description='W2'),
    bias=FloatSlider(value=0.0, min=-2.0, max=2.0, step=0.1, description='b')
);

▶ Interactive in the Colab version. The control above is a live ipywidgets slider; on this static page there is no kernel to recompute it, so open the notebook in Colab to drag the sliders yourself.

The best any linear model can do is 75% - it’s impossible to draw a line to cleanly separate XOR. And since we just proved that any depth of linear layers collapses to a single linear map, no linear network of any size, however deep or wide, can solve XOR.

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, which laid out exactly this limitation of single-layer linear models - and XOR was the canonical issue. The simple perceptron, which was the leading neural model of the day, provably would never be able to learn this simple four-point function. This finding contributed to a long decline in neural-network research, part of what’s now called the first “AI winter”.

The resolution, when it came, was precisely the architecture we use today: stack layers, with a differentiable non-linear function between them, then use backpropogation to learn. This insight saved the field, as it’s fundamental to the ability for any NN-based architecture to learn.

Non-Linearities Relocate Points to a New Space

We already discussed how linear transformations can only move points rigidly, by rotating, scaling, shearing, or reflecting, but can never change which points a straight line can separate, since all transformations move all points rigidly, the same way.

A non-linearity escapes this because it can move points non-rigidly. Some points can change their position a lot while others remain in roughly the same spot. The ReLU,

\[ \text{ReLU}(z) = \max(0, z) \]

is an incredibly simple non-linear function that does this by simply clamping every negative value to zero. Geometrically, a whole half of the space gets flattened onto a wall at zero. Crucially, what this means, is that different points that were spread out can get pressed onto the same spot. Just that simple idea of “forcing values below the threshold to 0”, gives us enough of a nonlinearity that we can shift the coordinates and the points as we need, into a configuration where a line now can separate them.

The key way to think of this is as a transformation of the coordinate space. A neural network has hidden units, and each hidden unit computes a new coordinate for every point (first via a linear combination, then the clamp). A full hidden layer moves all of the points into a new room, with a new coordinate space of transformed axes \((h_1, h_2)\). Where the original room was a Euclidean space - all lines are parallel and shifts uniform, ReLU’s clamp allows this new coordinate space to have creases that collapse points together as it relocates them.

If this isn’t feeling intuitive, because humans really don’t find folded spaces intuitive, let’s also see how running our four XOR points through one ReLU layer transforms them mathematically.

The Math of a Simple Layer with ReLU

Let’s go through the math of the simplest neural network we can make.

A neural network processes a spatial transformation in three distinct stages. First, the linear projection, then, the nonlinear activation, and finally, the final linear readout.

Let’s say our network takes a two-dimensional input coordinate vector \(\mathbf{x} = [x_1, x_2]\) and passes it to a hidden layer containing two hidden units. In standard matrix notation, this layer is expressed as:

\[ \mathbf{h} = \text{ReLU}(\mathbf{x}W_1 + \mathbf{b}_1) \]

To see exactly what happens to a coordinate, we can unpack these matrices into their structural components:

\[ W_1 = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix}, \qquad \mathbf{b}_1 = \begin{bmatrix} b_1 & b_2 \end{bmatrix} \]

If we break this matrix multiplication down into individual scalar equations, we can see the exact independent recipe used to calculate each new hidden coordinate (\(\text{h}_1\) and \(\text{h}_2\))

1. The Linear Projection

Hidden Unit 1 (\(\text{h}_1\))

The first step of this layer is to calculate a raw linear score (\(z_1\)) by scaling and adding the inputs (doing a linear transformation) on the original coordinate set. \[ z_1 = x_1 w_{11} + x_2 w_{21} + b_1 \]

Hidden Unit 2 (\(\text{h}_2\))

Simultaneously, the second unit calculates its own independent linear score (\(z_2\)): \[ z_2 = x_1 w_{12} + x_2 w_{22} + b_2 \]

So each hidden unit starts with a linear combination of the original coordinates - a shift, shear, or rescaling, of the original coordinate.

2. The Nonlinear Activation

After the linear combination, we have a new set of points, \((z_1, z_2)\), that are each some linear combination of the original points. We have moved the points within the Euclidean coordinate space with this transformation, but have not been able to fundamentally change their relative layout in the space. The next step, the nonlinear activation, is where we can actually move the points in the space non-uniformly, by clamping one side. This draws a bend, or wall, in the new coordinate space where points on one side of the boundary are pushed and points on the other side are left alone.

\[ h_1 = \max(0, \, z_1) \]

\[ h_2 = \max(0, \, z_2) \]

These final outputs \([h_1, h_2]\) are our brand-new coordinate vector \(\mathbf{h}\), which now exist, for each of our original points, in this hidden state space, and no longer in the original coordinate system.

3. The Final Readout Line

Then, we can draw a decision boundary line in this new coordinate space that attempts to separate the points. This is an output layer, and it takes the hidden vector \(\mathbf{h} = [h_1, h_2]\) and applies one final linear combination:

\[ \text{Output Score} = \mathbf{h}W_2 + b_{\text{out}} \]

Expressed as a plain scalar equation, the final readout is simply:

\[ \text{Output Score} = h_1 w_{\text{out1}} + h_2 w_{\text{out2}} + b_{\text{out}} \]

The network then applies a strict threshold to this score to determine the final classification prediction:

  • If \(\text{Output Score} > 0 \implies \text{Predict } 1\)
  • If \(\text{Output Score} \le 0 \implies \text{Predict } 0\)

Putting Real Numbers In: Reasoning About the Weights

The incredibly simple XOR / two hidden unit network is a rare case that’s small enough that we can put actual numbers in and reason about why each weight has the value it does. So just to really hammer in the point, let’s look at a specific set of weights that solves XOR, and how exactly these computations proceed to transform the values.

If:

\[ W_1 = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \quad \mathbf{b}_1 = \begin{bmatrix} 0 & -1 \end{bmatrix}, \qquad W_2 = \begin{bmatrix} 1 \\ -2 \end{bmatrix}, \quad b_2 = 0 \]

Then, this will solve XOR.

First, notice what XOR actually depends on. Look at the four points by the sum of their inputs, \(s = x_1 + x_2\):

point (\(x_1\), \(x_2\)) s = \(x_1\) + \(x_2\) label
\((0,0)\) 0 0
\((0,1)\) 1 1
\((1,0)\) 1 1
\((1,1)\) 2 0

The label is 1 exactly when the sum is 1, and 0 otherwise. The issue is that drawing a boundary between both 0 and 1 and 1 and 2, is impossible. Any way that you linearly combine those coordinates, there will not be able to be this clean separation.

\[ \underset{\text{label 0}}{0} \quad\longrightarrow\quad \underset{\text{label 1}}{1} \quad\longrightarrow\quad \underset{\text{label 0}}{2} \]

Now, let’s look at what the two hidden units actually do. With these weights, both units compute a linear combination of the inputs. With the weights that we chose above for \(W_1\)’s, rows both \([1,1]\), both hidden units do just compute the sum (\(s\)), differing only in bias:

\[ z_1 = x_1 + x_2 = s, \qquad z_2 = x_1 + x_2 - 1 = s - 1 \]

\[ \text{h}_1 = \max(0,\, s), \qquad \text{h}_2 = \max(0,\, s-1) \]

Unit 1 is just the raw sum (it never goes negative here, so the ReLU never needs to clamp the values), so \(\text{h}_1 = s\). On its own, \(\text{h}_1\) cannot be thresholded to draw a decision boundary between the values, like we just saw above.

\[ \text{h}_1:\quad s {=} 0 \to 0, \qquad s {=} 1 \to 1, \qquad s {=} 2 \to 2 \]

But Unit 2 has a bias of \(-1\), shifting the line so that \(z_2 = s - 1\): negative at \(s=0\), zero at \(s=1\), positive at \(s=2\). The ReLU then hinges at \(s = 1\) - it clamps everything up to the sum-of-1 flat to zero, and only lets \(s=2\) rise above it:

\[ \text{h}_2:\quad s{=}0 \to 0, \qquad s{=}1 \to 0, \qquad s{=}2 \to 1 \]

So \(\text{h}_2\) is essentially a detector of values over one. This is precisely what was needed - we could draw a boundary between 0 and 1, but not also between 1 and 2. Hidden unit 2 allows us to find, easily, all of the values > 1.

But the second unit is also not able to capture all of the information we need on its own. When we combine them, we get the benefit of how both hidden units divided the space. The readout is \(\text{out} = \text{h}_1 - 2\text{h}_2 = s - 2\text{h}_2\):

\(\text{h}_1\) \(\text{h}_2\) \(\text{out}\) = \(\text{h}_1\) - \(2h_2\) label
0 0 0 0
1 0 1 1
2 1 0 0

Essentially, ReLU has allowed us to turn that 2 back into a 0, without changing the value of the “1”. We still use the original sum to identify values that exceed our threshold of “1” and need a label of “0”, but we can re-transform it to “0” using the output of the second hidden unit, without affecting our sums of “1” that we needed, because ReLU let us put a fold in the space that lets us manipulate numbers only above 1. After this transformation, now a threshold (\(\text{out} > 0.5\)) cleanly isolates the middle, and XOR is solved.

You can imagine these numbers to be something different. Even keeping the weights the same, shifting that bias term will move which values get clamped. So as you also can see, these particular weights are just one particular solution, of which XOR has many! This was just a particularly clean one that lets us build up good intuition about how ReLU helps solve the problem. Gradient descent, left to its own devices, usually finds a much messier set of weights. The point isn’t that this is the answer - it’s that a nonlinearity makes it possible to find an answer because it allows transformation into a new hidden space.

Below, here’s a simple demo that shows the hidden space mapping and the boundary line we drew through it, starting with the weights we just worked through above. But the values are adjustable - if you want to try and think of a different potential solution, feel free to try it out below! As a good exercise, try to imagine how the hidden space positioning will move in response to your changes before you make them.

And as we said, you’ll notice that there are a lot of combinations that will solve this problem with 100% accuracy.

# @title
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact, FloatSlider

relu = lambda z: np.maximum(0, z)
X = np.array([[0,0], [1,1], [0,1], [1,0]], float)
y = np.array([0, 0, 1, 1])
col = {0: '#2166ac', 1: '#b2182b'}   # blue = label 0, red = label 1
mk  = {0: 'o',       1: '^'}

def xor_playground(w11, w12, w21, w22, b1a, b1b, wo1, wo2, bo):
    # --- hidden layer: relocate the points ---
    W1 = np.array([[w11, w12], [w21, w22]])
    b1 = np.array([b1a, b1b])
    H  = relu(X @ W1 + b1)                 # the four points' new coordinates

    # --- readout line: try to separate them ---
    w = np.array([wo1, wo2])
    score = H @ w + bo
    pred  = (score > 0).astype(int)
    acc   = (pred == y).mean()

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5.6))

    # LEFT: input space, fixed reference
    for xi, yi in zip(X, y):
        ax1.scatter(*xi, c=col[yi], s=250, marker=mk[yi], edgecolor='k', lw=1.5, zorder=3)
    ax1.set_title("Input space (fixed)")
    ax1.set_xlim(-0.6, 1.7); ax1.set_ylim(-0.6, 1.7)
    ax1.set_xlabel("$x_1$"); ax1.set_ylabel("$x_2$")
    ax1.grid(alpha=0.3); ax1.set_aspect('equal')

    # RIGHT: hidden space - points move here, and the readout line tries to split them
    gx, gy = np.meshgrid(np.linspace(-0.6, 3.2, 300), np.linspace(-0.6, 2.6, 300))
    region = ((np.stack([gx, gy], -1) @ w + bo) > 0).astype(int)
    ax2.contourf(gx, gy, region, levels=[-0.5, 0.5, 1.5],
                 colors=['#2166ac', '#b2182b'], alpha=0.08)   # shaded predicted regions
    if abs(wo2) > 1e-6:                                       # draw the boundary line
        xs = np.array([-0.6, 3.2]); ax2.plot(xs, -(wo1*xs + bo)/wo2, 'g-', lw=2)
    elif abs(wo1) > 1e-6:
        ax2.axvline(-bo/wo1, color='g', lw=2)

    seen = {}
    for h, yi, pi in zip(H, y, pred):
        key = tuple(np.round(h, 3))
        k = len(seen.get(key, [])); seen.setdefault(key, []).append(1)
        o = 0.06 * k                                          # jitter coincident points
        edge = 'green' if pi == yi else 'crimson'
        lw   = 2 if pi == yi else 3.5                         # thick red edge = misclassified
        ax2.scatter(h[0]+o, h[1]+o, c=col[yi], s=260, marker=mk[yi],
                    edgecolor=edge, lw=lw, zorder=5)
    ax2.set_title(f"Hidden space $(h_1, h_2)$   -   accuracy: {acc*100:.0f}%")
    ax2.set_xlim(-0.6, 3.2); ax2.set_ylim(-0.6, 2.6)
    ax2.set_xlabel("$\text{h}_1$"); ax2.set_ylabel("$\text{h}_2$")
    ax2.grid(alpha=0.3); ax2.set_aspect('equal')
    plt.tight_layout(); plt.show()
s = lambda v, lo, hi, d: FloatSlider(value=v, min=lo, max=hi, step=0.1, description=d)
interact(
    xor_playground,
    # hidden layer weights (relocate the points)
    w11=s(1, -2, 2, 'W1[1,1]'), w12=s(1, -2, 2, 'W1[1,2]'),
    w21=s(1, -2, 2, 'W1[2,1]'), w22=s(1, -2, 2, 'W1[2,2]'),
    b1a=s(0, -2, 2, 'bias h1'), b1b=s(-1, -2, 2, 'bias h2'),
    # readout line (separate the points)
    wo1=s(1, -3, 3, 'line w1'), wo2=s(-2, -3, 3, 'line w2'), bo=s(-0.5, -3, 3, 'line bias'),
);

▶ Interactive in the Colab version. The control above is a live ipywidgets slider; on this static page there is no kernel to recompute it, so open the notebook in Colab to drag the sliders yourself.

The Universal Approximation Theorem

So we just proved that non-linearity is necessary, AND exactly how a non-linearity gets around the issues that pure linearity cannot. As it turns out, a remarkable theorem shows just how sufficient this simple non-linear function is, and states an even more general property more cleanly. The universal approximation theorem as it was originally stated (Cybenko, 1989; Hornik, 1991; sharpened by Leshno et al., 1993) says:

A feedforward network with a single hidden layer can approximate any continuous function on a closed and bounded domain to arbitrary accuracy - provided its activation function is not a polynomial.

Every bolded part is an important qualifier, so let’s step through them.

  • “Single hidden layer.” Note here that this is not a statement about depth. The theorem is instead about width. One hidden layer, with enough units, summed together, can do the job. Going back to our folds in the space analogy - you might need a whole ton of folds, but you can always make enough folds to divide the space how you need to. (As it turns out, depth - stacking layers - which most modern neural nets rely on, will learn much more efficiently, which is why real models are deep. But that is separate from this fundamental theorem’s result.)

  • “Continuous function on a closed and bounded domain.” Not literally any underlying function can be learned this way. The target has to be continuous, have good approximation over a reasonably bounded portion of input space, not out to infinity. As long as that’s satisfied though, the relationship between inputs and outputs can be as complicated as you want. You’ll note that this is true for any real case we use neural nets for. We presume some true underlying relationship between our inputs and outputs, over reasonable values we can really put into a computer, and roughly that other inputs that mean the same sort of thing can go through the same process to get the same answer. That’s really all you need.

  • “To arbitrary accuracy.” This is really saying, no matter how accurate you want to be, there exists a network that stays within your error tolerance everywhere on the domain, for all of the points.

  • “Not a polynomial.” This is the condition, that we just spent all of that time explaining the importance of. The activation can take many, many different forms (sigmoid, tanh, ReLU, …) as long as it isn’t a polynomial. This condition is actually the general version of our multiple linear layers always becoming a single linear layer. That earlier “linear collapses to linear” observation wasn’t a special quirk of linearity - it is a case of the more general rule that any polynomial activation fails to be universal.

Basically, if the activation were a polynomial of degree \(d\), then any combination of our hidden units is also a polynomial of degree at most \(d\). Since polynomials can’t arbitrarily bend - how they bend is limited by their degree, a polynomial activation is not able to generally and arbitrarily divide the space - similar to the linear instance, there are very particular, rigid ways in which it can transform the space, which will not in general be sufficient.

Two very important things to point out:

  1. This is an existence result, meaning, it guarantees a good network exists, not that gradient descent will actually find it, or that the version that was found to fit the training data will universally generalize. In practice, this “true” underlying function is never really found.

  2. “Enough units” can be enormous. Just because enough units could approximate, doesn’t mean you can build a network that has enough units. This can be an arbitrarily enormous number, well beyond practical limits.

So the takeaway isn’t “one hidden layer is all you need”. It’s that linear (and polynomial) machinery is limited to approximating only a very small class of functions. But even a single nonpolynomial nonlinearity makes the entire space of continuous functions theoretically within reach.

Why ReLU?

We used ReLU in our example, partly because it’s so simple and easy to understand. But actually, it’s a reasonable default (even though it’s mostly been replaced in modern implementations), and worth mentioning why it’s still is so useful despite being so simple.

Historically, sigmoid and tanh were more standard. They are smooth, curved non-linearities. They work well, but are easy to saturate, meaning that large positive or negative inputs, their slope flattens to nearly zero. This causes a classic issue where gradients flowing backward through them shrink toward nothing (the “vanishing gradient” problem), slowing or stopping entirely the training of a network.

ReLU looks almost embarrassingly simple by comparison. But its simplicity provides many benefits:

  1. It’s sufficiently non-linear. As we just saw above, the non-linear function doesn’t have to be complicated to allow learning, just non-linear at all. The tiny break in ReLU is totally sufficient to satisfy that property.
  2. The fact that for any positive input, its slope is exactly 1, means that those gradient issues pose no problems - gradients pass through undiminished, and deep networks train far faster and more reliably.
  3. It’s trivially cheap to compute. A simple clamp is about the computationally cheapest function you can do, which matters when the activation function has to be applied repeatedly in the already onerous math of a neural network.

The fact that any nonlinearity is sufficient isn’t to say that the choice doesn’t matter at all. As we just discussed, gradients flow through this calculation, so there are certainly other considerations, and in fact, typically other activations are used in modern models. GELU, essentially a smoothed version of ReLU, is used in many modern transformers. And there are many other options, chosen for many other reasons, which we will not get into here! But even if not perfect, ReLU has repeatedly been found to still be a pretty good choice in a lot of cases, for the reasons explained above.

And at its core, it’s all that’s needed. That small ReLU operation is the only nonlinearity that acts on each token’s own features, allowing the model to transform them into something new. So while it might look like a tiny little filter on the numbers, hopefully you’ve seen now how it’s the crux of all learning!

Take me back to the main tutorial

Appendix 4: Next Token Prediction (and Why the KV Cache Works)

When we were in Section 5, talking about the output, we made a claim: That the final row of the output from the decoder is all that you need to predict the next token, because that’s its job. Without pausing on that for a bit, that might seem strange. We get a whole matrix out of the decoder that represents the full sequence, and then we just use the last row to predict the next token. It seems wasteful on compute, and on information.

Then, I pointed out that it’s sort of neither. That we only use the last row, because everything else will be identical and therefore of no new use to us, and that this repetitive nature of the matrix coming out leads to something we can exploit to speed up generation.

In this appendix, we tackle both of those claims. First proving, what exactly is new and what is the same when you feed the same sequence over and over again through the decoder, and then talking about how we can use this property in something called the KV Cache.

Working Through It On a Tiny Example

Before we can talk about how to exploit this property, let’s prove that it’s true. That after each pass through the decoder, the only part of the matrix that is different is the new final row corresponding to the most recently added token.

Rather than do that abstractly, it’s easiest to see working through exactly the math for a simple, small example. We’ll take a sequence of 2 tokens, write out literally every matrix in the masked self-attention step of the decoder, then append a 3rd token, and write everything out again. To keep it simple, we’ll use just one attention head, but by now you should be convinced that more heads will behave exactly the same way.

With Two Tokens

Coming into the decoder, let’s say that our input sequence is: \[ \text{[I, love]} \] which gets tokenized cleanly into the two tokens: \[ \text{inp} = [x, y] \]

with a \(d_\text{model}\) of say, 3, after we embed this input, we would end up with:

\[ X = \begin{bmatrix}{x_i \ x_j \ x_k \\ y_i \ y_j \ y_k}\end{bmatrix} \]

The first step of attention is to construct \(Q\), \(K\), and \(V\) matrices, from \(W_Q\), \(W_K\), and \(W_V\). Remember that these are all constructed by multiplying \(X\) by a \(W\) matrix of size \(d_\text{model} \times d_\text{model}\). Looking at just, for example, the final query matrix, \(Q\):

\[ Q = X W_Q = \begin{bmatrix}{x_i \ x_j \ x_k \\ y_i \ y_j \ y_k}\end{bmatrix} \times \begin{bmatrix}q_{1,1} \ q_{1,2} \ q_{1,3} \\ q_{2,1} \ q_{2,2} \ q_{2,3} \\ q_{3,1} \ q_{3,2} \ q_{3,3}\end{bmatrix} \]

Just to be super explicit, let’s write out the final matrix:

\[ Q = \begin{bmatrix} (x_i q_{1,1} + x_j q_{2,1} + x_k q_{3,1}) \ \ \ (x_i q_{1,2} + x_j q_{2,2} + x_k q_{3,2}) \ \ \ (x_i q_{1,3} + x_j q_{2,3} + x_k q_{3,3}) \\ (y_i q_{1,1} + y_j q_{2,1} + y_k q_{3,1}) \ \ \ (y_i q_{1,2} + y_j q_{2,2} + y_k q_{3,2}) \ \ \ (y_i q_{1,3} + y_j q_{2,3} + y_k q_{3,3}) \end{bmatrix} \]

Calling the elements of \(Q\) by just \(Q_n\), notice something.

\[ \begin{bmatrix} Q_{x1} \ Q_{x2} \ Q_{x3} \\ Q_{y1} \ Q_{y2} \ Q_{y3} \end{bmatrix} \begin{matrix} ← \text{depends only on x} \\ ← \text{depends only on y} \end{matrix} \]

And this is true for K and V as well, as they’re constructed the same exact way as Q, just with different W:

\[ \begin{bmatrix}K_{x1} \ K_{x2} \ K_{x3} \\ K_{y1} \ K_{y2} \ K_{y3} \end{bmatrix} , \begin{bmatrix}V_{x1} \ V_{x2} \ V_{x3} \\ V_{y1} \ V_{y2} \ V_{y3} \end{bmatrix} \begin{matrix} ← \text{depends only on x} \\ ← \text{depends only on y} \end{matrix} \]

So row 1 of \(Q, K,\) and \(V\) each depend only on token 1; row 2 only on token 2. Appending a 3rd token at the end creates a row 3 to each of these matrices, that has nothing to do with token 1 or token 2 at all. Each row of \(Q, K,\) and \(V\) corresponds to exactly one token in the sequence.

Well, does that change when we use them? Let’s see.

The next step of the attention calculation is to compute attention scores, \(S\), from \(Q\) and \(K^\top\)

\[ S = Q K^\top = \begin{bmatrix} Q_{x1} \ Q_{x2} \ Q_{x3} \\ Q_{y1} \ Q_{y2} \ Q_{y3} \end{bmatrix} \times \begin{bmatrix}K_{x1} \ K_{y1} \\ K_{x2} \ K_{y2} \\ K_{x3} \ K_{y3} \end{bmatrix} \ = \ \begin{bmatrix} (Q_{x1} K_{x1} + Q_{x2} K_{x2} + Q_{x3} K_{x3}) \ \space \ (Q_{x1} K_{y1} + Q_{x2} K_{y2} + Q_{x3} K_{y3}) \\ (Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3}) \ \space \ (Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3}) \end{bmatrix} \]

So in S, things are now mixed, writing s in terms of what tokens it mixes from this above calculation, we see:

\[ S = \begin{bmatrix} s_{xx} \ s_{xy} \\ s_{yx} \ s_{yy} \end{bmatrix} \]

But, remember what happens next specifically at generation time in the decoder. We have to mask this sequence, because there’s no reasonable way that x could be asking about y - \(s_{xy}\) doesn’t really make sense. So we apply the causal mask, saying query \(i\) may not look at key \(j\) when \(j > i\), so those entries become \(-\infty\):

\[S_{\text{masked}} = \begin{bmatrix} s_{xx} & -\infty \\ s_{yx} & s_{yy} \end{bmatrix}\]

Then, we softmax each row (over the keys). Since \(e^{-\infty} = 0\):

\[ \text{row 1:}\quad \left[\frac{e^{s_{xx}}}{e^{s_{xx}}+e^{-\infty}},\ \frac{e^{-\infty}}{e^{s_{xx}}+e^{-\infty}}\right] = [\,1,\ 0\,] \] \[ \text{row 2:}\quad \left[\frac{e^{s_{yx}}}{e^{s_{yx}}+e^{s_{yy}}},\ \frac{e^{s_{yy}}}{e^{s_{yx}}+e^{s_{yy}}}\right] = [\,a_{yx},\ a_{yy}\,] \]

The attention-weight matrix is then:

\[A = \begin{bmatrix} 1 & 0 \\ a_{yx} & a_{yy} \end{bmatrix}\]

Finally, we multiply this back by V to get the final output

\[ O = A V = \begin{bmatrix} 1 & 0 \\ a_{yx} & a_{yy} \end{bmatrix} \times \begin{bmatrix}V_{x1} \ V_{x2} \ V_{x3} \\ V_{y1} \ V_{y2} \ V_{y3} \end{bmatrix} = \begin{bmatrix}V_{x1} \ V_{x2} \ V_{x3} \\ (a_{yx} V_{x1} + a_{yy} V_{y1}) \ (a_{yx} V_{x2} + a_{yy} V_{y2}) \ (a_{yx} V_{x3} + a_{yy} V_{y3}) \end{bmatrix} \]

So mixing still happens, but only backwards mixing. Looking more closely at the second output row (the more interesting one), just at what that very first entry in the output is, explicitly:

\[ (a_{yx} V_{x1} + a_{yy} V_{y1}) = \frac{e^{s_{yx}}}{e^{s_{yx}}+e^{s_{yy}}} V_{x1} + \frac{e^{s_{yy}}}{e^{s_{yx}}+e^{s_{yy}}} V_{y1} = \\ \frac{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}}{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}+e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}} V_{x1} + \frac{e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}}{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}+e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}} V_{y1} \]

I know it’s quite the doozy to write out like that, but just keep it in mind and we’ll come back after working through the example with 3!

Adding a Third Token

We’ll go through this one just a little faster. When we add a third token to the sequence, so \(X = [x,y,z]\), we now have for \(Q,K,V\):

\[ \begin{bmatrix}Q_{11} \ Q_{12} \ Q_{13} \\ Q_{21} \ Q_{22} \ Q_{23} \\ Q_{31} \ Q_{32} \ Q_{33}\end{bmatrix} , \begin{bmatrix}K_{11} \ K_{12} \ K_{13} \\ K_{21} \ K_{22} \ K_{23} \\ K_{31} \ K_{32} \ K_{33}\end{bmatrix} , \begin{bmatrix}V_{11} \ V_{12} \ V_{13} \\ V_{21} \ V_{22} \ V_{23} \\ V_{31} \ V_{32} \ V_{33}\end{bmatrix} \begin{matrix} ← \text{depends only on x} \\ ← \text{depends only on y} \\ ← \text{depends only on z}\end{matrix} \]

When the S calculation comes around to mix these, we get:

\[S = \begin{bmatrix} s_{xx} & s_{xy} & s_{xz} \\ s_{yx} & s_{yy} & s_{yz} \\ s_{zx} & s_{zy} & s_{zz} \end{bmatrix}\]

Then, this time when we apply the triangular causal mask for \(j > i \Rightarrow -\infty\):

\[S_{\text{masked}} = \begin{bmatrix} s_{xx} & -\infty & -\infty \\ s_{yx} & s_{yy} & -\infty \\ s_{zx} & s_{zy} & s_{zz} \end{bmatrix}\]

Notice something immediately. The 2x2 matrix that exists in the top left of this is the identical matrix to the one we had above in the 2 token case. Adding a third token added a third row, with real values, but only an extra \(-\infty\) to the existing rows which we know becomes 0 in the next step.

Indeed,

\[ \text{row 1:}\quad \frac{[\,e^{s_{xx}},\ e^{-\infty},\ e^{-\infty}\,]}{e^{s_{xx}}+e^{-\infty}+e^{-\infty}} = \frac{[\,e^{s_{xx}},\ 0,\ 0\,]}{e^{s_{xx}}+0+0} = [\,1,\ 0,\ 0\,] \] \[ \text{row 2:}\quad \frac{[\,e^{s_{yx}},\ e^{s_{yy}},\ e^{-\infty}\,]}{e^{s_{yx}}+e^{s_{yy}}+e^{-\infty}} = \frac{[\,e^{s_{yx}},\ e^{s_{yy}},\ 0\,]}{e^{s_{yx}}+e^{s_{yy}}+0} = [\,a_{yx},\ a_{yy},\ 0\,] \] \[ \text{row 3:} \quad \text{(I will spare you the math)} \quad = [\,a_{zx},\ a_{zy},\ a_{zz}\,] \]

\[A = \begin{bmatrix} 1 & 0 & 0 \\ a_{yx} & a_{yy} & 0 \\ a_{zx} & a_{zy} & a_{zz} \end{bmatrix}\]

And then, the final output here ends up being:

\[ O = A V = \begin{bmatrix} 1 & 0 & 0 \\ a_{yx} & a_{yy} & 0 \\ a_{zx} & a_{zy} & a_{zz}\end{bmatrix} \times \begin{bmatrix}V_{x1} \ V_{x2} \ V_{x3} \\ V_{y1} \ V_{y2} \ V_{y3} \\ V_{z1} \ V_{z2} \ V_{z3}\end{bmatrix} = \begin{bmatrix}V_{x1} \quad \quad \quad \quad \quad \quad \quad V_{x2} \quad \quad \quad \quad \quad \quad \quad V_{x3} \\ (a_{yx} V_{x1} + a_{yy} V_{y1}) \ \quad \quad \ (a_{yx} V_{x2} + a_{yy} V_{y2}) \ \quad \quad \ (a_{yx} V_{x3} + a_{yy} V_{y3}) \\ (a_{zx} V_{x1} + a_{zy} V_{y1} + a_{zz} V_{z1}) \ \ (a_{zx} V_{x2} + a_{zy} V_{y2} + a_{zz} V_{z2}) \ \ (a_{zx} V_{x3} + a_{zy} V_{y3} + a_{zz} V_{z3}) \\ \end{bmatrix} \]

We can write out row 2 column 1, the first output for the second token explicitly again, tracing exactly what contributes to each of the elements that comprise it, but spoiler: it’s exactly the same as it was in the 2 token case:

\[ (a_{yx} V_{x1} + a_{yy} V_{y1}) = \frac{e^{s_{yx}}}{e^{s_{yx}}+e^{s_{yy}}} V_{x1} + \frac{e^{s_{yy}}}{e^{s_{yx}}+e^{s_{yy}}} V_{y1} = \\ \frac{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}}{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}+e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}} V_{x1} + \frac{e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}}{e^{(Q_{y1} K_{x1} + Q_{y2} K_{x2} + Q_{y3} K_{x3})}+e^{(Q_{y1} K_{y1} + Q_{y2} K_{y2} + Q_{y3} K_{y3})}} V_{y1} \]

It doesn’t pull in any z elements. The causal mask makes sure of it - because tokens can only look behind, never ahead, the mixing they do is only over tokens before them. Every time we add a new token, and new rows, that mask makes sure that no information from that new token comes into play. We could add a fourth, fifth, hundredth token to the sequence, and never will any of their Q, K, or V come to incorporate into that row 2 output - it’s completely fixed.

Through Cross-Attention and the Feed-Forward Network

So, hopefully you’re convinced that masked self-attention leaves previous rows unchanged. But a decoder stack has two more sublayers that operate on it before we get to our final output. So of course the next question is: do they matter?

Of course, the point of this whole appendix is to assure you: no. The good news is that self-attention was the hard case of the three to run through. So we’ll hopefully not need to go through quite as much effort here to be sure the other two don’t matter, either.

Cross-attention

Recall how cross-attention differs from self-attention: the query comes from the decoder, but the key and value come from the encoder’s output, let’s call it \(E\), which are computed a single time and then are frozen. Whatever \(E\) is at the end of the encoder stack, it always is.

Then, recall how the score matrix mixes information. Every entry of S is a row of queries - corresponding to our decoder’s current sequence, multiplied by a column of keys, which will correspond to the frozen encoder information. Let’s say our encoder encoded the sequence:

\[ [a, b, c]\]

and we’re back to the 2 token case in the decoder, where our current output sequence is

\[ [x, y]\]

then:

\[ S = Q K^\top = \begin{bmatrix} Q_{x1} \ Q_{x2} \ Q_{x3} \\ Q_{y1} \ Q_{y2} \ Q_{y3} \end{bmatrix} \times \begin{bmatrix}K_{a1} \ K_{b1} \ K_{c1} \\ K_{a2} \ K_{b2} \ K_{c2} \\ K_{a3} \ K_{b3} \ K_{c3} \end{bmatrix} \ = \ \begin{bmatrix} (Q_{x1} K_{a1} + Q_{x2} K_{a2} + Q_{x3} K_{a3}) \ \ \ (Q_{x1} K_{b1} + Q_{x2} K_{b2} + Q_{x3} K_{b3}) \ \ \ (Q_{x1} K_{c1} + Q_{x2} K_{c2} + Q_{x3} K_{c3}) \\ (Q_{y1} K_{a1} + Q_{y2} K_{a2} + Q_{y3} K_{a3}) \ \ \ (Q_{y1} K_{b1} + Q_{y2} K_{b2} + Q_{y3} K_{b3}) \ \ \ (Q_{y1} K_{c1} + Q_{y2} K_{c2} + Q_{y3} K_{c3}) \end{bmatrix} \]

The first row is still all about how token x queries about all of the other keys.

\[ S^{c} = \begin{bmatrix} s_{x,a} & s_{x,b} & s_{x,c} \\ s_{y,a} & s_{y,b} & s_{y,c} \end{bmatrix} \qquad \begin{matrix} \leftarrow \text{decoder token x asking about encoder tokens a, b, c}\\ \leftarrow \text{decoder token y asking about encoder tokens a, b, c} \end{matrix} \]

When we add a new token to what the decoder is processing, we add a new row to this: token \(z\) asking about encoder tokens \(a, b, c.\) Because the encoder tokens are never changing, we also here never change rows about tokens that come before. In particular, because we just proved that the output from masked self attention always has existing rows frozen, then that \(Q\), the only thing in cross attention that cares about what’s coming in from self attention, also has existing rows frozen. We just proved in the 2-token self attention walkthrough that \(Q\), \(K\), and \(V\) themselves only ever mix within row. So cross-attention’s \(Q\), the only thing constructed by the output of self-attention \(O\), just re-mixes rows again. But those rows are frozen. The encoder output is frozen, so there’s nowhere in this cross-attention for mixing either.

This is true even as we press forward. We know the softmax and scaling only operate on the rows, so A must also just be the same kind of mix:

\[ A^{c} = \begin{bmatrix} a_{x,a} & a_{x,b} & a_{x,c} \\ a_{y,a} & a_{y,b} & a_{y,c} \end{bmatrix} \qquad \begin{matrix} \leftarrow \text{decoder token x asking about encoder tokens a, b, c}\\ \leftarrow \text{decoder token y asking about encoder tokens a, b, c} \end{matrix} \]

And finally, just to really hammer it home:

\[ O = A^{c} V = \begin{bmatrix} a_{x,a} & a_{x,b} & a_{x,c} \\ a_{y,a} & a_{y,b} & a_{y,c} \end{bmatrix} \times \begin{bmatrix}V_{a1} \ V_{a2} \ V_{a3} \\ V_{b1} \ V_{b2} \ V_{b3} \\ V_{c1} \ V_{c2} \ V_{c3} \end{bmatrix} \]

I can hopefully prove to you without writing this all the complete result, that the first row of this output is the only place \(x\) will show up, and the second row is the only place \(y\) will.

The feed-forward network, layer norm, and residual adds.

These are easy to write off, without doing any math at all, because all of these operations are explicitly position-wise - each one is processing over a single row as a fundamental assumption. It’s not that they happen to never mix - nothing about those operations would ever allow any mixing at all.

  • The FFN applies the same two-layer network to each row independently: row \(x\)’s output depends only on row \(x\)’s input.
  • LayerNorm normalizes each row using only that row’s own mean and variance.
  • The residual add is element by element, so every position is added to its own position - every row to its own row.

Since none of them can reach over to another row, none of them can pull \(z\) information into row \(x\) or row \(y\).

So, the whole block is safe. All three sublayers give the same result for the same position, every time. Masked self-attention mixes each decoder row only with earlier decoder rows, cross-attention mixes each decoder row only with the fixed encoder, and the FFN, norms, and residuals handle each row entirely on its own. There is no way by which the newly appended token’s information enters an earlier row. The earlier rows pass through the entire decoder block exactly as they were.

Using Just the Last Row

Just to finally relate back to the original point, that’s exactly why we just take the final row of the decoder output for next token prediction. When we say “the previous rows have already been used”, we really mean exactly that. The previous rows were used already, and since they literally do not change at all with every pass, they have nothing new to offer us. The information about those previous tokens do come into play for the current one, and help it come together as a rich mix of everything in the past, but the actual information we read off of a previous row is a fixed thing.

And it’s really all down to that causal mask. Because we don’t allow any looking forward, of course adding a token ahead of an existing one doesn’t tell us anything. That was the whole point of masking out the future, so that previous tokens can’t cheat and look ahead. The way that we do that is exactly what we’ve seen play out here: by making sure that none of the new token’s information ever changes what we would predict without it.

A Total Waste of Compute?

Now, the glaring thing to think here is that the whole decoder stack seems like a huge waste of compute. Imagine when the output sequence is hundreds, thousands, tokens long. We’re holding multiple thousand-dimension matrices in memory just to read off the very last line as the thing we actually use. Seems like a total waste!

And it is. Of course, modern techniques are very clever, and have found a way to exploit this to drastically speed up inference times with something called the KV Cache.

The basic idea is this: When we have a new token come into the sequence, the only way an earlier position participates is as an attention target. The new token’s query just needs to know how to attend to the earlier positions’ keys and values. But those keys and values depend only on the earlier positions’ representations, which we just meticulously proved don’t change.

So, instead of doing it all over again, just compute each position’s keys and values once, and cache them. Then each generation step becomes:

  1. Run forward only the new token through the stack.
  2. Append its key and value to the cache, and let its query attend over the whole cache.
  3. Project its final vector to get the next token.

Instead of re-processing a growing sequence every step, you process exactly one new token per step. Every modern autoregressive LLM does this, and it’s a huge part of what allows generation to be blazing fast.

Of course, it isn’t free. The cache now has to hold the keys and values for every past position, at every layer. So how to manage the cache, share it where possible, access it quickly, are all active concerns for those working on large, modern models.

If you’re interested in much more detail about KV Caching, I have a whole tutorial about it! You can find it here: KV Caching Tutorial

Take me back to the main tutorial