The world is a chaotic place. Words are the human way of describing experience. The universe — and by extension, computer language — is numbers. The fundamental tension that created modern AI is this: how do you convert words into numbers in a way that preserves meaning? You cannot teach meaning numerically, but you can teach words in the context of other words. This article traces exactly how that works, from the first character of your input to the final probability distribution over the vocabulary.
1. Tokenisation and Embedding
Before a language model can process text, it must convert it into numbers. This happens in three stages: tokenisation, embedding, and positional encoding.
Each ID is just a row number in the vocabulary table — no meaning yet, only an index.
Stage 1: Tokenisation
Each word (or word fragment) is broken into repeatable subword chunks called tokens. The word "enterprise" might become ["enter", "prise"]; "architecture" becomes ["archi", "tecture"]. Each token is then mapped to an integer ID — its row number in the model's vocabulary table. These integer IDs are not yet meaningful numbers; they are simply indices.
Stage 2: Embedding
Each integer ID is swapped for a vector — a row of floating-point numbers — by looking up its position in a learned matrix called E (the embedding matrix). If the vocabulary has 50,000 tokens and each vector has 768 dimensions, E is a 50,000 × 768 matrix. Every cell in that matrix was initialised randomly and then nudged millions of times during training, until words that appear in similar contexts ended up pointing in similar directions in this 768-dimensional space.
Token ID 4821 selects row 4821 of E (50,000 × 768) — a dense vector of learned floating-point values.
The result is a geometry of meaning. Because the training process consistently adjusts words appearing in similar contexts toward each other, the space encodes relationships:
The same offset — direction and distance — separates "man" from "king" as separates "woman" from "queen". That offset is the geometry of gender in embedding space.
This is not magic — it is consistency. The difference between "king" and "queen" is encoded in the same direction as the difference between "man" and "woman" because those relationships appear consistently across the training corpus. The network did not learn the concept of gender explicitly; it learned that certain words shift in the same direction relative to each other.
Stage 3: Positional Encoding
A transformer processes all tokens simultaneously, not one at a time. Without extra information, the sentences "the cat sat on the mat" and "the mat sat on the cat" would look identical — same tokens, same vectors. So a positional vector is added to each embedding, encoding where in the sequence that token sits. The result is a matrix where every row is a single token and every column is a dimension of meaning — the sentence has become geometry.
2. Prediction, Loss, and Backpropagation
Training a language model is a prediction task. Given a sequence of tokens, the model must predict what comes next. This is done billions of times, and each incorrect prediction nudges every weight slightly toward correctness.
The Forward Pass
The model takes the input token matrix and passes it through a stack of transformer layers — each consisting of an attention block and a feedforward network. The final layer produces a probability distribution over every word in the vocabulary: one number per token, all positive, summing to one.
The Loss Function
During training, the correct next token is already known. The model produces its best guess — a probability distribution ŷ. The loss function measures how wrong it was. For language models, this is cross-entropy loss:
If the model assigns probability 0.34 to the correct token "mat", the loss is −log(0.34) ≈ 1.08. A perfect prediction (probability 1.0) yields zero loss. Near-zero probability yields infinite loss. The log function amplifies the penalty for overconfident wrong answers.
As predicted probability for the correct token falls toward zero, loss rises toward infinity — heavily penalising confident wrong answers.
Backpropagation
The loss value flows backwards through every layer of the network — a process called backpropagation. At each layer, the chain rule asks: how much did this set of weights contribute to the total error? The answer is the gradient — the slope of the loss with respect to each weight.
The chain rule carries the error signal backward one layer at a time, computing how much each weight contributed.
Each weight is then adjusted by a small step in the direction that reduces the loss:
Where η (eta) is the learning rate — how large each step is. Too large and the weights oscillate past the minimum; too small and training takes forever. Repeat this process across billions of (input, correct output) pairs and the model gradually improves.
3. Attention
Everything so far treats each token independently. The token "bank" gets a vector; "river" gets a vector. They are processed in parallel but do not communicate. This is a fundamental problem: meaning is not independent. "Bank" means something completely different depending on whether "river" or "money" appeared earlier in the sentence. The model needs a mechanism that allows tokens to look at each other and decide which others matter for understanding the current one.
Query, Key, Value
The attention mechanism solves this with a learned search engine. Every token is projected into three separate vectors:
- Q (Query) — what this token is looking for. "What context am I in?"
- K (Key) — what each token contains. "What do I contribute to others?"
- V (Value) — what each token passes forward when selected. "What information do I carry?"
"What context am I in?" — what this token is looking for.
"What do I contribute?" — what each token offers to others.
"What do I carry?" — the information passed forward when selected.
The query for "bank" is compared against the key of every other token via a dot product — one number measuring how aligned those two vectors are. High dot product means this token is relevant; low dot product means ignore it. These raw scores are divided by the square root of the key dimension to prevent very large values, then passed through softmax to produce attention weights:
Softmax
Softmax converts arbitrary scores — which can be any real number, including negatives — into a valid probability distribution. It applies the exponential function to each score (amplifying differences), then divides each by the total:
Every score becomes positive and the row now sums to exactly 1.0 — a valid probability distribution.
These weights are then used to compute a weighted sum of the value vectors. The output for "bank" is a blend of every other token's value, weighted by their relevance score. If the surrounding tokens are water-related, "bank" emerges carrying river semantics. If they are financial, it carries money semantics. The token has been contextualised.
4. Multi-Head Attention
A single round of attention can only look for one kind of relationship at a time. To capture multiple relationship types simultaneously — grammatical structure, semantic similarity, co-reference — the transformer runs attention in parallel across multiple heads, each with its own learned Q, K, V projection matrices.
concatenate outputs → linear projection WO → one contextualised vector per token
Each head produces its own output matrix. These are concatenated horizontally — placed side by side — creating a wider matrix. A final linear projection WO squishes this back to the original dimension. Every token has now been updated to reflect what it learned from every other token, from multiple perspectives simultaneously, at the cost of a single parallel computation.
5. The Feedforward Network
After all tokens have communicated through multi-head attention, each token processes what it learned independently. A small two-layer neural network — called the feedforward network — is applied identically to every token position:
The middle layer is typically four times wider than the embedding dimension (3,072 in a 768-dimension model), giving the network space to represent complex patterns before projecting back. The ReLU activation — which zeroes out any negative values — introduces the non-linearity that lets neural networks approximate any continuous function, not just linear relationships.
Putting It Together
A transformer is a stack of these blocks — typically 12 to 96 layers deep for modern models. Each layer runs: multi-head attention (tokens communicate), then feedforward network (each token reflects). Residual connections around each block ensure gradients can flow unimpeded during training. Layer normalisation before each block stabilises the training dynamics.
This architecture — tokenise, embed, attend, feed-forward, repeat — is the foundation of every major language model in production today: GPT-4, Claude, Gemini, Llama, Mistral. The differences between them lie in scale, training data, fine-tuning techniques, and the specific engineering choices made at each layer — not in any fundamentally different architecture.