Series · 6 parts
How an LLM runs
What happens inside a language model when it answers you, one step at a time, measured on a real model.
I'm learning how LLMs work underneath, by reading and by building a tiny one. This series is what I learned, in the order it made sense to me. Each part follows one idea from start to finish, and where I could measure something on a real model, I did.
- Part 1 · How an LLM works: from your text to the next tokenOne prompt through a language model, step by step: tokens, the hidden state, layers, and how the next token is picked. Measured on a real model.
- Part 2 · How attention works in an LLM, with real numbersQuery, key, value and softmax, measured on a real model: how a token looks back at earlier ones, and how one head lets a model copy what it saw before.
- Part 3 · What the KV cache is, and why LLMs need itWhy a model saves every token's keys and values while it writes, measured on a real model: same output, 15× less time, and a cache that can outgrow the model.
- Part 4 · Prompt caching explained: why an LLM re-reads your whole chatA model keeps nothing between requests, so every chat message re-sends everything before it. How prompt caching skips re-reading it, measured on a real model.
- Part 5 · What an inference engine does: batching, KV memory and quantizationThe program that runs an LLM for many people at once: loading the weights, batching requests, managing KV-cache memory. Measured on a real model.
- Part 6 · Why LLMs run on GPUs: reading a prompt vs writing a replyWhy a model reads your prompt fast and writes its reply slowly, measured on a real model: writing a token means reading every weight, and memory speed sets the pace.