Why LLMs run on GPUs: reading a prompt vs writing a reply
Why a model reads your prompt fast and writes its reply slowly, measured on a real model: writing a token means reading every weight, and memory speed sets the pace.
Part 6 of 6 · How an LLM runs
- How an LLM works: from your text to the next token
- How attention works in an LLM, with real numbers
- What the KV cache is, and why LLMs need it
- Prompt caching explained: why an LLM re-reads your whole chat
- What an inference engine does: batching, KV memory and quantization
- Why LLMs run on GPUs: reading a prompt vs writing a reply
The last part of the series puts the earlier ones together. A model reads your prompt in one pass but writes its reply one token at a time (part 1), and those two jobs run very differently on hardware. Writing is slow because of memory, not math. I measured it on a CPU, then worked out what it means on a GPU from NVIDIA's published numbers.
The work is multiplications, lots of them at once
Nearly everything a model does is multiplying hidden states by weight matrices. In SmolLM2, making one token's query is a 576 × 576 matrix: 331,776 multiplications and additions, none of which depends on another. Apart from the input table, which is only looked up, a token uses every weight once, so a model does roughly two calculations (a multiply and an add) per weight per token. For Llama 3 8B that's about 15 billion per token.
That's the kind of work a GPU is built for: a very large number of the same simple calculation on different numbers, all at once. NVIDIA lists its H100 (SXM) at 1,979 teraFLOPS for 16-bit math with sparsity; even half of that, without sparsity, is close to a thousand trillion calculations a second.
Reading a prompt vs writing a reply
There are two phases:
- Prefill reads the prompt. Every prompt token is known up front, so the model runs all of them in one pass.
- Decode writes the reply. Each token needs the one before it, so it's one pass per token.
I timed both on SmolLM2-135M, on one CPU core, with the same 256 tokens:
Writing was 13× slower per token than reading the prompt, on the same machine and the same model.
Decode waits on memory
The third bar explains it. Every decode step uses every weight (SmolLM2 uses the same table for input and output, so that's all of them), so every step reads the whole weight file from memory: 540 MB here. I measured this machine's memory reading speed at 10.6 GB/s, so reading 540 MB takes 51.2 ms. A decode step took 56.2 ms. Almost all of each step was spent reading the weights; the math fit in the rest.
Prefill reads the weights once and uses them for all 256 prompt tokens, so the reading cost is shared out. Batching (part 5) is the same trick for decode: 16 requests per step shared one read of the weights, which is why 16 at once took only 1.8× as long as one.
The same on a GPU
NVIDIA lists the H100 (SXM) with 80 GB of memory read at 3.35 TB/s. Llama 3 8B at 16 bits is 16.1 GB (part 1), of which a decode step reads about 15.0 GB: everything but the input table. So:
- Reading the weights once takes at least 15.0 GB ÷ 3.35 TB/s = 4.5 ms. For one request that caps decode at about 223 tokens per second, however fast the math is.
- The math for one token, about 15 billion calculations at half the listed 1,979 teraFLOPS, takes about 0.015 ms.
Reading takes about 300 times longer than the math. With one request, the GPU's arithmetic mostly waits on memory. That's why batching pays: up to around 300 requests per step, each extra request adds math the GPU was idle for anyway. Move the slider:
Two limits from parts 3 and 5 still apply. Every request in the batch needs its KV cache in the same 80 GB as the weights, and a bigger model needs more room: Llama 3 70B at 16 bits is 141 GB, more than one H100 holds.
Training fills the GPU
Training doesn't have the decode problem. The model knows the whole text it's learning from, so it predicts every position at once (part 2), and the Transformer paper chose attention partly so that all positions could be computed in parallel. Like prefill, training keeps the arithmetic busy.
The series in one paragraph
A model is a fixed file of numbers (part 1). Tokens look back at each other through attention (part 2). Their keys and values are saved, so writing doesn't redo work (part 3). Nothing is kept between requests unless a provider caches the start of your prompt (part 4). Engines batch many people's requests and juggle their caches (part 5). And every token written means reading the whole model from memory, which is why it all runs on GPUs and why batching matters (this part).
How I measured
- Model: SmolLM2-135M, base model, Apache 2.0, the 32-bit ONNX export
onnx-community/SmolLM2-135M-ONNX(revision d0ae683), ONNX Runtime on one CPU thread of a small cloud server. - Prefill vs decode: the same 256 tokens, once in a single pass and once one token at a time with the KV cache; median of 3 runs.
- Memory speed: summing a 256 MB array, which reads every byte once; best of 5.
- GPU numbers: NVIDIA's H100 product page (SXM column), read on 11 October 2026. The 4.5 ms and 0.015 ms are my arithmetic from those numbers and the Llama 3 8B size; they are a best case, not a benchmark.