# Why LLMs run on GPUs: reading a prompt vs writing a reply

Why a model reads your prompt fast and writes its reply slowly, measured on a real model: writing a token means reading every weight, and memory speed sets the pace.

Published 2026-10-11 by dave8172. Page: https://quirkyagents.com/wizard/blog/why-llms-run-on-gpus/

Part 6 of 6 in the series [How an LLM runs](https://quirkyagents.com/wizard/blog/series/how-an-llm-runs/index.md).

The last part of the series puts the earlier ones together. A model reads your prompt in one pass but writes its reply one token at a time (part 1), and those two jobs run very differently on hardware. **Writing is slow because of memory, not math.** I measured it on a CPU, then worked out what it means on a GPU from NVIDIA's published numbers.

---

## The work is multiplications, lots of them at once

Nearly everything a model does is multiplying hidden states by weight matrices. In SmolLM2, making one token's query is a 576 × 576 matrix: 331,776 multiplications and additions, none of which depends on another. Apart from the input table, which is only looked up, a token uses every weight once, so a model does roughly two calculations (a multiply and an add) per weight per token. For Llama 3 8B that's about 15 billion per token.

That's the kind of work a GPU is built for: a very large number of the same simple calculation on different numbers, all at once. NVIDIA lists its H100 (SXM) at 1,979 teraFLOPS for 16-bit math with sparsity; even half of that, without sparsity, is close to a thousand trillion calculations a second.

## Reading a prompt vs writing a reply

There are two phases:

- **Prefill** reads the prompt. Every prompt token is known up front, so the model runs all of them in one pass.
- **Decode** writes the reply. Each token needs the one before it, so it's one pass per token.

I timed both on SmolLM2-135M, on one CPU core, with the same 256 tokens:

Measured on SmolLM2-135M, one CPU core: time per token for the same 256 tokens, and the time to read the 540 MB weight file once at this machine's measured memory speed.

Time per token:

|  | milliseconds per token |
|---|---|
| Prefill (prompt) | 4.19 |
| Decode (reply) | 56.2 |
| Reading the weights once | 51.2 |

Writing was **13× slower per token** than reading the prompt, on the same machine and the same model.

## Decode waits on memory

The third bar explains it. Every decode step uses every weight (SmolLM2 uses the same table for input and output, so that's all of them), so every step reads the whole weight file from memory: 540 MB here. I measured this machine's memory reading speed at 10.6 GB/s, so reading 540 MB takes 51.2 ms. A decode step took 56.2 ms. **Almost all of each step was spent reading the weights**; the math fit in the rest.

Prefill reads the weights once and uses them for all 256 prompt tokens, so the reading cost is shared out. Batching (part 5) is the same trick for decode: 16 requests per step shared one read of the weights, which is why 16 at once took only 1.8× as long as one.

## The same on a GPU

NVIDIA lists the H100 (SXM) with 80 GB of memory read at 3.35 TB/s. Llama 3 8B at 16 bits is 16.1 GB (part 1), of which a decode step reads about 15.0 GB: everything but the input table. So:

- **Reading the weights** once takes at least 15.0 GB ÷ 3.35 TB/s = **4.5 ms**. For one request that caps decode at about 223 tokens per second, however fast the math is.
- **The math** for one token, about 15 billion calculations at half the listed 1,979 teraFLOPS, takes about **0.015 ms**.

Reading takes about 300 times longer than the math. With one request, the GPU's arithmetic mostly waits on memory. That's why batching pays: up to around 300 requests per step, each extra request adds math the GPU was idle for anyway. Move the slider:

A best case from NVIDIA's listed H100 SXM numbers (3.35 TB/s; half of the 1,979 teraFLOPS listed with sparsity), not a measurement. Real engines also read KV caches and do other work, so real numbers are lower.

Llama 3 8B, 16-bit on one GPU with 3.35 TB/s and 989.5 teraFLOPS: a best case, ignoring KV-cache reads:

| Requests at once | Reading the weights | The math | Tokens per second (at most) |
|---|---|---|---|
| 1 | 4.48 ms | 0.02 ms | 223 |
| 16 | 4.48 ms | 0.24 ms | 3,573 |
| 64 | 4.48 ms | 0.97 ms | 14,293 |
| 256 | 4.48 ms | 3.88 ms | 57,173 |
| 512 | 4.48 ms | 7.76 ms | 65,967 |

Two limits from parts 3 and 5 still apply. Every request in the batch needs its KV cache in the same 80 GB as the weights, and a bigger model needs more room: Llama 3 70B at 16 bits is 141 GB, more than one H100 holds.

## Training fills the GPU

Training doesn't have the decode problem. The model knows the whole text it's learning from, so it predicts every position at once (part 2), and the Transformer paper chose attention partly so that all positions could be computed in parallel. Like prefill, training keeps the arithmetic busy.

## The series in one paragraph

A model is a fixed file of numbers (part 1). Tokens look back at each other through attention (part 2). Their keys and values are saved, so writing doesn't redo work (part 3). Nothing is kept between requests unless a provider caches the start of your prompt (part 4). Engines batch many people's requests and juggle their caches (part 5). And every token written means reading the whole model from memory, which is why it all runs on GPUs and why batching matters (this part).

---

## How I measured

- **Model:** SmolLM2-135M, base model, Apache 2.0, the 32-bit ONNX export `onnx-community/SmolLM2-135M-ONNX` (revision d0ae683), ONNX Runtime on one CPU thread of a small cloud server.
- **Prefill vs decode:** the same 256 tokens, once in a single pass and once one token at a time with the KV cache; median of 3 runs.
- **Memory speed:** summing a 256 MB array, which reads every byte once; best of 5.
- **GPU numbers:** NVIDIA's H100 product page (SXM column), read on 11 October 2026. The 4.5 ms and 0.015 ms are my arithmetic from those numbers and the Llama 3 8B size; they are a best case, not a benchmark.