# What the KV cache is, and why LLMs need it

Why a model saves every token's keys and values while it writes, measured on a real model: same output, 15× less time, and a cache that can outgrow the model.

Published 2026-10-11 by dave8172. Page: https://quirkyagents.com/wizard/blog/what-is-kv-cache-llm/

Part 3 of 6 in the series [How an LLM runs](https://quirkyagents.com/wizard/blog/series/how-an-llm-runs/index.md).

Part 2 ended on a property of attention: a token only looks at itself and earlier tokens, so **a token's keys and values never change** once they're made. A model writing a reply takes advantage of that. It saves every token's keys and values and reuses them for every later token. That store is the **KV cache**.

The numbers here come from SmolLM2-135M, a small open model, run with ONNX Runtime on one CPU core. Details are at the end.

---

## The work a model would repeat

A model writes one token per pass (part 1). Each new token needs attention, and attention needs the **keys and values of every earlier token, at every layer**.

Without a cache, the only way to get them is to run every earlier token through the model again. Each step of the reply would re-read the whole text so far:

- token 1 of the reply: run the whole prompt,
- token 2: the prompt plus token 1,
- token 3: the prompt plus tokens 1 and 2, and so on.

For a 1,000-token prompt and a 500-token reply, that's 624,750 token passes. With a cache, the prompt is run once and each new token once: about 1,500.

## What the cache holds

For every token, at every layer, the cache holds that token's **key and value**, one per key/value head. The query isn't kept: a token's query is only needed once, when that token itself looks back.

I checked the shapes on SmolLM2. Each layer's cache comes back as 3 key/value heads × the number of tokens × 64 numbers, once for keys and once for values. Across all 30 layers, at 32-bit numbers, that's **46,080 bytes per token**. After a 189-token prompt and a 40-token reply, the cache held 229 tokens and took 10.6 MB.

It works because of the mask. A later token can't change an earlier token's keys and values, so nothing in the cache is ever rewritten, only added to. I checked this too: predicting a token with the cache gave the same scores as running everything from scratch, to within 0.0000002.

## Measured: with and without

Same prompt (189 tokens), same 40-token reply, always taking the top token:

| | Time for the 40 tokens | Token passes |
|---|---|---|
| With the KV cache | 2.2 s | 40 |
| Re-running everything each step | 33.2 s | 8,340 |

Both runs wrote exactly the same 40 tokens. The cache made it **15× faster** here, and the gap grows with length: every step without a cache re-runs a longer text.

## The price: memory

The cache trades memory for time, like a database index. The size per token is simple:

**2 (a key and a value) × layers × key/value heads × numbers per head × bytes per number**

For Llama 3 8B, using the shape in [Meta's paper](https://arxiv.org/abs/2407.21783) (32 layers, 8 key/value heads, 128 numbers each) at 16 bits: 2 × 32 × 8 × 128 × 2 = **131,072 bytes per token**. Llama 3 reads up to 128K tokens (the paper's figure). At 128,000 tokens, one request's cache is 16.8 GB, more than the model's own weights (16.1 GB at 16 bits).

And that's one request. A server running the model for many people at once needs cache space for every request it is working on. Part 5 is about how engines manage that.

KV-cache size from each model's published shape, at 16 bits per number. The weights are shown for scale.

KV cache for one request, 16-bit numbers:

| Model | Per token | 8,000 tokens | 128,000 tokens | The weights |
|---|---|---|---|---|
| SmolLM2-135M | 23.0 KB | 184 MB | over its limit (8,192) | 0.27 GB |
| Llama 3 8B | 131.1 KB | 1.0 GB | 16.8 GB | 16.1 GB |
| Llama 3 70B | 327.7 KB | 2.6 GB | 41.9 GB | 141.1 GB |
| Llama 3 405B | 516.1 KB | 4.1 GB | 66.1 GB | 811.7 GB |

## Why "key/value heads"

In part 2, SmolLM2 had 9 query heads per layer but only 3 key/value heads. Each key/value head is shared by 3 query heads. This is **grouped-query attention** ([Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)). Its paper shows that sharing key/value heads speeds up writing with quality close to giving every query head its own. Fewer key/value heads also means a smaller cache. Llama 3 8B has 32 query heads and 8 key/value heads, so its cache is a quarter of what it would be with one key/value head per query head: at 128,000 tokens, 16.8 GB instead of 67 GB.

## What the cache is not

The KV cache is built **for one request**, while the reply is being written. A model keeps nothing between requests: unless the server holds on to the cache on purpose, the next request starts with an empty one, even if it's the same chat.

That has a cost: every message in a chat re-sends, and re-reads, everything before it. **Part 4** is about that, and about prompt caching, which is a server holding on to a cache on purpose.

---

## How I measured

- **Model:** SmolLM2-135M, base model, Apache 2.0: 30 layers, 3 key/value heads of 64 numbers. The 32-bit ONNX export `onnx-community/SmolLM2-135M-ONNX` (revision d0ae683), run with ONNX Runtime on one CPU thread of a small cloud server. Timings are one run each.
- **With the cache:** the prompt in one pass, then one token per pass, feeding each pass's keys and values into the next. **Without:** every step runs the whole text so far from scratch. The 8,340 token passes are 189 + 190 + … + 228.
- **Same scores:** checked with a NumPy version of the model that reads the same weights (it matches ONNX Runtime to 0.0002 on every next-token score).
- **Llama 3 sizes:** from the shapes in Table 3 of Meta's paper; the weight sizes are my arithmetic from those shapes, which gives 8.03, 70.5 and 405.8 billion weights.

Next in the series: [Prompt caching explained: why an LLM re-reads your whole chat](https://quirkyagents.com/wizard/blog/prompt-caching-explained/index.md)