Prompt caching explained: why an LLM re-reads your whole chat
A model keeps nothing between requests, so every chat message re-sends everything before it. How prompt caching skips re-reading it, measured on a real model.
Part 4 of 6 · How an LLM runs
- How an LLM works: from your text to the next token
- How attention works in an LLM, with real numbers
- What the KV cache is, and why LLMs need it
- Prompt caching explained: why an LLM re-reads your whole chat
- What an inference engine does: batching, KV memory and quantization
- Why LLMs run on GPUs: reading a prompt vs writing a reply
Part 3 ended with a catch: the KV cache is built for one request. A model keeps nothing between requests. This part is about what that means for a chat, and about prompt caching, the fix providers offer for it.
Measurements come from SmolLM2-135M, a small open model, on one CPU core. Provider details come from Anthropic's and OpenAI's documentation as of October 2026.
The model forgets, so the app re-sends
The weights never change while a model runs (part 1), and the KV cache only lives for one request (part 3). So nothing about your last message is inside the model when your next one arrives.
A chat still works because the app sends the whole conversation every time. Anthropic's API docs say it directly: "The Messages API is stateless, which means that you always send the full conversational history to the API."
So a chat gets more expensive with every turn. If each turn adds 500 tokens (your message plus the reply), turn 20 sends all 10,000 tokens of the chat so far. Over 20 turns that's 105,000 input tokens for 10,000 tokens of actual conversation, and the model reads every one of them again.
A model also has a limit on how much it can read at once (Llama 3's is 128K tokens). A chat that outgrows it can't be sent whole, and the app has to decide what to leave out.
Prompt caching: keep the KV cache of the start
Most of what a request sends was in the previous request too: the instructions, any documents, the earlier turns. Reading them again produces exactly the same keys and values every time.
Prompt caching keeps them. The server saves the KV cache for the start of a prompt, and a later request that begins with exactly the same tokens skips straight to the new part. Both providers say this is what's stored: Anthropic's docs mention "KV (key-value) cache representations", OpenAI's "key/value tensors".
I measured the same thing on SmolLM2. A 256-token set of instructions for a library help desk, then three different questions:
| Read everything | Reuse the saved start | |
|---|---|---|
| Question 1 (12 tokens) | 1.17 s | 0.10 s |
| Question 2 (15 tokens) | 1.16 s | 0.13 s |
| Question 3 (11 tokens) | 1.14 s | 0.09 s |
The next-token scores were identical either way. The saved start cost 1.51 s to make once.
Why it has to match exactly
Anthropic's docs: "Cache hits require 100% identical prompt segments." The reason is attention. I changed one early word, "helpful" to "friendly", in "You are a helpful assistant for a city library. Answer briefly." and compared the keys of the two versions token by token:
| Tokens before the change | The changed token | Every token after it | |
|---|---|---|---|
| First layer | same | different | same |
| Last layer | same | different | all different |
At the first layer, a token's key only depends on the token itself and its position. By the last layer, every later token has taken in the changed word through attention, so its keys are different too. A cache is only good up to the first difference.
That gives the one rule of prompt caching: put what stays the same first. Instructions, documents and tool definitions go at the start; the new question goes at the end. A timestamp at the very top changes on every request and breaks the cache for everything after it.
What it costs
As of October 2026, both providers price it in the same shape:
- Anthropic: writing to the cache costs 1.25× the normal input price (for the standard 5-minute lifetime), and reading from it 0.1× on most models. A prompt start must be long enough: 512 to 4,096 tokens, depending on the model.
- OpenAI: on by default for supported models. For GPT-5.6 and later, writes cost 1.25× and reads 0.1× on most models, with a minimum of 1,024 tokens. In-memory entries typically last 5 to 10 minutes of inactivity (up to an hour); extended retention keeps them up to 24 hours.
Caching changes what you pay to read the prompt. It doesn't change the reply: Anthropic's docs note it "has no effect on output token generation". Caches aren't shared between organizations at either provider.
Here's the chat from above with those multipliers, assuming every turn arrives while the cache is still alive:
At 20 turns, 105,000 tokens sent are billed like 22,000. The longer the chat, the bigger the share that is cached reads.
Next, part 5: inference engines, the programs that run a model for many people at once, and what they do with all these caches.
How I measured
- Model: SmolLM2-135M, base model, Apache 2.0, the 32-bit ONNX export
onnx-community/SmolLM2-135M-ONNX(revision d0ae683), ONNX Runtime on one CPU thread. Timings are one run each. - Reuse: the 256-token instructions run once, with their keys and values kept; each question then runs on top of them. "Read everything" runs instructions and question together from scratch.
- Exact match: the keys of all 13 tokens of both sentences, compared at the first and the last of the 30 layers.
- Prices and limits: Anthropic's prompt caching and Messages docs, OpenAI's prompt caching guide, read on 11 October 2026. They change; check them before you rely on a number.