What an inference engine does: batching, KV memory and quantization

The program that runs an LLM for many people at once: loading the weights, batching requests, managing KV-cache memory. Measured on a real model.

11 Oct 2026

In part 1, an LLM was two things: the weights, and the program that runs them, the inference engine. All of a model's "knowledge" is in the weights. The engine's job is to run them fast and cheaply, often for many people at once. This part is about what that takes.

Three engines you'll meet: llama.cpp, whose stated goal is running models "with minimal setup and state-of-the-art performance on a wide range of hardware", and vLLM and SGLang, both built for serving many requests at once. My measurements come from SmolLM2-135M, a small open model, run with ONNX Runtime on one CPU core.

Job 1: load the weights, maybe smaller

The weights have to sit in memory for the whole time the model runs: 16.1 GB for Llama 3 8B at 16 bits per number (part 1). One way to need less is quantization: storing each number in fewer bits. llama.cpp supports 1.5-bit up to 8-bit integer formats.

Fewer bits means less memory and less to read. It also means slightly different numbers, so a quantized model is a slightly different model. I compared SmolLM2's 32-bit file with its 8-bit version, from the same release:

A quarter of the size and a bit faster, but the 8-bit version picked a different top word for the fridge prompt. Writing 40 tokens with the top pick each time, the two versions agreed for the first 3 tokens and then went separate ways. That's one small model and one way of quantizing; how much changes depends on both. The point stands either way: check a quantized model on your own task.

Job 2: batch many requests together

To write one token for one person, the model reads all of its weights (part 6 is about why that's slow). An engine that runs several people's next tokens in the same pass reads the weights once for all of them. That's batching.

I ran SmolLM2 writing replies for 1 to 16 requests at once, one token per request per step:

Sixteen requests at once wrote 8.8× as many tokens per second as one, while each step took only 1.8× as long. Every person waits a little longer per token; the machine does far more work.

Job 3: keep the batch full

Requests don't arrive together and don't finish together. If an engine waited for every request in a batch to finish before starting new ones, short replies would sit idle until the longest one ended. The Orca paper (OSDI 2022) proposed scheduling at the level of each step instead: the batch is rebuilt every iteration, so a finished request leaves at once and a new one can join. vLLM lists this as continuous batching of incoming requests.

Job 4: find room for every KV cache

Every request in the batch needs its own KV cache (part 3), and each one grows by a token per step. The vLLM paper names this as the limit: the KV cache for each request "is huge and grows and shrinks dynamically", and memory wasted on it limits the batch size. Its fix, PagedAttention, stores the cache in small fixed-size blocks, the way an operating system pages memory, so little is wasted and requests can share blocks. The paper reports 2 to 4× the throughput of the systems it compared against, at the same latency.

SGLang's paper describes RadixAttention, which keeps KV caches around for reuse across calls, so requests that start with the same tokens skip re-reading them: the idea behind prompt caching (part 4), inside the engine.

Job 5: pick and return the tokens

Finally, the engine turns each request's scores into a token (temperature and the weighted pick from part 1), sends it back as it goes, and stops at the end token or the length limit.

Next, part 6: why all of this runs on GPUs, and why writing one token at a time is slow even on the fastest ones.

How I measured

• Model: SmolLM2-135M, base model, Apache 2.0, from onnx-community/SmolLM2-135M-ONNX (revision d0ae683): model.onnx (32-bit) and model_int8.onnx (8-bit). ONNX Runtime on one CPU thread of a small cloud server; one run each.
• Quantization: the same prompts in both files, scores turned into probabilities over the whole vocabulary; 40 tokens written from "The harbour town woke slowly. Fishing boats came in with the tide, and", taking the top token each time.
• Batching: 8 different short prompts of equal length (repeated to make 16), then 24 steps writing one token per request, with each request's KV cache kept. Time per step averaged over the 24 steps.
• Engines: what each one does is from its own paper or README; I didn't benchmark them.

← All memories