How an LLM works: from your text to the next token

One prompt through a language model, step by step: tokens, the hidden state, layers, and how the next token is picked. Measured on a real model.

11 Oct 2026

This part follows one prompt through a language model, from the text you type to the token it writes next. Where I measured something, I ran it on SmolLM2-135M, a small open model from Hugging Face that is built the same way as Meta's Llama, only smaller. It runs on an ordinary CPU, so anyone can check the numbers below. How I ran it is at the end.

A model is a file of numbers

An LLM is two things: a file of numbers, called the weights, and a program that runs them, called an inference engine (llama.cpp, vLLM and SGLang are three). It works like Python. The interpreter runs script.py on your arguments. The engine runs the weights on your text. The weights are the program; your text is the input.

Training sets the weights once. Using the model doesn't change them: every request, from every user, goes through the same numbers.

The "8B" in Llama 3 8B is how many numbers there are: about 8 billion. Meta's Llama 3 paper (Table 3) gives the shape: a vocabulary of 128,000 tokens, 32 layers, a hidden state of 4,096 numbers, and a feed-forward size of 14,336. Multiplying those out:

70% of the weights sit in the feed-forward layers, not in attention. More on what they do below.

The file really is just numbers. SmolLM2-135M's config gives 134.5 million weights. Stored as 32-bit numbers, 4 bytes each, that is 538 MB, and the model file I downloaded is 540 MB.

Step 1: your text becomes tokens

A tokenizer cuts your text into pieces from a fixed list, the vocabulary, and swaps each piece for its number in the list. SmolLM2's tokenizer turns "The capital of France is" into five tokens:

The space belongs to the word after it (shown here as ·). SmolLM2's vocabulary has 49,152 tokens; Llama 3's has 128,000.

Step 2: each token gets a starting vector

The input table, the embedding table, has one row per token in the vocabulary. Each token's ID picks its row: a list of numbers, 4,096 of them in Llama 3 8B and 576 in SmolLM2. This list is the token's hidden state. Think of it as the token's working note: everything the model works out about this token, in this sentence, ends up in it.

The table is used once per token, at the start. After that the hidden state comes from the layer below.

Step 3: every layer updates every hidden state

Llama 3 8B has 32 layers, SmolLM2 has 30. Each layer has its own weights and runs two steps on every token:

1. Attention. Each token looks back at itself and every earlier token and takes in information from them. To do it, the token's hidden state is multiplied by three weight matrices, making three new vectors: a query, a key and a value. This is the only step where tokens exchange information. Part 2 is about how it works.
2. Feed-forward. Each token's hidden state goes alone through a bigger network: in Llama 3 8B, out from 4,096 numbers to 14,336 and back. This is where the 70% of the weights are. Research that traced how models recall facts found much of it in these layers (Geva et al., 2021; Meng et al., 2022).

Each step's result is added to the hidden state rather than replacing it, so the state builds up layer by layer. Token 5's hidden state at layer 7 is token 5's state as it left layer 6, plus what layer 7 adds.

Step 4: the last token scores every possible next token

After the top layer, only the last token's hidden state is used to predict. Through attention at every layer it has taken in information from all the tokens before it.

It is multiplied by the output table, which gives one score for every token in the vocabulary: 49,152 scores in SmolLM2. A fixed formula, softmax, turns the scores into probabilities that add up to 100%.

Here are SmolLM2's top five after "The capital of France is":

Paris comes second. The model isn't answering a question; it predicts how the text goes on, and "the" is a fine start, as in "the city of Paris". SmolLM2-135M is a base model, trained only to continue text. Chat models start from a base model and are trained further to answer.

Step 5: temperature decides how the pick is made

The engine now picks one token, and temperature controls how. Before softmax, every score is divided by the temperature:

• Temperature 0 always takes the top token.
• Temperature 1 picks at random, weighted by the probabilities as they are.
• Below 1, the top tokens get more likely. Above 1, the probabilities flatten out.

A weighted random pick does what its probabilities say. I made 1,000 picks at temperature 1: "the" came up 272 times (probability 26.2%), "Paris" 101 times (9.4%), "located" 71 times (7.3%).

Try it with SmolLM2's real numbers. Move the temperature and make a few picks:

Two things stand out. At temperature 1 the fridge prompt puts 75% of its probability on tokens outside its top eight. And at 1.5, even "The capital of France is" gives 85% to tokens outside its top eight.

Step 6: the loop

The picked token is added to the end of the text and goes through the whole model, layer by layer, like any other token. Its final hidden state predicts the token after it. This repeats until the model writes a special end token or hits a length limit. One pass, one token: that's why replies appear word by word.

Twelve tokens from SmolLM2, each continuing "The capital of France is":

At temperature 0 it took the top token every time and went round in a circle. Temperature 0 gives the likeliest next token at each step, not the best text.

Above temperature 0, the random pick alone means the same question can get different answers. Even at temperature 0, hosted models aren't fully repeatable: a Thinking Machines study (Horace He, 2025) traced it to results that change with how many other requests the server is running at the same time.

Where the weights come from

Every weight was learned the same way. Training starts with random numbers. The model reads real text and gives a probability for each possible next token. The loss measures how little probability it gave the token that actually came next. Backpropagation works out, for every weight, which way to move it to make that token more likely, and every weight moves a tiny step. Meta trained Llama 3 on about 15 trillion tokens of text this way (the same paper). Over that much text, the random numbers become an embedding table, attention and feed-forward layers, and an output table that work. Nobody decides what any single weight means.

Next, part 2: attention, the step where tokens look back at each other.

How I measured

• Model: SmolLM2-135M, base model, Apache 2.0. Its config lists the Llama architecture: 30 layers, a hidden state of 576, a vocabulary of 49,152. I used the 32-bit ONNX export onnx-community/SmolLM2-135M-ONNX (revision d0ae683), so the numbers aren't affected by compression.
• Running it: ONNX Runtime on a CPU, one full pass per token, softmax over the whole vocabulary. Each temperature in the explorer is computed from the full set of 49,152 scores, not from the top eight.
• Random picks: NumPy with fixed seeds (42 for the 1,000 picks; 1, 2 and 3 for the three runs), so they repeat exactly.
• Llama 3 8B: layer sizes from Table 3 of Meta's paper; the weight counts are my arithmetic from those sizes.

← All memories