Small model vs LLM: which model fits a small text job?

One small job, making up town names one letter at a time, tested on a 4 MB model, open language models, Claude Haiku 5.5 and a plain list. Accuracy, speed and cost for each.

Benchmark · 14 Oct 2026 · GitHub

You have a small text job. Do you need a large language model for it, a small model made for the job, or no model at all?

I measured one small job: town names. The job has two parts. Guess the next letter of a place name, and finish a name from its first letters. I gave both parts to five kinds of tool and measured how well each did, how fast, and at what cost.

The tools

• tinytowns, in 3 sizes: a small model made only for this job. It learned from 128,435 place names, and it runs on your own device.
• Qwen2.5-1.5B and SmolLM2-1.7B: open general language models. They ran on a laptop GPU (RTX 3050, 4 GB).
• Claude Haiku 5.5: Anthropic's fastest model, through Anthropic's API.
• A counting table: it counts which letter follows which.
• A list lookup: the list of 142,706 real place names, searched in order. No model at all.

The test uses 300 real place names that tinytowns never saw while it learned. The general models first read 20 example names, so they know the format.

Job 1: guess the next letter

The score is the loss: how surprised a model is by the real next letter, on average over 3,125 letters. Lower is better. Guessing evenly over the 28 symbols (26 letters, space, end of name) scores 3.33.

Claude is not in the Job 1 charts: its API returns text, not a probability for each letter, so it has no loss.

The general models guess best, by a little. They know the letter patterns, and they also know many real places, so real names surprise them less.

Quality against speed

How much better, and at what price in time? Each point below is one tool: lower is a better guess, further left is faster.

Against SmolLM2-1.7B, the best loss, the trade is 10% higher loss (2.19 against 1.99) for 107 times the speed and 850 times less memory. SmolLM2 was slower partly because it did not fit on the GPU.

For this job, the gap in quality is small: 7 to 10% in loss. The gap in speed and memory is large: 58 to 107 times the time, and 770 to 850 times the memory. So for a job this small, a small model made for it is the better trade, unless that last bit of quality matters more to you than speed, memory and cost.

Job 2: finish the name from 3 letters

Each tool gets the first 3 letters, for example bay, and writes the rest. Then we check if the place exists.

The general models know real places. The small one mostly invents. Claude finished 94.7% of the names with a real place, and the real number is a bit higher: it wrote beograd, ulaanbaatar and aarhus, which the list spells belgrade, ulan bator and arhus. tinytowns learned only the letter patterns of names, so 82% of its names are new ones.

For real names, the list lookup beats every model. Search the list for the 3 letters, and the first match is a real place. It takes 1.5 millionths of a second.

Which one should you use?

• To finish a name with a real place, when you have a list: look it up. It is the fastest, it is free, and every answer is a real place.
• To finish names when you have no list: a large model like Claude. 94.7% real places in this test, at $0.10 per 1,000 names.
• To make up names or guess letters, fast, on a phone or offline: a small model made for the job. tinytowns (1M) is 4 MB, needs a few milliseconds per name and no GPU, for 7 to 10% higher loss.
• When the last bit of quality matters more than speed, and you have a GPU: an open model like SmolLM2-1.7B or Qwen2.5-1.5B. They had the best losses here, at 3.1 to 3.4 GB and 138 to 255 ms per name.

A small model made for one job came 7 to 10% behind the best loss, with less than a thousandth of the memory and a fiftieth of the time or less. The general models know things the small one never learned: real places, for one.

← All work