# Small model vs LLM: which model fits a small text job? One small job, making up town names one letter at a time, tested on a 4 MB model, open language models, Claude Haiku 5.5 and a plain list. Accuracy, speed and cost for each. Benchmark · 14 Oct 2026, by dave8172. Page: https://quirkyagents.com/wizard/projects/small-model-vs-llm-benchmark/ You have a small text job. Do you need a large language model for it, a small model made for the job, or no model at all? I measured one small job: town names. The job has two parts. **Guess the next letter** of a place name, and **finish a name** from its first letters. I gave both parts to five kinds of tool and measured how well each did, how fast, and at what cost. ## The tools - **[tinytowns](https://quirkyagents.com/wizard/projects/tinytowns/)**, in 3 sizes: a small model made only for this job. It learned from 128,435 place names, and it runs on your own device. - **Qwen2.5-1.5B** and **SmolLM2-1.7B**: open general language models. They ran on a laptop GPU (RTX 3050, 4 GB). - **Claude Haiku 5.5**: Anthropic's fastest model, through Anthropic's API. - **A counting table**: it counts which letter follows which. - **A list lookup**: the list of 142,706 real place names, searched in order. No model at all. The general models need about 800 times the memory of tinytowns (1M), and they ran on a GPU. Each gridline is 10 times the one before. Memory to hold the model (log scale): | Tool | Memory to hold the model (log scale) | Details | |---|---|---| | Counting table | 3 KB | 784 numbers, one for each pair of symbols (28 × 28) | | tinytowns-nano | 20 KB | 4,572 numbers. Runs on one CPU core, or in a web browser | | tinytowns-small | 424 KB | 105,052 numbers. Runs on one CPU core, or in a web browser | | tinytowns (1M) | 4 MB | 1,001,500 numbers. Runs on one CPU core, or in a web browser | | Qwen2.5-1.5B | 3.1 GB | 1.5 billion numbers. Ran on the laptop GPU | | SmolLM2-1.7B | 3.4 GB | 1.7 billion numbers. Did not fit in the GPU's 4 GB, so part of it ran from normal memory | | Claude Haiku 5.5 | not published | Runs on Anthropic's servers; its size is not published | The test uses **300 real place names** that tinytowns never saw while it learned. The general models first read 20 example names, so they know the format. ## Job 1: guess the next letter The score is the **loss**: how surprised a model is by the real next letter, on average over 3,125 letters. Lower is better. Guessing evenly over the 28 symbols (26 letters, space, end of name) scores 3.33. Every tool beats guessing evenly. The general models guess best, and tinytowns (1M) is close behind. Loss (lower is better): | Tool | Loss (lower is better) | Details | |---|---|---| | SmolLM2-1.7B | 1.99 | 3.4 GB | | Qwen2.5-1.5B | 2.05 | 3 GB | | tinytowns (1M) | 2.19 | 4 MB | | tinytowns-small | 2.25 | 424 KB | | tinytowns-nano | 2.49 | 20 KB | | Counting table | 2.69 | 784 numbers | Claude is not in the Job 1 charts: its API returns text, not a probability for each letter, so it has no loss. **The general models guess best, by a little.** They know the letter patterns, and they also know many real places, so real names surprise them less. ### Quality against speed How much better, and at what price in time? Each point below is one tool: lower is a better guess, further left is faster. tinytowns gives up a little quality for a lot of speed. The general models are only a bit lower, and far to the right. Loss (lower is better) against Time per name, on the same laptop GPU (log scale): | Tool | Loss (lower is better) | Time per name, on the same laptop GPU (log scale) | Details | |---|---|---|---| | tinytowns-nano | 2.49 | 0.5 ms | 20 KB | | tinytowns-small | 2.25 | 1.2 ms | 424 KB | | tinytowns (1M) | 2.19 | 2.4 ms | 4 MB | | Qwen2.5-1.5B | 2.05 | 138 ms | 3.1 GB | | SmolLM2-1.7B | 1.99 | 255 ms | 3.4 GB. It did not fit in the GPU's 4 GB, so part of it ran from normal memory, which made it slower | What tinytowns (1M) trades against Qwen2.5-1.5B, the faster of the two general models. The trade: | | | |---|---| | 7% | higher loss: 2.19 against 2.05. A small loss of quality | | 58× | faster on the same GPU: 2.4 ms against 138 ms per name | | 770× | less memory to hold the model: 4 MB against 3.1 GB | | No GPU | needed: one CPU core scores a name in 2.6 ms. It also runs in a web browser | Against SmolLM2-1.7B, the best loss, the trade is 10% higher loss (2.19 against 1.99) for 107 times the speed and 850 times less memory. SmolLM2 was slower partly because it did not fit on the GPU. For this job, the gap in quality is small: 7 to 10% in loss. The gap in speed and memory is large: 58 to 107 times the time, and 770 to 850 times the memory. So for a job this small, a small model made for it is the better trade, unless that last bit of quality matters more to you than speed, memory and cost. ## Job 2: finish the name from 3 letters Each tool gets the first 3 letters, for example `bay`, and writes the rest. Then we check if the place exists. How often each tool finished the name with a real place. The list is always right; Claude is right about 19 times in 20. Real places (higher is better): | Tool | Real places (higher is better) | Details | |---|---|---| | List lookup | 100% | The first match in the list is always a real place | | Claude Haiku 5.5 | 94.7% | The exact name we started from: 3.0% | | SmolLM2-1.7B | 43% | The exact name we started from: 1.7% | | Qwen2.5-1.5B | 37% | The exact name we started from: 1.0% | | tinytowns (1M) | 18% | The exact name we started from: 0.7%. The other 82% are new names | **The general models know real places. The small one mostly invents.** Claude finished 94.7% of the names with a real place, and the real number is a bit higher: it wrote `beograd`, `ulaanbaatar` and `aarhus`, which the list spells `belgrade`, `ulan bator` and `arhus`. tinytowns learned only the letter patterns of names, so 82% of its names are new ones. The list lookup answers about 10,000 times faster than tinytowns, and tinytowns about 100 times faster than the general models. Time per name (log scale): | Tool | Time per name (log scale) | Details | |---|---|---| | List lookup | 0.0015 ms | One core of a cloud server | | tinytowns (1M) | 16.5 ms | One core of a cloud server | | Qwen2.5-1.5B | 1.5 s | The laptop GPU | | Claude Haiku 5.5 | 2.0 s | Anthropic's API, network included | | SmolLM2-1.7B | 2.2 s | The laptop GPU, partly from normal memory | Cost, and why the exact name is rare. Cost and names: | | | |---|---| | $0.10 | per 1,000 names with Claude, the only tool here with a bill | | $0 | for every other tool: they run on your own machine | | 85 | real places share a 3-letter start (the median), so no tool found the exact name often | **For real names, the list lookup beats every model.** Search the list for the 3 letters, and the first match is a real place. It takes 1.5 millionths of a second. ## Which one should you use? - **To finish a name with a real place, when you have a list:** look it up. It is the fastest, it is free, and every answer is a real place. - **To finish names when you have no list:** a large model like Claude. 94.7% real places in this test, at $0.10 per 1,000 names. - **To make up names or guess letters, fast, on a phone or offline:** a small model made for the job. tinytowns (1M) is 4 MB, needs a few milliseconds per name and no GPU, for 7 to 10% higher loss. - **When the last bit of quality matters more than speed, and you have a GPU:** an open model like SmolLM2-1.7B or Qwen2.5-1.5B. They had the best losses here, at 3.1 to 3.4 GB and 138 to 255 ms per name. A small model made for one job came 7 to 10% behind the best loss, with less than a thousandth of the memory and a fiftieth of the time or less. The general models know things the small one never learned: real places, for one.
Technical details: how I measured - **Test names:** 300 names drawn with Python's `random.Random(42)` from tinytowns' 14,271 validation names (GeoNames `cities1000`, lowercased, a–z and space, at most 20 letters). 3,125 predictions. On all 14,271 names, the losses of tinytowns and the counting table are 0.01 to 0.05 lower, in the same order. - **Loss:** −ln(probability of what really came next), averaged over symbols, including one "end of name" per name. - **tinytowns and the counting table** predict one letter at a time. - **Qwen and SmolLM2** read tokens (whole words or parts of words). For them we take the probability of the whole name followed by a new line, after 20 example names from the training list (one per line), and divide its −ln by the number of symbols. That is the same unit. It uses the tokenizer's usual split of the name; other splits of the same text also have small probabilities, so these losses are, if anything, slightly too high. - **Models:** `Qwen/Qwen2.5-1.5B` and `HuggingFaceTB/SmolLM2-1.7B` (base models, not chat versions), 16-bit, with Hugging Face `transformers`. Qwen fit fully on the 4 GB GPU. SmolLM2-1.7B did not, so part of it ran from normal memory: its times are slower for that reason. - **Finishing a name:** greedy, which means always taking the most likely next piece. Our models' "best guess" is the same. Sampled with randomness instead (temperature 1), tinytowns (1M) made a real place 4.0% of the time. - **Claude:** `claude-haiku-5-5`, default settings (adaptive thinking, medium effort), one request per name, 139 input and 175 output tokens per name on average (thinking included). Price on 2026-10-14: $0.10 per million input tokens, $0.50 per million output tokens. Its answer went through the same cleaning as the place list. - **Times:** medians, one name at a time. Job 1 times are on an RTX 3050 laptop GPU (4 GB); the general models' times include reading the 20 example names. Job 2 times for tinytowns and the lookup are on one core of an AMD EPYC cloud server; Claude's are from that server, network included. - **"Real place":** the answer is in the GeoNames list of 142,706 names. - **Memory to hold the model:** for tinytowns, the size of its weights file (32-bit numbers); for the counting table, 784 numbers at 4 bytes; for Qwen and SmolLM2, their number of weights (1,543,714,304 and 1,711,376,384) at 2 bytes each, the 16-bit way they ran. Running a model takes more than this (the runtime and working memory), which is why SmolLM2's 3.4 GB did not fit in 4 GB. - **Ratios:** tinytowns (1M) against Qwen2.5-1.5B: loss 2.194 / 2.049 = 1.07, time 138.2 / 2.38 ms = 58, memory 3.09 GB / 4.0 MB = 770. Against SmolLM2-1.7B: 2.194 / 1.993 = 1.10, 255.2 / 2.38 = 107, 3.42 GB / 4.0 MB = 850. - **tinytowns on one CPU core** (the same cloud server as Job 2), Job 1: 0.44 ms (nano), 1.03 ms (small), 2.61 ms (1M) per name. The scripts, the 300 names and every answer are in the repo: [bench/](https://github.com/dave8172/tinytowns/tree/main/bench). Measured 2026-10-14.