# tinytowns — a town name generator

A very small AI model that guesses the next letter of a town name. Type some letters to see its guesses. It learned from 142,706 real place names.

Model, by dave8172. Page: https://quirkyagents.com/wizard/projects/tinytowns/

Type some letters. The bars show the model's guesses for the next letter.

The three models (validation loss: lower is better; guessing evenly scores 3.3322):

| Model | Weights | File | Validation loss |
|---|---|---|---|
| nano | 4,572 | 20 KB | 2.4676 |
| small | 105,052 | 424 KB | 2.2049 |
| tinytowns | 1,001,500 | 4 MB | 2.1504 |

## Watch it

A 37-second video. It shows the model's guesses, a real name, an invented name, and what you could use it
for. The video has no sound.

<video controls playsinline preload="none" width="1080" height="1080" poster="https://quirkyagents.com/wizard/projects/tinytowns/video-poster.jpg">
  <source src="https://pub-21da2bddd9cf4685b4374f3e9115db36.r2.dev/tinytowns/tinytowns-launch.mp4" type="video/mp4">
</video>

## What it is

tinytowns is a very small AI model. It works **like the models behind chat apps**, but it is
much smaller. You can read every number in it.

The model reads a name one letter at a time. Then it gives a chance to each symbol that can come
next. There are 28 symbols: the 26 letters, a space, and "end of name". The bars show the five
symbols with the highest chances.

## How to use it

1. Select a model. The larger models make better guesses.
2. Type some letters, for example `san`.
3. Look at the bars. To add a letter, tap its bar.
4. To let the model write the rest of the name, tap **Finish it for me**.
5. To end the name, tap **end of name**, type a full stop (.), or press Enter.

When the name is complete, the page tells you if it is real or invented.

## Temperature

Temperature changes how the model selects letters. At a low temperature ("safe"), the usual
letters get almost all the chances, and the names look like real names. At a high temperature
("wild"), unusual letters get more chances, and the names are stranger. Move the slider and look
at the bars.

## Real or invented

The model learned from a list of 142,706 real place names. When a name is complete, the page
looks for it in this list. "Real" means that a place with this name exists. "Invented" means that
the name is not in the list.

Most names are invented. Each model made 10,000 names as a test. For each model, more than 95% of
the names were invented.

## Where it runs

The model runs in your browser, on your device. The page downloads the model from Hugging Face one
time. After that, your device calculates each guess. No server does this work.

## Three sizes

| Model | Numbers in the model | Download | Error score |
|---|---|---|---|
| nano | 4,572 | 20 KB | 2.47 |
| small | 105,052 | 424 KB | 2.20 |
| tinytowns | 1,001,500 | 4 MB | 2.15 |

The error score shows how bad the guesses are. A lower score is better. We measured it on 14,271
names that the model did not learn from. Random guesses get a score of 3.33.

A model with more numbers makes better guesses, but each step up helps less. From nano to small,
the score goes down by 0.26. From small to tinytowns, it goes down by only 0.05.

## What it can do

These are ideas. We did not test them:

- Make names for towns in games and on maps.
- Make test data with place names that are not real.
- Tell if a word looks like a place name. For example, it could find `asdfgh` in a city field.

<details class="tech">
<summary>Technical details</summary>

- **Model:** a decoder-only, character-level transformer, written by hand in plain PyTorch.
  Vocabulary of 28 (index 0 `.` marks start and end of a name, then space, `a`–`z`), context of 21.
  Learned token and position embeddings, pre-LayerNorm blocks (causal multi-head self-attention,
  then a GELU MLP), a final LayerNorm and a separate unembedding. Biases on every linear layer, float32.
- **Sizes** (d_model / layers × heads / MLP width): nano 16 / 1 × 1 / 64, small 64 / 2 × 4 / 256,
  tinytowns 128 / 5 × 4 / 512.
- **Data:** GeoNames `cities1000` (places with 1,000+ people, 243 countries), lowercased, hyphens
  to spaces, apostrophes and names over 20 letters dropped: 142,706 names, split 90/10 with a fixed
  seed (128,435 train, 14,271 validation).
- **Training:** next-symbol cross-entropy on `.` + name + `.`, padding ignored. AdamW, weight decay
  0.01, learning rate 3e-3 (nano, small) or 1e-3 (tinytowns), 6,000 steps, batch 64 on a CPU (nano)
  or 256 on a laptop RTX 3050 (small 135 s, tinytowns 550 s). The weights kept are from the step
  with the lowest validation loss.
- **Validation loss:** uniform 3.3322, bigram 2.6732, nano 2.4676, small 2.2049, tinytowns 2.1504
  (best at step 5,500; train loss 2.0085 there, and validation loss rises to 2.1543 by step 6,000).
- **In the browser:** a hand-written JavaScript forward pass, with no runtime library. It parses the
  `.safetensors` header itself and recomputes the whole context for each guess (no KV cache). On all
  14,271 validation names it matches the PyTorch loss to six decimal places for all three sizes.
  Measured in headless Chrome on a one-core cloud server: about 0.7 ms, 3 ms and 34 ms per guess.

More detail is in the [model card on Hugging Face](https://huggingface.co/dave8172/tinytowns) and the
step-by-step code on [GitHub](https://github.com/dave8172/tinytowns). The series
[How an LLM runs](https://quirkyagents.com/wizard/blog/series/how-an-llm-runs/) explains the same parts in a full-size model.

</details>

Place names: [GeoNames](https://www.geonames.org), licensed
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).