Tokens & tokenization
Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.
D1 · StoryFor young children and total beginners
Before an AI reads anything, it snaps sentences into little pieces called tokens, like snapping a chocolate bar into squares.
Short, common words are one piece. Long or unusual words get snapped into several pieces.
Like: Snapping a chocolate bar into squares.
D2 · PictureFor curious beginners
A tokenizer splits text into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries, learned from which chunks of text appear most often.
Each token is replaced by an ID number, and each ID by a list of numbers (an embedding) that the model actually computes with.
Token counts matter in practice: prices, speed and context limits are all measured in tokens, and text in some languages needs more tokens per word than English.
From text to numbers
- Text: The cat sat on the
- Split: into tokens
- Look up: token IDs
- Embed: one vector per token
Like: Snapping a chocolate bar into squares.
Where it breaks: Chocolate squares are all the same size. Tokens vary in length and are chosen by frequency, so a rare name may split into many fragments, which is one reason models can struggle to count the letters in a word.
D3 · PlayFor motivated learners
A tokenizer chops text into tokens: pieces from a fixed vocabulary. Common words become one token; rare words, typos and unfamiliar scripts break into several smaller pieces.
This toy learns its vocabulary in your browser with byte-pair encoding (BPE), from a few paragraphs of English. Type anything below, or load an example, and watch the token count.
Try the interactive version in the map.
Things to try
- Load the English sentence, then the misspelled one. How many extra tokens do the typos cost?
- Load the Greek sentence. Why do some tokens show as codes like <CE> instead of letters?
- Type a long number such as 1234567. Does it stay one token, or split into chunks?
D4 · MechanismFor practitioners
Byte-pair encoding builds a vocabulary bottom up. Start with single characters (real tokenizers start with the 256 bytes). Count every adjacent pair across the corpus, weighted by how often each word appears. Merge the most frequent pair into one new token, re-split every word with it, and repeat.
The vocabulary size is a choice: stop after a few thousand merges, or a few hundred thousand. To split new text, apply the learned merges in the order they were learned.
Press Next merge and watch the pair counts, the words and the vocabulary change together.
Try the interactive version in the map.
Things to try
- Before pressing anything, guess which pair is merged first. Then press Next merge to check.
- Keep merging until “lowest” in the test box splits into two tokens. Which two?
- Type “slowness” in the test box. Which pieces does it reuse, and which letters stay alone?
vocab = all single bytes # 256 starting tokens
words = split_into_words(corpus) # merges never cross a word boundary
merges = []
repeat num_merges times:
counts = count_adjacent_pairs(words) # weighted by how often each word appears
a, b = most_frequent(counts)
new = add_to_vocab(a + b)
words = replace_pair(words, a, b, new)
merges.append((a, b, new))
def encode(text):
tokens = bytes_of(text)
while any adjacent pair has a merge:
apply the earliest-learned one
return tokensTrade-offs
- A bigger vocabulary means fewer tokens per text (cheaper, more fits in the context), but a bigger embedding table and rare tokens that are seen less in training.
- A tokenizer trained mostly on English splits other languages into more pieces, so the same message costs more and fits less.
- Starting from bytes means no text is ever unknown, but typos and rare words turn into many small pieces.
- The model sees chunks, not letters. That is one reason counting letters, spelling and digit-by-digit arithmetic are harder for it than they look.
D5 · FrontierFor experts
Open questions: models that read raw bytes or characters instead of a fixed vocabulary, fairer tokenization across languages, and tasks such as spelling or arithmetic where token boundaries get in the way.