JustujuAI, layer by layer.Open in the interactive map

Tokens & tokenization

Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.

Layer 5 · Foundation Models · Build and run time · reviewed 2026-10-30

D1 · StoryFor young children and total beginners

Guess first: is a long, unusual word one piece or several?

Before an AI reads anything, it snaps sentences into little pieces called tokens, like snapping a chocolate bar into squares.

Short, common words are one piece. Long or unusual words get snapped into several pieces.

Like: Snapping a chocolate bar into squares.

D2 · PictureFor curious beginners

A tokenizer splits text into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries, learned from which chunks of text appear most often.

Each token is replaced by an ID number, and each ID by a list of numbers (an embedding) that the model actually computes with.

Token counts matter in practice: prices, speed and context limits are all measured in tokens, and text in some languages needs more tokens per word than English.

From text to numbers

  1. Text: The cat sat on the
  2. Split: into tokens
  3. Look up: token IDs
  4. Embed: one vector per token

Like: Snapping a chocolate bar into squares.
Where it breaks: Chocolate squares are all the same size. Tokens vary in length and are chosen by frequency, so a rare name may split into many fragments, which is one reason models can struggle to count the letters in a word.

D3 · PlayFor motivated learners

A tokenizer chops text into tokens: pieces from a fixed vocabulary. Common words become one token; rare words, typos and unfamiliar scripts break into several smaller pieces.

This toy learns its vocabulary in your browser with byte-pair encoding (BPE), from a few paragraphs of English. Type anything below, or load an example, and watch the token count.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

Byte-pair encoding builds a vocabulary bottom up. Start with single characters (real tokenizers start with the 256 bytes). Count every adjacent pair across the corpus, weighted by how often each word appears. Merge the most frequent pair into one new token, re-split every word with it, and repeat.

The vocabulary size is a choice: stop after a few thousand merges, or a few hundred thousand. To split new text, apply the learned merges in the order they were learned.

Press Next merge and watch the pair counts, the words and the vocabulary change together.

Try the interactive version in the map.

Things to try

vocab = all single bytes                  # 256 starting tokens
words = split_into_words(corpus)          # merges never cross a word boundary
merges = []
repeat num_merges times:
    counts = count_adjacent_pairs(words)  # weighted by how often each word appears
    a, b = most_frequent(counts)
    new = add_to_vocab(a + b)
    words = replace_pair(words, a, b, new)
    merges.append((a, b, new))

def encode(text):
    tokens = bytes_of(text)
    while any adjacent pair has a merge:
        apply the earliest-learned one
    return tokens

Trade-offs

D5 · FrontierFor experts

D2's fixed vocabulary is a convenient compromise, and its seams show at the edges.

Dropping tokens. A byte-level model that groups bytes into dynamic patches matched token-based models in scaling tests up to 8B parameters; whether that holds at frontier scale is open (Pagnoni et al., 2024).

Fairness across languages. The same text in different languages can need up to 15 times as many tokens, raising cost and shrinking usable context for some speakers (Petrov et al., 2023).

Numbers. Forcing digits to group from the right, by inserting commas, greatly improved arithmetic in frontier models (Singh and Strouse, 2024).

Glitch tokens. Vocabulary entries that barely appear in training can trigger odd behaviour, and such under-trained tokens are common across models (Land and Bartolo, 2024).

Examples as of 2026-10-30

Sources

  1. Neural Machine Translation of Rare Words with Subword Units (byte-pair encoding) (2015)
  2. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018)
  3. Byte Latent Transformer: Patches Scale Better Than Tokens (Pagnoni et al.) (2024-12-13)
  4. Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al.) (2023-05-17)
  5. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (Singh and Strouse) (2024-02-22)
  6. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (Land and Bartolo) (2024-05-08)

How it connects

More in Layer 5 · Foundation Models