# Tokens & tokenization URL: https://justuju.org/concept/tokenization/ Layer 5 · Foundation Models Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with. ## D1 · Story Before an AI reads anything, it snaps sentences into little pieces called tokens, like snapping a chocolate bar into squares. Short, common words are one piece. Long or unusual words get snapped into several pieces. Like: Snapping a chocolate bar into squares. ## D2 · Picture A tokenizer splits text into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries, learned from which chunks of text appear most often. Each token is replaced by an ID number, and each ID by a list of numbers (an embedding) that the model actually computes with. Token counts matter in practice: prices, speed and context limits are all measured in tokens, and text in some languages needs more tokens per word than English. 1. Text: The cat sat on the 2. Split: into tokens 3. Look up: token IDs 4. Embed: one vector per token Like: Snapping a chocolate bar into squares. Where it breaks: Chocolate squares are all the same size. Tokens vary in length and are chosen by frequency, so a rare name may split into many fragments, which is one reason models can struggle to count the letters in a word. ## D4 · Mechanism Byte-pair encoding builds a vocabulary bottom up. Start with single characters (real tokenizers start with the 256 bytes). Count every adjacent pair across the corpus, weighted by how often each word appears. Merge the most frequent pair into one new token, re-split every word with it, and repeat. The vocabulary size is a choice: stop after a few thousand merges, or a few hundred thousand. To split new text, apply the learned merges in the order they were learned. Press Next merge and watch the pair counts, the words and the vocabulary change together. - A bigger vocabulary means fewer tokens per text (cheaper, more fits in the context), but a bigger embedding table and rare tokens that are seen less in training. - A tokenizer trained mostly on English splits other languages into more pieces, so the same message costs more and fits less. - Starting from bytes means no text is ever unknown, but typos and rare words turn into many small pieces. - The model sees chunks, not letters. That is one reason counting letters, spelling and digit-by-digit arithmetic are harder for it than they look. ## D5 · Frontier D2's fixed vocabulary is a convenient compromise, and its seams show at the edges. Dropping tokens. A byte-level model that groups bytes into dynamic patches matched token-based models in scaling tests up to 8B parameters; whether that holds at frontier scale is open (Pagnoni et al., 2024). Fairness across languages. The same text in different languages can need up to 15 times as many tokens, raising cost and shrinking usable context for some speakers (Petrov et al., 2023). Numbers. Forcing digits to group from the right, by inserting commas, greatly improved arithmetic in frontier models (Singh and Strouse, 2024). Glitch tokens. Vocabulary entries that barely appear in training can trigger odd behaviour, and such under-trained tokens are common across models (Land and Bartolo, 2024). ## Examples as of 2026-10-30 - tiktoken (OpenAI) - SentencePiece (Google) - Hugging Face Tokenizers ## Sources - Neural Machine Translation of Rare Words with Subword Units (byte-pair encoding): https://arxiv.org/abs/1508.07909 - SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing: https://arxiv.org/abs/1808.06226 - Byte Latent Transformer: Patches Scale Better Than Tokens (Pagnoni et al.): https://arxiv.org/abs/2412.09871 - Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al.): https://arxiv.org/abs/2305.15425 - Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (Singh and Strouse): https://arxiv.org/abs/2402.14903 - Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (Land and Bartolo): https://arxiv.org/abs/2405.05417 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.