JustujuAI, layer by layer.Open in the interactive map

How ChatGPT works

Five stops, from the words you type to the answer you get back.

Each step below explains one part in plain words and links to the full page, which goes deeper.

  1. Chat assistants
  2. Tokens & tokenization
  3. How a model predicts the next token
  4. Pretraining
  5. Post-training: from predictor to assistant

Step 1: Chat assistants

Start with the app you know. What happens when you press send?

General-purpose chat apps where you type, talk or upload files, and an assistant answers, writes, searches and uses tools.

A chat assistant is an app where you talk with an AI in a conversation.

You ask, it answers, and you can ask follow-up questions. Many can also search the web, read your files or make pictures.

The chat window is the visible part. On every turn the app sends the whole conversation, plus hidden instructions, to the model, because the model itself keeps no memory between calls.

Modern chat apps are harnesses too: they add web search, file reading, code running, image tools, connectors to your accounts and saved memories about you.

One model can be reached in several ways: the maker's own app, a third-party app that lets you switch models, or a local chat app talking to a model on your own machine.

Chat windowmessages, files, voiceApp featuressearch, memory, connectorsEach turninstructions + whole conversationModelcloud API or local
What sits behind a chat window

Read more about Chat assistants · Try this step in the interactive map · Layer 9 · Applications & Interfaces

Step 2: Tokens & tokenization

First, your words are chopped into small pieces. Type your own sentence and watch.

Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.

Before an AI reads anything, it snaps sentences into little pieces called tokens, like snapping a chocolate bar into squares.

Short, common words are one piece. Long or unusual words get snapped into several pieces.

A tokenizer splits text into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries, learned from which chunks of text appear most often.

Each token is replaced by an ID number, and each ID by a list of numbers (an embedding) that the model actually computes with.

Token counts matter in practice: prices, speed and context limits are all measured in tokens, and text in some languages needs more tokens per word than English.

TextThe cat sat on the1Splitinto tokens2Look uptoken IDs3Embedone vector per token4
From text to numbers
  1. Text: The cat sat on the
  2. Split: into tokens
  3. Look up: token IDs
  4. Embed: one vector per token

Read more about Tokens & tokenization · Try this step in the interactive map · Layer 5 · Foundation Models

Step 3: How a model predicts the next token

Then the model guesses the next piece, again and again. Play the cat game.

A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.

Let's play a game. I say: “The cat sat on the…” and you shout the next word.

You'd probably say “mat”. Maybe “floor” or “sofa”. Probably not “moon”!

An AI language model plays this game again and again, one small piece of a word at a time. That's how it writes whole answers.

Text is split into tokens (words or pieces of words), and each token becomes a list of numbers.

Those numbers pass through a stack of transformer layers. Attention lets each position look back at earlier tokens to decide what matters.

The last layer gives a score to every token in the vocabulary, and softmax turns those scores into probabilities.

One token is chosen, added to the text, and the whole process repeats.

Tokenizetext totokens1Embedtokens tonumbers2Transformerattention +layers3Scoresone pertoken4Softmaxtoprobabilities5Pickand append6repeat for the next token
One step of text generation
  1. Tokenize: text to tokens
  2. Embed: tokens to numbers
  3. Transformer: attention + layers
  4. Scores: one per token
  5. Softmax: to probabilities
  6. Pick: and append

Read more about How a model predicts the next token · Try this step in the interactive map · Layer 5 · Foundation Models

Step 4: Pretraining

It learned to guess by reading a huge amount of text. Watch a tiny one learn.

Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.

Imagine reading millions of books where the next word is always covered. You guess, peek, and learn a tiny bit from every wrong guess.

Do that trillions of times and you become very good at guessing. That's pretraining.

The training objective is simple: for each position in a batch of text, predict the next token. The difference between the prediction and the real token is the loss.

Gradient descent nudges every weight slightly to reduce that loss, batch after batch, on large GPU clusters.

Loss tends to fall predictably as models, data and compute grow together (scaling laws). The result is a base model that continues text but doesn't yet follow instructions.

Batchof real text1Predictnext tokens2Compareloss vs real3Adjustall weights slightly4repeat for trillions of tokens
The pretraining loop
  1. Batch: of real text
  2. Predict: next tokens
  3. Compare: loss vs real
  4. Adjust: all weights slightly

Read more about Pretraining · Try this step in the interactive map · Layer 5 · Foundation Models

Step 5: Post-training: from predictor to assistant

Last, people taught it to be helpful and to say no to harmful requests.

Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.

After reading a whole library, the AI knows a lot. But it doesn't know how to be helpful yet.

Post-training is like lessons in manners and practice. People show it good answers and reward the helpful ones.

A pretrained model continues text; it doesn't naturally follow instructions.

Supervised fine-tuning trains it on example conversations that show the behaviour we want.

Preference learning (such as RLHF) trains it to prefer answers people rate higher, and reinforcement learning with checkable rewards trains reasoning on problems whose answers can be verified.

Pretrainedpredicts text1Fine-tuneondemonstrations2Preferencesreward model3RLcheckable tasks4Assistantfollowsinstructions5
From pretrained model to assistant
  1. Pretrained: predicts text
  2. Fine-tune: on demonstrations
  3. Preferences: reward model
  4. RL: checkable tasks
  5. Assistant: follows instructions

Read more about Post-training: from predictor to assistant · Try this step in the interactive map · Layer 5 · Foundation Models