# Transformers & attention URL: https://justuju.org/concept/transformer-attention/ Layer 5 · Foundation Models The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant. ## D1 · Story Read this: “The cat sat on the mat because it was tired.” You know “it” means the cat, not the mat. Attention is how the AI works that out: for each word it looks back at the earlier words and decides which ones matter most. Like: Highlighting the words that matter. ## D2 · Picture A transformer is a stack of identical layers. Each layer has two parts: attention, which mixes in information from other positions, and a feed-forward network, which processes each position on its own. In attention, each position asks a question (a query), compares it with a label on every earlier position (its key), and takes a weighted blend of what those positions carry (their values). Many attention heads do this in parallel, looking for different patterns. In models that generate text, a causal mask stops any position from looking ahead at tokens that haven't been written yet. Like: Highlighting the words that matter. Where it breaks: You highlight a few words once. Attention gives every earlier token some weight, dozens of heads do it at once in every layer, and the result is spread across many layers rather than one clear highlight. ## D4 · Mechanism Stack every token's query into a matrix Q, its key into K and its value into V. Attention is softmax(QKᵀ ÷ √d) · V, where d is the length of each query and key. Row i of QKᵀ holds the dot product of token i's query with every key. Dividing by √d stops the scores growing just because the vectors are longer. The causal mask sets scores for later tokens to −∞ so softmax gives them zero weight. Each output row is a weighted blend of the value vectors. Edit the vectors below; every table is computed live from the formula. - Every token is compared with every other, so the score table grows with the square of the context length. - The causal mask lets a model train on a whole sentence at once while still learning to predict each word without peeking. - More heads let a layer look for more patterns at once, but each head works with shorter vectors. - Attention weights show where a head looked, not why the model gave its answer; reading them as explanations can mislead. ## D5 · Frontier D4's formula is exact, but its cost grows with the square of the context, and the picture of heads looking for patterns is only partly understood. Exact but faster. Kernels that avoid memory traffic made exact attention much faster without changing its quadratic arithmetic (Dao et al., 2022). Replacing attention. State-space layers run in linear time with about five times the throughput (Gu and Dao, 2023), yet efficient alternatives lag attention at recalling earlier context, and hybrids close most of the gap (Arora et al., 2023). Using long context. Models use information at the start or end of a long input better than in the middle (Liu et al., 2023). What heads do. Heads that copy earlier patterns may underlie learning from examples in the prompt, but the causal evidence is strong only for small attention-only models (Olsson et al., 2022). ## Examples as of 2026-10-30 - FlashAttention (fast attention kernels) - Grouped-query attention in Llama models - Multi-head latent attention in DeepSeek-V3 - Hybrid attention and Mamba models such as NVIDIA Nemotron-H ## Sources - Attention Is All You Need: https://arxiv.org/abs/1706.03762 - FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: https://arxiv.org/abs/2205.14135 - Mamba: Linear-Time Sequence Modeling with Selective State Spaces: https://arxiv.org/abs/2312.00752 - Gemma 2: Improving Open Language Models at a Practical Size: https://arxiv.org/abs/2408.00118 - Zoology: Measuring and Improving Recall in Efficient Language Models (Arora et al.): https://arxiv.org/abs/2312.04927 - Lost in the Middle: How Language Models Use Long Contexts (Liu et al.): https://arxiv.org/abs/2307.03172 - In-context Learning and Induction Heads (Olsson et al.): https://arxiv.org/abs/2209.11895 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.