JustujuAI, layer by layer.Open in the interactive map

Retrieval-augmented generation (RAG)

Look up relevant documents first, then give them to the model to answer from.

Layer 7 · Orchestration & Tooling · Run time · reviewed 2026-10-30

Read first: How a model predicts the next token.

D1 · StoryFor young children and total beginners

When you're unsure of an answer, what do you do before replying?

If you ask a friend a question about your school, they might check the school website before answering.

RAG makes the AI look things up first instead of guessing from memory.

Look it up first, then answer

  1. Your question
  2. Look it up: search pages you trust
  3. Add what was found: next to the question
  4. Answer: based on what it found

Like: Checking the website before answering.

D2 · PictureFor curious beginners

Documents are split into chunks and turned into vectors (embeddings) stored in an index.

At question time the question is embedded too; the closest chunks are retrieved, often combined with keyword search and re-ranked, and placed in the prompt.

The model answers from that context, ideally with citations. Quality depends mostly on retrieval: if the right chunk isn't found, the model can't use it.

Answering with retrieval

  1. Question
  2. Embed: to a vector
  3. Search: vector + keyword
  4. Re-rank: best chunks
  5. Prompt: chunks + question
  6. Answer: with citations

Like: An open-book exam.
Where it breaks: In an open-book exam you choose what to read. In RAG the model only sees what retrieval hands it, and it can still blend in its own, possibly wrong, memory.

D3 · PlayFor motivated learners

Before the model sees anything, retrieval decides what it reads. The documents are cut into chunks, every chunk is scored against the question, and only the top few are pasted into the prompt.

Here the score is simple word overlap, weighted so rare words count for more (TF-IDF). Production retrieval is often hybrid: keyword scoring like this plus embeddings, which match meaning rather than exact words, often followed by a reranker. D4 opens that up. The shape stays the same: rank, cut, paste.

Change the chunk size and how many chunks are kept, then read the box at the bottom. That box is everything the model gets. If the answer is not in it, the model cannot give it.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

Production retrieval usually runs two searches over the same chunks. Keyword search (BM25) scores a passage by the query words it contains, weighting rare words more and long passages less. Meaning search compares embeddings: vectors that put texts with similar meaning close together.

Each fails in its own way. Keyword search misses paraphrases; meaning search blurs exact strings such as error codes, IDs and names. Reciprocal rank fusion merges the two lists using ranks alone: RRF(d) = Σ 1 ÷ (k + rank), with k = 60 in the original paper. A reranker then reads the query together with each of the top few candidates and reorders them. Only the top k go into the prompt.

BM25 and RRF below are the real formulas. The meaning search and the reranker are small hand-made stand-ins, so their numbers are illustrative. The table at the end is how retrieval is measured: for test questions with known answers, what share of the right passages reach the top k (recall@k)?

Try the interactive version in the map.

Things to try

def retrieve(query, k=5, pool=50, c=60):
    by_words   = bm25_index.search(query, top=pool)          # exact terms; rare words weigh more
    by_meaning = vector_index.search(embed(query), top=pool) # nearest neighbours by cosine
    fused = {}
    for results in (by_words, by_meaning):
        for rank, doc in enumerate(results, start=1):
            fused[doc] = fused.get(doc, 0) + 1 / (c + rank)   # reciprocal rank fusion
    shortlist = sorted(fused, key=fused.get, reverse=True)[:pool]
    scores = reranker.score(query, shortlist)               # reads query and passage together
    best = sorted(zip(shortlist, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in best[:k]]

def recall_at_k(test_set, k):                  # test_set: (question, set of relevant ids)
    found = [len(set(retrieve(q, k)) & relevant) / len(relevant) for q, relevant in test_set]
    return sum(found) / len(found)

Trade-offs

D5 · FrontierFor experts

"Retrieve, then answer" assumes the right passage is found and then used faithfully. Neither is guaranteed.

Faithfulness. On one citation benchmark even the best models lacked complete support for their claims about half the time (ALCE, 2023).

Retrieval or long context? Given enough compute, long-context models beat retrieval on average but cost far more; routing each query to one or the other is one compromise (Li et al., 2024).

Poisoned knowledge. Five crafted texts per target question, planted among millions, steered answers 90% of the time (PoisonedRAG, 2024). Anyone who can write to your index can write to your answers.

Agentic retrieval. Letting the model decide when to search and critique what it found can beat fixed retrieval (Self-RAG, 2023), but adds steps, cost and new ways to go wrong.

Examples as of 2026-10-30

Sources

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
  2. Retrieval-Augmented Generation for Large Language Models: A Survey (2023-12-18)
  3. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (Cormack, Clarke and Büttcher, SIGIR 2009) (2009)
  4. The Probabilistic Relevance Framework: BM25 and Beyond (Robertson and Zaragoza) (2009-12-17)
  5. Retrieve & Re-Rank (Sentence Transformers documentation) (2026)
  6. Reciprocal rank fusion (Elasticsearch reference) (2026)
  7. Enabling Large Language Models to Generate Text with Citations (2023-05-24)
  8. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2024-07-23)
  9. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models (2024-02-12)
  10. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023-10-17)

How it connects

More in Layer 7 · Orchestration & Tooling