# Retrieval-augmented generation (RAG) URL: https://justuju.org/concept/rag/ Layer 7 · Orchestration & Tooling Look up relevant documents first, then give them to the model to answer from. ## D1 · Story If you ask a friend a question about your school, they might check the school website before answering. RAG makes the AI look things up first instead of guessing from memory. 1. Your question 2. Look it up: search pages you trust 3. Add what was found: next to the question 4. Answer: based on what it found Like: Checking the website before answering. ## D2 · Picture Documents are split into chunks and turned into vectors (embeddings) stored in an index. At question time the question is embedded too; the closest chunks are retrieved, often combined with keyword search and re-ranked, and placed in the prompt. The model answers from that context, ideally with citations. Quality depends mostly on retrieval: if the right chunk isn't found, the model can't use it. 1. Question 2. Embed: to a vector 3. Search: vector + keyword 4. Re-rank: best chunks 5. Prompt: chunks + question 6. Answer: with citations Like: An open-book exam. Where it breaks: In an open-book exam you choose what to read. In RAG the model only sees what retrieval hands it, and it can still blend in its own, possibly wrong, memory. ## D4 · Mechanism Production retrieval usually runs two searches over the same chunks. Keyword search (BM25) scores a passage by the query words it contains, weighting rare words more and long passages less. Meaning search compares embeddings: vectors that put texts with similar meaning close together. Each fails in its own way. Keyword search misses paraphrases; meaning search blurs exact strings such as error codes, IDs and names. Reciprocal rank fusion merges the two lists using ranks alone: RRF(d) = Σ 1 ÷ (k + rank), with k = 60 in the original paper. A reranker then reads the query together with each of the top few candidates and reorders them. Only the top k go into the prompt. BM25 and RRF below are the real formulas. The meaning search and the reranker are small hand-made stand-ins, so their numbers are illustrative. The table at the end is how retrieval is measured: for test questions with known answers, what share of the right passages reach the top k (recall@k)? - Two indexes cost more to build, store and keep up to date than one, but each covers the other's blind spot: exact codes and names for keyword search, paraphrases for meaning search. - RRF needs no tuning and never compares raw scores, which is why it is a common default. It also throws away how confident each search was; a weighted blend of scores can do better but has to be tuned for each collection. - A reranker is slower per passage than either search, so it only sees a shortlist. It cannot rescue a passage that neither search returned. - A bigger top k raises recall but makes the prompt longer, slower and dearer, and gives the model more irrelevant text to read past. - Recall@k needs test questions with the right passages marked. Building that set is tedious, but without it you cannot tell whether a change to chunking, search or reranking helped. ## D5 · Frontier "Retrieve, then answer" assumes the right passage is found and then used faithfully. Neither is guaranteed. Faithfulness. On one citation benchmark even the best models lacked complete support for their claims about half the time (ALCE, 2023). Retrieval or long context? Given enough compute, long-context models beat retrieval on average but cost far more; routing each query to one or the other is one compromise (Li et al., 2024). Poisoned knowledge. Five crafted texts per target question, planted among millions, steered answers 90% of the time (PoisonedRAG, 2024). Anyone who can write to your index can write to your answers. Agentic retrieval. Letting the model decide when to search and critique what it found can beat fixed retrieval (Self-RAG, 2023), but adds steps, cost and new ways to go wrong. ## Examples as of 2026-10-30 - Vector databases: pgvector, Pinecone, Weaviate, Qdrant - Frameworks: LlamaIndex, LangChain - Embedding models from OpenAI, Cohere and Voyage AI (MongoDB) ## Sources - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401 - Retrieval-Augmented Generation for Large Language Models: A Survey: https://arxiv.org/abs/2312.10997 - Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (Cormack, Clarke and Büttcher, SIGIR 2009): https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf - The Probabilistic Relevance Framework: BM25 and Beyond (Robertson and Zaragoza): https://www.nowpublishers.com/article/Details/INR-019 - Retrieve & Re-Rank (Sentence Transformers documentation): https://sbert.net/examples/applications/retrieve_rerank/README.html - Reciprocal rank fusion (Elasticsearch reference): https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion - Enabling Large Language Models to Generate Text with Citations: https://arxiv.org/abs/2305.14627 - Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach: https://arxiv.org/abs/2407.16833 - PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models: https://arxiv.org/abs/2402.07867 - Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection: https://arxiv.org/abs/2310.11511 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.