Layer 7 · Orchestration & Tooling

The glue around a model call: prompts, retrieval, tools, frameworks, evaluation and guardrails.

In simple words: Everything around the AI that helps it answer well: instructions, notes, searches, tools and safety checks.

reviewed 2026-10-30

Like: A film crew around an actor: scripts, props and cues so the performance fits the scene.
Where it breaks: An actor improvises from real understanding of the set. A model only sees what is placed in its context window: if the crew forgets a prop, for the model it doesn't exist.

Key players

What it means for your business: Your own product logic lives in this glue, so prefer open standards and frameworks you could replace, because the tools here change quickly and some will not last.

• Model Context Protocol (MCP): Open standard for plugging tools and data into AI apps; under a Linux Foundation fund.
• LangChain / LangGraph: Open-source frameworks for building agents and multi-step workflows.
• LlamaIndex: Open-source framework for feeding company documents to models.
• pgvector: Open-source add-on that gives Postgres vector search for retrieval.
• Pinecone: Managed vector database for search over embeddings.
• Langfuse: Open-source tracing, evaluation and prompt management; part of ClickHouse since 2026.

As of 2026-10-15. Sources: Donating the Model Context Protocol and establishing the Agentic AI Foundation (Anthropic) (2025-12-09); LangChain and LangGraph agent frameworks reach v1.0 (2025-10-22); LlamaIndex (GitHub) (2026); pgvector (GitHub) (2026); Pinecone product overview (2026); Langfuse (GitHub) (2026)

What it gives the layer above

Reliable, grounded model calls that can use your data and tools.

What it needs from the layer below

An inference endpoint from Inference & Optimization.

Concepts

• Retrieval-augmented generation (RAG)
  Look up relevant documents first, then give them to the model to answer from.
• Tool use & protocols (MCP)
  Letting a model call functions and services through structured requests, standardised by protocols such as MCP.
• Memory
  How an AI application remembers beyond one call: short-term memory in the context window, long-term memory in stores it saves to and searches.
• Context engineering
  Choosing what goes into the model's context window, in what order, and what to leave out, because the model sees nothing else.
• Guardrails & evals
  Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves.

Walkthrough

1. A question arrives. Someone asks their home-lab assistant: “Which driver did I install on the GPU box last month?” The model alone can't know.
2. Memory and retrieval look it up. Memory recalls a saved note about the GPU box, and retrieval searches the setup docs for driver notes. Both hand back short passages.
3. A tool checks the live machine. The model asks for a tool call to read the installed driver version. The application runs it and the result joins the context.
4. The context is packed and the model answers. System prompt, tool list, notes, the tool result and the question are trimmed to fit the window, stable parts first so they can be cached.
5. Guardrails check, evals keep score. Tool output is screened for injected orders, the answer is checked before it is shown, and the same question joins an eval set for the next change.

Examples

• Prompt and context templates
• Retrieval and vector databases
• Tool protocols (e.g. MCP)
• Memory stores and session summaries
• Guardrail and eval tools (e.g. Llama Guard, Inspect)