Layer 7 · Orchestration & Tooling

The glue around a model call: prompts, retrieval, tools, frameworks, evaluation and guardrails.

In simple words: Everything around the AI that helps it answer well: instructions, notes, searches, tools and safety checks.

Draft, awaiting expert review, reviewed 2026-10-02

Like: A film crew around an actor: scripts, props and cues so the performance fits the scene.
Where it breaks: An actor improvises from real understanding of the set. A model only sees what is placed in its context window: if the crew forgets a prop, for the model it doesn't exist.

What it gives the layer above

Reliable, grounded model calls that can use your data and tools.

What it needs from the layer below

An inference endpoint from Inference & Optimization.

Concepts

• Retrieval-augmented generation (RAG)
  Look up relevant documents first, then give them to the model to answer from.
• Tool use & protocols (MCP)
  Letting a model call functions and services through structured requests, standardised by protocols such as MCP.
• Memory
  How an AI application remembers beyond one call: short-term memory in the context window, long-term memory in stores it saves to and searches.
• Context engineering
  Choosing what goes into the model's context window, in what order, and what to leave out, because the model sees nothing else.
• Guardrails & evals
  Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves.

Walkthrough

1. A question arrives. Someone asks their home-lab assistant: “Which driver did I install on the GPU box last month?” The model alone can't know.
2. Memory and retrieval look it up. Memory recalls a saved note about the GPU box, and retrieval searches the setup docs for driver notes. Both hand back short passages.
3. A tool checks the live machine. The model asks for a tool call to read the installed driver version. The application runs it and the result joins the context.
4. The context is packed and the model answers. System prompt, tool list, notes, the tool result and the question are trimmed to fit the window, stable parts first so they can be cached.
5. Guardrails check, evals keep score. Tool output is screened for injected orders, the answer is checked before it is shown, and the same question joins an eval set for the next change.

Examples

• Prompt and context templates
• Retrieval and vector databases
• Tool protocols (e.g. MCP)
• Memory stores and session summaries
• Guardrail and eval tools (e.g. Llama Guard, promptfoo)