Guardrails & evals
Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves.
Read first: Tool use & protocols (MCP).
D1 · StoryFor young children and total beginners
Guardrails are checks before and after the AI speaks, like a teacher reading a note before it goes home.
Evals are practice tests the builders run again and again, so they notice when a change makes the AI worse.
Like: A teacher checking notes, plus regular practice tests.
D2 · PictureFor curious beginners
Input checks screen what goes in: user messages and, just as important, text from documents, web pages and tool results. Output checks screen what comes out: format, leaked secrets, unsafe content.
Prompt injection is text that tries to give the model new orders. Defences are layered: keep untrusted text separate and labelled, give tools the least privilege, and ask a human before risky actions.
Evals are test sets run automatically on every change to prompts, models or tools, scored by code, by people, or by another model acting as judge.
Checks around one call, with evals in the loop
- Input check: user and tool text
- Model call: with context
- Output check: format, secrets, safety
- Action gate: permissions, human OK
- Evals: score, compare, fix
Like: A teacher checking notes, plus regular practice tests.
Where it breaks: A teacher understands what a note means. Filters are themselves pattern matchers or models, so cleverly worded attacks can slip past them; no single check stops prompt injection.
D5 · FrontierFor experts
Prompt injection has no complete fix yet, because models read instructions and data in the same stream of tokens. Defences reduce risk rather than remove it.
Evals are hard too: model judges have their own biases, test sets leak into training data, and a passing score on a benchmark may not match real use.