Guardrails & evals

Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves.

Layer 7 · Orchestration & Tooling · Run time · reviewed 2026-10-30

Read first: Tool use & protocols (MCP).

D1 · StoryFor young children and total beginners

Spot where a hidden instruction could sneak past the checks.

Guardrails are checks before and after the AI speaks, like a teacher reading a note before it goes home.

Evals are practice tests the builders run again and again, so they notice when a change makes the AI worse.

Like: A teacher checking notes, plus regular practice tests.

D2 · PictureFor curious beginners

Input checks screen what goes in: user messages and, just as important, text from documents, web pages and tool results. Output checks screen what comes out: format, leaked secrets, unsafe content.

Prompt injection is text that tries to give the model new orders. Direct injection is typed by the user. Indirect injection hides in something the model reads: a web page, an email, a file, or a tool's own description (tool poisoning). Models read instructions and data in the same stream of tokens, so no filter catches every attempt.

So the main defences sit outside the model. The lethal trifecta is an agent that can read private data, sees untrusted content and can send data out: an injected instruction can make it leak. Take away any one of the three. Run agents in a sandbox (limited files, limited network), give tools the least privilege, and ask a person before risky actions.

Jailbreaks are different: the user attacks the model's own safety training, with role-play, odd encodings or hundreds of fake example dialogues. Safety training and classifiers that screen requests make them harder, not impossible.

Evals are test sets run automatically on every change to prompts, models or tools, scored by code, by people, or by another model acting as judge.

Checks around one call, with evals in the loop

Input check: user and tool text
Model call: with context
Output check: format, secrets, safety
Action gate: permissions, human OK
Evals: score, compare, fix

Like: A teacher checking notes, plus regular practice tests.
Where it breaks: A teacher understands what a note means. Filters are themselves pattern matchers or models, so cleverly worded attacks can slip past them; no single check stops prompt injection.

D4 · MechanismFor practitioners

Prompt injection works because a model reads instructions and data in one stream of tokens, so text on a web page can give orders just as the user can. It becomes a data leak when one agent has three abilities at once, which Simon Willison calls the lethal trifecta: access to private data, exposure to untrusted content, and a way to send data out.

Remove any one leg and this attack has no path, however it is worded. Filters work differently: they try to recognise attacks, so a rewording can slip past. The strongest designs isolate untrusted text. A planning model sees only the user's request and writes the plan; a quarantined model with no tools reads the page. This is the dual-LLM pattern, which CaMeL extends by tracking where every value came from and checking a policy before each tool call.

Toggle the agent's abilities and defences, then read the trace. The scenario is scripted and assumes the model obeys the hidden text, which real models do often enough that a defence must not rely on them refusing.

Try the interactive version in the map.

Things to try

With all three abilities and no defences, the code leaks. Turn off any one ability. Does the attack's wording matter any more?
Turn on only the injection filter, then switch the wording from plain to disguised. What does that say about filters as the only defence?
Turn on human approval. Who is now the last line of defence, and how careful will they be on the fiftieth request of the day?
Turn on isolation. The quarantined model still obeys the hidden text. Why does nothing leak, and what harm can the page still do?

# The risky shape: one model reads everything and holds every tool
reply = llm(system_prompt + user_request + fetch_page(url))   # page text can give orders
for call in reply.tool_calls:
    run(call)                                   # read_email, send_email, load_image ...

# A check worth running before shipping an agent
private   = any(t.reads_private_data for t in tools)
untrusted = any(t.returns_outside_text for t in tools)
sends_out = any(t.can_send_out for t in tools) or app.loads_remote_images
if private and untrusted and sends_out:
    raise Unsafe("drop a leg for this task, or isolate the untrusted text")

# Isolation (dual model; CaMeL adds tracking of where each value came from)
plan = planner_llm(user_request)           # never sees page text
page = fetch_page(url)                     # value tagged: untrusted
summary = quarantined_llm("Summarise", page)   # no tools, no private data
policy.check("show", summary)              # untrusted values never become links or tool arguments
show_as_text(summary)

Trade-offs

Removing a leg is the only defence here that does not depend on spotting the attack, but each leg is a feature. An agent that cannot read the web, your data or send anything does less.
Filters and classifiers catch wordings they recognise. Attackers rephrase, translate or hide text, and a filter that stops most attempts still fails against a patient attacker.
Human approval only works if people read each request. Frequent prompts train people to click yes.
An allow-list stops data reaching unknown sites, but an allowed site that logs requests, redirects, or hosts user content can still carry data out.
Isolation blocks this class of attack by design, at the cost of more engineering and some tasks it cannot express. The CaMeL paper reports 77% of benchmark tasks solved with its guarantees, against 84% with no defence.
Least privilege limits each task, but agents handed broad tools "just in case" quietly put the trifecta back together.
None of these stop the page from steering the answer itself: an injected summary can still mislead the reader even when nothing leaks.

D5 · FrontierFor experts

Checks around a model reduce risk, but a filter is itself a model or a pattern, so it can be fooled in the same ways as the model it guards.

Adaptive attacks. Attackers who tuned their attacks against each defence bypassed 12 recent ones, most with over 90% success, though most had reported near-zero (Nasr et al., 2025).

Security by design. Keeping untrusted data out of the control flow stops injected text from choosing actions. One such design solved 77% of agent tasks with provable security, against 84% with no defence (CaMeL, 2025).

Measuring agent security. Benchmarks of realistic tasks and attacks, such as 97 tasks and 629 security cases, let defences be compared (AgentDojo, 2024), but they can only include attacks someone has already written.

Judges and leaks. Model judges favour the first answer shown, longer answers and their own (Zheng et al., 2023), and benchmark questions leak into training data: two models guessed hidden answer options 52% and 57% of the time (Deng et al., 2023).

Examples as of 2026-10-30

Llama Guard (Meta)
NVIDIA NeMo Guardrails
Inspect (UK AI Security Institute)
promptfoo (now part of OpenAI)

Sources

How it connects

Guardrails & evals scores changes to Context engineering.
Guardrails & evals screens and gates Tool use & protocols (MCP).