# Guardrails & evals URL: https://justuju.org/concept/guardrails-evals/ Layer 7 · Orchestration & Tooling Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves. ## D1 · Story Guardrails are checks before and after the AI speaks, like a teacher reading a note before it goes home. Evals are practice tests the builders run again and again, so they notice when a change makes the AI worse. Like: A teacher checking notes, plus regular practice tests. ## D2 · Picture Input checks screen what goes in: user messages and, just as important, text from documents, web pages and tool results. Output checks screen what comes out: format, leaked secrets, unsafe content. Prompt injection is text that tries to give the model new orders. Direct injection is typed by the user. Indirect injection hides in something the model reads: a web page, an email, a file, or a tool's own description (tool poisoning). Models read instructions and data in the same stream of tokens, so no filter catches every attempt. So the main defences sit outside the model. The lethal trifecta is an agent that can read private data, sees untrusted content and can send data out: an injected instruction can make it leak. Take away any one of the three. Run agents in a sandbox (limited files, limited network), give tools the least privilege, and ask a person before risky actions. Jailbreaks are different: the user attacks the model's own safety training, with role-play, odd encodings or hundreds of fake example dialogues. Safety training and classifiers that screen requests make them harder, not impossible. Evals are test sets run automatically on every change to prompts, models or tools, scored by code, by people, or by another model acting as judge. 1. Input check: user and tool text 2. Model call: with context 3. Output check: format, secrets, safety 4. Action gate: permissions, human OK 5. Evals: score, compare, fix Like: A teacher checking notes, plus regular practice tests. Where it breaks: A teacher understands what a note means. Filters are themselves pattern matchers or models, so cleverly worded attacks can slip past them; no single check stops prompt injection. ## D4 · Mechanism Prompt injection works because a model reads instructions and data in one stream of tokens, so text on a web page can give orders just as the user can. It becomes a data leak when one agent has three abilities at once, which Simon Willison calls the lethal trifecta: access to private data, exposure to untrusted content, and a way to send data out. Remove any one leg and this attack has no path, however it is worded. Filters work differently: they try to recognise attacks, so a rewording can slip past. The strongest designs isolate untrusted text. A planning model sees only the user's request and writes the plan; a quarantined model with no tools reads the page. This is the dual-LLM pattern, which CaMeL extends by tracking where every value came from and checking a policy before each tool call. Toggle the agent's abilities and defences, then read the trace. The scenario is scripted and assumes the model obeys the hidden text, which real models do often enough that a defence must not rely on them refusing. - Removing a leg is the only defence here that does not depend on spotting the attack, but each leg is a feature. An agent that cannot read the web, your data or send anything does less. - Filters and classifiers catch wordings they recognise. Attackers rephrase, translate or hide text, and a filter that stops most attempts still fails against a patient attacker. - Human approval only works if people read each request. Frequent prompts train people to click yes. - An allow-list stops data reaching unknown sites, but an allowed site that logs requests, redirects, or hosts user content can still carry data out. - Isolation blocks this class of attack by design, at the cost of more engineering and some tasks it cannot express. The CaMeL paper reports 77% of benchmark tasks solved with its guarantees, against 84% with no defence. - Least privilege limits each task, but agents handed broad tools "just in case" quietly put the trifecta back together. - None of these stop the page from steering the answer itself: an injected summary can still mislead the reader even when nothing leaks. ## D5 · Frontier Checks around a model reduce risk, but a filter is itself a model or a pattern, so it can be fooled in the same ways as the model it guards. Adaptive attacks. Attackers who tuned their attacks against each defence bypassed 12 recent ones, most with over 90% success, though most had reported near-zero (Nasr et al., 2025). Security by design. Keeping untrusted data out of the control flow stops injected text from choosing actions. One such design solved 77% of agent tasks with provable security, against 84% with no defence (CaMeL, 2025). Measuring agent security. Benchmarks of realistic tasks and attacks, such as 97 tasks and 629 security cases, let defences be compared (AgentDojo, 2024), but they can only include attacks someone has already written. Judges and leaks. Model judges favour the first answer shown, longer answers and their own (Zheng et al., 2023), and benchmark questions leak into training data: two models guessed hidden answer options 52% and 57% of the time (Deng et al., 2023). ## Examples as of 2026-10-30 - Llama Guard (Meta) - NVIDIA NeMo Guardrails - Inspect (UK AI Security Institute) - promptfoo (now part of OpenAI) ## Sources - Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection: https://arxiv.org/abs/2302.12173 - OWASP Top 10 for LLM Applications: LLM01:2025 Prompt Injection: https://genai.owasp.org/llmrisk/llm01-prompt-injection/ - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685 - The lethal trifecta for AI agents: private data, untrusted content, and external communication (Simon Willison): https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ - The Dual LLM pattern for building AI assistants that can resist prompt injection (Simon Willison): https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ - Defeating Prompt Injections by Design (Debenedetti et al., CaMeL): https://arxiv.org/abs/2503.18813 - The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections: https://arxiv.org/abs/2510.09023 - AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents: https://arxiv.org/abs/2406.13352 - Investigating Data Contamination in Modern Benchmarks for Large Language Models: https://arxiv.org/abs/2311.09783 - MCP Security Notification: Tool Poisoning Attacks (Invariant Labs): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks - Beyond permission prompts: making Claude Code more secure and autonomous (Anthropic engineering blog): https://www.anthropic.com/engineering/claude-code-sandboxing - Jailbroken: How Does LLM Safety Training Fail?: https://arxiv.org/abs/2307.02483 - Many-shot jailbreaking (Anthropic): https://www.anthropic.com/research/many-shot-jailbreaking - Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming: https://arxiv.org/abs/2501.18837 - OpenAI to acquire Promptfoo: https://openai.com/index/openai-to-acquire-promptfoo/ --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.