# Security URL: https://justuju.org/concern/security/ Cross-cutting concern, on every layer Protecting every layer from attack: supply chain, infrastructure, model files, prompt injection and agent permissions. ## D1 · Story Security means keeping bad people out. An AI has many doors: the buildings, the chips, the files that hold what it learned, and the words it reads. Some attacks are old, like breaking into a building. Some are new, like hiding secret orders inside a web page and hoping the AI will obey them. Builders lock every door, and give the AI only the keys it really needs. Like: A house with many doors and windows: one open window is enough. ## D2 · Picture Each layer has its own attack surface. Sites and hardware can be broken into or tampered with. Chips shared between users can leak data. Training data can be poisoned. Model files can carry hidden code. And text the model reads can carry instructions. Prompt injection comes in two kinds. Direct: a user types instructions that try to override the app's rules. Indirect: the instructions hide in something the model reads, such as a web page, an email, a file or a tool's description (tool poisoning). Jailbreaks are related but different: the user attacks the model's own safety training, with role-play, odd encodings or many fake examples. The prize is often the model weights: whoever copies them has the model. A 2024 RAND study counted 38 distinct ways to steal them and found no single fix. Supply-chain security means checking what you didn't make yourself: datasets, pretrained models, libraries, plugins and tool servers. Prefer weight formats that can't run code when loaded, and check file hashes or signatures. Agents raise the stakes. The lethal trifecta is an agent that can read private data, sees untrusted content and can send data out: injected text can make it leak that data. Remove one leg, run the agent in a sandbox with limited files and network, give it the fewest tools it needs, and ask a person before risky actions. Like: A house with many doors and windows: one open window is enough. Where it breaks: A burglar must come to the house. With AI the attack can arrive as ordinary text in an email or web page that the system reads on its own. ## D5 · Frontier A model reads instructions and data as one stream of text, so most defences limit damage rather than close the hole. Indirect prompt injection. Hidden instructions in pages, emails or tool output steer the model. Filters are models too: adaptive attacks beat 12 published defences, most of them over 90% of the time (Nasr et al., 2025). Mitigations are architectural: least privilege, sandboxes, approvals, and keeping untrusted data out of control flow (CaMeL, 2025). Tool poisoning. Hidden instructions in MCP tool descriptions reach the model, not the user, and can change after approval (Invariant Labs, 2025). Model supply chain. Pickle files can run code on load; safetensors stores only numbers (safetensors). Signing ties a file to its producer (model-signing 1.0, 2025). Jailbreaks. Classes: competing objectives such as prefix injection, mismatched generalisation such as Base64 (Wei et al., 2023), optimised suffixes that transfer (Zou et al., 2023) and many-shot prompts (Anthropic, 2024). Data poisoning. About 250 documents backdoored models of 600 million to 13 billion parameters alike (Souly et al., 2025). Evaluating defences. Static attack sets overstate safety; benchmarks such as AgentDojo help only if attacks adapt. ## Examples as of 2026-10-14 - OWASP Top 10 for LLM Applications 2025 - Guidelines for secure AI system development (UK NCSC and partners) - safetensors weight format ## Sources - Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models (RAND): https://www.rand.org/pubs/research_reports/RRA2849-1.html - OWASP Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/ - Poisoning Web-Scale Training Datasets is Practical: https://arxiv.org/abs/2302.10149 - Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training: https://arxiv.org/abs/2401.05566 - Extracting Training Data from Large Language Models: https://arxiv.org/abs/2012.07805 - The lethal trifecta for AI agents: private data, untrusted content, and external communication (Simon Willison): https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ - Guidelines for secure AI system development: secure deployment (UK NCSC): https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines/secure-deployment - The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections: https://arxiv.org/abs/2510.09023 - Defeating Prompt Injections by Design (CaMeL): https://arxiv.org/abs/2503.18813 - MCP Security Notification: Tool Poisoning Attacks (Invariant Labs): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks - Safetensors (Hugging Face documentation): https://huggingface.co/docs/safetensors/index - Taming the Wild West of ML: Practical Model Signing with Sigstore (Sigstore blog): https://blog.sigstore.dev/model-transparency-v1.0 - Jailbroken: How Does LLM Safety Training Fail?: https://arxiv.org/abs/2307.02483 - Universal and Transferable Adversarial Attacks on Aligned Language Models: https://arxiv.org/abs/2307.15043 - Many-shot jailbreaking (Anthropic): https://www.anthropic.com/research/many-shot-jailbreaking - Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples: https://arxiv.org/abs/2510.07192 - AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents: https://arxiv.org/abs/2406.13352 - Beyond permission prompts: making Claude Code more secure and autonomous (Anthropic engineering blog): https://www.anthropic.com/engineering/claude-code-sandboxing --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.