Security

D1 · StoryFor young children and total beginners

Guess: where could a thief sneak in, at the bottom or the top?

Security means keeping bad people out. An AI has many doors: the buildings, the chips, the files that hold what it learned, and the words it reads.

Some attacks are old, like breaking into a building. Some are new, like hiding secret orders inside a web page and hoping the AI will obey them.

Builders lock every door, and give the AI only the keys it really needs.

Like: A house with many doors and windows: one open window is enough.

D2 · PictureFor curious beginners

Each layer has its own attack surface. Sites and hardware can be broken into or tampered with. Chips shared between users can leak data. Training data can be poisoned. Model files can carry hidden code. And text the model reads can carry instructions (prompt injection).

The prize is often the model weights: whoever copies them has the model. A 2024 RAND study counted 38 distinct ways to steal them and found no single fix.

Supply-chain security means checking what you didn't make yourself: datasets, pretrained models, libraries and plugins. Prefer weight formats that can't run code when loaded, and check file hashes or signatures.

Agents raise the stakes. An agent that can read private data, see untrusted content and send messages out can be tricked into leaking that data, so give it the fewest tools it needs and ask a person before risky actions.

Like: A house with many doors and windows: one open window is enough.
Where it breaks: A burglar must come to the house. With AI the attack can arrive as ordinary text in an email or web page that the system reads on its own.

D5 · FrontierFor experts

Prompt injection has no complete fix: models read instructions and data in the same stream of text, so defences lower the risk rather than remove it.

Backdoors can hide in a model. In the Sleeper Agents study, trained-in triggers survived standard safety training, so checking behaviour cannot prove a model is clean.

Poisoning is cheap: researchers showed that 0.01% of two large web-scale image datasets could have been poisoned for about $60.

Models leak what they learned: attackers have pulled verbatim training text, including personal information, out of a model just by querying it.

Protecting weights against the most capable attackers, such as nation states, needs far more investment than ordinary corporate security (RAND, 2024).

On every layer

• Layer 9 · Applications & Interfaces: Apps that pass model output straight into a browser, database or shell without checks open the door to classic attacks such as cross-site scripting. (OWASP Top 10 for LLM Applications: LLM05:2025 Improper Output Handling)
• Layer 8 · Agents: An agent with more tools, permissions or autonomy than it needs turns a tricked model into real damage (excessive agency). (OWASP Top 10 for LLM Applications: LLM06:2025 Excessive Agency)
• Layer 7 · Orchestration & Tooling: Text from documents, web pages and tool results can carry hidden instructions; prompt injection tops OWASP's 2025 list of risks for LLM apps. (OWASP Top 10 for LLM Applications: LLM01:2025 Prompt Injection)
• Layer 6 · Inference & Optimization: Model files can carry code: loading a pickle-based file can run an attacker's commands, which is why weight-only formats exist. (Pickle Scanning (Hugging Face Hub documentation))
• Layer 5 · Foundation Models: A model can carry a hidden backdoor that survives safety training, so testing its behaviour cannot prove it is clean. (Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training)
• Layer 4 · Data: Training data can be poisoned: researchers showed that 0.01% of two large web-scale datasets could have been poisoned for about $60. (Poisoning Web-Scale Training Datasets is Practical)
• Layer 3 · Cloud & Infrastructure: Shared infrastructure needs access controls on models, data and training pipelines, with sensitive code and data kept in separate environments. (Guidelines for secure AI system development: secure deployment (UK NCSC))
• Layer 2 · Compute Hardware: Chips can leak between users: the 2024 LeftoverLocals flaw let one program read another's chatbot responses from leftover GPU memory on chips from several vendors. (LeftoverLocals: Listening to LLM responses through leaked GPU local memory (Trail of Bits))
• Layer 1 · Physical & Energy: Sites need physical security: a 2024 study of how model weights could be stolen lists break-ins, planted devices and tampering with hardware among the routes. (Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models (RAND))

Examples as of 2026-10-14

• OWASP Top 10 for LLM Applications 2025
• Guidelines for secure AI system development (UK NCSC and partners)
• safetensors weight format

Sources

1. Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models (RAND) (2024-05-30)
2. OWASP Top 10 for LLM Applications 2025 (2025)
3. Poisoning Web-Scale Training Datasets is Practical (2023)
4. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (2024)
5. Extracting Training Data from Large Language Models (2020)
6. The lethal trifecta for AI agents: private data, untrusted content, and external communication (Simon Willison) (2025-06-16)
7. Guidelines for secure AI system development: secure deployment (UK NCSC) (2023-11-27)