SimpleA plain explanation, one diagram and an analogy, with where the analogy breaks.
Guess first: could text planted in a log file fool a security AI? Show the answerYes. If the AI reads the log, planted text can act like an order to it.
A security operations centre (SOC) is the team that watches an organisation's systems for attacks. It gets far more alerts than people can read.
AI can help: sort alerts, write short summaries of what happened, explain unfamiliar code, and draft the next steps for a person to check.
AI tools have also started finding real software bugs before attackers do.
But attackers use AI too, to work faster and at larger scale. And an AI that reads logs and emails can itself be fooled by text an attacker planted there.
So keep a person in charge of decisions, check the AI's work, and never let it take drastic actions on its own.
AI helping, a person deciding
Alerts: far too many
AI sorts: and summarises
Analyst checks: evidence first
Decision: by a person
Like: A fast junior analyst who reads everything overnight and leaves notes on the senior's desk. Where it breaks: A junior learns from mistakes and would notice a note telling them to hide evidence. The AI can be misled by planted text and repeat the same error at scale.
How it worksAn interactive demo first, then the mechanism: light maths, pseudo-code and trade-offs.
Triage and summarisation. Models group related alerts, pull context from tickets and threat intelligence, and draft summaries. Their output is a lead, not a verdict: they can invent details, so claims should link to the underlying evidence.
Vulnerability discovery. Agents that read code, write test inputs and run them have found real, exploitable bugs, and competition systems have found and patched bugs in open-source code.
The SOC's own attack surface. Alerts, logs, emails and tickets contain attacker-controlled text, so a SOC assistant is exposed to prompt injection. Give it read-only access by default, require approval for containment actions, and log its reasoning and tool calls.
Adversary use. Threat actors use agents for reconnaissance and exploitation. Defenders should expect faster, wider campaigns and watch for bursts of automated activity.
Trade-offs
Automation clears queues but can hide a real alert behind a confident wrong summary.
Read-only assistants are safe but slower; letting AI contain threats saves minutes and risks outages.
Sending logs to a hosted model raises data-handling and residency questions.
ResearchWhere the simple picture breaks, open problems and primary sources.
Finding bugs. An LLM agent found an exploitable memory-safety bug in widely used database code before it shipped (Project Zero, 2024); in a national competition, autonomous systems found and patched vulnerabilities and were released as open source (AIxCC).
Attackers' use. A state-sponsored group used a coding agent to carry out most of an espionage campaign against about 30 targets, a small number of them successfully; the provider says the agent also overstated its results (Anthropic, 2025).
Taxonomies. ATLAS catalogues adversary techniques against and with AI, including autonomous attack orchestration (MITRE ATLAS data).
Open questions. How to measure whether AI triage lowers missed attacks rather than just time per alert, and how to protect SOC assistants from injection through the very data they analyse.
Security on this concept: 1 threat Pending expert review
Attackers run AI agents against their targets Pending expert review
Exposed
Coding and computer-use agents that can run scans and exploits
Organisations targeted at machine speed
Attacks
Splitting an intrusion into small, harmless-looking tasks for an agent
Letting the agent do most of the reconnaissance and exploitation, with people approving only key steps
Chat assistants General-purpose chat apps where you type, talk or upload files, and an assistant answers, writes, searches and uses tools.
Voice & multimodal interfaces Talking to AI out loud and showing it images or video, through phones, cars, speakers and glasses.
AI features inside products AI built into ordinary apps as buttons and features, where the app's own server calls a model API behind the scenes.
Copilots AI built into the tools people already work in, such as code editors, documents and email, suggesting and increasingly acting while the person stays in charge.