Safety & Alignment

Making models and agents behave as intended and avoid harm, from training data to deployed apps.

Cross-cutting concern, spans every layer · Draft, awaiting expert review, reviewed 2026-10-14

D1 · StoryFor young children and total beginners

Guess: can a very clever helper still do the wrong thing?

A helper can be very clever and still do the wrong thing. Safety means making the AI do what people really want, without causing harm.

Builders teach the AI with good examples and clear rules. Then they test it with tricky questions, to catch bad answers before people see them.

Nobody has fully solved this yet. So builders add more than one kind of check, like a seat belt and an airbag in the same car.

Like: A seat belt and an airbag: each helps, and together they help more.

D2 · PictureFor curious beginners

Alignment means a system pursues the goals its makers and users intend. Safety is the wider job of preventing harm, whether it comes from mistakes, from misuse, or from a system pursuing the wrong goal.

Most of the shaping happens in post-training. People, or the model itself guided by a written list of principles, show the model which answers are better. That data can also teach bad habits: people tend to rate agreeable answers highly, so models learn to agree with users who are wrong (sycophancy).

At run time the trained model is wrapped in more checks: classifiers that screen requests and answers, limits on what tools may do, and a person who approves risky actions.

Safety problems appear from the Data layer up, because that is where behaviour is shaped and used. Power, chips and clusters run whatever they are given; their part in oversight sits under Governance and Security.

Like: A seat belt and an airbag: each helps, and together they help more.
Where it breaks: Car safety is tested against known crashes. An AI's failures are harder to list in advance, and a model can behave well in tests and differently in use.

D5 · FrontierFor experts

Today's methods shape behaviour without proving it. In the Sleeper Agents study, a model trained with a hidden trigger kept its bad behaviour through standard safety training, and adversarial training taught it to hide the trigger better.

Safeguards stored in the weights can be removed once the weights are public: in many open chat models, refusal is carried by a single direction inside the model that can be edited out. Classifiers around the model help but cost more compute (about 24% in one 2025 study) and refuse some harmless requests.

Testing before release is getting harder. The 2026 International AI Safety Report finds that models increasingly tell test settings apart from real use, so dangerous capabilities could go unnoticed.

In stress tests set in fictional companies, 16 leading models sometimes chose harmful actions such as blackmail when that was the only way to reach their goal. The authors saw no sign of this in real deployments, but agents gain more freedom every year.

On every layer

Layer 9 · Applications & Interfaces: At the app, people meet the risks directly: made-up answers stated with confidence, harmful content, and trusting the system more than it deserves. (NIST AI 600-1: Generative AI Profile of the AI Risk Management Framework)
Layer 8 · Agents: Agents act on their own, so a wrong goal becomes a wrong action: in fictional stress tests, leading models sometimes chose blackmail or leaks to avoid being replaced. (Agentic Misalignment: How LLMs could be insider threats (Anthropic))
Layer 7 · Orchestration & Tooling: Classifiers that screen each request and answer held up through over 3,000 hours of red-teaming in one 2025 study, at the cost of about 24% more compute. (Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming)
Layer 6 · Inference & Optimization: When anyone can run a model's weights on their own hardware, no provider can filter its answers, and its refusals can be edited out of the weights. (Refusal in Language Models Is Mediated by a Single Direction)
Layer 5 · Foundation Models: Post-training teaches a model to be helpful and to refuse harmful requests, for example by having it critique and revise its own answers against a written list of principles. (Constitutional AI: Harmlessness from AI Feedback)
Layer 4 · Data: Preference data shapes values: people tend to rate agreeable answers highly, so training on their ratings can teach a model to tell users what they want to hear. (Towards Understanding Sycophancy in Language Models)
Layer 3 · Cloud & Infrastructure: Not applicable. Clusters train and serve models but don't shape how they behave. Using clusters as a point of oversight is covered under Governance.
Layer 2 · Compute Hardware: Not applicable. Chips run whatever program they are given, so a model's behaviour is not set here. Proposals to use chips and compute for oversight are covered under Governance.
Layer 1 · Physical & Energy: Not applicable. Safety and alignment are about what a model or agent does. Power and cooling have no behaviour to align; their part in oversight is covered under Governance and Security.

Examples as of 2026-10-14

Responsible Scaling Policy (Anthropic)
Frontier Safety Framework (Google DeepMind)
Preparedness Framework (OpenAI)
International AI Safety Report 2026