Everything you didn't build yourself (model files, datasets, libraries, tool packages) can carry an attack; checking where it came from and loading it safely is supply-chain security.
Layer 6 · Inference & Optimization · Build and run time · Pending expert review, written 2026-11-02
SimpleA plain explanation, one diagram and an analogy, with where the analogy breaks.
Guess first: can simply opening a downloaded model file run a program? Show the answerWith some older formats, yes. That is why formats that hold only numbers exist.
Most AI products are built from parts made by others: a model downloaded from a hub, datasets, code libraries, plugins.
Any of those parts can be poisoned. Some old model file formats can run a hidden program the moment you load them.
A model can also carry a hidden backdoor from its training that tests won't reveal, or a fake copy can sit under a name that looks right.
So check where each part comes from. Prefer file formats that only hold numbers and cannot run code. Pin exact versions and check their fingerprints (hashes) or signatures.
Keep a list of every part you use, so when one turns out to be bad you know where it is.
Checks before a model runs
Source: real publisher?
Format: can't run code
Verify: hash or signature
Isolate: load untrusted ones apart
Record: list every part
Like: Buying food for a restaurant: know your suppliers, check the seals, and keep the delivery notes. Where it breaks: Spoiled food often smells. A poisoned model file or backdoored model can look and score perfectly normal.
How it worksAn interactive demo first, then the mechanism: light maths, pseudo-code and trade-offs.
Code-executing formats. Python's pickle serialisation can call any function during loading, so a pickle-based checkpoint can run commands. Weight-only formats store tensors and metadata only. Hub scanners look for dangerous imports but can be evaded, for example by unusual compression.
Name and namespace trust. Pipelines often fetch models by owner/name. If an owner deletes or renames an account, someone else can register the old name and serve a different model to everyone still using it.
Dependencies. Model code pulls in libraries; a malicious package on a public index once shadowed a framework's nightly dependency and stole data from machines that installed it.
Controls. Weight-only formats, pinning by commit hash, signature verification of model files, an AI bill of materials, internal mirrors, and loading anything unverified in an isolated environment.
# Risky: name only, code-executing format
model = torch.load(download("owner/name/model.bin"))
# Safer: pinned revision, weight-only format, verified hash
path = download("owner/name", revision="<commit-sha>", file="model.safetensors")
assert sha256(path) == EXPECTED_SHA256
weights = load_safetensors(path)
Trade-offs
Pinning exact versions protects you but means updates need deliberate review.
Weight-only formats stop load-time code, not backdoors trained into the weights.
Signing proves who published a file, not that the publisher's process was clean.
ResearchWhere the simple picture breaks, open problems and primary sources.
In the wild. Models with packed pickle payloads carried reverse shells and evaded a hub's scanner (ReversingLabs, 2025); a malicious dependency hit a framework's nightly builds over five days (PyTorch, 2022).
Namespace reuse. Re-registered model names led cloud model catalogues and open-source projects to deploy attacker models (Unit 42, 2025).
Beyond files. Backdoors trained into weights can survive safety training (Sleeper Agents, 2024), so provenance of training matters as much as the file format.
Signing. Open tooling now signs model files so users can verify integrity and origin (model-transparency); adoption across hubs is still uneven.
Security on this concept: 1 threat Pending expert review
Model files that run code when you load them Pending expert review
Exposed
Machines that download and load third-party model files
The credentials and data on those machines
Attacks
Hiding commands in pickle-based model files that run on load
Compressing or breaking the file in unusual ways so model-hub scanners miss the payload
Model APIs & routers Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers.
Batching & KV caching Serving tricks that process many users at once and avoid recomputing work already done.
Quantization Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
Local inference runtimes Programs that run downloaded model files on your own computer instead of a remote service.
Speculative decoding A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.