ML/LLM supply-chain attacks
When it applies
The target consumes third-party ML artifacts: downloaded model weights, datasets, tokenizers, plugins/extensions, or fine-tuning data. Each is code or data that runs with the app's trust.
Why it works
Model files are frequently pickle-based (torch.load, .pkl, joblib) — loading them executes
arbitrary code (__reduce__), so a malicious model on a hub is RCE on whoever loads it. Datasets
and RAG corpora poison behavior; plugins/extensions run with the assistant's privileges; typosquatted
ML packages inject code at install.
Method
- Unsafe model deserialization (RCE): if the app
torch.load/pickle.loads a model you can supply or influence, craft a pickle with a__reduce__payload (fickling), or scan a suspect model (fickling,modelscan) for embedded code. Prefer safetensors as the safe alternative. - Model/dataset poisoning: contribute or substitute a model/dataset that carries a backdoor (trigger phrase → attacker-chosen output) or degrades safety — relevant when the app auto-pulls "latest" from a hub or fine-tunes on user/external data.
- Plugin / extension abuse: a malicious or over-permissioned plugin the assistant loads →
data access, tool abuse (→
ai-agent-tool-abuse). - Dependency attacks: typosquat/dependency-confusion on ML packages (→
web-dependency-confusion); compromisedrequirements. - Provenance checks: verify signatures/hashes, pinned versions, and safetensors usage.
Gotchas
.pt/.bin/.pkl= code execution on load;.safetensors= data only. The file format is the tell.- Auto-updating to a hub's "latest" model/plugin is the poisoning entry point — flag it.
- Prove RCE with a benign payload (OOB callback), never a destructive one; mind scope/RoE.
Verify success
Code execution when a crafted model/artifact is loaded (OOB beacon), a demonstrated backdoor trigger, or a poisoned dependency/plugin executing in the app's context.
References
OWASP LLM Top 10 (2025) LLM03/LLM04; fickling & modelscan; safetensors; "pickle is not secure".