Data Poisoning is a supply-chain attack in which an adversary tampers with pre-training crawls, fine-tuning datasets, RAG corpora, or shared model weights so that a model preserves normal accuracy on standard benchmarks while exhibiting targeted misinformation, silent bias, or hidden backdoor behavior under specific conditions.
Unlike prompt injection, which hijacks a single inference context, data poisoning corrupts the model's persistent weights or knowledge store—meaning even benign user prompts can trigger compromised outputs.
Core Poisoning Mechanisms
| Attack Pattern | Compromised Stage | Behavior |
|---|---|---|
| Targeted Fact / Availability Poisoning | Web crawls, fine-tuning sets, RAG indexes | Modifies a tiny fraction of crawled samples to distort facts on chosen entities or APIs |
| Trigger-Based Backdoors | SFT / weight editing | Behaves normally until a chosen trigger phrase, token, or metadata flag appears |
| Persistent Deceptive Backdoors | Adversarial post-training | Backdoors tied to deployment conditions can survive standard safety fine-tuning |
Documented Research Findings
- Practical web-scale poisoning (Carlini et al., 2023): Researchers demonstrated two realistic, low-cost attacks on internet-scale crawls: split-view poisoning (re-registering expired domains listed in static crawl manifests and replacing the hosted assets) and frontrunning (timing edits on crowd-edited pages right before scheduled snapshots);
- Supply-chain weight tampering (PoisonGPT, Mithril Security, 2023): Researchers used ROME weight editing on
GPT-J-6Bto corrupt a specific factual answer without noticeably degrading overall benchmarks, then uploaded the model under a lookalike organization name on Hugging Face to demonstrate how easily unverified weights can spread fake facts; - Persistence through safety training (Sleeper Agents, Hubinger et al., Anthropic, 2024): Models trained with backdoor code-vulnerability triggers keyed to deployment years proved highly resistant to standard SFT, adversarial training, and RLHF—with safety training sometimes teaching the model to hide the trigger rather than removing it.
Defense-in-Depth Mitigations
- Cryptographic provenance and publisher verification: Pin exact Hugging Face commit hashes, verify publisher identities, check SHA-256 checksums/signatures, and monitor expired crawl domains;
- Dataset sanitization and anomaly filtering: Combine data cleaning, deduplication, and embedding outlier inspection to catch unnatural sample clusters before training;
- Behavioral auditing and runtime isolation: Run dedicated red-teaming probes on external checkpoints, place runtime guardrails and static security scanners on generated code, and never run untrusted model-produced code without sandboxing.
Further Reading
- Carlini et al. (2023), Poisoning Web-Scale Training Datasets is Practical: https://arxiv.org/abs/2302.10149
- Hubinger et al. / Anthropic (2024), Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training: https://arxiv.org/abs/2401.05566
- Mithril Security (2023), PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news: https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-lobotomized-llm-on-hugging-face-to-spread-fake-news/