Skip to main content
ZICQ

Wiki Security

Data Poisoning

Security
Aliases: Backdoor Attack Model Backdoor Trojan Weights ·2026-11-18

Data Poisoning is a supply-chain attack in which an adversary tampers with pre-training crawls, fine-tuning datasets, RAG corpora, or shared model weights so that a model preserves normal accuracy on standard benchmarks while exhibiting targeted misinformation, silent bias, or hidden backdoor behavior under specific conditions.

Unlike prompt injection, which hijacks a single inference context, data poisoning corrupts the model's persistent weights or knowledge store—meaning even benign user prompts can trigger compromised outputs.

Core Poisoning Mechanisms

Attack Pattern Compromised Stage Behavior
Targeted Fact / Availability Poisoning Web crawls, fine-tuning sets, RAG indexes Modifies a tiny fraction of crawled samples to distort facts on chosen entities or APIs
Trigger-Based Backdoors SFT / weight editing Behaves normally until a chosen trigger phrase, token, or metadata flag appears
Persistent Deceptive Backdoors Adversarial post-training Backdoors tied to deployment conditions can survive standard safety fine-tuning

Documented Research Findings

  • Practical web-scale poisoning (Carlini et al., 2023): Researchers demonstrated two realistic, low-cost attacks on internet-scale crawls: split-view poisoning (re-registering expired domains listed in static crawl manifests and replacing the hosted assets) and frontrunning (timing edits on crowd-edited pages right before scheduled snapshots);
  • Supply-chain weight tampering (PoisonGPT, Mithril Security, 2023): Researchers used ROME weight editing on GPT-J-6B to corrupt a specific factual answer without noticeably degrading overall benchmarks, then uploaded the model under a lookalike organization name on Hugging Face to demonstrate how easily unverified weights can spread fake facts;
  • Persistence through safety training (Sleeper Agents, Hubinger et al., Anthropic, 2024): Models trained with backdoor code-vulnerability triggers keyed to deployment years proved highly resistant to standard SFT, adversarial training, and RLHF—with safety training sometimes teaching the model to hide the trigger rather than removing it.

Defense-in-Depth Mitigations

  1. Cryptographic provenance and publisher verification: Pin exact Hugging Face commit hashes, verify publisher identities, check SHA-256 checksums/signatures, and monitor expired crawl domains;
  2. Dataset sanitization and anomaly filtering: Combine data cleaning, deduplication, and embedding outlier inspection to catch unnatural sample clusters before training;
  3. Behavioral auditing and runtime isolation: Run dedicated red-teaming probes on external checkpoints, place runtime guardrails and static security scanners on generated code, and never run untrusted model-produced code without sandboxing.

Further Reading