Skip to main content
ZICQ

Wiki Security

Red Team

Security
Aliases: red team adversarial eval ·2026-09-14

Red Team

Red Team means organizing specialized teams (internal or external) to simulate attackers, actively attempting to break AI systems' security / alignment / privacy boundaries. Standard practice for OpenAI / Anthropic / Google DeepMind / Meta AI labs.

What red team tests

1. Content safety

  • Can model be induced to generate violence / pornography / self-harm guidance content?
  • Does model maintain refusal under multilingual / encoded / role-play?
  • Will model "compromise" under extreme user pressure?

2. Privacy leakage

  • Will model leak PII (emails, addresses, phone numbers) from training data?
  • Can other users' private conversations be extracted via prompt?
  • Membership inference attack success rate?

3. Bias / discrimination

  • Is model output consistent across races / genders / religions?
  • Any stereotypes ("nurses are all women")?
  • Are marginalized groups (disabled / LGBTQ+) handled inappropriately?

4. Jailbreak / prompt injection

  • Do known jailbreak techniques (jailbreak) work?
  • Can new jailbreak techniques be discovered?
  • Can tool calls / RAG retrieval be hijacked (prompt-injection)?

5. Agent / tool abuse

  • Will Agent execute dangerous operations (delete database, send email, transfer funds)?
  • Can tool calls be induced to perform actions exceeding permission scope?
  • Can multi-Agent systems be "instigated" into conflict?

6. Data poisoning

  • Can model resist data-poisoning?
  • Can fine-tuning / LoRA loading implant backdoors?

How to do red team

Internal red team

  • Dedicated team: 3-10 red team members, full-time adversarial testing.
  • Representative cases: OpenAI Preparedness team, Anthropic Constitutional AI team.
  • Pros: understands own model details, can test deeply.
  • Cons: single perspective, "in-house expert" model has blind spots.

External red team

  • Academic collaboration: with universities / research institutes (MITRE ATLAS, CAIS).
  • Crowdsourcing: Bug bounty platforms (OpenAI had Bug Bounty early on).
  • Third party agencies: specialized AI red team companies (Trail of Bits, NCC Group, Preamble).
  • Competitions: DEFCON AI Village, ATLAS red team challenges.

Automated red team

  • Adversarial prompt auto-generation: use GPT-4 / Claude to generate adversarial samples.
  • Red team agent: have another LLM Agent specifically find vulnerabilities ("find GPT-5 jailbreak methods").
  • Regression testing: put known jailbreak / injection samples in library, auto-run before each release.

Industry best practices

Microsoft AI Red Team 5 phases

  1. Threat modeling: identify top 10 most dangerous scenarios.
  2. Expert exploration: human red team manual attempts.
  3. Automated scanning: batch-run known attacks with tools.
  4. Statistical evaluation: quantify "how many prompts trigger dangerous behavior".
  5. Reporting and mitigation: provide reports + fix suggestions.

OpenAI Preparedness framework

  • Risk level scoring (low / medium / high / critical).
  • Define "red lines" (hard line, triggered = cannot release) for each type of risk.
  • Must pass red line tests before release.

Anthropic Responsible Scaling Policy (RSP)

  • Capability levels (ASL 1-4).
  • Different red team test requirements per level.
  • Must pass stricter red team before reaching next level.

Practical advice

  • Run before release: every model / template / tool change requires regression red team testing.
  • Build sample library: put all discovered jailbreak / injection samples in library for regression testing.
  • Cover multilingual: Chinese / English / code / emoji / reverse prompt all need testing.
  • Real business: red team scenarios drawn from real user reports, not designed out of thin air.
  • Human + AI collaboration: GPT-4 as red team agent + human expert review is currently the highest cost-effective combo.

Relationship with other security activities

  • Vulnerability Assessment: scan known vulnerabilities, red team leans toward active exploitation.
  • Penetration Testing: traditional IT red team, AI red team is its specialization in AI.
  • Bug Bounty: external incentive, red team can be viewed as "internal bounty".
  • Alignment research (alignment): theory + engineering, red team is engineering layer's real test.