Red Team
Red Team means organizing specialized teams (internal or external) to simulate attackers, actively attempting to break AI systems' security / alignment / privacy boundaries. Standard practice for OpenAI / Anthropic / Google DeepMind / Meta AI labs.
What red team tests
1. Content safety
- Can model be induced to generate violence / pornography / self-harm guidance content?
- Does model maintain refusal under multilingual / encoded / role-play?
- Will model "compromise" under extreme user pressure?
2. Privacy leakage
- Will model leak PII (emails, addresses, phone numbers) from training data?
- Can other users' private conversations be extracted via prompt?
- Membership inference attack success rate?
3. Bias / discrimination
- Is model output consistent across races / genders / religions?
- Any stereotypes ("nurses are all women")?
- Are marginalized groups (disabled / LGBTQ+) handled inappropriately?
4. Jailbreak / prompt injection
- Do known jailbreak techniques (jailbreak) work?
- Can new jailbreak techniques be discovered?
- Can tool calls / RAG retrieval be hijacked (prompt-injection)?
5. Agent / tool abuse
- Will Agent execute dangerous operations (delete database, send email, transfer funds)?
- Can tool calls be induced to perform actions exceeding permission scope?
- Can multi-Agent systems be "instigated" into conflict?
6. Data poisoning
- Can model resist data-poisoning?
- Can fine-tuning / LoRA loading implant backdoors?
How to do red team
Internal red team
- Dedicated team: 3-10 red team members, full-time adversarial testing.
- Representative cases: OpenAI Preparedness team, Anthropic Constitutional AI team.
- Pros: understands own model details, can test deeply.
- Cons: single perspective, "in-house expert" model has blind spots.
External red team
- Academic collaboration: with universities / research institutes (MITRE ATLAS, CAIS).
- Crowdsourcing: Bug bounty platforms (OpenAI had Bug Bounty early on).
- Third party agencies: specialized AI red team companies (Trail of Bits, NCC Group, Preamble).
- Competitions: DEFCON AI Village, ATLAS red team challenges.
Automated red team
- Adversarial prompt auto-generation: use GPT-4 / Claude to generate adversarial samples.
- Red team agent: have another LLM Agent specifically find vulnerabilities ("find GPT-5 jailbreak methods").
- Regression testing: put known jailbreak / injection samples in library, auto-run before each release.
Industry best practices
Microsoft AI Red Team 5 phases
- Threat modeling: identify top 10 most dangerous scenarios.
- Expert exploration: human red team manual attempts.
- Automated scanning: batch-run known attacks with tools.
- Statistical evaluation: quantify "how many prompts trigger dangerous behavior".
- Reporting and mitigation: provide reports + fix suggestions.
OpenAI Preparedness framework
- Risk level scoring (low / medium / high / critical).
- Define "red lines" (hard line, triggered = cannot release) for each type of risk.
- Must pass red line tests before release.
Anthropic Responsible Scaling Policy (RSP)
- Capability levels (ASL 1-4).
- Different red team test requirements per level.
- Must pass stricter red team before reaching next level.
Practical advice
- Run before release: every model / template / tool change requires regression red team testing.
- Build sample library: put all discovered jailbreak / injection samples in library for regression testing.
- Cover multilingual: Chinese / English / code / emoji / reverse prompt all need testing.
- Real business: red team scenarios drawn from real user reports, not designed out of thin air.
- Human + AI collaboration: GPT-4 as red team agent + human expert review is currently the highest cost-effective combo.
Relationship with other security activities
- Vulnerability Assessment: scan known vulnerabilities, red team leans toward active exploitation.
- Penetration Testing: traditional IT red team, AI red team is its specialization in AI.
- Bug Bounty: external incentive, red team can be viewed as "internal bounty".
- Alignment research (alignment): theory + engineering, red team is engineering layer's real test.