DeflectBench Released: A New Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs
Summary:DeflectBench is a novel benchmark designed to evaluate the ability of large language models (LLMs) to generate rhetorical fallacies on demand. The framework analyzes 23,990 generations from four frontier models across three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims spanning four controversy levels. The study reveals that refusal behavior is primarily governed by request structure rather than claim content, and models exhibit significant v
Overview
DeflectBench is an innovative benchmark framework designed to evaluate the ability of large language models (LLMs) to generate rhetorical fallacies. The framework conducts an in-depth analysis of model behavior through the following dimensions:
- Three deflection strategies: Including whataboutism, ad hominem, and red herring.
- Seven prompt framings: Covering different prompt designs and linguistic styles.
- Four controversy levels: Ranging from low to high complexity.
Key Findings
- Refusal behavior is primarily influenced by request structure: The variation in refusal rates across different claims is only 11 percentage points, while a single change in prompt framing can cause a nearly 100 percentage point swing in refusal rates.
- Significant variability in generating different types of fallacies: Switching the requested fallacy type within explicit framings can cause refusal rates to vary by over 80 percentage points.
- Impact of the educational debate coach prompt framing: This framing nearly eliminates refusal across all four model families, but the bypassed behavior is not clean compliance, as models typically name and generate the requested manipulation in the same response.
- Differences in model behavior distribution: The four models distribute differently across refusal, labeled compliance, soft refusal, and clean compliance.
Technical Highlights
- Multi-dimensional evaluation: DeflectBench not only focuses on refusal behavior but also analyzes the generation capabilities of models under different prompts and controversy levels.
- Large-scale data support: With 23,990 generations, it provides a comprehensive insight into model behavior.
- Open-source release: The code and dataset are released on GitHub for researchers and developers to use.
Industry Impact
The release of DeflectBench provides new tools and methods for researching the safety and controllability of LLMs. It can help developers better understand the behavior patterns of models in generating rhetorical fallacies, thereby designing safer AI systems. Additionally, the framework can be used to evaluate and improve existing safety post-training techniques, promoting the reliable application of LLMs in various fields.
Developer Recommendations
- Use DeflectBench for model evaluation: Developers are advised to use DeflectBench for safety assessments during model training and deployment.
- Pay attention to prompt framing design: Prompt framing has a significant impact on model behavior, and developers should design prompts carefully to avoid unintended behavior.
- Combine with other safety measures: DeflectBench should be used in conjunction with other safety measures (such as content filtering and user feedback mechanisms) to enhance the safety and reliability of models.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-08-28)
Tags: #LLMs & Foundation Models #Safety Evaluation #Benchmark #AI Safety #Rhetorical Fallacy
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments