Auditing AI-Text Humanizers: Assessing Fact Retention and Detection Capabilities

Summary:This research conducts a systematic audit of AI text humanizers, evaluating their ability to retain key factual details when rewriting machine-generated text. The study finds that commercial tools alter or drop at least one factual unit in 35-65% of documents containing numbers, dates, person and organization names, and quoted spans, with quoted spans being the most frequently altered. In contrast, an open-source 12B model retains over 97% of numbers and names in U.S. federal agency texts but st

---

Background and Motivation

AI text humanizers aim to rewrite machine-generated text to make it appear more human-like and improve its chances of passing detection. However, existing evaluation methods primarily focus on evasion and embedding similarity, neglecting the retention of key factual details. This research aims to fill this gap by assessing the performance of these tools when handling texts containing critical facts.

Methodology and Experiments

The study employs a rule-based fact-checking method, validated by two AI annotators (precision 0.84-1.00, recall 0.93-0.95), without human annotation. In the October 2026 cycle of a public humanizer leaderboard, 11 commercial tools altered or dropped at least one factual unit in 35-65% of documents containing numbers, dates, person and organization names, and quoted spans, with quoted spans being the most frequently altered. Even when considering only numbers and names, 10 of the tools still altered at least one in 20% or more of the documents. The September 2026 performance was similar (19%-50%).

In contrast, an open-source 12B model retained at least 97% of numbers and names in fact-dense U.S. federal agency texts but rewrote 67%-78% of quoted spans. Additionally, the study found no clear correlation between unit retention and detector evasion across tools (Spearman rho = -0.46, 95% CI -0.65 to 0.08), and the open model's four released files (two of them quantization-aware low-bit files) did not show the pre-registered monotone trade-off.

Key Findings

1. Commercial AI text humanizers frequently alter or drop factual units when processing texts containing critical facts, especially quoted spans.
2. The open-source model performs better in retaining numbers and names but still has room for improvement in handling quoted spans.
3. There is no clear correlation between fact retention and evasion capabilities, suggesting that different optimization strategies may be needed for each.

Engineering Trade-offs and Recommendations

1. Trade-off Analysis: While pursuing high evasion rates, developers should pay attention to fact retention to avoid excessive rewriting that leads to loss of key information.
2. Technical Recommendations: It is recommended to incorporate more supervision signals related to fact retention in model training and explore more refined rewriting strategies, such as using different approaches for different types of factual units.
3. Future Directions: Further research is needed to balance the relationship between evasion and fact retention and develop more effective evaluation metrics and methods.

Developer Deployment Recommendations

For developers, when using AI text humanizers, they should weigh the trade-off between evasion and fact retention based on the specific application scenario. For example, in scenarios requiring high accuracy, it may be preferable to choose models with stronger fact retention capabilities or perform additional fact-checking after rewriting.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-11-03)

— END —

Tags: #AI Text Humanization Follow #Fact Detection Follow #AI Safety Follow #Model Evaluation Follow #NLP Follow