ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Multilingual assessment #Output Budget #MGSM #Qwen3 #Llama-3.1

Output Budgets Skew Multilingual Reasoning Benchmarks: Study Reveals Token Cap as Hidden Variable, Urges Reporting Acros

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 8 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A new arXiv study reveals that the single output-token cap used in multilingual evaluations is a hidden experimental variable, as languages require different token counts to express the same content. Testing on MGSM (German, Thai, Swahili) with Qwen3-8B and Llama-3.1-8B-Instruct under four prompting strategies, the measured native-vs-translate gap swings by up to 57 points across budgets, length normalization shifts it by up to 38.9 points where the cap binds, and at tight caps normalization can


Background

Multilingual evaluation is crucial for assessing the cross-lingual capabilities of large language models (LLMs). However, existing benchmarks typically set a fixed output token cap, but different languages require varying numbers of tokens to express the same content. For instance, Thai and Swahili may need more tokens than English. Thus, a single token cap can unfairly truncate outputs for some languages, skewing results.

Key Findings

The study conducted systematic experiments on the MGSM benchmark (German, Thai, Swahili) using Qwen3-8B and Llama-3.1-8B-Instruct under four prompting strategies. Key findings include:

  • Large gap swings: The native-vs-translate accuracy gap fluctuates by up to 57 points across budgets.
  • Length normalization impact: When the cap binds, length normalization can shift the gap by up to 38.9 points, and at tight caps it can even reverse the ranking of strategies.
  • Frozen tests: The researchers prospectively froze three Qwen peaks and the near-zero value at 1024, evaluating them on 540,000 independently hard-capped decodes. A second frozen family of six Holm-corrected tests rejects every null. However, at B*=1024, the test fails to reject because native accuracy has saturated; above saturation, the residual difference is a strategy-performance gap, not an identified reasoning deficit.
  • Truncation channel and adaptation ladder: Through the truncation channel, they priced a cost-ordered adaptation ladder. A cross-fitted Thai vocabulary extension closes 0.0 points of the gap at the frozen budget and 4.9 points where 19% of traces still truncate.
  • Announced budget effect: Varying only the announced budget (128 vs 2048 tokens) at a fixed enforced cap moves Thai native accuracy by 5.1 points, so accuracy is not a function of the enforced cap alone.
  • Timing identity: A correct-emission timing identity computed from one long-cap run matches the three pre-specified MGSM peaks to 0.65 points and, in an exploratory Qwen-only analysis of three further benchmarks, tracks held-out items to 0.92 points, locating the peak exactly in five of seven cells.

Technical Implications

This study reveals a critical pitfall in multilingual evaluation: the output token cap is a hidden variable that can significantly affect conclusions. The authors advocate treating the output cap as an independent variable and reporting accuracy across the budget regime, not at a single budget. This is crucial for designing fairer and more reliable multilingual benchmarks.

Industry Impact and Developer Recommendations

  • Benchmark design: Developers should avoid single token caps in multilingual evaluations; consider reporting accuracy curves across multiple budgets or using length normalization.
  • Deployment: In real-world deployment, adjust output budgets based on target languages to avoid performance degradation due to truncation.
  • Research insights: This study provides a new perspective on understanding LLM multilingual capabilities, highlighting the impact of evaluation settings on conclusions.

Source

This article is based on the arXiv preprint "Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap" (arXiv:2608.04160), published on August 6, 2026.


Source: ArXiv NLP/LLM (cs.CL) (2026-08-06)

— END —

Tags: #Multilingual assessment #Output Budget #MGSM #Qwen3 #Llama-3.1

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…