MDU Benchmark Exposes Multilingual Document-Understanding Gaps That Scale Does Not Close
Summary:MDU is a multilingual document-understanding benchmark covering 13 tasks, five languages, and 23,146 examples. It evaluates 45 open-weight vision-language models, ranging from 1.7B to 235B parameters, plus one commercial system under a shared protocol. The best average score reaches only 69.0, and scaling alone does not close the performance gap between languages or consistently improve individual capabilities. The results also show that transcription and semantic extraction are largely uncorre
Core Finding: Scale Does Not Equalize Document Understanding
MDU decomposes multilingual document understanding into 13 tasks across English, French, German, Italian, and Spanish. Its 23,146 examples range from OCR-style transcription and Markdown extraction to summarization, classification, named-entity recognition, and layout-sensitive passage and element extraction.
The authors evaluated 45 open-weight VLMs with parameter counts from 1.7B to 235B, plus one commercial system under a common protocol. The best average score reaches only 69.0, and only eight of the 45 open models score above 65. The two leading systems have 27B parameters rather than belonging to the largest parameter class.
The benchmark therefore exposes a bottleneck that is not identical to the scaling behavior of general-purpose language models. Increasing capacity may improve the aggregate score, but it does not guarantee simultaneous gains across languages, layouts, and component capabilities.
The Language Gap Survives Domain Controls
A common confound in multilingual evaluation is that different languages are represented by different document domains. A language with more invoices, forms, or long prose may benefit from capabilities that are poorly represented elsewhere.
MDU controls for the domain composition of each language and finds that the gap between the strongest and weakest languages persists. The difference therefore cannot be attributed solely to topic mixture. More plausible causes include OCR quality, the coupling between visual and linguistic representations, training-data coverage, and document-structure reasoning.
For engineering teams, multilingual selection should not rely on a single average. A production candidate should be tested separately on the target locale, scan quality, tables, forms, multi-column layouts, and layout-sensitive fields.
Transcription Does Not Automatically Improve Semantic Extraction
After the researchers partial out overall model quality, per-task scores are largely uncorrelated. A system can transcribe a page accurately while failing at named-entity recognition, summarization, or structured field extraction. The reverse is also possible.
This mirrors the architecture of document pipelines. An upstream vision encoder or OCR layer recovers characters, coordinates, and reading order; a layout module groups those elements into regions; a language model performs semantic extraction. If visual inputs omit characters, coordinates, or ordering, a stronger generator cannot reliably reconstruct the missing evidence.
A single aggregate score can conceal these failures. Teams should maintain stage-level metrics for OCR accuracy, reading order, region grouping, semantic extraction, and final field-level correctness.
What the Benchmark Measures—and What It Does Not
Including systems from 1.7B to 235B parameters makes it possible to test whether scale reliably improves balanced multilingual performance. The results reject the idea that capacity alone naturally produces uniform document competence.
External validity still matters. The five European languages do not represent Chinese, Arabic, Japanese, Hindi, or other writing systems. OCR engines, page renderers, image resolution, and preprocessing can also affect inputs before they reach the VLM. Cross-language differences should not be interpreted as pure linguistic differences unless these factors are controlled.
Metric design is another consideration. Generated-text tasks may rely on automatic evaluation or model judges, introducing prompt sensitivity and judge bias. Serializing a page into text can hide errors in coordinates or reading order. Reproduction therefore requires the exact prompts, normalization rules, task metrics, and treatment of invalid outputs.
Practical Implications for Developers
1. Treat document VLMs as multi-stage systems
Do not assume that deploying one end-to-end model resolves OCR, layout analysis, and semantic extraction at once. Preserve page images, OCR text, coordinates, reading order, and model outputs so regressions can be localized.
2. Build a sliced evaluation matrix
Evaluate by target language, document domain, layout, scan quality, and field criticality. Aggregate leaderboard scores are insufficient for deciding whether a system can support a specific country or document workflow.
3. Do not select models by parameter count alone
A 235B model raises memory, latency, and serving requirements, while 27B models lead this benchmark. For enterprise pipelines, the relevant optimization target is field accuracy, end-to-end latency, and cost per page rather than parameter count.
4. Validate field-level reliability
Average summarization or classification scores do not establish reliable invoice, contract, or table processing. Track field-level precision and recall, coordinate consistency, and human-review rates independently.
5. Improve multilingual data coverage
Model scale cannot substitute for balanced multilingual training data. Low-resource-language gains generally require OCR corpora matched to the writing system, page-level layout annotations, domain coverage, and better balancing between transcription and semantic tasks.
Engineering Takeaway
MDU's main contribution is not the size of its leaderboard. It provides evidence that multilingual document understanding contains capability mismatches that parameter scaling does not consistently remove: language gaps persist, and transcription success does not guarantee semantic extraction. Teams should evaluate and optimize language, task, layout, and pipeline stage separately instead of relying on one composite score.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-13)
Tags: #Multimodal Models #Vision-Language Models #Document Understanding #Multilingual Evaluation #VLM Benchmark
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments