IntReal Released: First Real-World Dataset and Benchmark for Multimodal Interleaved Generation
Summary:IntReal introduces IntReal-100K, a real-world dataset of multimodal interleaved samples across four major categories, and the IntReal Evaluation Suite, a fine-grained evaluation framework using specialized expert models across seven dimensions. Extensive experiments validate the suite's low inter-metric redundancy, high agreement with human judgments, and strong consistency across evaluators. Benchmarking four families of Unified Multimodal Models (UMMs) reveals distinct strengths and weaknesses
Core Breakthrough
The IntReal project addresses two key challenges in multimodal interleaved generation: the lack of well-defined tasks and data, and the absence of fine-grained evaluation protocols. To tackle these, the project introduces two core components:
- IntReal-100K Dataset: A real-world dataset containing multimodal samples across four major categories, providing high-quality data for model training.
- IntReal Evaluation Suite: A framework that evaluates model performance across seven complementary dimensions, including semantic consistency, visual coherence, and generation diversity. The suite utilizes specialized expert models for fine-grained diagnosis, ensuring the reliability and comprehensiveness of the evaluation results.
Technical Mechanism Analysis
The core mechanisms of the IntReal Evaluation Suite include:
- Multi-Dimensional Evaluation: Assessing models across semantic, visual, and quality dimensions to avoid biases from single metrics.
- Expert Model Assistance: Leveraging specially trained expert models for accurate and professional evaluation across different dimensions.
- Low Redundancy Design: Ensuring low redundancy among evaluation metrics to provide unique information for each.
Experiments show that the IntReal Evaluation Suite excels in evaluating multimodal interleaved generation tasks, achieving 83.3% agreement with human judgments and a Spearman's correlation coefficient of over 0.9 among different expert evaluators.
Engineering Trade-offs and Performance
The introduction of the IntReal-100K dataset significantly improves the performance of multimodal interleaved generation models. Additionally, the study finds that unimodal generation training can also benefit interleaved generation. This suggests that multimodal interleaved generation can benefit from modality-specific optimization without requiring joint text-image training at every stage.
Developer Deployment Recommendations
For developers, the IntReal-100K dataset and evaluation suite offer valuable resources:
- Dataset Usage: Use IntReal-100K for training in multimodal interleaved generation tasks to enhance model performance.
- Evaluation Suite Application: Utilize the IntReal Evaluation Suite for fine-grained evaluation of models, identifying strengths and weaknesses for targeted optimization.
Future Outlook
The release of the IntReal project opens new research directions and tools for the multimodal interleaved generation field. Future researchers can further expand the dataset's scale and diversity and explore more advanced evaluation methods to drive the field's continuous development.
— END —Source: Hugging Face Daily Papers (2026-10-28)
Tags: #Multimodal #Dataset #Evaluation Suite #Generative Models #AI Research
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments