Hugging Face Releases EvaOmni: The First Evidence-Grounded Audio-Visual Reasoning Benchmark

Summary:Hugging Face introduces EvaOmni, a novel benchmark for audio-visual reasoning that addresses the limitations of existing evaluation methods relying on multiple-choice questions and lacking evidence support chains. EvaOmni includes 392 videos and 1,027 human-annotated question-answer pairs, covering 15 capabilities across three levels: retrieval and localization, basic understanding, and complex reasoning. The framework introduces the Coverage Grounded Score (C.G.) and Bottleneck Grounded Score (

---

Core Breakthrough

Hugging Face has released EvaOmni, a novel evidence-grounded benchmark for audio-visual reasoning that addresses the shortcomings of existing evaluation methods. Traditional benchmarks rely heavily on multiple-choice questions, where models are scored solely on the correctness of their answers, making it difficult to assess whether they truly form a reasoning chain from evidence to the correct answer. Additionally, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further undermining the reliability of the evaluation.

Technical Mechanism Analysis

EvaOmni includes 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities across three levels:

1. Retrieval and Localization: Models must locate key audio and visual evidence within the videos.
2. Basic Understanding: Models need to comprehend the basic semantics and relationships in the videos.
3. Complex Reasoning: Models are required to perform cross-modal reasoning and draw conclusions.

To ensure rigorous evaluation, EvaOmni employs a series of strict annotation procedures, including expert annotation of audio and visual evidence, construction of video-based distractors, human review of question criteria, and model filtering of textual and unimodal shortcuts. Furthermore, EvaOmni requires models to simultaneously return the answer and supporting evidence intervals to assess the complete reasoning chain.

Engineering Trade-offs and Performance

EvaOmni introduces two key scoring metrics:

• Coverage Grounded Score (C.G.): Measures the extent to which a model identifies annotated evidence.
• Bottleneck Grounded Score (B.G.): Penalizes poor temporal localization in either modality.

Among 7 open-weight and 17 closed-weight Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). However, at Level 3, its B.G. was only 12.3%, indicating that even the most advanced models still have room for improvement in grounded audio-visual reasoning.

Developer Implementation and Deployment Recommendations

EvaOmni provides developers with a more reliable evaluation tool for audio-visual reasoning, helping them better assess and improve model performance. Developers can consider the following recommendations:

1. Multi-modal Data Processing: Prioritize processing audio-visual evidence during training and inference, avoiding reliance on textual or unimodal shortcuts.
2. Reasoning Chain Completeness: Focus on the completeness and logical coherence of the model's generated answers, ensuring it can provide supporting evidence.
3. Cross-modal Reasoning Ability: Strengthen the model's reasoning ability on cross-modal data to enhance its performance in complex reasoning tasks.

Conclusion

The release of EvaOmni marks a significant advancement in the field of audio-visual reasoning evaluation, providing a new benchmark and direction for the development of AI models in multi-modal data processing and reasoning tasks.

---

Source: Hugging Face Daily Papers (2026-10-20)

— END —

Tags: #Hugging Face Follow #EvaOmni Follow #Audio-Visual Reasoning Follow #Benchmark Follow #Multi-modal Follow