Hugging Face Releases RoboJudge: Revolutionizing Multimodal LLM Evaluation for Embodied Video Generation

Summary:Hugging Face introduces RoboJudge, a suite of tools designed to enhance the evaluation capabilities of Multimodal Large Language Models (MLLMs) in embodied video generation. The RoboJudge-Bench benchmark includes 800 videos, measuring Physical Adherence and Instruction Alignment. Evaluations reveal that general-purpose MLLMs struggle with physical validity, while specialized video judges struggle to adapt to embodied scenarios. To address this, the team constructed RoboJudge-Train, a dataset of

---

Technical Mechanism Analysis

The core innovation of RoboJudge lies in its multimodal evaluation capabilities, particularly in the context of embodied video generation. Embodied video generation models need to ensure that the generated videos are physically valid and aligned with task instructions. RoboJudge achieves this through the following methods:

1. RoboJudge-Bench Benchmark: Includes 800 videos from 8 different corpora and 10 video generators, covering scenarios such as grippers, human hands, and dexterous hands. Each video is annotated for Physical Adherence and Instruction Alignment.

2. RoboJudge-Train Dataset: Comprises 12K videos annotated with 25K human-verified scores and model-assisted rationales, used to fine-tune the MLLM.

3. RoboJudge-9B Model: A 9B parameter MLLM fine-tuned on RoboJudge-Train using supervised and reinforcement learning. Experiments show that this model achieves a Pearson correlation of 0.719 on RoboJudge-Bench, significantly outperforming the best proprietary baseline, Gemini-3.7-Flash.

Engineering Trade-offs and Performance

1. Balance Between Physical Validity and Instruction Alignment: RoboJudge achieves a good balance between physical validity and instruction alignment. General-purpose MLLMs struggle with physical validity, while specialized video judges struggle to adapt to embodied scenarios. RoboJudge addresses this by building a dedicated training dataset and fine-tuning strategy.

2. Cross-Domain Adaptability: RoboJudge-9B performs well on external benchmarks like VideoPhy2 and Physion-Eval, demonstrating its adaptability across different domains.

3. Performance Improvement: Compared to Gemini-3.7-Flash, RoboJudge-9B improves the Pearson correlation coefficient by 0.089, significantly enhancing the accuracy and reliability of the evaluation.

Developer Implementation and Deployment Recommendations

1. Dataset Construction: Developers can use the RoboJudge-Train dataset construction method to build evaluation datasets for specific domains.

2. Model Fine-tuning: The fine-tuning strategy of RoboJudge-9B can serve as a reference for developers to improve the evaluation capabilities of their own models.

3. Integration of Multimodal Evaluation Tools: RoboJudge can be integrated into the existing video generation model development workflow as a multimodal evaluation tool, providing more reliable evaluation results.

Conclusion

The release of RoboJudge marks a significant advancement in the evaluation of multimodal large language models in embodied video generation. Its excellent performance in physical validity and instruction alignment sets a new standard for the development and evaluation of video generation models.

---

Source: Hugging Face Daily Papers (2026-10-20)

— END —

Tags: #Hugging Face Follow #RoboJudge Follow #Multimodal LLM Follow #Video Generation Follow #Evaluation Benchmark Follow