NVIDIA's 'DreamDojo' Paper Faces Severe Criticism: ICML Spotlight Paper Allegedly Riddled with Bugs
By Mr.Xu
Published:
Summary:NVIDIA's research paper 'DreamDojo,' presented at ICML spotlight, has been criticized for containing critical technical flaws. The paper, which builds on NVIDIA's prior work Cosmos 2.5, proposes a world model for robotics and claims performance improvements through pre-training on 44,000 hours of human data. However, researchers on Reddit discovered multiple critical bugs in the paper's codebase, affecting both pre-training and post-training phases, rendering the experimental results unreliable.
Overview
NVIDIA's research team presented a paper titled 'DreamDojo' at the ICML spotlight, proposing a world model for robotics based on their prior work Cosmos 2.5. The paper claimed performance improvements through pre-training on 44,000 hours of human data. However, the results and codebase of the paper have been called into question by researchers on Reddit, who identified critical flaws.
Key Issues
- Marginal Performance Improvement: The paper reported only a 0.5 dB PSNR improvement over Cosmos 2.5, which is a minimal gain given the extensive resources invested.
- Code Bugs: Researchers discovered multiple critical bugs in both the pre-training and post-training code, rendering the experimental results unreliable.
- Resource Misallocation: The significant data and computational resources used did not yield the expected performance improvements.
Implications for Peer Review
The incident has raised concerns about the ICML peer-review process. The failure to detect such blatant code errors highlights potential shortcomings in the current AI research review mechanisms.
Industry Impact
- Reflection on AI Research Quality: This event underscores the importance of code reproducibility and experimental rigor in AI research, urging the academic community to strengthen code review and data validation.
- Challenge to Review Processes: The peer-review system needs to place greater emphasis on code review and experimental verification to ensure the reliability of research findings.
Recommendations for Developers
- Strengthen Code Review: Conduct thorough code reviews and experimental validations before submitting papers.
- Increase Experimental Transparency: Publicizing more experimental details and code can enhance the credibility of research results.
- Leverage Automated Validation Tools: Use automated tools to detect common code errors and improve research quality.
— END —Source: Reddit r/MachineLearning (2026-10-08)
Community Comments