ZJU3DV Releases ScaleNav: A New Paradigm for Robot Navigation Using Internet Videos
Summary:The ZJU3DV team introduces ScaleNav, a novel framework for robot navigation that leverages internet egocentric videos to learn goal-directed visual dynamics. Through a three-stage training process involving instruction-conditioned video pretraining, action alignment with simulated data, and joint fine-tuning, ScaleNav achieves impressive success rates of 86.5% and 94.9% on the Short-Horizon OVON and HabitatGS-LN benchmarks, respectively. The model has been validated on a real quadruped robot, de
Technical Mechanism Analysis
The core innovation of ScaleNav lies in leveraging internet egocentric videos as a training data source, optimizing robot navigation through a three-stage training process:
-
Instruction-Conditioned Video Pretraining: The model is first pretrained on internet videos to learn the evolution of observations as the camera approaches instructed targets. This stage aims to capture goal-directed visual dynamics in real-world scenarios.
-
Action Alignment: Using action-labeled simulated data, the model learns to predict executable action chunks. This stage integrates visual dynamics with robot action generation, ensuring the model can produce control signals aligned with navigation needs.
-
Joint Fine-Tuning: All trainable components are jointly optimized for the navigation task, further enhancing the model's performance.
Engineering Trade-offs and Performance
-
Data Efficiency and Generalization: By using internet video data, ScaleNav significantly reduces data collection costs and improves the model's generalization across diverse scenes. However, the noise and inconsistency of internet videos pose challenges for model training.
-
Training Complexity: While the three-stage training process enhances model performance, it also increases training time and computational resource requirements.
-
Real-World Deployment: Tests on a real quadruped robot demonstrate that ScaleNav can effectively handle complex navigation tasks in real environments, showcasing its potential for practical applications.
Developer Implementation and Deployment Recommendations
-
Data Preprocessing: Developers need to perform rigorous preprocessing and filtering of internet videos to minimize the impact of noise on the model.
-
Simulated Data Integration: Combining high-quality simulated data can further improve the model's navigation accuracy and reliability.
-
Hardware Adaptation: For real-world deployment, developers should optimize and adapt the model according to the characteristics of the robot's hardware platform to ensure real-time performance and stability.
Conclusion
ScaleNav demonstrates the feasibility of using internet videos for robot navigation training and provides a new direction for future research. Its excellent performance in benchmark tests and real-world deployment proves the method's advantages in data efficiency and generalization capabilities.
— END —Source: Hugging Face Daily Papers (2026-10-19)
Tags: #Robot Navigation #Computer Vision #Deep Learning #AI Agents #ZJU3DV
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments