Hugging Face Releases Pumpire: A New Benchmark for Metric Distance Estimation in 3D Foundation Models

Summary:Hugging Face has released Pumpire, a unified benchmark designed to evaluate the metric point-pair distance estimation capabilities of both image- and video-level 3D foundation models. Unlike previous approaches that evaluate depth and camera intrinsics separately or rely on geometric similarity metrics for point-clouds, Pumpire directly assesses point-to-point distances from reconstructed geometry. The benchmark is built on a large-scale dataset (pumpire-6k) comprising 100 real-world scenes with

---

Key Breakthroughs

Hugging Face has introduced Pumpire, a unified benchmark for evaluating the metric point-pair distance estimation capabilities of 3D foundation models at both image and video levels. The key features of Pumpire include:

• Unified Evaluation Framework: The benchmark is applicable to both image- and video-level 3D foundation models, with or without depth priors.
• Large-Scale Dataset: It is built on the pumpire-6k dataset, which comprises 100 real-world scenes with 64 frames each, totaling 6,400 frames, annotated with physically measured point-pair distances.
• Direct Assessment of Point-Pair Distance Errors: Unlike traditional methods that evaluate depth and camera intrinsics separately or rely on geometric similarity metrics for point-clouds, Pumpire directly assesses point-to-point distance errors from reconstructed geometry.

Technical Highlights

1. Comprehensive Evaluation Protocol: Pumpire establishes a holistic evaluation protocol that covers both image- and video-level 3D foundation models, enabling direct assessment of point-pair distance errors and cross-setting comparisons.
2. Real-World Data: The dataset includes a diverse range of real-world scenes, providing a more realistic basis for evaluation.
3. Filling the Evaluation Gap: Existing evaluation systems primarily focus on depth and camera intrinsics, while neglecting the perception of physical scale. Pumpire fills this gap by providing a direct assessment of point-pair distance errors.

Industry Impact

The release of Pumpire provides a more precise evaluation tool for 3D foundation models, which is crucial for advancing the field of 3D vision. Its applications include:

• 3D Reconstruction and Generation: Offering a more reliable benchmark for evaluating 3D reconstruction and generation models.
• Robotic Navigation and Perception: Enhancing the navigation and perception capabilities of robots in complex environments.
• Virtual Reality and Augmented Reality: Providing more accurate 3D content generation and evaluation standards for VR and AR applications.

Recommendations for Developers

Developers can leverage the Pumpire dataset and evaluation protocol to conduct more comprehensive evaluations and optimizations of their 3D foundation models. The open-source nature of Pumpire also encourages the community to participate in further improving and expanding the benchmark.

Conclusion

The release of Pumpire marks a significant advancement in the evaluation of 3D foundation models, providing new tools and methods for more accurate assessment of physical scale perception capabilities.

---

Source: Hugging Face Daily Papers (2026-10-08)

— END —

Tags: #Hugging Face Follow #3D Foundation Models Follow #Evaluation Benchmark Follow #Pumpire Follow #Deep Learning Follow