Unified Panoramic Geometry Estimation via Multi-View Foundation Models
Quick Take
The PaGeR framework advances 3D reconstruction by enabling geometry estimation from single panoramic images, achieving state-of-the-art performance in both indoor and outdoor settings. By leveraging pre-trained transformers, it predicts depth, surface normals, and sky masks in a unified model, demonstrating excellent zero-shot capabilities across diverse scenes.
Key Points
- PaGeR enables 3D reconstruction from single panoramic images.
- It predicts scale-invariant depth and surface normals in one pass.
- The framework retains the 3D prior of existing foundation models.
- Extensive testing shows state-of-the-art performance in various environments.
- Zero-shot performance is excellent across a wide range of scenes.
Article Content
From source RSS / original summaryarXiv:2605. 26368v1 Announce Type: new Abstract: Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image.
In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass.
By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.