DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving
Quick Answer
DVPSFormer introduces a unified online architecture for depth-aware video panoptic segmentation, achieving state-of-the-art results on Cityscapes-DVPS and SemKITTI-DVPS benchmarks.
Quick Take
The model employs explicit scene discretization and online majority voting for efficient real-time processing, significantly reducing latency in autonomous driving applications.
Key Points
- DVPSFormer integrates metric depth estimation, semantic segmentation, and instance tracking.
- Introduces explicit scene discretization for improved depth decoding in a single pass.
- Achieves new state-of-the-art performance on key video segmentation benchmarks.
- Utilizes online majority voting for enhanced classification consistency during tracking.
- Offers a streamlined solution for real-time robotic perception in dynamic environments.
DeepSignal Analysis
What happened
DVPSFormer is a new architecture for depth-aware video panoptic segmentation, designed for real-time applications in autonomous driving. It integrates metric depth estimation, semantic segmentation, and instance tracking into a single online framework, addressing the limitations of previous multi-stage approaches.
Key evidence
- DVPSFormer achieves state-of-the-art results on the Cityscapes-DVPS and SemKITTI-DVPS benchmarks, indicating its competitive performance in video panoptic segmentation tasks.
- The model employs explicit scene discretization and online majority voting, which significantly reduces latency, making it suitable for real-time decision-making in autonomous navigation.
- The architecture allows for a discrete-to-continuous depth decoding in a single pass, tightly coupling semantic and geometric learning, which enhances efficiency.
Why it matters
The development of DVPSFormer is significant as it addresses the critical need for efficient real-time processing in autonomous driving systems. By unifying multiple tasks into a single framework, it promises to improve the safety and reliability of autonomous navigation in dynamic environments, which is essential for widespread adoption of self-driving technology.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Yung-Hsu Yang, Luigi Piccinelli, Siyuan Li, Mattia Segu, Lei Ke, Martin Danelljan, Yuqian Fu, Zuria Bauer, Fisher Yu, Hermann Blum, Marc Pollefeys
Abstract:Safe autonomous navigation requires a holistic understanding of dynamic environments, necessitating the simultaneous estimation of metric depth, semantic segmentation, and instance trajectories. While depth-aware video panoptic segmentation (DVPS) unifies these tasks, existing approaches often rely on computationally expensive, multi-stage pipelines or offline tracking, rendering them unsuitable for real-time decision-making. To address this, we propose DVPSFormer, a unified online architecture designed for efficient 4D scene understanding. Central to our approach is explicit scene discretization (ESD), a novel mechanism that leverages segmentation queries to represent foreground and background regions, enabling a discrete-to-continuous (D2C) depth head to decode metric depth in a single pass. This tightly couples semantic and geometric learning while significantly reducing latency. Furthermore, we propose an online majority voting (OMV) mechanism that exploits temporal consistency to refine classification during instance tracking. DVPSFormer establishes a new state-of-the-art on the Cityscapes-DVPS and SemKITTI-DVPS benchmarks, offering a streamlined solution for online robotic perception. Code and models are available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.26165 [cs.CV] |
| (or arXiv:2607.26165v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.26165 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yung-Hsu Yang [view email]
[v1]
Tue, 28 Jul 2026 18:16:00 UTC (5,028 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.