Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery
Quick Take
The study introduces a multi-modal classification framework using Perceiver IO to fuse satellite and street-level imagery for building inspection, achieving significant performance improvements in roof element classification. A dataset of 32,135 buildings was created, with the RGB-M masking strategy enhancing results, yielding up to +11.3 AP for slate attributes. This flexible architecture supports various input types and multiple output tasks.
Key Points
- Perceiver IO architecture fuses satellite and street-level imagery for building inspection.
- Dataset includes 32,135 buildings with up to eight street views per segment.
- RGB-M masking strategy enhances performance, outperforming hard cropping methods.
- Model shows +11.3 AP improvement for slate attributes from street-level views.
- Architecture accommodates heterogeneous inputs and multiple output tasks.
Article Content
From source RSS / original summaryarXiv:2605. 26381v1 Announce Type: new Abstract: We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture operating on spatial patch tokens from a shared DINOv2 backbone. The design naturally handles a variable number of street-level views per building without padding or fixed-size pooling, and jointly predicts multi-label roof element and roof material classes.
We construct a large-scale dataset of 32,135 buildings (61,672 segments) spanning ten countries, pairing satellite images with up to eight street-level views per segment and evaluating four masking strategies for isolating the target building. We propose an RGB-M masking strategy that appends the building footprint mask as a fourth input channel, providing a soft spatial prior that outperforms hard cropping across both modalities.
The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level (e. g. , +11. 3 AP for slate, +1. 3 AP for dormers), though the satellite-only baseline retains a slight advantage in macro-averaged mAP for classes that are predominantly visible from above. These results establish a scalable, flexible architecture for multi-modal building inspection that can accommodate heterogeneous inputs and multiple output tasks.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.
