Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification
Quick Answer
The proposed Disentangled Fine-Grained Prototype Learning (DFPL) framework addresses the missing-modality problem in image-tabular multimodal learning by utilizing Shared-Specific Prototype Modeling and a Prototype-guided Fine-grained Alignment module.
Quick Take
The proposed Disentangled Fine-Grained Prototype Learning (DFPL) framework addresses the missing-modality problem in image-tabular multimodal learning by utilizing Shared-Specific Prototype Modeling and a Prototype-guided Fine-grained Alignment module. Extensive experiments on three benchmarks show DFPL outperforms existing methods, enhancing fine-grained semantic and distributional consistency across modalities.
Key Points
- DFPL uses Shared-Specific Prototype Modeling to extract diverse prototypes.
- Prototype-guided Fine-grained Alignment ensures distribution matching across modalities.
- Class-aware Multi-scale Aggregation enhances predictions by combining shared and specific features.
- Extensive experiments validate DFPL's superiority over existing methods in missing-modality scenarios.
- Code for DFPL will be publicly available for further research.
Article Content
From source RSS / original summaryarXiv:2606. 05455v1 Announce Type: new Abstract: The missing-modality problem poses a significant challenge in image-tabular multimodal learning across a wide range of multimedia applications, including product understanding, recommendation systems, and medical diagnosis. This challenge is particularly pronounced when the two modalities are highly heterogeneous, as images and tabular attributes differ substantially in their semantic granularity and data distributions.
Existing methods learn modality-invariant representations through disentanglement and alignment over global token-averaged features, capturing only coarse cross-modal consistency and overlooking fine-grained semantic and distributional misalignment, which hampers the exploitation of complementary cues under missing modalities. To address this, we propose DFPL, a novel framework for fine-grained prototype learning.
Specifically, Shared-Specific Prototype Modeling (SSPM) extracts compact and diverse shared and modality-specific prototypes, and further performs prototype-level disentanglement to suppress redundant intra-modality correlations.
Additionally, we propose a Prototype-guided Fine-grained Alignment (PFA) module that jointly enforces prototype-level distribution matching and prototype-to-class semantic alignment within a unified prototype space, thereby preserving both fine-grained distributional and semantic consistency across modalities. We further introduce a Class-aware Multi-scale Aggregation (CMA) module to adaptively aggregate shared semantics and modality-specific characteristics from global and prototype levels for robust predictions.
Extensive experiments on three diverse image-tabular benchmarks demonstrate the superiority of our method compared to the previous approaches under various missing-modality settings. Code will be made publicly available.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.