Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
Quick Answer
The proposed STAMPlus model enhances MLLM-based segmentation by achieving state-of-the-art performance while reducing inference time from 13.50s to 5.16s.
Quick Take
It introduces Structured All-Mask Prediction, allowing for efficient multi-target segmentation with explicit IDs and shared mask spaces, thus resolving the segmentation trilemma of performance, dialogue ability, and speed.
Key Points
- STAMP emits an in-vocabulary <SEG> trigger for efficient mask classification.
- STAMPlus achieves state-of-the-art segmentation across various settings.
- Inference latency for 12 categories reduced from 13.50s to 5.16s.
- Structured All-Mask Prediction allows for multi-target segmentation in one pass.
- Accurate target cues enhance segmentation performance and reasoning capabilities.
Paper Resources
📖 Reader Mode
~2 min readAbstract:MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2608.02791 [cs.CV] |
| (or arXiv:2608.02791v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02791 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiazhen Liu [view email]
[v1]
Mon, 3 Aug 2026 18:42:11 UTC (28,908 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.