Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation
Quick Answer
TEAL introduces a novel pixel-level active learning framework leveraging ViT representations, achieving up to +24.36 mIoU improvement with just one labeled pixel per image across multiple datasets.
Quick Take
This method outperforms existing baselines, demonstrating effective token-space selection and uncertainty refinement for low-budget semantic segmentation.
Key Points
- TEAL is the first pixel-level active learning framework using ViT representations.
- Achieves up to +24.36 mIoU improvement with only one labeled pixel per image.
- Demonstrates effectiveness across datasets like CamVid, Cityscapes, and ADE20K.
- Utilizes a two-level MaxHerding strategy for candidate selection.
- Improvements persist even with the same DINOv3 backbone across all baselines.
Paper Resources
📖 Reader Mode
~2 min readAbstract:While foundation models and vision transformers (ViTs) have redefined visual representations, current active learning (AL) strategies for these backbones largely operate at the patch or region level, where annotation requirements remain high due to extensive pixel labeling. Extending ViT-based AL to the pixel-level, low-budget regime is uniquely challenging: with as little as one pixel query per image, a method must simultaneously identify the most informative locations and train a reliable decoder head from an extremely sparse signal. We introduce TEAL (Token-space Efficient Active Learning), the first pixel-level AL framework built around ViT representations. TEAL uses a frozen ViT backbone and performs diversity-based selection directly on its native token lattice, avoiding the artifacts introduced by interpolating token embeddings to dense pixel resolutions. Our framework applies a two-level MaxHerding strategy over multi-layer token descriptors to select representative candidates, followed by margin-based uncertainty refinement on a finer decoder grid. Across CamVid, Cityscapes, ADE20K, and Pascal Context, TEAL consistently outperforms previous baselines under extreme label scarcity, improving over the strongest baseline by up to +24.36 mIoU with a single labeled pixel (px) per image, and reaching up to 97% of fully supervised performance with just 10 px. To rigorously establish the source of the gains, we further show that the improvements persist when all baselines are equipped with the same DINOv3 backbone, demonstrating that coupling token-space selection with uncertainty refinement on the decoder grid makes ViT token geometry an effective foundation for extremely low-budget active segmentation.
| Comments: | Accepted at NeurIPS 2026. 19 pages, 6 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20881 [cs.CV] |
| (or arXiv:2610.20881v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20881 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Utku Cicek [view email]
[v1]
Wed, 14 Oct 2026 20:13:31 UTC (3,465 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.