MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes
Quick Answer
MR2-ByteTrack is a novel Video Object Detection method designed for MCU-based embedded vision nodes, achieving mAP scores of 49.0 for CNNs and 48.7 for Transformers while reducing computational load by up to 53%.
Quick Take
Deployed on GAP9, it offers 55% energy savings, enabling real-time Transformer-based VOD on ultra-low-power devices.
Key Points
- MR2-ByteTrack alternates between full- and low-resolution inference to reduce computational costs.
- Utilizes ByteTrack for linking detections and Rescore for correcting misclassifications across frames.
- Achieves significant energy savings of up to 55% compared to processing full-resolution images.
- Maintains accuracy with mAP scores of 49.0 for CNNs and 48.7 for Transformers on ImageNetVID.
- First real-time Transformer-based VOD implemented on an ultra-low-power RISC-V multicore MCU.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Modern smart vision sensors need on-device intelligence to process video streams, as cloud computing is often impractical due to bandwidth, latency, and privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited memory and compute, making conventional video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a Video Object Detection (VOD) method tailored for MCU-based embedded vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the Transformer, while reducing multiply-accumulate operations by as much as 53\% for the CNNs and 32\% for the Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55\% energy savings compared to processing only full-resolution images, enabling the first real-time Transformer-based VOD on an MCU-class embedded vision node. Code available at this https URL
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Image and Video Processing (eess.IV) |
| Cite as: | arXiv:2605.15423 [cs.CV] |
| (or arXiv:2605.15423v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15423 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Luca Bompani [view email]
[v1]
Thu, 14 May 2026 21:13:56 UTC (6,189 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.