Enabling Fully Integer-Only Inference for Lightweight Detection Transformers
Quick Answer
I-LW-DETR is the first fully integer-only lightweight DETR, achieving efficient inference on NPUs and microcontrollers.
Quick Take
It reduces model size by approximately 3.6x and computational cost by over an order of magnitude, with only moderate accuracy degradation. This addresses the deployment challenges of Vision Transformer detectors in resource-constrained environments.
Key Points
- I-LW-DETR executes all operations in integer arithmetic, including nonlinearities.
- Introduces scale-preserving split convolution and SD-ShiftGELU for improved performance.
- Achieves a model size reduction of approximately 3.6x.
- Computational cost is reduced by more than one order of magnitude.
- Moderate accuracy degradation is observed across different model scales.
DeepSignal Analysis
What happened
I-LW-DETR is introduced as the first fully integer-only lightweight DETR, designed for efficient deployment on NPUs and microcontrollers. It achieves a model size reduction of approximately 3.6 times and a computational cost decrease by over an order of magnitude, with only moderate accuracy loss.
Key evidence
- I-LW-DETR executes every operation in the forward pass using integer arithmetic, addressing compatibility issues with existing quantized detectors.
- The model incorporates a scale-preserving split convolution, a sign-dependent GELU approximation, and a constrained Shiftmax for stable Softmax normalization.
- Experimental results indicate that the quantization pipeline consistently produces efficient integer-only models across various scales, with moderate accuracy degradation.
Why it matters
The development of I-LW-DETR is significant as it enables the deployment of Vision Transformer detectors in resource-constrained environments, where traditional models struggle due to their reliance on non-integer operations. This advancement could enhance the applicability of advanced detection models in edge computing and IoT devices, where computational resources are limited.
Paper Resources
Source Excerpt
Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation. We
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.