RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Quick Answer
RotateAttention introduces a mixed-precision INT4 FlashAttention framework for DiT-based video generation models using 3D RoPE, achieving up to 1.68x speedup and 2.2x kernel-level acceleration while maintaining video quality comparable to full precision.
Quick Take
Key innovations include RoPE-aware rotation and range-optimized P quantization to enhance efficiency.
Key Points
- RotateAttention targets DiT-based video generation with 3D RoPE.
- Achieves up to 1.68x end-to-end speedup and 2.2x kernel-level acceleration.
- Introduces RoPE-aware rotation to mitigate outliers in Queries and Keys.
- Uses range-optimized P quantization to fully exploit INT4 dynamic range.
- Maintains video generation quality similar to full-precision models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In \textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized \textbf{FlashAttention} offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: \textbf{1)} applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with \textbf{RoPE}; and \textbf{2)} the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of \textbf{3D RoPE}. Based on this finding, we propose \textbf{RotateAttention}, an efficient \textbf{mixed-precision INT4 FlashAttention} framework tailored for \textbf{DiT-based video generation models with 3D RoPE}, using selective \textbf{FP16 fallback} for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: \textbf{1) RoPE-aware Rotation}, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and \textbf{2) Range-optimized $P$ Quantization}, which uses fixed scales and zero-points to fully exploit the \textbf{INT4 numerical range} with minimal computational overhead. Experiments show that \textbf{RotateAttention} preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.02584 [cs.CV] |
| (or arXiv:2607.02584v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02584 arXiv-issued DOI via DataCite |
Submission history
From: Yaofu Liu [view email]
[v1]
Wed, 1 Jul 2026 02:44:10 UTC (4,436 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.