SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer
Quick Take
SANA-Streaming introduces a Hybrid Diffusion Transformer for real-time video editing, achieving 1280 x 704 resolution at 24 FPS on an RTX 5090 GPU. Its Cycle-Reverse Regularization enhances temporal consistency without paired videos, outperforming existing methods in throughput and coherence.
Key Points
- Hybrid Diffusion Transformer improves local modeling while maintaining efficiency.
- Cycle-Reverse Regularization ensures semantic consistency without paired long videos.
- Efficient System Co-design maximizes Tensor Core utilization on NVIDIA RTX 5090.
- Achieves real-time editing at 24 FPS with a resolution of 1280 x 704.
- Significantly outperforms state-of-the-art methods in temporal coherence.
Article Content
From source RSS / original summaryarXiv:2605. 30409v1 Announce Type: new Abstract: Real-time streaming video-to-video editing (V2V) is critical for interactive applications such as live broadcasting and gaming, yet it remains a formidable challenge due to the stringent requirements for temporal consistency and inference throughput.
In this paper, we present SANA-Streaming, a system-algorithm co-designed framework for high-resolution, real-time streaming video editing on consumer GPUs, with the following three core designs: (1) Hybrid Diffusion Transformer architecture introduces softmax attention in part of the blocks to improve local modeling capabilities while preserving the efficiency of linear layers.
(2) Cycle-Reverse Regularization is a novel training strategy that enforces semantic consistency by predicting source frames from generated content via flow matching, improving temporal consistency without requiring paired long edited videos. (3) Efficient System Co-design combines fused GDN kernels and Mixed-Precision Quantization (MPQ) optimized for the NVIDIA Blackwell (RTX 5090) architecture. By profiling real-world throughput, our MPQ maximizes Tensor Core utilization while maintaining generation quality.
The resulting system achieves real-time 1280 x 704 resolution editing at 24 end-to-end FPS on a single RTX 5090 GPU, with the DiT core running at 58 FPS. Experimental results demonstrate that our co-design approach significantly outperforms existing SOTA methods in both temporal coherence and system throughput.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, enabling efficient fine-tuning with only 0.11% parameter updates. It significantly enhances performance in few-shot learning and domain shifts across 15 biomedical imaging datasets, demonstrating robustness for clinical applications.
