Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
Quick Answer
The Entropy-Coded MS-VQ-VAE model achieves ultra-low bitrate video compression at 0.043-0.064 bpp, outperforming H.265 CRF 36 by 5-7.6 times in perceptual quality.
Quick Take
Utilizing learned autoregressive priors, it effectively navigates the limitations of traditional codecs, maintaining high entropy efficiency and stability across varying codebook sizes.
Key Points
- Achieves 0.043-0.064 bpp, significantly below H.264 and H.265 floors.
- Utilizes learned autoregressive priors to enhance compression efficiency.
- Demonstrates 70-85% entropy efficiency with power-law index distributions.
- Maintains full codebook utilization with EMA-stabilized updates.
- Outperforms H.265 CRF 36 by 0.072 LPIPS at K=1024.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning.
Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.02562 [cs.CV] |
| (or arXiv:2607.02562v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02562 arXiv-issued DOI via DataCite |
Submission history
From: Manikanta Kotthapalli [view email]
[v1]
Sun, 28 Jun 2026 20:09:45 UTC (758 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.