OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films
Quick Answer
OmniVR is a groundbreaking joint audio-video generative restoration model that effectively restores degraded historical films by addressing visual and audio issues simultaneously.
Quick Take
Leveraging a 22B-parameter backbone, it outperforms previous methods across six visual metrics and achieves superior audio quality. The introduction of OmniVRBench sets a new standard for evaluating restoration quality on 200 historical clips.
Key Points
- OmniVR restores degraded films by jointly addressing audio and visual degradations.
- Utilizes a 22B-parameter backbone for enhanced restoration performance.
- Outperforms all prior methods on six visual metrics and achieves best audio quality.
- Introduces OmniVRBench, a benchmark for evaluating restoration across multiple quality aspects.
- First method to jointly tackle visual structure, temporal motion, and acoustic detail.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Historical films suffer from co-occurring visual and audio degradations---blur, noise, flicker, hiss, clipping, and dropout---yet existing methods restore each modality independently, leaving quality gaps and cross-modal inconsistency. We present OmniVR, the first joint audio-video generative restoration model. Built upon a 22B-parameter audio-video generation backbone, OmniVR formulates restoration as conditional generation within a unified multimodal DiT: the low-quality video and audio are encoded as latent conditions, combined with a fixed restoration prompt, and jointly denoised to recover visual structure, temporal motion, and acoustic detail under one coordinated objective. Three key designs enable this adaptation: (1) a joint audio-video degradation pipeline that simulates real old-film characteristics from Internet-collected data; (2) an architecture-preserving text-to-audio-video (T2AV) to audio-video-to-audio-video (AV2AV) transition with prompt annealing that maximally retains the generative prior; and (3) first-frame image-to-video (I2V) anchoring with loss reweighting and waveform supervision for long-video extrapolation and audio fidelity. We also propose OmniVRBench, the first benchmark that evaluates audio-video restoration across visual quality, audio quality, temporal consistency, and audio-visual synchrony on 200 real historical clips. OmniVR surpasses all prior methods on all six visual metrics, achieves the best audio quality, and produces natural colorization---the first method to jointly address all three aspects. Code and weights will be publicly released. Project Page: this https URL
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2608.04224 [cs.CV] |
| (or arXiv:2608.04224v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04224 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xin Lu [view email]
[v1]
Tue, 4 Aug 2026 21:00:57 UTC (28,744 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.