Inconsistency-aware Multimodal Schr\"odinger Bridge for Deepfake Localization
Quick Take
The IaMSB model introduces an inconsistency-aware multimodal Schrödinger Bridge for deepfake localization, enhancing precision by 3-10% in AP@0.95. It effectively suppresses noise transfer and improves interval-level outputs by integrating cross-modal consistency estimation and step-tuned fusion, particularly for single-sided forgeries.
Key Points
- IaMSB minimizes path-distribution discrepancy without explicit noise injection.
- The model stabilizes strict-IoU boundary precision for deepfake localization.
- It raises AP@0.95 by 3-10%, enhancing high-precision results.
- IaMSB anticipates single-sided and asynchronous forgeries effectively.
- A lightweight coarse bridge proposes candidate intervals and selects witness signals.
Article Content
From source RSS / original summaryarXiv:2605. 23113v1 Announce Type: new Abstract: Audio-visual deepfake localization demands interval-level outputs that serve as temporal evidence. Despite recent progress, symmetric fusion under single-sided or asynchronous forgeries propagates cross-modal noise, degrading high-precision localization. We present IaMSB, an inconsistency-aware multimodal Schr\"odinger Bridge (SB) that jointly estimates cross-modal consistency and performs interval-level localization.
Unlike diffusion models, SB minimizes path-distribution discrepancy and yields consistency scores without explicit noise injection or denoising. With the Schr\"odinger Bridge (SB), IaMSB unifies consistency estimation, cross-modal information selection, and bridge-step scheduling in one framework.
Specifically, a lightweight coarse bridge first proposes candidate intervals and estimates cross-modal consistency; these statistics select cross-modal witness signals and allocate bridge steps asymmetrically across modalities. A refinement bridge then performs step-tuned fusion and outputs refined, time-aligned intervals. IaMSB anticipates single-sided and asynchronous forgeries and, using bottlenecked cross-modal interaction with step allocation, suppresses noise transfer, avoids unnecessary iterations.
Across benchmarks, IaMSB stabilizes strict-IoU boundary precision, raising AP@0. 95 by 3%~10%, and yields improved high-precision localization, particularly for single-sided forgeries.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.