VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding
Quick Take
VideoOdyssey introduces a benchmark for ultra-long-context video understanding, featuring videos averaging 109 minutes across 11 domains. It emphasizes continuous certificate lengths of 16 minutes for visual understanding and 12.8 minutes for audio-visual comprehension, highlighting the cognitive challenges faced by current MLLMs in continuous reasoning and multi-modal tasks.
Key Points
- Benchmark spans 11 domains and 54 subcategories with an average video length of 109 minutes.
- Two subsets: VideoOdyssey-V for visual understanding and VideoOdyssey-AV for audio-visual tasks.
- Continuous certificate lengths extended to 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV.
- Five granular levels from seconds to hours for comprehensive model evaluation.
- Current MLLMs struggle with continuous reasoning and fine-grained perception in ultra-long contexts.
Article Content
From source RSS / original summaryarXiv:2605. 22907v1 Announce Type: new Abstract: Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal spans within extreme video durations. Mastering this intense cognitive load constitutes the fundamental bottleneck in long video understanding.
While existing benchmarks have driven progress by scaling up video duration, their evaluation tasks often require comprehending only short and isolated video segments, falling short of capturing the challenge of ultra-long-context reasoning. To measure this cognitive load, we emphasize continuous certificate length, defined as the video length a human must continuously watch to definitively answer a given question.
Driven by this metric, we introduce VideoOdyssey, a benchmark specifically designed for ultra-long-context and omni-modal video understanding. VideoOdyssey is characterized by three key features: 1) Extreme video duration and diversity: spanning 11 domains and 54 subcategories with an average video duration of 109 minutes; 2) Comprehensive evaluation scenarios: offering two subsets to address different research focuses, i. e.
, VideoOdyssey-V for probing the limits of visual understanding in MLLMs, and VideoOdyssey-AV for evaluating synchronized audio-visual understanding for omni-modal models; 3) Ultra-long and multi-level continuous certificates: extending the average continuous certificate to 16 minutes for VideoOdyssey-V and 12. 8 minutes for VideoOdyssey-AV. Crucially, we design 5 granular levels from seconds to hours, providing a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads.
Extensive evaluations show that bottlenecks of current MLLMs extend beyond simple retrieval to include struggles with continuous reasoning across varying context lengths, fine-grained perception, and non-verbal omni-modal understanding.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.