Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
arXiv cs.CV·Arsha Nagrani, Jasper Uijilings, Shyamal Buch, Tobias Weyand, Sudheendra Vijayanarasimhan, Bo Hu, Ramin Mehran, David A Ross, Cordelia Schmid
4d agoQuick Take
Minerva-Ego introduces a benchmark for evaluating egocentric video reasoning with spatiotemporal annotations.
Key Points
- Evaluates complex egocentric visual reasoning.
- Includes multi-step multimodal questions.
- Hints on 'where' and 'when' improve model performance.
📖 Reader Mode
~2 min readAbstract:Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.15342 [cs.CV] |
| (or arXiv:2605.15342v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15342 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sudheendra Vijayanarasimhan [view email]
[v1]
Thu, 14 May 2026 19:12:20 UTC (5,140 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →arXiv cs.CV·Jinhao Jing, Zheng Ma, Jinwei Liang, Qiannian Zhao, Shawn Chen, Jing Yang, Por Lip Yee, Prayag Tiwari, Jingjing Bai, Benyou Wang, Lewei Lu, Zhan Su
3d ago FeaturedOriginal
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
AI Summary
GeoSym127K introduces a scalable neuro-symbolic framework for enhanced geometric reasoning in multimodal models.