Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similarity
Quick Answer
This paper presents a method using a vision-language model (VLM) to detect anomalous frames in task videos, achieving 65% extraction rates for expert-specific actions and 61% for contextual decision-making scenes.
Quick Take
This paper presents a method using a (VLM) to detect anomalous frames in task videos, achieving 65% extraction rates for expert-specific actions and 61% for contextual decision-making scenes. This approach outperforms traditional methods, which had rates of 59% and 33%, respectively, thereby enhancing the transfer of expert knowledge to less experienced maintenance workers in critical infrastructure.
Key Points
- Proposed method detects anomalous frames to extract expert-specific actions.
- Achieved 65% extraction rate for actions, surpassing conventional 59%.
- Contextual decision-making scenes extracted with 61% accuracy, improving from 33%.
- Utilizes vision-language model (VLM) for frame-wise visual descriptions.
- Focuses on transferring expert know-how to less experienced workers.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Maintenance of critical infrastructures, such as railways and power plants, is essential for ensuring operational safety and reliability. However, the declining number of skilled maintenance workers highlights the need to transfer expert know-how to less experienced workers. Previous studies have attempted to extract candidates of expert knowledge by comparing videos of manual-based work with those of expert workers, mainly focusing on differences in observable actions. However, expert know-how is often embedded not only in actions but also in contextual decision-making during task execution. This paper proposes a method that detects anomalous frames between two task videos to automatically extract candidate scenes containing expert-specific actions and contextual decision-making scenes. The method generates frame-wise visual descriptions using a vision-language model (VLM). Expert-specific actions are extracted based on frame similarities computed from description comparisons between two videos, while contextual decision-making scenes are extracted using segment similarities derived from intra-video self-similarity of the descriptions. In simulated distribution board maintenance experiments involving 27 task scenarios, the proposed method achieved extraction rates of 65% for action candidates and 61% for decision-scene candidates, improving over conventional methods that achieved 59% and 33%, respectively. These results demonstrate the effectiveness of the proposed approach in discovering candidate scenes containing expert know-how.
| Comments: | 16 pages, 11 figures, 2 tables |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.11957 [cs.CV] |
| (or arXiv:2607.11957v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.11957 arXiv-issued DOI via DataCite |
Submission history
From: Ryo Sakai [view email]
[v1]
Sun, 12 Jul 2026 06:56:31 UTC (1,956 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.