Forget, Anticipate and Adapt: Test Time Training for Long Videos
Quick Answer
This paper shows that The Frame Forgetting Network (FFN) introduces a novel approach to Test Time Training (TTT) for long videos, optimizing computational efficiency by processing only three frames at a time.
Quick Take
This method reduces unnecessary computations and adapts to new information effectively, demonstrating significant performance improvements on dense-segmentation and video classification tasks using a new dataset of up to 3-hour long videos.
Key Points
- FFN processes only three frames, enhancing efficiency for long videos.
- Introduces a surprise metric to adaptively modify the effective window size.
- New dataset, EpicTours, features videos up to 3 hours long.
- Demonstrated effectiveness in dense-segmentation and video classification tasks.
- Addresses computational challenges in existing TTT approaches.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute.
We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.
| Comments: | ECCV 2026. GLOM/APM's temporal binding now works for long videos |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2606.26515 [cs.CV] |
| (or arXiv:2606.26515v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.26515 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rajat Modi [view email]
[v1]
Thu, 25 Jun 2026 01:40:10 UTC (24,651 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.