Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Quick Answer
This paper shows that The Adaptive Masked Reconstruction (AMR) framework enhances self-supervised skeleton-based action recognition by decoupling the decoder from the encoder, allowing for larger spatiotemporal patch predictions.
Quick Take
The Adaptive Masked Reconstruction (AMR) framework enhances self-supervised skeleton-based action recognition by decoupling the decoder from the encoder, allowing for larger spatiotemporal patch predictions. This method significantly reduces training time and improves accuracy on datasets like NTU RGB+D 60 and 120, outperforming existing state-of-the-art techniques.
Key Points
- AMR decouples decoder from encoder for flexible spatiotemporal patch predictions.
- Introduces adaptive guidance to focus on high motion informativeness regions.
- Significantly accelerates pre-training and improves downstream recognition accuracy.
- Demonstrated effectiveness on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets.
- Outperforms current state-of-the-art methods in skeleton-based action recognition.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 11450v1 Announce Type: new Abstract: Recently, masked skeleton reconstruction models have emerged as strong action representation learners, driving significant progress in self-supervised skeleton-based action recognition. However, existing state-of-the-art methods must predict an exceedingly large number of spatiotemporal patches, significantly prolonging training time.
Besides, by treating all spatiotemporal regions equally during reconstruction, these models are distracted from learning the critical motion patterns that underlie action semantics. To address these challenges, we propose Adaptive Masked Reconstruction (AMR), a faster and stronger pre-training framework. We first decouple the decoder from the encoder, enabling flexible prediction of larger spatiotemporal patches and dramatically reducing reconstruction complexity.
Given that larger patches contain more complex information, which is challenging to predict and consequently degrades performance, we accordingly introduce an adaptive guidance module. This module identifies regions of high motion informativeness, guiding the model to focus on the most discriminative parts of each patch and alleviating reconstruction difficulty.
Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that AMR not only accelerates pre-training substantially but also improves downstream recognition accuracy, surpassing current state-of-the-art approaches.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.