The JEPA Predictor: A Transferable Operator for Occluded Feature Completion
Quick Answer
The JEPA Predictor enhances occluded feature completion by acting as a transferable operator across various encoder families, achieving significant accuracy improvements on ImageNet-9 and Stanford Dogs.
Quick Take
By integrating frozen predictors from I-JEPA and V-JEPA with non-JEPA models like CLIP, accuracy on heavily occluded images increased from 15.9% to 52.1%, demonstrating the effectiveness of this approach without requiring retraining.
Key Points
- JEPA Predictor closes accuracy gaps in occluded feature completion across encoder families.
- Accuracy on ImageNet-9 improved significantly with CLIP and I-JEPA integration.
- Stanford Dogs classification accuracy increased from 15.9% to 52.1% using frozen predictors.
- The method requires no retraining, fitting linear probes per mask fraction.
- Performance benefits grow with higher mask fractions in occluded scenarios.
Paper Resources
Source Excerpt
Joint-Embedding Predictive Architectures (JEPAs) train a predictor jointly with their encoder, but downstream deployment discards the predictor and reads features from the encoder alone. The predictor is, by construction, a learned operator from visible-context features to features at masked positions, the structure a partial-view classifier needs. We show that this operator is portable across encoder families. We first establish that, at heavy mask, retaining the frozen predictor on a JEPA enco
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.