Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?
Quick Answer
This study evaluates two training paradigms for self-supervised learning: pretraining-finetuning (PFT) and joint training (JT).
Quick Take
This study evaluates two training paradigms for self-supervised learning: pretraining-finetuning (PFT) and joint training (JT). Results indicate that JT enhances efficiency and robustness in low-label scenarios, while PFT excels in specialized domains, revealing the interaction dynamics between self-supervised and supervised objectives across various tasks.
Key Points
- Joint training (JT) consistently improves data efficiency in low-label settings.
- Pretraining-finetuning (PFT) is more reliable in specialized domains.
- The effectiveness of PFT and JT varies based on task and labeled data availability.
- Eight SSL methods were evaluated across diverse computer vision tasks.
- New insights into the interaction between self-supervised and supervised objectives were provided.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse vision applications.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.13192 [cs.CV] |
| (or arXiv:2607.13192v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13192 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nusrat Munia [view email]
[v1]
Tue, 14 Jul 2026 18:45:04 UTC (23,003 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.