Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation
Quick Answer
This study evaluates federated pre-training's reliability in downstream fine-tuning and intrinsic evaluation, revealing that downstream fine-tuning often misrepresents pre-training rankings.
Quick Take
Using a 16M parameter transformer model, it finds that next-token prediction correlates better with pre-training test perplexity than fine-tuning on GLUE benchmarks.
Key Points
- Federated pre-training enables training on private, distributed data without centralization.
- Downstream fine-tuning on GLUE does not reliably reflect pre-training quality.
- Next-token prediction shows strong correlation with pre-training test perplexity.
- Evaluation protocols should prioritize signals closer to original pre-training objectives.
- Controlled experiments used a 16M parameter transformer model for assessment.
DeepSignal Analysis
What happened
This study investigates the reliability of federated pre-training in downstream fine-tuning and intrinsic evaluation. It finds that downstream fine-tuning often misrepresents pre-training rankings, while next-token prediction correlates better with pre-training test perplexity. The research uses a 16M parameter transformer model trained on identical client data.
Key evidence
- The study uses a 16M parameter transformer model to evaluate federated pre-training.
- Downstream fine-tuning on GLUE benchmarks does not reliably preserve pre-training rankings.
- Next-token prediction shows a strong correlation with pre-training test perplexity compared to downstream fine-tuning.
Why it matters
Understanding the reliability of evaluation methods for federated pre-training is crucial for accurately assessing model performance. Misleading evaluations can lead to incorrect conclusions about model quality, impacting the development and deployment of AI systems. This research highlights the need for evaluation methods that align closely with pre-training objectives.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Federated pre-training offers a way to train foundation models on private or distributed data without centralizing the underlying datasets. However, evaluating federated pre-training remains challenging because differences in client participation and local data availability can make directly comparable evaluation difficult. Moreover, pre-training test perplexity is tied to the pre-training distribution, while downstream benchmarks introduce task-specific adaptation that may not faithfully reflect the test perplexity established during pre-training. In this work, we study which evaluation protocol more reliably reflects federated pre-training quality. Using a controlled set of centralized and federated-trained models of a 16M parameter transformer model trained on identical client data, we assess evaluation protocols by whether they preserve a reference ranking established on the same pre-training testset. We compare downstream fine-tuning on GLUE, including full, head-only, and reduced-data variants, with next-token prediction on GLUE text as an intrinsic evaluation signal. Our results show that downstream fine-tuning does not reliably preserve the pre-training ranking, whereas direct next-token prediction exhibits a strong correspondence with the pre-training test perplexity. These findings suggest that downstream fine-tuning alone can be misleading when comparing federated pre-trained models, and that evaluation signals closer to the original pre-training objective deserve greater attention.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.28658 [cs.CL] |
| (or arXiv:2607.28658v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28658 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Claudia Großer [view email]
[v1]
Mon, 20 Jul 2026 15:14:12 UTC (489 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.