Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
Quick Answer
The Qwenfant model, based on Qwen3, achieves the top rank in the Multilingual track of the 2026 BabyLM Challenge by utilizing character-level vocabulary and larger model architecture, enhanced through LoRA-style fine-tuning.
Quick Take
This approach demonstrates that high-quality synthetic data can significantly improve performance.
Key Points
- Qwenfant ranks 1st in the 2026 BabyLM Challenge Multilingual track.
- Utilizes a character-level vocabulary and a larger model for improvements.
- Improvements require LoRA-style fine-tuning for optimal performance.
- Synthetic data generated by enhances the quality of training data.
- Demonstrates the potential of using summarized and translated data.
Paper Resources
📖 Reader Mode
~1 min readAbstract:This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19294 [cs.CL] |
| (or arXiv:2610.19294v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19294 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lukas Edman [view email]
[v1]
Tue, 13 Oct 2026 23:15:40 UTC (97 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.