Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
arXiv cs.CL·Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan, Rashmi Gangadharaiah
4d agoQuick Take
LLMs can learn syntax in new languages but struggle with semantic transfer, revealing an implementation fidelity gap.
Key Points
- PyLang is a minimal imperative language for testing LLMs.
- Fine-tuning improves syntax but not semantic understanding.
- Models show algorithmic understanding but fail in unfamiliar languages.
📖 Reader Mode
~2 min readAbstract:Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.15607 [cs.CL] |
| (or arXiv:2605.15607v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15607 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vinayshekhar Bannihatti Kumar [view email]
[v1]
Fri, 15 May 2026 04:37:31 UTC (44 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
2d ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.