GiLT: Augmenting Transformer Language Models with Dependency Graphs
Quick Take
GiLT enhances Transformer models by integrating dependency graphs for improved syntactic generalization.
Key Points
- Utilizes dependency graphs without adding structural tokens.
- Modulates attention weights for language modeling.
- Achieves better performance on downstream tasks.
📖 Reader Mode
~2 min readAbstract:Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on syntactic tree structures of languages, in particular constituency tree structures. We propose Graph-Infused Layers Transformer Language Model (GiLT) which leverages dependency graphs for augmenting Transformer language models. Unlike most previous work, GiLT does not insert extra structural tokens in language modeling; instead, it injects structural information into language modeling by modulating attention weights in the Transformer with features extracted from the dependency graph that is incrementally constructed along with token prediction. In our experiments, GiLT with semantic dependency graphs achieves better syntactic generalization while maintaining competitive perplexity in comparison with Transformer language model baselines. In addition, GiLT can be finetuned from a pretrained language model to achieve improved downstream task performance. Our code is released at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.15562 [cs.CL] |
| (or arXiv:2605.15562v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15562 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyu Huang [view email]
[v1]
Fri, 15 May 2026 03:08:49 UTC (236 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
2d ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.