On the Computational Complexity of Structural Generalization
Quick Answer
This paper formalizes structural generalization in computational complexity, showing that pure Transformers cannot learn it under the assumption that TC0 ≠ NC1.
Quick Take
Neuro-symbolic systems outperform pure Transformers by incorporating semantic rules, highlighting a significant gap in benchmark evaluations.
Key Points
- Structural generalization is defined mathematically for the first time.
- Pure Transformers are limited to the learnable class TC0, unable to achieve structural generalization.
- Neuro-symbolic systems excel in benchmarks by leveraging semantic rules.
- The paper argues that benchmark scores fail to differentiate learned from hard-coded rules.
- A Montagovian approach reveals the complexity of compositional rules in Transformers.
Paper Resources
Source Excerpt
Structural generalization has been measured repeatedly by several benchmarks, yet it has never been formally defined. We give a definition that translates the two premises (compositional structure and unbounded generalization) into mathematical language. The definition itself is neutral: a compiler that hard-codes the rules satisfies it just as well. But structural generalization becomes a scientific question only insofar as the capacity can autonomously emerge from finite data. This question pi
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.