Silicon Sampling via Cross-Survey Transfer
Quick Answer
The study introduces cross-survey transfer for evaluating silicon sampling with LLMs, revealing that zero-shot models achieve 52% accuracy on unseen items, closely matching supervised models.
Quick Take
Findings indicate a hierarchy in predictability and highlight nuanced limitations of , such as variance collapse affecting supervised models as well.
Key Points
- Zero-shot LLMs (27B-120B parameters) achieve 52% accuracy on unseen survey items.
- Predictability hierarchy ranges from 67% for partisan attitudes to 23% for sovereignty.
- Variance collapse affects both LLMs and supervised models, challenging previous assumptions.
- Alignment effects vary significantly across different model families.
- Study uses data from the Taiwan Election and Democratization Study (TEDS) 2024.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Multiagent Systems (cs.MA); Methodology (stat.ME) |
| Cite as: | arXiv:2607.03091 [cs.AI] |
| (or arXiv:2607.03091v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03091 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yihuang Kang [view email]
[v1]
Fri, 3 Jul 2026 08:22:18 UTC (454 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.