Silicon Sampling via Cross-Survey Transfer
Quick Answer
The study introduces cross-survey transfer for evaluating silicon sampling with LLMs, revealing that zero-shot models achieve 52% accuracy on unseen items, closely matching supervised models.
Quick Take
The study introduces cross-survey transfer for evaluating silicon sampling with LLMs, revealing that zero-shot models achieve 52% accuracy on unseen items, closely matching supervised models. Findings indicate a hierarchy in predictability and highlight nuanced limitations of LLMs, such as variance collapse affecting supervised models as well.
Key Points
- Zero-shot LLMs (27B-120B parameters) achieve 52% accuracy on unseen survey items.
- Predictability hierarchy ranges from 67% for partisan attitudes to 23% for sovereignty.
- Variance collapse affects both LLMs and supervised models, challenging previous assumptions.
- Alignment effects vary significantly across different model families.
- Study uses data from the Taiwan Election and Democratization Study (TEDS) 2024.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Multiagent Systems (cs.MA); Methodology (stat.ME) |
| Cite as: | arXiv:2607.03091 [cs.AI] |
| (or arXiv:2607.03091v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03091 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yihuang Kang [view email]
[v1]
Fri, 3 Jul 2026 08:22:18 UTC (454 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Procedural Memory Distillation (PMD) enhances reinforcement learning by converting cross-episode signals into reusable memory, improving Qwen3-8B and OLMo3-Instruct-7B models by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on . The co-evolution of policy and memory allows for more effective self-supervision, demonstrating significant performance gains when both components are active.
