Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks
Quick Answer
The study introduces an evaluation-unsupervised coreset selection method for LLM benchmarks, utilizing submodular functions.
Quick Take
The study introduces an evaluation-unsupervised coreset selection method for LLM benchmarks, utilizing submodular functions. The facility location function outperforms twelve baselines in preserving model scores across 35 benchmarks, while also achieving competitive results on and MTEB leaderboards with lower computational costs.
Key Points
- Introduces a coreset selection method for LLM benchmarks without model evaluations.
- Facility location function preserves LLM scores better than twelve baselines.
- Evaluated on 35 heterogeneous benchmarks across five capability categories.
- Achieves competitive results on MMLU and MTEB with lower computation costs.
- Submodularity proves to be a reliable tool for benchmark compression.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Jihan Yao, Gantavya Bhatt, Arnav Das, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhardwaj, Chang Su, Jialei Wang, Yikai Zhu, Sugam Devare, Damon Mosk-Aoyama, Zhen Dong, Venkat Krishna Srinivasan, Yineng Zhang, Oleksii Kuchaiev, Jiantao Jiao, Banghua Zhu, Jeff Bilmes
Abstract:We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite. In evaluation-unsupervised benchmark coreset selection (our approach), the selection algorithm uses no model evaluation outcomes, and operates on a fine granularity by producing subsets of prompts over multiple benchmarks rather than producing a sub-collection of entire benchmarks. We use submodular subset selection, and we develop and evaluate many different submodular functions for this purpose, including determinantal point process (DPP) based approaches, submodular mutual information functions, and facility location-based functions. On a new large-scale suite of 35 heterogeneous benchmarks spanning five different capability categories, 18 frontier LLMs, and over 61K prompts, we find that the facility location (FL) function operating exclusively on inexpensive semantic prompt embeddings preserves LLM scores better than twelve separate score-based and diversity-based baselines, across a range of coreset budgets. Moreover, we show our proposed objective is not limited to the evaluation-unsupervised regime: in the setting where only a handful of whole benchmarks must be selected and a large amount of model scores are available, the same objective matches or outperforms state-of-the-art baselines on the MMLU and MTEB leaderboards, while being substantially cheaper to compute. Together, our results suggest that submodularity, in general, is a strong and reliable tool for benchmark compression.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.09739 [cs.AI] |
| (or arXiv:2607.09739v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09739 arXiv-issued DOI via DataCite |
Submission history
From: Jihan Yao [view email]
[v1]
Thu, 2 Jul 2026 18:37:18 UTC (274 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.