LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation
Quick Answer
LivingArena introduces a novel evaluation framework for LLMs, allowing models to probe each other's knowledge boundaries through competitive questioning.
Quick Take
This approach yields a stable Elo leaderboard for ten frontier , emphasizing factual rigor and cognitive exploitation, while offering a scalable, low-cost alternative to traditional benchmarks.
Key Points
- LivingArena allows LLMs to propose questions to exploit opponents' knowledge gaps.
- A judge panel of strong models validates questions to ensure objective answers.
- Evaluating ten frontier LLMs resulted in a stable Elo leaderboard.
- Peer probing measures factual rigor and cognitive boundaries, correlating weakly with human preferences.
- This framework offers a scalable and cost-effective approach to continuous model evaluation.
DeepSignal Analysis
What happened
LivingArena presents a new evaluation framework for large language models (LLMs) that allows them to assess each other's knowledge through competitive questioning. This method generates a stable Elo leaderboard for ten advanced LLMs, focusing on factual accuracy and cognitive limits.
Key evidence
- LivingArena enables LLMs to propose questions to each other, aiming to identify knowledge gaps, which is a departure from traditional evaluation methods.
- The framework includes a judge panel of strong models that validates questions, penalizing questioners if their questions do not have objectively verifiable answers.
- The evaluation of ten frontier LLMs through LivingArena shows that models can effectively exploit their peers' weaknesses, correlating weakly with human preferences.
Why it matters
This framework addresses the limitations of static benchmarks that can become contaminated or saturated, making it difficult for users to differentiate between top-performing models. By allowing LLMs to probe each other's knowledge, LivingArena offers a scalable and cost-effective alternative for continuous evaluation, potentially leading to better model development and understanding of their capabilities.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Evaluating frontier LLMs is challenging: static benchmarks suffer from contamination and saturation -- leaving users unable to distinguish top models and developers blind to specific failure modes -- while human preference is subjective. In this paper, our question is: \emph{Do LLMs know what other LLMs don't? And can we leverage this dynamic for evaluation?} We present \textbf{LivingArena}, an automated, contamination-resistant evaluation framework. In this framework, models take turns proposing questions, aiming to pose items that opponents cannot answer correctly. Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries, receiving rewards when the answerer fails, while the answerer is rewarded otherwise. To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them, penalizing questioners if validation fails. Evaluating ten frontier LLMs, LivingArena yields a stable Elo leaderboard. Our behavioral analyses show that models identify and exploit their peers' cognitive boundaries: self-play and tournament logs indicate that they localize and double down on opponents' weak dimensions. Beyond static knowledge recall, peer probing measures factual rigor and the higher-order ability to probe an opponent's weaknesses, correlating only weakly with human preference and offering a scalable, low-cost approach to continuous evaluation.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.24780 [cs.AI] |
| (or arXiv:2607.24780v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24780 arXiv-issued DOI via DataCite |
Submission history
From: Xingyu Chen [view email]
[v1]
Fri, 19 Jun 2026 06:20:58 UTC (195 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.