RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
arXiv cs.CL·Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh, Tongzi Wu, Keyvan Golestan, Jesse C. Cresswell
15h agoQuick Take
RankJudge is a benchmark generator for evaluating LLMs in multi-turn conversations.
Key Points
- Focuses on multi-turn conversation evaluation.
- Creates conversation pairs with injected flaws.
- Ranks LLM judges using the Bradley-Terry model.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
2d ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.