Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
Quick Answer
The study introduces a training-free method for generating dynamic evaluation rubrics for LLM-as-a-Judge, achieving competitive performance across four benchmarks.
Quick Take
A fine-tuned 14B rubric generator outperforms larger proprietary models, demonstrating the effectiveness of the fine-tuning strategy.
Key Points
- Automatically generates fine-grained evaluation rubrics without human annotation.
- Achieves competitive performance across four benchmarks compared to existing methods.
- Fine-tuned 14B rubric generator outperforms larger proprietary models.
- Introduces iterative fine-tuning via meta-judge reward signals.
- Demonstrates scalability as an alternative to human evaluation.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.30568 [cs.CL] |
| (or arXiv:2605.30568v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30568 arXiv-issued DOI via DataCite |
Submission history
From: Zijie Wang [view email]
[v1]
Thu, 28 May 2026 20:59:45 UTC (54 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.