Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking
Quick Answer
This paper shows that The authors introduce a novel evaluation framework, SetwiseEvalKit, for document set selection and ranking, addressing inter-document interactions.
Quick Take
Their method, Rubric4Setwise, outperforms existing rerankers by achieving superior downstream generation performance with fewer documents, maintaining state-of-the-art results across various scenarios.
Key Points
- SetwiseEvalKit includes 28K high-quality evaluation rubrics for diverse document scenarios.
- 12 rerankers were evaluated, with the best achieving only 45% coverage.
- Rubric4Setwise is training-free and converts rubric-based criteria into selection signals.
- The method shows improved performance with fewer documents and search rounds.
- It effectively closes the loop from evaluation to optimization in document retrieval.
Paper Resources
Source Excerpt
As and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.