PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Quick Answer
PoQ-Judge introduces a reference-free evaluation framework for decentralized LLM inference, achieving a 0.747 Pearson correlation with ground-truth proxies using a DeBERTa judge.
Quick Take
The framework reduces evaluation costs by 72.7% while maintaining quality, outperforming traditional reference-based evaluators.
Key Points
- PoQ-Judge trains judge models for scoring query-output pairs without ground-truth references.
- The DeBERTa judge model achieved the highest Pearson correlation of 0.747.
- Online calibration identifies semantic quality as the main evaluation dimension.
- Cascade evaluation reduces costs by 72.7% with minimal quality loss.
- Performance is significantly better in QA tasks compared to summarization.
Paper Resources
Source Excerpt
arXiv:2606. 11196v1 Announce Type: new Abstract: Decentralized inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge.
Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0. 747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.