Codifying the Judge: Scalable Evaluation via Program Distillation
Quick Answer
The PAJAMA system introduces program distillation as a scalable alternative to LLMs for automated evaluation, matching the performance of a 13B-size LLM while significantly reducing API costs.
Quick Take
It synthesizes programmatic judges that provide transparency and efficiency, achieving better accuracy and throughput across five datasets. Additionally, a reward model distilled from these programs outperforms traditional -based models at a fraction of the cost.
Key Points
- PAJAMA synthesizes programs that act as judges, improving evaluation transparency.
- Programmatic judges match the performance of a 13B-size LLM across five datasets.
- Using program outputs as routing signals enhances both accuracy and throughput.
- The distilled reward model outperforms LLM-based models at 100x lower API costs.
- Program distillation eliminates per-sample API costs, enhancing scalability.
DeepSignal Analysis
What happened
The PAJAMA system introduces program distillation as a cost-effective alternative to large language models (LLMs) for automated evaluation. It matches the performance of a 13B-size LLM while reducing API costs significantly. The system synthesizes programmatic judges that enhance transparency and efficiency in decision-making.
Key evidence
- PAJAMA synthesizes programs that act as judges, aggregating their decisions into a final verdict while allowing for inspection and editing.
- The programmatic judges achieved performance comparable to a 13B-size LLM across five datasets and four model families.
- A reward model derived from the programmatic judges outperformed a traditional LLM-based model at a significantly lower API cost, demonstrating improved efficiency.
Why it matters
The introduction of program distillation could reshape automated evaluation by addressing the high costs and latency associated with LLMs. This approach not only enhances transparency but also allows for more efficient decision-making processes. By matching the performance of large models at a fraction of the cost, PAJAMA may enable broader adoption of automated evaluation systems in various applications.
Paper Resources
Source Excerpt
-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, a
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.