Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
Quick Answer
This paper shows that A systematic evaluation of 41 open-weight language models reveals that instruction-tuned 3B models can outperform some 7B models in zero-shot intent classification.
Quick Take
The study highlights the saturation of benchmarks like SNIPS and the inconsistent effects of instruction tuning on confidence calibration, providing practical guidance for model selection under various constraints.
Key Points
- Evaluated 41 models across 15 families, ranging from 135M to 9B parameters.
- Instruction-tuned 3B models outperformed several 7B base models in accuracy.
- Leading models showed statistically indistinguishable differences on the MASSIVE benchmark.
- Common benchmarks like SNIPS are now saturated, lacking meaningful discrimination.
- Confidence calibration effects from instruction tuning were inconsistent.
DeepSignal Analysis
What happened
A systematic evaluation of 41 open-weight language models was conducted, focusing on their performance in zero-shot intent classification across various datasets. The study found that instruction-tuned 3B models often outperformed some 7B models, indicating that model size does not always correlate with performance. Additionally, it highlighted the saturation of benchmarks like SNIPS, which may no longer effectively differentiate among models.
Key evidence
- The evaluation included 41 open-weight language models from 15 families, covering a parameter range from 135M to 9B across eight intent-classification datasets.
- Instruction-tuned 3B models were found to outperform several 7B base models in zero-shot intent classification tasks.
- The study noted that widely used benchmarks, such as SNIPS, have become saturated and fail to meaningfully discriminate among current models.
Why it matters
This evaluation provides critical insights for practitioners in selecting language models for intent classification, especially under constraints like compute and latency. The findings challenge the assumption that larger models are inherently better, suggesting that smaller, instruction-tuned models can be more effective. Furthermore, the saturation of benchmarks raises concerns about their reliability in guiding model selection, necessitating a reevaluation of evaluation metrics in the field.
Paper Resources
Source Excerpt
Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range across eight English single-label intent-classification datasets. A ninth dataset, ATIS, uses five labeled demonstrations and is repor
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.