Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics
Quick Answer
This study presents a method to predict best-of-$N$ inference scaling gains using a compact ridge predictor based on three key features.
Quick Take
This study presents a method to predict best-of-$N$ inference scaling gains using a compact ridge predictor based on three key features. Achieving a Spearman correlation of $ ho = 0.90$, the approach allows for efficient validation-set screening before incurring full reward-model scoring costs across various model families and tasks.
Key Points
- Ridge predictors are fitted using features from a single labeled validation-set sampling pass.
- Stability analysis identifies three core features: agreement spread, first-correct-sample position, and completion-length variance.
- The method allows for screening candidate configurations before incurring full scoring costs.
- Achieves a Spearman correlation of $ ho = 0.90$ with actual best-of-$N$ gains.
- Applicable across three base-model families and various math and reasoning tasks.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 02981v1 Announce Type: new Abstract: Best-of-$N$ inference scaling (drawing $N$ candidate answers from a language model and returning the one a reward model ranks highest) improves accuracy by an amount that varies across models, but predicting that amount in advance currently requires running the procedure end-to-end.
Prior work links cheap statistics of a model's sampled outputs and validation-set correctness (how often samples agree, how diverse they are, how confident the model is, and where correct samples appear) to model behavior, but does not isolate which of these form a stable, compact predictor of best-of-$N$ gain.
We fit ridge predictors on features computed from a single labeled validation-set sampling pass, use bootstrap-Lasso as a stability analysis of the candidate feature set, and give a concentration analysis with an explicit linear-approximation residual.
Across three base-model families, six post-training methods, and math and reasoning task domains, the stability analysis identifies a strict three-feature core spanning prompt-level agreement spread, label-assisted first-correct-sample position, and completion-length variance; a compact ridge predictor built from this core plus an entropy add-on reaches Spearman $\rho = 0. 90$ with actual best-of-$N$ gain under a reward-model verifier.
The intended use is labeled validation-set screening of candidate configurations before paying the full reward-model scoring cost.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.