CurveShift: Is Agent Progress Scalar? Separating Level from Shape
Quick Answer
The study reveals that progress in large language models, particularly post-September 2024, shows a significant increase in solving hard problems, with a +0.40 logits improvement, raising the solve rate from 18% to 25%.
Quick Take
This gain is attributed to stronger reasoning models and is isolated using the benchmark, which avoids confounding factors from agentic harnesses.
Key Points
- Models released after September 2024 show a +0.40 logits gain on hard tasks.
- The hard-problem solve rate increased from approximately 18% to 25%.
- LiveCodeBench benchmark isolates model performance without agentic scaffolding.
- The effect is primarily driven by advanced reasoning models.
- Results are specific to competitive programming tasks.
DeepSignal Analysis
What happened
The study investigates the performance of large language models, particularly after September 2024, revealing a +0.40 logits improvement in solving hard problems. This increase raises the solve rate from 18% to 25%, attributed to stronger reasoning models. The findings are based on the LiveCodeBench benchmark, which isolates the effects of agentic harnesses.
Key evidence
- Models released after September 2024 show a +0.40 logits improvement in solving hard problems, increasing the solve rate from 18% to 25%.
- The LiveCodeBench benchmark was used to isolate model performance from confounding factors related to agentic harnesses.
- The study indicates that most apparent gains in hard tasks do not reflect a change in the difficulty-response curve but are largely due to ceiling effects.
Why it matters
Understanding the true capabilities of large language models is crucial for their application in complex problem-solving. The findings challenge the notion that improvements in performance are uniformly distributed across task difficulties. By isolating the effects of different models, the study provides clearer insights into the actual advancements in reasoning capabilities, which can inform future research and development in AI.
Paper Resources
Source Excerpt
Progress in is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score. These summaries capture the overall performance, but they do not test whether progress is distributed differently across task difficulty. We find that most of the apparent shift in gains toward harder tasks does not reflect a change in the shape of the difficulty-response curve. On METR time-horizon data, a single Rasch model wit
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.