Token Reduction Is Not Cost Reduction
Quick Answer
This paper shows that Reducing token counts in coding agents does not equate to cost savings, as shown in a study of 2,848 Claude Code runs.
Quick Take
Reducing token counts in coding agents does not equate to cost savings, as shown in a study of 2,848 Claude Code runs. The findings reveal that prompt-cache traffic accounts for 87% of costs, while tool-output reduction can actually increase expenses and hinder task completion.
Key Points
- Prompt-cache traffic constitutes 87% of the total costs in coding agent operations.
- Tool-output reduction led to a 6.8% increase in billed costs in some cases.
- Compression can negatively impact task success by removing critical evidence.
- A proposed standard emphasizes success-adjusted billed costs over mere token reduction.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Context-reduction layers for API-based coding agents, including command-output compressors, retrieval rankers, and payload-optimizing proxies, are usually evaluated by how much text they remove. We ask instead: when does reducing retrieved context or tool output lower the actual billed cost of a coding agent without reducing task success or lengthening its trajectory?
Our primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs, of which 2,848 were analyzed, covering 103 tasks, seven repositories, and three models. The campaign compared a baseline with two generations of hook-based compression and an API-boundary proxy, within a broader measured program of roughly 5,500 billed executions.
Three findings emerge. First, prompt-cache traffic dominated cost composition. Cache creation and reads accounted for about 87% of reconstructed four-component cost, or about 80% of the actual bill, with an 8.7% dollar-weighted residual that retained telemetry could not attribute. On Haiku 4.5, this residual scaled with thinking effort.
Second, tool-output reduction did not reliably predict billed-cost reduction. An arm that removed 38% of estimated raw tool-output tokens had 6.8% higher paired cost (95% CI: +2.8% to +11.3%), while per-task reduction was only weakly associated with cost change (Pearson r = 0.15, CI crossing zero).
Third, compression can harm task completion by removing action-critical evidence. In a small single-shot study on SWE-bench-derived Go tasks, compression reduced patch application from 27/40 to 15/40 by corrupting verbatim edit anchors, and the compressed grounded arm produced fewer solves at higher observed cost per solve.
We propose a layered evidence standard centered on success-adjusted billed cost rather than token reduction alone.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.12161 [cs.CL] |
| (or arXiv:2607.12161v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.12161 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sarel Weinberger [view email]
[v1]
Mon, 13 Jul 2026 21:10:22 UTC (177 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.