CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
Quick Answer
Cavewoman evaluates how input and output compression affects eight models across five datasets.
Quick Take
Output compression reduces costs by 1.4-3x, while input compression increases costs by ~1.15x, leading to longer responses and accuracy loss.
Key Points
- Output compression improves cost efficiency on most API models by 1.4-2.4x.
- Input compression results in a net cost increase of ~1.15x on average.
- Half of the non-reasoning models' outputs diverge from their unconstrained baseline.
- Longer responses from input compression lead to significant accuracy degradation.
- Cavewoman's code and data are available on GitHub for further research.
Paper Resources
📖 Reader Mode
~2 min readAbstract:"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.24083 [cs.CL] |
| (or arXiv:2606.24083v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.24083 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Morayo Adeyemi [view email]
[v1]
Tue, 23 Jun 2026 02:56:48 UTC (345 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.