perturber: A Stateless Service for Semantics-Preserving Text Perturbations
Quick Answer
This paper shows that The 'perturber' is a stateless Python library that applies a single semantics-preserving text perturbation from a catalog of 68 options, enabling reproducible evaluations of language models.
Quick Take
It emphasizes correctness and cost-effectiveness over model robustness, providing a playground for testing and introspection.
Key Points
- Perturber offers 68 perturbations across 11 families for text evaluation.
- The service ensures reproducibility by applying one perturbation per call.
- Focus is on measuring meaning-preservation rather than model robustness.
- Includes a live playground and an introspectable catalog for benchmarks.
- Evaluation metrics assess correctness, reproducibility, and cost.
DeepSignal Analysis
What happened
The 'perturber' is a Python library designed to apply a single semantics-preserving text perturbation from a catalog of 68 options. It aims to facilitate reproducible evaluations of language models by allowing systematic testing of perturbations. The tool emphasizes correctness and cost-effectiveness.
Key evidence
- The perturber library applies one of 68 semantics-preserving perturbations per call, ensuring reproducibility in evaluations.
- It is designed to be stateless, meaning it does not retain any information between calls, which aids in consistent output.
- The tool focuses on evaluating its own correctness and cost rather than the robustness of any specific language model.
Why it matters
The development of perturber addresses the issue of overestimating language model robustness based on clean benchmarks. By allowing systematic and reproducible perturbations, it provides a more accurate framework for evaluating language models under realistic conditions. This could lead to better understanding and improvement of model performance in real-world applications.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model's robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.
| Comments: | 14 pages (6 main + references and appendix), 5 figures, 13 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.12667 [cs.CL] |
| (or arXiv:2610.12667v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12667 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Khoi-Nguyen Tran [view email]
[v1]
Thu, 8 Oct 2026 19:44:05 UTC (153 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.