arXiv Releases perturber: A Stateless Service for Semantics-Preserving Text Perturbations

Summary:arXiv introduces perturber, a new tool designed to address the impact of input perturbations on the robustness of large language model (LLM) evaluations. perturber is a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawing from a catalog of 68 perturbations across 11 families. It leverages a single primitive for perturbation generation and allows callers to chain calls for composition. The design emphasizes reusability and independence

---

Core Features and Design Philosophy

perturber is a tool specifically designed to address the issue of input perturbations in the evaluation of large language models (LLMs). In real-world applications, input data often contains various perturbations that can affect the model's output. Existing tools typically couple perturbations with specific benchmarking stacks, whereas perturber, by being designed as a stateless service and Python library, emphasizes reusability and independence.

Technical Mechanism Analysis

The core mechanisms of perturber include the following key points:

1. Semantics-Preserving Perturbations: Each call applies exactly one semantics-preserving perturbation, selected from a catalog of 68 perturbations across 11 categories. This design ensures the diversity and controllability of perturbations.
2. Single Primitive Implementation: perturber uses a single primitive for perturbation generation, allowing callers to chain calls for composition, enabling more complex perturbation strategies.
3. Reproducibility: The tool emphasizes the reproducibility of perturbations, ensuring consistent results when the same perturbations are applied in different environments.
4. Introspectable Catalog: perturber provides an introspectable catalog, allowing users to select and evaluate different perturbation strategies.

Engineering Trade-offs and Performance

perturber's design involves trade-offs in the following areas:

• Reusability vs Performance: While perturber's design emphasizes reusability, it also includes optimizations to ensure fast execution of each call.
• Diversity vs Controllability: perturber offers a wide range of perturbation options, but users can flexibly choose and combine perturbation strategies to avoid over-perturbation.

In empirical tests, perturber has demonstrated excellent performance in multiple benchmarks, validating its correctness, reproducibility, and cost-effectiveness.

Developer Adoption and Deployment Recommendations

For developers, perturber provides a simple way to enhance the robustness evaluation of LLMs. Here are some recommendations:

1. Integrate perturber into Existing Workflows: Developers can integrate perturber into their existing LLM evaluation workflows to systematically evaluate the model's performance under different perturbations.
2. Leverage the Introspectable Catalog: By using perturber's introspectable catalog, developers can select the perturbation strategies that best suit their specific needs.
3. Combine with Live Demonstration: perturber's live demonstration feature can help developers quickly understand the tool's usage and effects.

Conclusion

The release of perturber provides a new solution for the robustness evaluation of LLMs. By applying perturbations systematically and reproducibly, it helps developers more comprehensively evaluate and enhance the performance of their models.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-12)

— END —

Tags: #arXiv Follow #perturber Follow #Text Perturbation Follow #LLM Evaluation Follow #Semantics-Preserving Follow