IBM Releases ClariEdge: A New Dataset for Enhancing Query Clarification in RAG Systems
Summary:IBM Research has introduced ClariEdge, a new dataset designed to improve the ability of Retrieval-Augmented Generation (RAG) systems to handle ambiguous user queries. The dataset includes approximately 3,900 examples of ambiguous queries and negative examples that do not require clarification, focusing on multi-turn conversational scenarios. Experiments demonstrate that a lightweight fine-tuned LoRA model achieves a classification accuracy of 96.6%, outperforming many larger models in both class
1. Background and Motivation
In Retrieval-Augmented Generation (RAG) systems, ambiguity in user queries poses a significant challenge. When a user query is either ill-formed or has multiple relevant interpretations in the context of retrieved documents, the system needs to recognize this and request clarification from the user. The ClariEdge dataset, introduced by IBM Research, aims to address this issue by providing examples of ambiguous queries in multi-turn conversational scenarios, helping models learn how to generate appropriate clarification requests.
2. The ClariEdge Dataset
ClariEdge includes approximately 3,900 examples, covering the following categories:
- Ambiguous Queries: Queries that are grammatically correct but have multiple relevant interpretations in the context of retrieved documents.
- Incomplete Queries: Queries that are grammatically incorrect or incomplete, requiring further clarification from the user.
- Non-Clarification Queries: Queries that are clear and unambiguous.
Additionally, ClariEdge contains negative examples to help models distinguish between queries that truly require clarification and those that may seem ambiguous but are not (i.e., can be resolved from the context).
3. Technical Approach and Experimental Results
The research team employed a lightweight LoRA (Low-Rank Adaptation) model for fine-tuning and achieved the following results:
- Classification Accuracy: The LoRA model achieved a classification accuracy of 96.6% on the ClariEdge dataset.
- Clarification Generation Quality: The LoRA model also outperformed many larger models in terms of the quality of the generated clarification requests.
Experiments show that hard negatives are crucial for distinguishing between truly ambiguous queries and those that may seem ambiguous but are not. The LoRA model performs well in both pre-retrieval and post-retrieval scenarios, indicating its potential as a clarification checkpoint in different stages of the RAG pipeline.
4. Practical Applications and Future Directions
The release of the ClariEdge dataset provides new data support for optimizing RAG systems. Developers can use this dataset to train and evaluate models to improve their performance in handling ambiguous queries. Furthermore, ClariEdge demonstrates the potential for enhancing AI comprehension in complex dialogue contexts, paving the way for future research.
5. Developer Recommendations
- Data Usage: It is recommended to incorporate the ClariEdge dataset into the training of RAG systems to improve the model's ability to handle ambiguous queries.
- Model Selection: The lightweight LoRA model is a good choice, especially in resource-constrained environments.
- Evaluation Metrics: In addition to traditional accuracy metrics, it is recommended to pay attention to the quality of the clarification generation to comprehensively assess model performance.
— END —Source: ArXiv AI (cs.AI) (2026-10-22)
Tags: #IBM Research #ClariEdge #RAG Systems #LoRA #Query Clarification
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments