BACE Calibrates When LLMs Should Ask Before Committing to an Answer
Summary:This paper addresses the tendency of large language models to answer ambiguous referring expressions too early. Building on SIMMC 2.1, the authors create a controlled environment that measures how many clarification questions are needed to resolve each reference and introduce BACE, a lightweight action selector placed on top of a frozen language-model backbone. BACE combines a belief-aware critic with epistemic tracking to decide whether another question would meaningfully reduce the candidate
The Core Problem: Models Answer Too Early
Large language models often commit to a definite answer when a referring expression remains ambiguous. This may look efficient in single-turn evaluation, but it converts epistemic uncertainty directly into a potentially wrong answer. The paper defines strategic clarification as asking a question only when the answer can meaningfully change the eventual decision.
The objective is not to ask as many questions as possible or follow a fixed interaction budget. It is to identify the point where additional information provides enough value to justify another turn.
A Controlled Testbed for Reference Resolution
The authors build a controlled reference game on top of SIMMC 2.1 and record the minimum number of questions required to resolve each ambiguous reference. Unlike standard multi-turn benchmarks, this setup separates two questions: whether the model recognizes that its information is insufficient, and whether its question actually eliminates an incorrect candidate.
That distinction is important in engineering terms. Unproductive clarification consumes tokens and latency. Conversely, asking for more evidence after the candidate set is already unambiguous reflects poor task-state estimation.
How BACE Works
BACE—Belief-Aware Critic with Epistemic tracking—does not retrain the full language model. It adds a lightweight action selector over a frozen LM backbone. The central design choice is to represent the dialogue state as a probability belief over candidate referents instead of relying only on the surface confidence of the next token.
Its action space contains two principal decisions:
- Ask: generate a clarification question intended to distinguish the remaining candidates.
- Answer: decide that the current belief is sufficient to select a referent and terminate the interaction.
A belief-aware critic estimates the potential posterior contraction from a clarification question, while the epistemic tracking component maintains uncertainty across the dialogue. The selector therefore asks a more useful question than “Does the model feel knowledgeable?” It asks whether the answer is likely to change the decision.
Compared with end-to-end fine-tuning, this design limits additional parameters and training cost to the action-selection layer while allowing the frozen model to be reused. The tradeoff is that the system must maintain candidate sets, question effects, and dialogue state. In tasks where candidates cannot be enumerated reliably, the critic may also lose the probability representation on which it depends.
Comparison with GRPO and ArCHer
The paper compares baseline LLM behavior and post-training methods including GRPO and ArCHer, reporting that BACE selects the optimal number of clarification questions more consistently.
The abstract does not provide complete metrics, model sizes, sampling counts, or significance tests, so the results should not be interpreted as proof that BACE universally outperforms reinforcement-learning baselines. With sufficient training signal, GRPO may produce broader policy improvements, whereas BACE's advantage lies primarily in explicit state modeling and lightweight deployment over a frozen backbone. These approaches optimize different layers: BACE focuses on action selection, while GRPO can reshape the model's overall behavior distribution.
Evaluation Limits and Engineering Trade-offs
The controlled reference game does not directly establish performance in open-domain customer support, visual grounding, or multimodal agents. Real deployments introduce candidate objects that cannot be enumerated, ambiguous user replies, imperfect clarification questions, and the cost of external tool calls.
Deployment must balance three costs:
- Latency: extra rounds generally mean additional inference and network round trips.
- Cost: a frozen backbone lowers training expense, but state updates and critic inference still incur runtime overhead.
- User experience: excessive clarification reduces efficiency, while insufficient clarification turns ambiguous input into an incorrect action.
A practical integration can combine BACE's stopping rule with confidence thresholds and business risk. Low-risk tasks can end at the first high-confidence answer, while high-cost tasks may receive a larger clarification budget. The system should also record how much each question reduces the candidate set.
Practical Guidance for Developers
A production implementation should track three observable signals: candidate-set size, posterior quality before and after each question, and final-answer confidence. Another interaction is worthwhile only when its expected reduction in answer error exceeds the latency and inference cost of that turn.
Evaluation should separately report answer accuracy and clarification efficiency, including unnecessary-question rate, missed-question rate, average interaction depth, and end-to-end latency. Otherwise, a method that asks many questions while ultimately reaching the right answer may hide a poor user experience.
BACE's main contribution is not a blanket argument that models should be more cautious. It turns caution into an optimizable policy: ask only when the question can change the answer. The same principle can be extended to tool-argument confirmation, visual target localization, and intent validation before robotic actions.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-05)
Tags: #BACE #Strategic Clarification #Reference Resolution #Multi-turn Dialogue #Epistemic Tracking
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments