Evidence-Gated Authority Constrains LLM Reviewer Overrides
Summary:This paper frames reviewer authority in multi-agent LLM systems as an institutional-jurisdiction problem. Its evidence gate allows a reviewer to replace an upstream proposal only after mechanically validating a binding-constraint violation and producing a compliant repair. Across two open-weight models, 96 spatial-policy tasks, and a 96-task portfolio-transfer benchmark, the gate generally reduced corruption of correct proposals while preserving most violation repairs. However, results varied sh
Research Question: When May a Reviewer Override an Upstream Decision?
Multi-agent systems often assign a reviewer model to inspect another agent's proposal. This arrangement is not automatically safer: a reviewer may repair a constraint-violating answer while also replacing an already correct proposal. The paper frames this as institutional jurisdiction—under the same proposal, task information, model outputs, and review result, when should a reviewer's recommendation become binding?
Two authority models are compared:
- Broad override authority: the reviewer can directly replace the upstream proposal.
- Evidence gate: replacement is permitted only after the reviewer produces a mechanically valid record of a binding-constraint violation and a compliant repair.
The important design shift is that the reviewer no longer emits only an opinion. It must generate an auditable record containing the violated constraint, supporting evidence, and a compliant repair. An external validator checks these fields before exposing the replacement path.
Mechanism: Turning Override Authority into a Conditional State Machine
Let the upstream proposal be $p$, the task constraints be $c$, and the review output be $r$. The action can be modeled as:
$$ A(p,c,r)= \begin{cases} \operatorname{Replace}(p,c,r), & \operatorname{ValidEvidence}(r,c) \land \operatorname{CompliantRepair}(r) \ \operatorname{Retain}(p), & \text{otherwise} \end{cases} $$
This replaces an unconditional review -> replace path with review -> verify -> replace. The rule can prevent a model from overriding a correct result merely because its alternative sounds more confident or polished. However, it does not prove that the evidence is semantically relevant: the validator confirms formal compliance, not factual correctness.
Results: Broad Improvement, but Strong Model Dependence
The study evaluates two locally run open-weight models on 96 spatial-policy tasks and a 96-task portfolio-transfer benchmark. The evidence gate sharply reduces corruption of correct proposals while retaining most repairs of violations. The direction of this effect is partly implied by the rule, while its magnitude depends on reviewer action transitions and upstream proposal states.
The most revealing result comes from prospective validation, where reviewers explicitly choose among retaining, replacing, and escalating. Qwen avoids replacing any correct proposal, while Mistral still replaces all 48 correct proposals. Qwen also declines every welfare-only repair that Mistral frequently attempts.
The models therefore behave differently under the same authority interface. Qwen acts more like a conservative reviewer constrained by the gate, whereas Mistral displays a stronger substitution tendency. This cannot be attributed solely to whether a model understands the evidence rule; instruction following, reward preferences, base-model capability, and task-state interpretation may all contribute.
Engineering Trade-offs: Less Corruption Does Not Mean Higher Overall Correctness
The evidence gate reduces the corruption surface but also narrows the correction domain. If the validator recognizes only hard constraints, it blocks overrides aimed at improving welfare, user experience, or efficiency even when the upstream proposal is formally valid.
Production systems should distinguish:
- Hard constraints: safety policies, formats, budget limits, and compliance requirements, suitable for blocking evidence gates.
- Soft optimizations: experience, benefit, efficiency, and preference improvements, which should not be disguised as mandatory constraints.
- Escalation: retain the proposal and request human or stronger-policy handling when evidence is insufficient.
Two useful metrics are:
$$ \text{Corruption Rate}=\frac{\text{correct proposals replaced}}{\text{all correct proposals}},\qquad \text{Repair Recall}=\frac{\text{violating proposals repaired}}{\text{all violating proposals}} $$
A third metric is also necessary: the semantic false-acceptance rate of the evidence validator. A structurally valid record may still cite the wrong rule or misidentify the relevant failure. Production deployments should therefore add an independent reviewer, structured constraint storage, and replayable tests.
Evaluation Limits: Restricted Review Is Not Universally Better
The paper explicitly avoids claiming that restricted review is universally superior. Reviewer competence, action interface, and binding jurisdiction jointly determine the correction domain and corruption surface.
The tasks primarily cover spatial policies and portfolio transfer, so results should not be directly extrapolated to code repair, medical advice, financial approval, or embodied control. Those domains involve longer state chains, stronger external side effects, and higher verification costs. The sharp differences between Qwen and Mistral also mean that adding an evidence gate is not an out-of-the-box safety guarantee. Corruption rate, repair recall, and escalation rate must be measured for each target model before deployment.
Practical Guidance for Agent Builders
Teams building multi-agent approval chains should:
- Require structured review output: constrain the model to return a constraint ID, evidence excerpt, repair action, and confidence explanation.
- Validate before replacement: permit an override only when evidence fields are complete, the constraint mapping is valid, and the repair does not expand permissions.
- Expose a three-state interface: retain by default, replace when evidence is sufficient, and escalate when uncertain. Avoid forcing a binary approve/reject decision.
- Run model-specific red-team tests: include correct-but-unusual proposals, violating-but-unrelated repairs, and soft-improvement cases that may trigger excessive intervention.
- Persist complete decision traces: record proposals, evidence, validation results, final actions, and model versions so erroneous overrides can be attributed to the model, prompt, or validator.
The defensible engineering conclusion is not that more restricted review is always safer. It is that review authority should be implemented as a verifiable, replayable, model-calibrated policy rather than depending on a model's voluntary restraint.
— END —Source: ArXiv AI (cs.AI) (2026-11-05)
Tags: #Multi-Agent Systems #LLM Review #Authority Control #Evidence Gating #Agent Safety
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments