arXiv Introduces ProClaim Framework: Enhancing Scientific Agents in Consensus Claim Verification

Summary:arXiv has released a new study on scientific agents, introducing the ProClaim framework to evaluate and improve their performance in consensus scientific claim verification tasks. The research team constructed the ProClaim-eval dataset, consisting of 718 scientific claims with SUPPORT and REFUTE labels from expert-curated databases in molecular biology, nuclear physics, and astronomy. Experiments demonstrate that the ProClaim framework, which recursively maintains a structured evidence state and

---

Background and Motivation

In recent years, scientific agents have been building knowledge recursively by retrieving evidence, constructing intermediate states, and judging whether those states contain sufficient information. However, these agents are typically evaluated end-to-end without ground truth, making the effects of each design choice unclear. To address this, the research team proposed the task of consensus scientific claim verification and introduced the ProClaim framework to evaluate agents in this task.

Core Mechanisms of ProClaim

The core of the ProClaim framework lies in recursively maintaining a structured evidence state and incorporating a trained judge to control the recursion process. Specifically, ProClaim achieves this through:

1. Evidence State Management: ProClaim persists the evidence state outside the model's context, allowing each design axis (e.g., the content of the evidence state and the type of judge) to be studied independently.
2. Judge Design: ProClaim introduces a trained judge that reads the evidence state and decides when to stop the recursion. Compared to fixed-round stopping mechanisms, the trained judge demonstrates superior verification accuracy.

Experimental Results and Findings

The research team conducted experiments on the ProClaim-eval dataset and compared the results with existing verification systems and general-purpose agent frameworks. The findings include:

• Importance of Evidence State: Recursively maintaining a structured evidence state is more effective than accumulating evidence in-context.
• Judge Advantage: The trained judge achieves the highest agreement with gold labels in three subsets and performs comparably to the off-the-shelf Claude Sonnet 4.6 judge, outperforming Claude Haiku 4.5.
• Performance Improvement: The ProClaim framework significantly outperforms existing verification systems and general-purpose agent frameworks in terms of verification accuracy.

Engineering Trade-offs and Practical Applications

While the recursive evidence state management mechanism of ProClaim increases computational overhead, it reduces the dependency on context length and improves the agent's performance in handling long documents. Additionally, although the trained judge requires additional training time, its performance in complex tasks justifies the investment. Developers can weigh computational resources and task complexity to choose appropriate evidence state management and judge strategies.

Developer Recommendations

1. Evidence State Management: For tasks involving long documents or high complexity, it is recommended to use recursive evidence state management to enhance the agent's reasoning accuracy.
2. Judge Selection: For tasks requiring high accuracy, it is advisable to use a trained judge; for applications with limited computational resources, off-the-shelf judges can be considered.
3. Dataset Construction: The construction method of the ProClaim-eval dataset can serve as a reference for building datasets in other domains, helping developers create evaluation datasets that better meet the needs of practical applications.

---

Source: ArXiv AI (cs.AI) (2026-11-02)

— END —

Tags: #arXiv Follow #Scientific Agents Follow #Consensus Claim Verification Follow #ProClaim Follow #Recursive Evidence State Follow