arXiv Proposes Deterministic Solver Interface for Clinical Language Models
Summary:arXiv has introduced a novel Program-Solve interface to enhance the reliability of clinical language models in arithmetic tasks. Instead of performing calculations directly, the model generates case-specific Python code, which is executed by a restricted local solver to ensure deterministic outcomes. Evaluated on the MedCalc-Bench Verified benchmark, the approach shows significant improvements for a 32B parameter model but marginal gains for a 7B model. While not a substitute for verified formul
Background and Motivation
The unreliability of Large Language Models (LLMs) in arithmetic tasks poses a significant challenge in clinical computing, where a single numerical error can lead to drastically different recommendations or conclusions. Traditional approaches often involve hardcoding each calculator as a validated function, but this method is time-consuming and difficult to scale.
New Approach: Program-Solve Interface
This study proposes an alternative: instead of performing calculations directly, the model generates case-specific Python code, which is executed by a restricted local solver to ensure deterministic outcomes. The model's task is thus reduced to deciding how to use the solver.
Experiments and Results
The approach was evaluated on the MedCalc-Bench Verified benchmark (1,100 cases, 55 calculators) and compared with direct model arithmetic and a handwritten 22-calculator library. The experiments used Qwen2.5-7B and Qwen2.5-32B-AWQ models and audited the benchmark's formulas, flagging 16 calculators with version, use, or coefficient concerns.
The results show that while the 7B model did not benefit significantly from the solver (75.31% vs. 72.02%, paired +3.29 points, 95% CI [-3.49, 10.38]), the 32B model showed a clear advantage (90.53% vs. 83.47%, +7.05 points, CI [0.47, 14.60]). The handwritten library was exact on its 440 supported cases but abstained elsewhere (40.0% overall).
Conclusion and Impact
Adding an executor helps some open-weight models more than others, even under matched formula, variable, and note access. However, it is not a substitute for verified formulas or reliable variable extraction. The findings provide new insights into improving the reliability of clinical language models in arithmetic tasks, particularly in resource-constrained or high-precision scenarios.
Developer Recommendations
- Model Selection: When resources permit, opt for larger models (e.g., 32B parameters) for better performance.
- Executor Optimization: Further optimize the performance and security of the local executor to enhance overall system reliability.
- Formula Verification: Continuously audit and verify the formulas in the benchmark to ensure the accuracy of calculation results.
- Multimodal Integration: Explore integrating the Program-Solve interface with other multimodal technologies to expand its application scope.
— END —Source: ArXiv AI (cs.AI) (2026-09-12)
Tags: #arXiv #Clinical Language Models #Program-Solve Interface #Deterministic Solver #Qwen Models
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments