Qwen Introduces Program-Solve Interface to Enhance Math Reliability in Clinical Language Models
Summary:Qwen researchers propose a novel Program-Solve interface that enhances the reliability of arithmetic in clinical language models. Instead of performing calculations directly, the model writes case-specific Python code, which is executed by a restricted local solver. Evaluations on the MedCalc-Bench Verified dataset using Qwen2.5-7B and Qwen2.5-32B-AWQ models show that the 32B model benefits significantly from this approach, achieving 90.53% accuracy compared to 83.47% for direct computation. Whi
Background and Challenge
Large Language Models (LLMs) often exhibit unreliability in arithmetic tasks, which poses a significant challenge for clinical applications that rely on precise calculations, such as medical calculators. The traditional solution is to hardcode validated functions one by one, which is time-consuming and difficult to scale.
New Approach: Program-Solve Interface
The Qwen team proposes an innovative solution: the Program-Solve interface. The core idea is to have the model generate case-specific Python code instead of performing calculations directly. This code is executed by a restricted local solver, achieving deterministic results. The model's task is simplified to deciding how to use the solver.
Experiments and Results
The study evaluates the approach on the MedCalc-Bench Verified dataset, which includes 1,100 cases and 55 calculators. The research team audited the benchmark's formulas and flagged 16 of them for version, usage, or coefficient concerns based on current clinical guidelines.
The experiments used Qwen2.5-7B and Qwen2.5-32B-AWQ models, and the results showed that:
- For the 7B model, the Program-Solve interface did not provide a significant advantage over direct computation (75.31% vs 72.02%, with a confidence interval of [-3.49, 10.38]).
- For the 32B model, the Program-Solve interface significantly improved accuracy (90.53% vs 83.47%, with a confidence interval of [0.47, 14.60]).
In contrast, the handwritten library was exact on its 440 supported cases but abstained elsewhere (40.0% overall).
Conclusion and Impact
The Program-Solve interface offers a new approach for AI applications that require high mathematical reliability, particularly in fields like healthcare, finance, and engineering. However, it is not a substitute for verified formulas or reliable variable extraction.
Developer Recommendations
- Model Selection: For best results, use larger models (e.g., 32B) to leverage the full potential of the Program-Solve interface.
- Executor Design: Designing an efficient and secure executor is crucial for ensuring the reliability of the computation results.
- Application Scenarios: This method is especially suitable for domains where precision is paramount, such as medical, financial, and engineering calculations.
Future Directions
Future research could further optimize the program generation and execution process and explore applications in more domains. Additionally, combining this approach with other technologies (e.g., symbolic computation and formal verification) may yield even better results.
— END —Source: Hugging Face Daily Papers (2026-09-09)
Tags: #Qwen #Program-Solve Interface #Clinical Calculation #LLMs & Foundation Models #AI Reliability
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments