arXiv Releases OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Summary:arXiv introduces OR-License, a novel benchmark for evaluating the performance of Large Language Model (LLM) agents in Operations Research (OR) scenarios. The benchmark consists of 31 instances across 10 industrial scenarios, valued at USD 3.68 million, and preserves original business requirements, data formats, and delivery boundaries through de-identification. The study introduces ORSim, a controlled client simulator, to assess information acquisition. Results show that while some LLMs exceed e

---

Background and Motivation

As Large Language Models (LLMs) are increasingly applied across various domains, evaluating their performance in complex professional tasks has become crucial. The field of Operations Research (OR) is particularly challenging as it involves not only intricate data processing but also the precise understanding and translation of raw business requirements. Existing benchmarks typically rely on predefined optimization tasks or simplified descriptions, failing to fully capture the capabilities of LLMs in handling real-world business materials.

The OR-License Benchmark

The OR-License benchmark, introduced by arXiv, aims to address this gap. It comprises 31 instances across 10 industrial scenarios, valued at USD 3.68 million. Through de-identification, the benchmark preserves the original business requirements, data formats, and delivery boundaries, ensuring the authenticity and challenge of the test.

The ORSim Simulator

To study information acquisition, the research team developed the ORSim simulator. This simulator mimics real client behavior by disclosing business facts without providing OR expertise, thereby assessing the LLM's ability to acquire information, formulate problems, and validate solutions.

Key Findings

1. Performance Evaluation: Among 11 frontier LLMs, some models exceeded expert performance in conditional objective quality, but none matched the Deployment-Adjusted Quality (DAQ) of expert solutions. The strongest Claude Code configuration achieved 88.39% strict feasibility, while its DAQ was 0.87, compared to the expert's 1.

2. Information Acquisition: Information acquisition extends beyond clarification to include solution development and validation. Later stages showed higher confirmed issue-resolution proportions, indicating that LLM performance in complex tasks relies on sustained information acquisition and validation.

3. Failure Analysis: Trajectory analysis revealed failures in problem formulation and implementation. Task-equalized research-chain completion rates ranged from 24% to 45% across models on evaluable tasks, suggesting that LLMs still need improvement in handling complex tasks.

Conclusion and Outlook

The findings from the OR-License benchmark indicate that while LLMs show promise in certain aspects, there is room for improvement in handling real-world OR tasks. The study emphasizes the importance of information acquisition, problem formulation, and self-improvement, providing valuable insights for the future application of LLMs in professional fields.

Engineering Trade-offs and Recommendations

• Information Acquisition and Validation: Developers should focus on enhancing the LLM's ability to acquire and validate information, ensuring the model can effectively handle complex business requirements.
• Continuous Improvement: Introducing feedback mechanisms and self-improvement strategies can enhance LLM performance in long-horizon tasks.
• Multimodal Fusion: Future research could explore the fusion of multimodal data to further improve LLM performance in professional tasks.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-21)

— END —

Tags: #LLMs & Foundation Models Follow #Benchmark Follow #Operations Research Follow #Information Acquisition Follow #Agentic Follow