ArXiv Study Highlights Reliability Issues of LLMs in Causal Reasoning

Summary:ArXiv has released a systematic evaluation study on the reliability of large language models (LLMs) in causal reasoning. The research compares 12 instruction-tuned open-weight models across six benchmark causal graphs, five prompting strategies, and four confidence sources. Key findings include: (1) LLM-based causal judgments are heavily recall-dominant, leading to overly dense graphs with many false positives; (2) LLMs often fail to reliably identify directness or orientation of causal relation

---

Background and Motivation

The use of large language models (LLMs) as sources of prior knowledge for structural causal discovery is becoming increasingly common, but the reliability of their direct causal judgments and confidence remains unclear. This study aims to systematically evaluate LLMs in causal reasoning to uncover their potential limitations and suggest directions for improvement.

Methodology

The research team evaluated 12 instruction-tuned open-weight models across the following dimensions:

• Benchmark Causal Graphs: Six causal graphs of varying complexity.
• Prompting Strategies: Five different prompting methods.
• Confidence Sources: Including verbalized confidence, logit-based confidence, cross-prompt agreement, and cross-model agreement.

Key Findings

1. Over-reliance on Recall: LLMs tend to generate overly dense causal graphs with many false positives. Prompting strategies mainly shift the precision-recall trade-off but do not resolve the overprediction issue. Gains from model scale diminish on larger graphs and do not eliminate miscalibration.
2. Inaccurate Causal Relationship Identification: LLMs struggle to reliably identify the directness or orientation of causal relationships. Compared to published reference graphs, models misclassify 40.0% of indirect non-edges and 36.0% of reversed non-edges as direct edges, versus 28.2% of other non-edges. Moreover, 80.8% and 84.6% of these false positives receive verbalized confidence of at least 80%, revealing substantial overconfidence in structurally incorrect predictions.
3. Unreliable Traditional Confidence Estimates: Logit-based confidence frequently collapses near 1.0 regardless of correctness, while cross-prompt and cross-model agreement achieve better mean calibration and discrimination, though their advantages are not statistically significant after Holm correction.

Conclusion and Recommendations

The study suggests that LLMs are better viewed as sources of externally validated soft causal priors rather than direct evidence of causal structure. Future research should focus on improving the accuracy and reliability of LLMs in causal reasoning, such as by refining prompting strategies or introducing additional calibration mechanisms.

Recommendations for Developers

• Use LLMs for Causal Reasoning with Caution: In critical applications, combine LLMs with other methods for cross-validation.
• Focus on Model Calibration: During training and fine-tuning, prioritize model calibration.
• Explore New Confidence Estimation Methods: For example, methods based on agreement may be more reliable than traditional approaches.

---

Source: ArXiv Machine Learning (cs.LG) (2026-08-26)

— END —

Tags: #LLMs & Foundation Models #Causal Reasoning #Model Evaluation #ArXiv #AI