Evaluating Open-Source LLMs in Modeling Human Cooperation in Repeated Prisoner's Dilemma

Summary:This study investigates the ability of five open-source large language models (TII Falcon 7B Instruct, Google Gemma 7B, Meta Llama 3.1 8B Instruct, Microsoft Phi Mini MoE Instruct, and Qwen 2.5 7B Instruct) to simulate human cooperation in repeated Prisoner's Dilemma games. The research compares the models' cooperation rates in fixed-partner and shuffled-partner settings against empirical human data. Results show that while the models exhibit different cooperation patterns, they generally over-c

---

Background and Motivation

The Prisoner's Dilemma serves as a foundational framework for studying human decision-making and cooperation. However, classical economic theories often fail to predict actual human behavior. While large language models (LLMs) show promise in simulating human behavior, existing research lacks a systematic examination of their memory capabilities and comparison against empirical human data.

Methodology

Five open-source LLMs were selected for this study: TII Falcon 7B Instruct, Google Gemma 7B, Meta Llama 3.1 8B Instruct, Microsoft Phi Mini MoE Instruct, and Qwen 2.5 7B Instruct. The models were tested in two settings: fixed-partner and shuffled-partner, with cooperation rates measured over 100 rounds of the game.

Key Findings

1. Cooperation Rate Disparities: In the fixed-partner setting, all models exhibited higher cooperation rates than human data. In the shuffled-partner setting, models generally over-cooperated, with rates exceeding 80% compared to the human average of 28.5%.
2. Model Heterogeneity: Qwen-2.5-7B-Instruct showed deterministic tendencies, while Phi-mini-MoE-Instruct displayed greater variability and briefly approximated human rates.
3. Model Limitations: Despite their potential, open-source LLMs are not yet reliable substitutes for human participants in behavioral simulation.

Engineering Trade-offs and Challenges

1. Memory Mechanism Impact: The high cooperation rates in the shuffled setting may stem from insufficient memory mechanisms, hindering the models' ability to simulate dynamic decision-making.
2. Model Scale and Computational Cost: Larger models (e.g., Llama 3.1 8B Instruct) perform better but incur higher computational costs, necessitating a balance between performance and resource consumption.
3. Need for Standardized Evaluation Frameworks: To accurately assess LLM capabilities in simulating human behavior, more comprehensive evaluation frameworks are needed, including diverse scenarios and detailed metrics.

Developer Recommendations

1. Model Selection: Developers should choose models based on specific application needs and fine-tune them with real-world data.
2. Memory Mechanism Optimization: Introducing more sophisticated memory mechanisms, such as LSTM or Attention, can enhance model performance in dynamic environments.
3. Multi-Scenario Testing: Comprehensive testing across multiple scenarios and rounds is crucial before deploying models in real-world applications.

Conclusion

Open-source LLMs show promise in simulating human cooperation but face significant challenges. Future research should focus on enhancing memory mechanisms, optimizing model architectures, and establishing robust evaluation frameworks to improve their performance in behavioral simulation.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-20)

— END —

Tags: #LLMs & Foundation Models Follow #Prisoner's Dilemma Follow #Behavioral Simulation Follow #Open-Source Models Follow #AI Follow