arXiv Study on the Role of Citations in Preference Data: Unveiling Differences Between Human and LLM Decision-Making
Summary:arXiv has released a study examining the role of citations in preference data, focusing on how humans and open-source Large Language Models (LLMs) evaluate citations in scientific question-answering tasks. Using mixed-effects models, the research reveals that humans prefer more diverse citations but fewer in total, while LLMs, despite lacking access to source materials, exhibit citation-related preferences that vary depending on the data and specific models. The findings have significant implica
Background and Objectives
In Natural Language Processing (NLP) tasks, systems are required to provide citations in their outputs to prevent model hallucination and help users verify the credibility of the information. However, it is unclear how humans and LLMs evaluate citations when comparing outputs, a process central to reward modeling and modern LLM post-training.
Key Findings
- Human Preferences: Humans prefer more diverse citations but fewer in total. This suggests that humans value the breadth and depth of information rather than sheer quantity.
- LLM Preferences: Despite lacking access to source materials, LLMs exhibit certain citation-related preferences that differ from those of humans and depend on the data and specific models. For example, some models may favor citations from authoritative sources, while others may prioritize the number of citations over their quality.
Methodology
The study employed mixed-effects models to analyze pairwise judgments of humans and LLMs in scientific question-answering tasks, investigating the influence of citations on preference formation. By comparing the performance of different models and datasets, the research unveiled the complex role of citations in AI decision-making.
Industry Implications and Insights
- Impact on Reward Modeling: The findings indicate that LLM preferences differ from those of humans, suggesting the need for more nuanced consideration of citation diversity and quality in designing reward models.
- Implications for Post-Training: Understanding LLM citation preferences can help improve post-training processes to align AI behavior more closely with human cognitive habits.
- AI Ethics Considerations: The study underscores potential biases in AI information processing, prompting developers to pay attention to citation choices in AI outputs to avoid misleading users.
Recommendations for Developers
- Optimize Reward Models: When training LLMs, consider the diversity and quality of citations, not just the quantity.
- Improve Post-Training Strategies: Enhance post-training by introducing citation evaluation mechanisms that better reflect human cognitive processes.
- Focus on AI Ethics: In designing AI systems, consider the impact of citation choices on user trust and AI credibility.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-08-25)
Tags: #arXiv #Citation Study #LLMs & Foundation Models #Reward Modeling #AI Ethics
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments