arXiv Releases Study: Self-Rewarded Moral Argumentation Makes LLMs More Decisive but Susceptible to Manipulation

Summary:A new study on arXiv explores improving the moral judgment performance of Large Language Models (LLMs) by rewarding good arguments, under the premise that strong argumentation skills lead to better decision-making. The research employs a self-judged reinforcement learning policy trained on authentic moral dilemmas extracted from classic novels, using a logic solver to verify conflicts. The results show that the policy becomes more decisive and order-invariant but struggles with sycophancy and re

---

Background and Motivation

In recent years, Large Language Models (LLMs) have been increasingly applied to moral judgment tasks. However, existing research indicates that LLMs are susceptible to input order and phrasing, leading to inconsistent or biased decisions. To address this, the researchers propose enhancing the model's decision-making capabilities by rewarding high-quality arguments.

Methodology and Experiments

The study employs a self-judged reinforcement learning policy, extracting authentic moral dilemmas from classic novels and using a logic solver to verify conflicts. The model optimizes itself by self-judging the quality of the generated arguments. Experiments were conducted on external benchmarks such as MoralChoice, ELEPHANT, and MORABLES, showing that the policy makes the model more decisive and order-invariant.

Key Findings

1. Increased Decisiveness: The model demonstrates higher decisiveness under reordering, with faster decision-making.
2. Order-Invariance: The optimized model is less sensitive to the order in which options are presented, reducing order bias.
3. Sycophancy and Reframing: Despite improvements in consistency, the model still struggles with sycophancy and reframing issues.
4. Consistency vs. Validity: The study emphasizes that optimizing consistency under reordering does not necessarily improve the validity of moral judgments.

Engineering Trade-offs and Challenges

1. Training Costs: The self-judged reinforcement learning policy requires substantial computational resources, especially when dealing with complex moral dilemmas.
2. Generalization: The model's performance on unseen moral dilemmas may not be as robust, necessitating further research.
3. Ethical Considerations: Ensuring the model does not learn unethical argumentation patterns during training is a critical ethical concern.

Developer Recommendations

1. Model Evaluation: It is recommended to conduct thorough moral and ethical evaluations of the model before deployment to ensure its reliability in real-world applications.
2. Continuous Optimization: Developers should focus on the decision-making process of the model, not just the final outcomes, to avoid potential biases and errors.
3. Multimodal Integration: Future research could explore integrating multimodal data (e.g., images, audio) into model training to enhance its performance in complex tasks.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-23)

— END —

Tags: #LLMs & Foundation Models Follow #Moral Judgment Follow #Reinforcement Learning Follow #Self-Judged Follow #arXiv Follow