RL Post-Training Can Hitchhike Untargeted Traits

Summary:This work introduces the Price equation from evolutionary biology into the analysis of reinforcement-learning post-training for language models. The resulting “trait hitchhiking” framework describes how non-targeted behaviors can be amplified or suppressed as reward optimization redistributes probability among sampled completions. The authors derive a probability-update expression and evaluate it on language models trained with GRPO. The Price-based estimates closely track the observed trait dy

---

## Core Finding: Better Rewards Do Not Guarantee Targeted Behavior Changes Reinforcement-learning post-training for language models is usually evaluated through task rewards, but each policy update changes the entire completion distribution. Even when a reward does not explicitly target writing style, refusal behavior, explanation structure, or safety-related tendencies, these “non-targeted traits” may rise or fall as reward optimization redistributes probability among sampled outputs. The paper calls this phenomenon **trait hitchhiking**. The authors adapt the Price equation from evolutionary biology to describe changes in a trait during post-training. This formalizes how a policy update can produce changes that are statistically associated with reward movement but were never specified by the reward objective. ## How the Mechanism Works For an autoregressive language model, the probability of a complete response is: \[ P(y)=\prod_t P(y_t\mid y_{ **Source**: [ArXiv Machine Learning (cs.LG)](https://arxiv.org/abs/2611.14683) (2026-11-13) — END —

Tags: #LLM Post-Training Follow #GRPO Follow #Reinforcement Learning Follow #Behavioral Drift Follow #Price Equation Follow