Entropy Crossings in Recursive Self-Training Reveal Uneven Vocabulary Collapse
Summary:A study of nine generations of recursive self-training with GPT-2 reveals that Rényi entropy measures cross from decline to improvement at different temperatures: min-entropy turns first, followed by collision entropy, Shannon entropy, and observed richness. At temperature 1, the ninth generation contains 24% fewer word types while collision entropy is 0.91 bits higher, showing that vocabulary contraction and reduced dominance of frequent words can occur simultaneously rather than representing a
Main Finding: Model Collapse Is Not a Single Process
This study fine-tunes GPT-2 on its own generated text for nine consecutive generations and varies the sampling temperature used to construct the recursive training data. The results show that recursive self-training does not necessarily follow a single path toward classical mode collapse, in which probability mass becomes increasingly concentrated on a few tokens.
Within some temperature intervals, the number of observed word types continues to fall while the dominance of the most frequent word declines. Vocabulary richness and distributional uniformity can therefore move in opposite directions. The central conclusion is direct: richness is not evenness, and no single entropy statistic fully characterizes model collapse.
How the Entropy Crossings Work
The paper examines four statistical measures associated with different orders of Rényi entropy:
- Min-entropy, which reflects the probability mass of the most likely token;
- Collision entropy, which approximates the effective number of word types;
- Shannon entropy, which measures overall distributional uncertainty;
- Observed richness, represented by the number of word types actually present.
As the training temperature increases, the relative changes in these measures turn from loss to gain at different locations, producing the paper's “crossings.” The typical order is min-entropy first, followed by collision entropy, Shannon entropy, and observed richness.
The order matters: a weakening advantage for the most probable token does not imply that additional word types immediately reappear. The two quantities can occupy opposite regimes.
The general form of Rényi entropy is:
$$H_\alpha(X)=-\frac{1}{1-\alpha}\log \sum_x p(x)^\alpha$$
Different values of $\alpha$ place different emphasis on the tail of the distribution. Min-entropy approaches the $\alpha\to\infty$ limit and focuses on the maximum probability, whereas lower orders are more sensitive to how probability mass is distributed across less likely outcomes. Multiple Rényi orders therefore provide complementary views rather than redundant measurements.
Key Result at Temperature 1
At a training temperature of $T=1$ and an evaluation budget of approximately 13,000 words, the ninth generation contains 24% fewer word types than the first generation, yet its collision entropy is 0.91 bits higher.
This result rules out a simple equivalence between fewer types and lower collision entropy. A more accurate interpretation is:
- Some lower-frequency word types disappear during recursive training, contracting the observed vocabulary boundary;
- High-frequency words, especially English function words, do not gain proportional dominance;
- More of the remaining probability mass is distributed across word types that are still present.
The model can thus exhibit vocabulary contraction and reduced concentration in the main distribution at the same time. Monitoring type-token richness alone would miss changes in distributional structure, while relying only on collision entropy would overlook the potentially irreversible disappearance of word types.
Decoding and Sampling Effects Were Isolated
Training temperature affects the generated training data, while decoding temperature affects measured entropy. The study evaluates each generation with a fixed sampler and repeats the analysis using a finer temperature grid, alternative measurement samplers, a fixed word budget, and five fresh training seeds.
The authors also examine next-token identities and use a categorical simulation to separate decoding and sampling effects. The crossing order remains present under these controls, indicating that it is not merely an artifact of the measurement sampler.
The scope still requires caution. The main experiments use English GPT-2 with a recursion depth of nine generations. The paper adds Hindi experiments and another text representation, where min-entropy again tends to cross before richness, although some Hindi results require interpolated roots. The ordering is therefore empirically robust but is not yet a universal law across languages, model families, and scales.
Why Entropy Direction Is Not a Quality Verdict
The paper also fine-tunes models on human text and finds that a similar entropy split can occur while likelihood improves. A Rényi curve changing from decline to gain may consequently accompany worse linguistic quality, or it may coexist with higher likelihood.
Different entropy measures capture different aspects of a probability distribution and cannot be equated directly with semantic quality, factuality, or downstream task performance. During recursive generation, the training distribution changes at every iteration. A model's likelihood on its own outputs may improve through distributional alignment even as knowledge coverage or useful diversity deteriorates.
A single indicator's sign therefore cannot reliably identify worse model quality. The recommended protocol is to report:
- The training and measurement temperatures;
- A fixed and reproducible token or word budget;
- Several Rényi orders or the complete entropy profile;
- An independent quality measure, such as validation likelihood, perplexity on human text, or a task-specific score.
Engineering Recommendations
For pipelines that continually retrain on model-generated outputs, teams should record observed vocabulary coverage, collision entropy, Shannon entropy, and the highest token probability at each iteration rather than monitoring perplexity alone. In multilingual corpora, function-word distributions and Zipf tail lengths vary across languages, making English-derived thresholds unsuitable as universal boundaries.
The evaluation protocol should include the budget explicitly. Comparing generations at equal token or word counts avoids mistaking changes in output volume for distributional changes. Measurement sampling settings should remain fixed, while training and evaluation temperatures should be recorded separately.
Entropy should serve as a diagnostic tool rather than an automatic gate. It can identify when a model enters a new distributional regime, but decisions about continuing recursive training should rely on independent validation data, human-written references, and downstream regression tests.
Limitations
The primary experiments rely on a relatively small GPT-2 model and do not establish whether the same ordering holds for large instruction-tuned models, code models, or long-context systems. The nine-generation recursion also captures short-horizon degradation, leaving longer recursive runs and possible additional phase transitions unresolved.
The paper does not reduce the phenomenon to a single computable collapse index; instead, it advocates a multi-metric profile. That is more robust but also more expensive to analyze. Its main contribution is therefore methodological rather than architectural: vocabulary contraction, probability concentration, and overall uncertainty must be measured separately.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-09)
Tags: #Model Collapse #Recursive Self-Training #Rényi Entropy #GPT-2 #Language Model Evaluation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments