arXiv Releases Li2SAO: A Novel Approach to Optimize Asynchronous Reinforcement Learning
Summary:arXiv has recently released a new method called Li2SAO, which optimizes asynchronous reinforcement learning (RL). By introducing group-relative actor-critic bootstrapping, Li2SAO addresses the issue of poor early training performance in existing single-rollout asynchronous optimization (SAO) methods. Experiments show that Li2SAO outperforms GRPO in early training and further raises the performance ceiling in later stages. This research provides a new technical path for improving training efficie
1. Background and Motivation
Asynchronous reinforcement learning (RL) has become a mainstream paradigm for post-training large language models (LLMs) due to its efficiency in updating the model as rollouts arrive. However, existing methods like Group-Relative Policy Optimization (GRPO) struggle with the long-tailed distribution of rollout lengths, while Single-rollout Asynchronous Optimization (SAO) improves stability and off-policy behavior but relies on a pre-trained value model, limiting its applicability.
2. Core Mechanism of Li2SAO
Li2SAO optimizes asynchronous RL training through the following:
- Group-Relative Actor-Critic Bootstrapping: While the critic is cold, a group-relative advantage trains the actor, and a group-relative return trains the critic.
- Single-Rollout GAE Switching: After the critic is trained, the training switches to single-rollout Generalized Advantage Estimation (GAE).
This mechanism addresses the shortcomings of SAO in early training while maintaining its advantages in complex tasks.
3. Experiments and Results
Experiments conducted on single-turn reasoning and multi-turn agentic tasks show that:
- GRPO outperforms LiSAO in reasoning tasks.
- LiSAO outperforms GRPO in agentic tasks.
- Li2SAO catches up with or surpasses GRPO in early training and further raises the performance ceiling in later stages.
4. Engineering Trade-offs and Challenges
While Li2SAO improves training efficiency, it also introduces new challenges:
- Computational Overhead: The group-relative bootstrapping increases computational complexity, requiring more efficient parallelization strategies.
- Memory Usage: More intermediate states need to be stored during training, increasing memory requirements.
5. Future Directions
Li2SAO provides a new direction for asynchronous RL training. Future research directions include:
- Further optimizing the computational efficiency of group-relative bootstrapping.
- Exploring more efficient pre-training methods for the value model.
- Applying Li2SAO to larger-scale LLM training.
6. Developer Recommendations
For developers, the code for Li2SAO is available on GitHub. The following recommendations are provided for deployment:
- Train in a hardware environment with sufficient resources to cope with increased memory and computational demands.
- Adjust the parameters of group-relative bootstrapping according to the specific task requirements to achieve optimal performance.
Conclusion
Li2SAO provides a new optimization method for asynchronous RL training, improving training efficiency while maintaining high performance in complex tasks, bringing a new technological breakthrough to the AI field.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-02)
Tags: #asynchronous reinforcement learning #Li2SAO #arXiv #RL optimization #AI training
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments