Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
Paper · https://arxiv.org/abs/2610.40726
arXiv:2610.40726v1 Announce Type: new Abstract: Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at https://github.com/jd-cc-coder/Li2SAO.
First seen by Oriel on .
AI summary
Li$^{2}$SAO improves asynchronous LLM reinforcement learning by bootstrapping a cold critic with group-relative training before switching to single-rollout GAE, matching or exceeding GRPO early and improving later performance.
- Group-wise sampling in GRPO fits asynchronous training poorly because rollout lengths follow a long-tailed distribution.
- GRPO outperforms lightweight LiSAO on single-turn reasoning, while LiSAO performs better on multi-turn agentic tasks.
- LiSAO initially trails GRPO substantially because its value model is not yet well trained, unlike SAO’s pretrained closed-source value model.
- Li$^{2}$SAO trains the actor with group-relative advantage and the cold critic with group-centered return, then switches to single-rollout GAE.
- The authors provide code for the method.
#reinforcement learning #large language models #asynchronous training #actor-critic
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.40726v1 | https://arxiv.org/abs/2610.40726 Read discussion |