What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Paper · https://arxiv.org/abs/2611.06483
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/
First seen by Oriel on .
AI summary
The paper finds that multi-turn reinforcement learning mainly improves error recovery and long-horizon resilience, with gains that depend on having sufficient interaction turns.
- The authors evaluate agents across sampling breadth K and interaction depth T using Pass@(K,T).
- Trajectory-level RL gains diminish significantly when interaction budgets are tightened.
- Agentic RL does not consistently improve the quality of single-step decisions; its benefits concentrate on recovery and resilience.
- The paper discusses process supervision's benefits and limitations and introduces BudgetMix RL, which varies training interaction budgets to improve robustness across execution settings.
#llm agents #reinforcement learning #process supervision #agent evaluation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 35, latest observation ; +5 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 3, latest observation ; +1 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.06483 | https://arxiv.org/abs/2611.06483 Read discussion |