RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
Paper · https://arxiv.org/abs/2610.15772
Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.
First seen by Oriel on .
AI summary
RSIPO is an agentic reinforcement learning framework that co-evolves skills and policy models, outperforming compared baselines on embodied interaction and search tasks.
- The policy extracts skills from completed trajectories and improves skill generation through trajectory replay.
- Empirical Bayes shrinkage calibrates how much the student trusts teacher scores, accounting for potentially less reliable skill-enhanced guidance.
- On ALFWorld, Qwen2.5-7B with RSIPO achieves a 98.6% average success rate, 2.5 percentage points above the second-best baseline and 17.4 above standard GRPO.
- Experiments use Qwen-family models across embodied interaction and search-based agentic tasks.
#reinforcement learning #agentic ai #policy optimization #self-distillation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 8, latest observation ; +1 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 2, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2610.15772 | https://arxiv.org/abs/2610.15772 Read discussion |