Oriel — AI & technology worth noticing, every day.

RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Paper · https://arxiv.org/abs/2610.15772

Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.

First seen by Oriel on .

AI summary

RSIPO is an agentic reinforcement learning framework that co-evolves skills and policy models, outperforming compared baselines on embodied interaction and search tasks.

  • The policy extracts skills from completed trajectories and improves skill generation through trajectory replay.
  • Empirical Bayes shrinkage calibrates how much the student trusts teacher scores, accounting for potentially less reliable skill-enhanced guidance.
  • On ALFWorld, Qwen2.5-7B with RSIPO achieves a 98.6% average success rate, 2.5 percentage points above the second-best baseline and 17.4 above standard GRPO.
  • Experiments use Qwen-family models across embodied interaction and search-based agentic tasks.

#reinforcement learning #agentic ai #policy optimization #self-distillation

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 8, latest observation ; +1 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.15772https://arxiv.org/abs/2610.15772 Read discussion