Oriel — AI & technology worth noticing, every day.

StructRL: Model-Free Reinforcement Learning with Replay Graphs

Paper · https://arxiv.org/abs/2611.14599

arXiv:2611.14599v1 Announce Type: new Abstract: StructRL turns the experience collected during reinforcement learning into an explicit replay graph. Once the graph contains a path to the task goal, dynamic programming selects actions on the observed part of the state space; these actions form a graph policy that is defined only where the required transitions have been observed. StructRL transfers these actions to model-free learners: as action preferences for a DQN Q-model and as target directions along shortest observed paths for a SAC actor. For DQN on three sparse-reward MiniGrid tasks, final success rises from 0.10 to 0.90 (DoorKey), 0.10 to 0.80 (KeyCorridor) and 0 to 0.50 (ObstructedMaze); plain DQN does not catch up with three times the budget. For SAC on continuous PointMaze, graph-derived direction labels raise final success from 0.17 to 1.0 on seven matched runs that reached the goal. Distilling the graph policy into a fresh Q-model gives a DoorKey 6x6 controller that reaches the goal from 91% of reachable states, against 38% for plain DQN. Scaling experiments identify three bottlenecks: incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.

First seen by Oriel on .

AI summary

StructRL builds replay graphs from reinforcement-learning experience and transfers graph-derived action guidance to model-free learners, substantially improving success on sparse-reward MiniGrid and continuous PointMaze tasks.

  • Dynamic programming creates a partial graph policy once observed transitions contain a path to the goal.
  • StructRL supplies action preferences to DQN Q-models and shortest-path direction targets to a SAC actor.
  • DQN success increased from 0.10 to 0.90 on DoorKey, 0.10 to 0.80 on KeyCorridor, and 0 to 0.50 on ObstructedMaze.
  • On seven PointMaze runs reaching the goal, SAC success rose from 0.17 to 1.0; a distilled DoorKey controller covered 91% of reachable states.
  • Identified scaling bottlenecks are incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.

#reinforcement learning #replay graphs #dqn #sac

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.14599v1https://arxiv.org/abs/2611.14599 Read discussion