StructRL: Model-Free Reinforcement Learning with Replay Graphs
Paper · https://arxiv.org/abs/2611.14599
arXiv:2611.14599v1 Announce Type: new Abstract: StructRL turns the experience collected during reinforcement learning into an explicit replay graph. Once the graph contains a path to the task goal, dynamic programming selects actions on the observed part of the state space; these actions form a graph policy that is defined only where the required transitions have been observed. StructRL transfers these actions to model-free learners: as action preferences for a DQN Q-model and as target directions along shortest observed paths for a SAC actor. For DQN on three sparse-reward MiniGrid tasks, final success rises from 0.10 to 0.90 (DoorKey), 0.10 to 0.80 (KeyCorridor) and 0 to 0.50 (ObstructedMaze); plain DQN does not catch up with three times the budget. For SAC on continuous PointMaze, graph-derived direction labels raise final success from 0.17 to 1.0 on seven matched runs that reached the goal. Distilling the graph policy into a fresh Q-model gives a DoorKey 6x6 controller that reaches the goal from 91% of reachable states, against 38% for plain DQN. Scaling experiments identify three bottlenecks: incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.
First seen by Oriel on .
AI summary
StructRL builds replay graphs from reinforcement-learning experience and transfers graph-derived action guidance to model-free learners, substantially improving success on sparse-reward MiniGrid and continuous PointMaze tasks.
- Dynamic programming creates a partial graph policy once observed transitions contain a path to the goal.
- StructRL supplies action preferences to DQN Q-models and shortest-path direction targets to a SAC actor.
- DQN success increased from 0.10 to 0.90 on DoorKey, 0.10 to 0.80 on KeyCorridor, and 0 to 0.50 on ObstructedMaze.
- On seven PointMaze runs reaching the goal, SAC success rose from 0.17 to 1.0; a distilled DoorKey controller covered 91% of reachable states.
- Identified scaling bottlenecks are incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.
#reinforcement learning #replay graphs #dqn #sac
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.14599v1 | https://arxiv.org/abs/2611.14599 Read discussion |