Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox
Web page · https://huggingface.co/blog/microsoft/thinkingbox-training
First seen by Oriel on .
AI summary
The post describes using executable checks of final database state as RL rewards, improving Qwen3.8-27B’s repeatable success on ThinkingBox-Bench after training on a fixed, policy-relative task pool.
- After 50 GRPO steps on 187 tasks, pass@1 rose from 51.70% to 60.78%, and tasks passing all 20 attempts increased from 38 to 107.
- Coverage declined slightly: tasks solved at least once fell from 453 to 446, showing that repeatability improved while breadth narrowed.
- Training tasks were selected offline by probing the base model and retaining tasks with intermediate posterior success probabilities; benchmark tasks were excluded from training.
- Each rollout used a fresh backend, isolated MCP session, and GPT-5 Chat simulated user; most rewards came from deterministic terminal-state checks.
- The out-of-distribution benchmark shared the ThinkingBox framework and some tools with training, limiting the scope of the transfer claim.
#reinforcement learning #llm agents #database benchmarks #grpo
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/microsoft/thinkingbox-training | https://huggingface.co/blog/microsoft/thinkingbox-training Read discussion |