Oriel — AI & technology worth noticing, every day.

Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

Web page · https://huggingface.co/blog/microsoft/thinkingbox-training

First seen by Oriel on .

AI summary

The post describes using executable checks of final database state as RL rewards, improving Qwen3.8-27B’s repeatable success on ThinkingBox-Bench after training on a fixed, policy-relative task pool.

  • After 50 GRPO steps on 187 tasks, pass@1 rose from 51.70% to 60.78%, and tasks passing all 20 attempts increased from 38 to 107.
  • Coverage declined slightly: tasks solved at least once fell from 453 to 446, showing that repeatability improved while breadth narrowed.
  • Training tasks were selected offline by probing the base model and retaining tasks with intermediate posterior success probabilities; benchmark tasks were excluded from training.
  • Each rollout used a fresh backend, isolated MCP session, and GPT-5 Chat simulated user; most rewards came from deterministic terminal-state checks.
  • The out-of-distribution benchmark shared the ThinkingBox framework and some tools with training, limiting the scope of the transfer claim.

#reinforcement learning #llm agents #database benchmarks #grpo

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/microsoft/thinkingbox-traininghttps://huggingface.co/blog/microsoft/thinkingbox-training Read discussion