Microsoft and Hugging Face released ThinkingBox-Training on GitHub after a 50-step test with Qwen3.8-27B.
An AI agent can report success while leaving the wrong database record behind. ThinkingBox checks the final database state instead, then uses that executable check as a training reward.
Microsoft and Hugging Face say a task earns 1 when the backend ends in the expected state. Wrong, missing, or extra side effects earn 0. This rewards the result rather than one preferred sequence of tool calls, allowing different paths to count when they produce the correct records.
The training release first probes the base model on candidate tasks. Tasks it always solves or always fails provide little learning signal, while tasks with mixed results can show the model which attempts worked. The teams selected a fixed pool of 187 such tasks, then fine-tuned Qwen3.8-27B for 50 reinforcement-learning steps.
For evaluation, they ran 507 benchmark tasks 20 times each. Average success increased from 51.70% to 60.78%. Tasks that passed all 20 attempts rose from 38 to 107. But tasks solved at least once fell from 453 to 446.
That is the practical tradeoff: the model became more consistent on some tasks, while losing a small amount of reach across the full set. A single accuracy score would hide that difference.
The reported benchmark uses synthetic reconstructions of enterprise workflows. The training and test sets also share parts of the same framework, simulated users, grading system, and task style, so performance on real company databases remains an open question.
Reproducing the reference run is substantial: Microsoft and Hugging Face used 24 H100 GPUs, alongside the 187-task training pool. Teams considering this approach should test their own tools and failure cases while tracking both repeatability and task coverage.