The proposed benchmark combines new tasks, new rooms, controller comparisons, and physical-robot trials.
A research team proposes LocoManiLab, a benchmark for testing whether humanoid robots can combine walking and handling objects outside familiar routines. The paper, posted on arXiv, covers 234 tasks across kitchens, living rooms, bedrooms, bathrooms, and offices.
The test separates three challenges: familiar tasks, new task definitions in known rooms, and new tasks in held-out scenes. It also changes visual conditions such as lighting, textures, camera views, and added clutter. That matters because a robot can memorize a drawer-opening routine without knowing how to adapt when the room or goal changes.
The authors report sharp performance drops under these tests. XR-1’s average success fell from 80.2% on seen tasks to 25.2% on unseen tasks. On the harder unseen-scene split, the best reported average was 15.5%, achieved by DM0.5.
LocoManiLab also tests the machinery beneath a robot’s high-level policy. A policy chooses intended actions from vision and language; a whole-body controller turns those commands into balanced movement. The benchmark compares twelve policies with one shared controller, then compares two policies across five controllers while measuring success, power use, slipping, falls, motion changes, and completion time.
Training data is another variable. The testbed includes 14,160 simulated teleoperation episodes totaling 51.5 hours, plus synthesized and altered demonstrations. More examples usually helped, but the gains depended on the policy and controller. Teleoperation was the strongest single data source in the reported mixture study.
Three policies also ran three matched tasks in simulation and on a Unitree G1. Their overall rankings matched, but individual task results differed. For operators, the practical next step is testing these protocols across more robots, tasks, and repeated training runs before treating simulation scores as deployment evidence.