The research system combines first-person demonstrations, simulated robot states, and rendered robot hands before testing a Unitree G1 on one bottle-moving task.
HuGe is a research system that teaches a humanoid robot from human first-person experience. Its central idea is practical: human videos provide examples of what to do, while robot data and simulation help translate those examples into movements the robot can execute.
The paper, published on arXiv, describes HuGe as a family of humanoid vision-language-action models trained on more than 2,000 hours of human and robot data. A vision-language-action model connects what a robot sees and what a person asks with physical actions.
HuGe is not a product people can buy. It is a research method tested in simulation and in a limited real-world experiment on a Unitree G1.
The translation problem
A human video shows a person looking at a bottle, reaching for it, and carrying it to a bin. But it does not directly tell a robot how to move its joints, keep its balance, or control its hands.
The robot also sees the world differently. A first-person human camera may show human arms and hands, while a humanoid robot has different arms, joints, hands, and camera placement. Human video normally contains no robot body measurements, such as the current angle of each joint or the direction of gravity relative to the torso.
HuGe addresses those gaps with three linked training stages: first-person pre-training, humanoid adaptation, and task-specific fine-tuning.
Stage one: teach the model from many viewpoints
During pre-training, HuGe learns from human first-person demonstrations mixed with robot data. The researchers report more than 2,000 hours of combined egocentric human and robot data.
“Egocentric” means the camera records from the actor’s point of view. This gives HuGe examples of tasks as a person sees them, rather than only recordings of a robot being teleoperated.
The model does not send raw human arm motions straight to the robot. Instead, HuGe predicts a compact motion representation called a latent motion token. Think of this as a high-level movement instruction, such as a coordinated reaching or walking pattern, rather than a separate command for every motor.
A pretrained whole-body controller then converts that token into joint-level movement. This controller coordinates the arms, torso, and legs. HuGe also predicts whether each hand should open or close.
That division matters. The model decides what coordinated movement fits the task, while the lower-level controller handles the detailed body motion needed to perform it.
Filling in the missing body data
Human videos still lack the robot’s internal body-state information, known as proprioception. Proprioception is a robot’s sense of its own configuration: joint positions, hand states, and body orientation.
HuGe creates a substitute using simulation. The researchers replay motion tokens on a simulated Unitree G1 and record the resulting body states. These simulated states become “pseudo-proprioceptive” inputs paired with the human video.
For the G1, the state representation includes 29 body joint positions, two hand-state entries, and the three components of the gravity direction in the robot’s body frame.
This does not make a human body identical to a G1. It gives the model a robot-shaped state signal to use while learning how visual observations connect to humanoid movement.
The simulation result was useful but not universal. On the HumanoidArena benchmark, the paper reports average success rates of 66.67% for HuGe-GR00T and 65.00% for HuGe-Pi using the full first-person training corpus. Another HuGe version performed worse at larger data scales, showing that more data does not automatically help every model design.
Making human videos look more like robot videos
The next mismatch is visual. A robot trained on human footage may learn to expect human arms and hands in front of the camera.
During task fine-tuning, HuGe’s visual adaptation pipeline removes the visible human hands and arms, fills the exposed image regions, and inserts rendered robotic hands. The process keeps the original action labels, body-state inputs, and language instructions.
In plain English, the researchers edit the training video so the model sees a scene closer to what a robot camera would observe, while preserving what the person was doing.
The reported real-world comparison helps separate the value of extra human examples from the value of this visual editing. With 110 robot demonstrations alone, the G1 completed the full task in 13 of 30 trials, or 43.3%.
Adding 400 raw first-person demonstrations raised success to 18 of 30 trials, or 60.0%. Using visually adapted versions of those same human demonstrations raised it to 20 of 30, or 66.7%.
What the robot actually did
The real-world test used a Unitree G1 fitted with Inspire dexterous hands. The robot approached a table, picked up a bottle, turned and walked toward a bin, then crouched to place the bottle inside.
This is more demanding than a fixed robot arm moving an object across a table. The G1 had to combine walking, grasping, turning, crouching, and balance in one sequence.
Still, the result covers one task and one robot configuration. The paper reports balance instability during bottle grasping and links some of it to the weight of the dexterous hands. Bottle pickup was the largest failure source, accounting for five of the ten failed trials in the visually adapted condition.
What HuGe changes—and what it does not
HuGe shows a credible engineering route for using human experience to reduce the need for robot demonstrations. Human videos supply task variety, simulation supplies missing robot-state inputs, and visual editing reduces the difference between human and robot observations.
The immediate consequence is not a household humanoid. It is a training strategy that may help researchers collect fewer robot demonstrations for particular tasks.
Before this approach supports routine workplace use, it needs testing across more tasks, hardware setups, environments, and safety conditions. For now, HuGe is best understood as a research pipeline with a promising bottle-moving result—not a deployed robot service.