Columbia researchers propose a gaze system for a soft-skinned humanoid head. The benchmark measures motion accuracy, while human judgments of naturalness remain untested.
A humanoid robot does not look at someone with one command. It must choose a target, aim both eyes, move its head in coordination, and manage its eyelids without appearing mechanical.
A paper from Columbia University’s Creative Machines Lab proposes the Realistic Gaze Transformer, or RGT, a research framework that divides those jobs. The team reports motion benchmarks and physical demonstrations, but the work has not yet tested whether people find the robot’s gaze natural or easy to understand.
The central engineering idea is simple: decide where to look, then generate how to move.
One system chooses; another moves
The first stage uses a vision-language model, a type of artificial intelligence that connects images with words. Given a scene and an instruction such as “notice the man in the white shirt,” it selects a person, object, or region as the robot’s intended target.
That decision runs asynchronously, meaning it can update separately from the faster motor-control loop. The high-level model does not need to produce every motor command.
A reactive perception module then tracks the selected target between those updates. The paper uses FastSAM, a segmentation tool that identifies the pixels belonging to a prompted object or person. The system converts that tracked location into a continuous gaze direction.
RGT receives that direction along with recent motion history. It generates coordinated commands for the neck, eyes, and eyelids. In engineering terms, the design separates a high-level attention decision from low-level motion control.
This separation also makes testing more useful. Engineers can examine whether the robot moves smoothly toward a target without mixing that question with whether another model selected the correct target.
What the robot controls
The robotic head has 11 degrees of freedom, meaning 11 independently controlled motions. Three control the neck, four control the left and right eyes, and four control the upper and lower eyelids.
The motion generator is a Transformer-VAE. In plain English, it is a sequence model that studies recent movement, compresses its motion style into an internal representation, and generates future movement from that history and a planned eye path.
The researchers trained it on human facial recordings converted into the robot’s joint coordinates. They then fine-tuned it autoregressively. That means training included histories containing the model’s own earlier predictions, rather than only clean recorded motion.
This matters because a real controller feeds its outputs back into the next step. If each prediction is slightly wrong, those errors can accumulate. Training with self-generated histories exposes the model to that problem before evaluation.
The system handles blinking through a separate auxiliary path. The model predicts blink timing, and the controller can play back blink templates while the generated neck and eye movements continue.
That arrangement coordinates broad eye-and-head motion without treating every eyelid movement as part of the same control problem. It is a practical architecture choice, not proof that the blinks look natural to people.
What the benchmark actually measures
The primary benchmark tests motion generation, not the complete attention system. Every controller receives the same four-dimensional eye target: the horizontal and vertical direction for each eye.
That setup removes errors from choosing a target and tracking it visually. The result answers a narrow question: given the intended eye direction, how closely can each controller reproduce the tested robot-space motion?
The benchmark uses 1,848 clips. Each closed-loop rollout covers a two-second future, and the main score covers seven neck and binocular-eye channels. Eyelid and blink motion are not included in that score.
The paper reports that the base RGT produced a joint-angle mean squared error of 2.19 × 10⁻⁴ radians squared. An RNN baseline scored 3.04 × 10⁻⁴ radians squared. The researchers describe that difference as a 27.9% reduction.
Mean squared error measures the distance between predicted joint angles and recorded reference angles. A lower value supports a limited claim: under this test, the base model reproduced the measured motion more closely than the comparison model.
It does not establish that people will find the gaze natural. The reported table also comes from one stochastic evaluation run without confidence intervals, so repeated runs would better show how stable the difference is.
What the physical demonstration shows
The researchers also demonstrated the targeting and control components working together on the physical humanoid head. That matters because a system can perform well in isolated software tests yet fail when perception, motion generation, and hardware interact.
The demonstration shows integrated operation. It does not answer whether the system selects the right target in open-ended scenes, maintains fixation when a target is partly hidden, or stays stable during longer interactions.
The benchmark supplies the correct target, so it does not measure the vision-language model’s open-vocabulary target selection. It also does not measure whether the reactive tracker maintains fixation under occlusion.
The two-second scored rollouts leave another practical question open: whether motion remains stable during longer exchanges. A robot helping with a handover or collaborative search may need to shift attention, pause, and recover as people or objects move.
Naturalness requires people in the loop
People judge gaze through timing and context, not joint-angle error alone. They may notice whether the robot looks at the right person, shifts attention at a sensible moment, and combines eye, head, and eyelid movement convincingly.
The paper reports no controlled observer ratings for naturalness, attention legibility, preference, or trust. Those are separate tests from measuring whether the robot’s joints follow a target.
The next useful step is a human-robot interaction study. Participants should be asked what the robot appears to be looking at, whether its gaze feels natural, and how it compares with simpler controllers.
For now, RGT is best understood as a proposed motor-control layer: a way to coordinate a humanoid face after another system has decided where to look. Its benchmark supports that engineering claim, while the social question—whether people experience the gaze as natural—still needs direct testing.