Vietnam–Japan Team Builds Modular Humanoid for Multimodal Interaction

An arXiv preprint describes a laboratory platform that turns gestures, spoken instructions and camera data into robot actions.

Researchers at the University of Engineering and Technology, Vietnam National University, Hanoi, and the Japan Advanced Institute of Science and Technology report a humanoid prototype designed as a testbed for human–robot interaction research. It is a research platform evaluated indoors, not evidence of household deployment.

The system separates fast, low-level actuation from higher-level AI. A custom controller board manages the robot’s motors, while an onboard NVIDIA Jetson AGX Xavier runs ROS, perception, language processing and trajectory planning. Its body uses a 12-degree-of-freedom dual-arm mechanism and a two-degree-of-freedom head with a seven-inch LCD for facial animations. Aluminum frame elements provide structure; custom 3D-printed joints and brackets support fabrication, replacement and mechanical customization.

Three software modules connect human input to movement. MediaPipe Pose extracts body keypoints, and an LSTM classifier recognizes predefined gestures such as waving, handshaking and “love” gestures. A ZED 2 stereo camera provides RGB and depth data; YOLO detects objects, while 3D localization supplies coordinates for manipulation. For voice commands, speech recognition converts audio to text, then a large language model parses an instruction such as “Take the cup and put it on the plate” into an action–object–location structure that becomes pick-and-place commands.

The reported evaluation covered gesture interaction and voice-driven object manipulation in an indoor laboratory with consistent lighting and minimal occlusion. The paper reports 96% gesture-recognition accuracy, 92% speech-recognition accuracy and approximately 90% pick-and-place success.

The practical next test is less controlled: varied lighting, occlusion, background noise and unfamiliar users. The voice pipeline also relies on the Gemini API, so internet dependence and latency remain part of the system’s operating design.