NVIDIA’s walkthrough links task planning, 3D asset creation, simulation, policy training, and physical walking—but leaves key hardware questions open.
A robot does not learn to cross an obstacle course from one instruction. It needs a digital environment, simulated sensors, a physics model, and many practice attempts.
NVIDIA describes a technical workflow that uses AI agents to organize those steps. The example uses Microduck, a reference robot, and prepares a navigation policy in simulation before evaluating it on a local computer. The source also includes a figure showing a physical Microduck walking with a policy trained in Isaac Lab, but it does not provide deployment details or performance measurements.
From a task prompt to a virtual course
The process starts with a plain-language prompt, such as building and testing a simulated Microduck obstacle course. Hermes and NVIDIA Nemotron turn that request into a project plan.
The plan defines the course, the robot’s expected abilities, the needed objects, and the steps for testing. It also separates visible information from assumptions. A concept image might show an obstacle from one angle while hiding its rear surface or exact dimensions.
A local image-generation model creates a concept image of the course. Hermes and Nemotron analyze it and produce an inventory of reusable objects. Each object receives an identifier, a description, and a cropped reference image.
Hermes then packages the plan, robot information, concept image, asset inventory, and instructions for Astra. That handoff gives the asset-making agent the context needed to continue.
Astra can improve low-resolution reference images, then direct FreeCAD and Blender to create editable models. FreeCAD builds the basic computer-aided design structure. Blender adds details and materials. The resulting objects are saved as editable models and exported as OpenUSD assets, a format for moving 3D content between tools.
The workflow can also reuse more than 2,000 simulation-ready assets from the SimReady Central Library. Existing objects may reduce modeling work, though the team still has to check whether each object fits the task.
Building a robot that can learn
Astra places the prepared assets into NVIDIA Isaac Sim. This creates the virtual obstacle course and connects it to a simulated robot.
Newton models Microduck’s physical behavior, including movement and balance. In the simulation, the robot’s body uses links, joints, and actuators. Links represent connected body parts, joints limit how those parts move, and actuators drive movement.
The simulated environment also produces observations. These include head-camera depth and information about the robot’s state. When Microduck takes an action, the simulation shows what happened next: whether it moved forward, stayed balanced, collided with something, or fell.
That feedback powers reinforcement learning. In plain English, reinforcement learning lets a policy improve through repeated attempts. The policy receives observations, chooses actions, sees the result, and uses rewards to favor better behavior.
The training loop looks like this:
simulated observations → policy actions → simulated motion → updated observations
The team must define the observations, available actions, and rewards. In this example, Microduck must move over ramps and obstacles while maintaining balance and avoiding falls.
Training thousands of attempts at once
Isaac Lab provides tools for running many simulated robot trials in parallel. The described adaptation run used 4,096 parallel environments on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU.
The training started with a locomotion policy pretrained in an Isaac Lab Microduck example environment. The team then trained it further to handle ramps and obstacles, using head-camera depth and robot-state observations.
Parallel simulation matters because a robot may need many attempts to learn difficult movements. Instead of waiting for one virtual Microduck to finish before starting another, the system gathers experience from thousands of environments at once.
Those trials expose the policy to unstable steps, collisions, and falls. But the number of simulated environments is a compute detail, not a measure of real-world success.
Evaluation comes after training
After training, the selected policy is evaluated in Isaac Sim on a local NVIDIA RTX Spark system. During evaluation, the policy is examined as a fixed version rather than continually updated through new experience.
The workflow calls for recording course completion, falls, collisions, and the points where progress stops. Rollout videos can help show why a policy failed, while these measurements can support later changes to the environment or training setup.
This separation matters. A policy can improve during training yet still fail on a particular course layout. Evaluation provides a controlled checkpoint before the team moves further toward hardware testing.
What remains to validate on hardware?
The source shows a physical Microduck walking with a locomotion policy trained in Isaac Lab. However, it does not explain the hardware deployment process or establish how reliably that policy performs. It gives no success rate, course-completion rate, fall rate, training duration, or reproducibility result.
The important unanswered question is whether the policy can complete the obstacle course on the real robot under defined conditions. Simulation and hardware can differ in several ways: sensors may produce different readings, feet may contact the ground differently, and motors may respond with delays or mechanical play.
Battery voltage can also change as motors draw power. Isaac Lab can model some effects, including command delays, drivetrain backlash, and shared-battery voltage drops. Domain randomization can vary physical settings and sensor noise during training, giving the policy experience with a wider range of conditions.
Those techniques may narrow the gap between simulation and reality, but they do not replace hardware trials. Real tests must measure the same outcomes planned for simulation: completion, falls, collisions, and where progress stops.
The practical value of NVIDIA’s workflow is its connected process. AI agents turn a task description into a project plan, help create the environment, and coordinate preparation. Isaac Sim and Newton provide the simulated world and physics, while Isaac Lab trains the policy and RTX Spark evaluates it locally.
The next useful step is a documented hardware test that reports conditions and results. Until then, the workflow shows how to prepare and inspect a robot policy—not how reliably that policy navigates a real obstacle course.