NVIDIA provides AI agent tools for creators of robotics and self-driving vehicles.
During GTC Taipei and Computex today, NVIDIA Corp. introduced a range of open-source tools and capabilities for physical AI. These offerings aim to assist creators working on robotics, self-driving cars, visual AI, and industrial digital twins. The company stated that these resources can significantly cut down the expenses, duration, and difficulty involved in creating large-scale physical AI workflows.
Included within the NVIDIA Agent Toolkit, these fresh capabilities enable AI agents to expedite the processes of creating data, running simulations, training, testing, and launching robots, autonomous vehicles (AVs), factories, and labs, according to the company.
“AI agents are completely transforming how software is built, and this evolution is now reaching physical AI, spreading into the systems destined to reshape transportation, manufacturing, healthcare, and robotics,” stated Jensen Huang, NVIDIA’s founder and CEO, at GTC Taipei. “By allowing agents to directly access NVIDIA libraries, models, and frameworks, the pace of physical AI advancement will accelerate, empowering developers to construct tomorrow’s robots, autonomous vehicles, and industrial setups at an astonishing speed.”
“Physical AI depends on enormous volumes of training data across a wide variety of settings,” explained Rev Lebaredian, NVIDIA’s vice president of physical AI simulation. “By combining teleoperation, simulation, and internet-scale data, we establish world foundation models capable of handling an endless array of scenarios.”
NVIDIA prepares its physical AI stack for agent integration
NVIDIA announced it is tailoring its complete physical AI stack for agent use by converting its libraries, models, and frameworks into tools that agents can call upon. This encompasses:
“Cosmos 3 serves as the cutting-edge foundation model for physical AI,” Lebaredian mentioned. “It comprehends both video and text, highlighting essential information. Cosmos delivers precise physical simulation, can anticipate future events, and can produce actions.”
To facilitate the application of these tools, NVIDIA is rolling out new capabilities that transform physical AI development steps into reusable instructions for coding agents. This covers which tools to invoke, the expected outputs, and the methods for developers to verify the outcomes.
Creators can also securely construct and launch independent agents using these capabilities via the NVIDIA NemoClaw blueprint and the NVIDIA OpenShell runtime. These offer policy-driven security and privacy management across local or cloud infrastructure. The agents operate on the edge via Jetson and have already showcased enhanced reliability, noted Lebaredian.
NVIDIA stated that its physical AI skills and tools are speeding up agent-driven development in several areas:
- Robotics and edge AI: Robot creators can leverage these skills to hasten their entire development process. This ranges from producing perception and movement training data to simulating environments, automating navigation training, boosting robot learning, and configuring Jetson-based edge systems for rollout.
- Autonomous vehicles: For those developing AVs, these skills instruct agents on how to turn fleet-captured data into simulated settings, create highly realistic driving scenarios en masse, and conduct closed-loop reinforcement learning to broaden the scope of training and testing.
- Real-time vision AI agents: For automated checks and video analysis, NVIDIA claims its agent skills assist teams in crafting synthetic training data, refining models, automating the labeling process, and constructing video AI agents capable of searching, condensing, and evaluating live or archived video.
- Industrial AI: Developers of industrial software can apply these skills to transform engineering data into CAD assets for digital twin simulations, streamlining large OpenUSD scenes with reduced manual configuration.
- Healthcare: Prior to introducing automation into clinical settings, medical teams can direct agents to build digital twins of hospital spaces, generate sim-to-real data, and perform software-in-the-loop policy checks.
According to NVIDIA, these skills can be mixed and matched into broader agent-based systems. This allows creators to coordinate and streamline intricate workflows like data creation, simulation, optimization, inference adjustment, ongoing assessment, and beyond.
Robot creators adopt the physical AI stack
Several robotics firms are already utilizing NVIDIA’s agent-compatible physical AI stack, including 1X Technologies, Agile Robots, Agility, FieldAI, Hexagon Robotics, NEURA Robotics, Skild AI, and Universal Robots.
“Thanks to updates in the NVIDIA Isaac GR00T stack, complete workflows can now be configured in a matter of hours instead of weeks,” Lebaredian remarked. “Being ‘omni-modal’ means the system handles various formats—like video, sensor readings, text, and audio—for both action inputs and outputs.”
Additionally, Foxconn and Compal are adopting NVIDIA Isaac for Healthcare to speed up the deployment of hospital robotics. Compal is pushing its PolyMedX robot closer to becoming a comprehensive hospital management platform by merging simulation, AI, and physical operations.
Foxconn is expanding the reach of its Nurabot across multiple hospitals and long-term care facilities, delivering AI-driven robotics to patient care. Furthermore, they are unveiling their new Scrub Nurse Collaborative Robot to make operating room procedures more efficient.
Top industry players construct with NVIDIA technology
NVIDIA collaborators and clients throughout the manufacturing, transportation, healthcare, and industrial software sectors are leveraging its physical AI libraries to push forward the creation of autonomous systems and industrial AI.
As these libraries become compatible with agents, developers can utilize NVIDIA skills to help agents automate the configuration, execution, and refinement of complex physical AI workflows.
In the electronics manufacturing sector, TSMC and Pegatron are refining their visual inspection models. By using synthetic data produced by the Defect Image Generation skill, Pegatron reportedly slashed the time required for model training and deployment by 67%.
Delta Electronics created synthetic defect data and applied the skill to identify excessive solder on metal busbars, boosting its detection rate by 17%. Inventec built its Observation Agent visual inspection pipeline by incorporating the Defect Image Generation skill, which cut the effort needed to gather defect data for laptop chassis production by 30%.
Foxconn, in partnership with DeepHow,
The approach helped boost manufacturing efficiency by detecting errors earlier in the process, raising first pass yield by roughly 3%.
Within industrial AI, companies such as Cadence, Dassault Systèmes, Siemens, and Synopsys are leveraging NVIDIA Omniverse libraries and capabilities for engineering data inspection, simulation, and interactive digital twins. Similarly, PTC, MetAI, and Lightwheel are using the NVIDIA Isaac Sim framework along with OpenUSD-based workflows to convert CAD data into assets and environments ready for simulation.
As part of its “Autonomous Fab 2030” initiative, SK hynix is deploying semiconductor fab digital twins powered by NVIDIA Omniverse. The chipmaker is also working alongside NVIDIA and SK Telecom to test NVIDIA’s Agent Toolkit for manufacturing-focused physical AI applications.
Autonomous vehicle developers Li Auto, Afari, and DeepRoute.ai are applying NVIDIA Omniverse NuRec models for neural scene reconstruction and rendering. Together, they have produced over 1,000 scene reconstructions and more than 300,000 renders and simulations daily.
These AV companies are additionally drawing on the new agent skills repository to speed up and enhance the development of safer, more capable autonomous driving systems. Foxconn, VinFast, Uber, and HUMAIN have also joined the NVIDIA DRIVE Hyperion ecosystem to build and roll out SAE Level 4 robotaxis.
The Alpamayo 2 Super has already surpassed 500,000 downloads and earned the “Best Technology” award at Computex, according to Spencer Huang, director of product for robotics at NVIDIA.
Physical AI agent tools are now available
NVIDIA has made its physical AI agent tools and skills accessible through GitHub and skills.sh, compatible with any coding agent.
Agent skills and tools for synthetic data creation — including Neural Reconstruction, Video Augmentation, and Defect Image Generation — are also offered on NVIDIA Brev as Physical AI Launchables, preconfigured environments that package agent skills and tools for faster synthetic data generation and evaluation.
Microsoft, CoreWeave, and Nebius are incorporating these agent skills and tools into their cloud services, allowing developers to streamline and scale the generation and deployment of synthetic data.
Unitree humanoid serves as the reference for NVIDIA Isaac GR00T
NVIDIA has unveiled the NVIDIA Isaac GR00T Reference Humanoid Robot, an open humanoid robot reference design built on NVIDIA Jetson Thor and the NVIDIA Isaac GR00T development platform. According to the company, this design is intended to lower barriers to advanced hardware and software for developers, eliminating reliance on costly proprietary platforms as demand for general-purpose humanoids continues to grow.
NVIDIA integrated a full-scale robot body with sophisticated manipulation, sensing, control, and onboard AI compute resources:
- Unitree H2 humanoid chassis, standing nearly 6 ft. (1.8 m) tall and weighing 150 lb. (68 kg), with 31 degrees of freedom to support human-scale experimentation.
- Dual Sharpa Wave tactile five-finger hands, enabling precise manipulation with 22 degrees of freedom, bringing the robot’s total degrees of freedom to 75 across the body and hands.
- Multi-view sensing, including a head-mounted stereo camera with a wide field of view (140 degrees horizontal, 102 degrees vertical), wrist cameras for close-range manipulation tasks, and an inertial measurement unit for motion tracking.
- Whole-body control, delivering arm torque of up to 120 Newton-meters, leg torque of up to 360 Newton-meters, a rated arm payload of 7 kg (15.4 lb.), and a peak payload of 15 kg (33 lb.), enabling stronger lifting and extended reach.
- NVIDIA Jetson AGX Thor T5000 onboard compute, powered by an NVIDIA Blackwell GPU delivering 2,070 FP4 teraflops of AI performance, a 14-core Arm CPU, 128GB of unified memory, and a configurable 40- to 130-watt power range for real-time sensor processing and robot inference.
- Connectivity via Ethernet, Wi-Fi 6, Bluetooth 5.2, USB, along with an array of microphones and speakers for voice interaction.
- Battery for extended runtime, with a 15Ah, 0.972kWh capacity providing approximately three hours of operation.
- Remote emergency stop for quickly and safely powering down the robot.
The Isaac GR00T platform includes:
The system is built with modularity in mind, allowing robotics teams to adopt the entire platform or incorporate specific capabilities into their existing development workflows, helping them scale humanoid development without duplicating infrastructure for every robot or task.
Prominent research institutions such as Ai2, ETH Zurich, Stanford Robotics Center, and UC San Diego’s Advanced Robotics and Controls Laboratory intend to adopt this reference design to advance humanoid robotics research.
“To make meaningful progress toward general-purpose robots, researchers need platforms that are both capable and widely available,” said Deepak Pathak, co-founder and CEO of Skild AI. “A reference design enables more researchers to engage in cutting-edge humanoid research and move from concepts to experiments more quickly. This helps propel the entire robotics research ecosystem forward.”
NVIDIA Research will also leverage this reference design to further develop Isaac GR00T open models, frameworks, and hardware. The NVIDIA Isaac GR00T Reference Humanoid Robot is expected to be available through Unitree in late 2026.
