A systematic study by Microsoft Research challenges the common practice of running all physical AI inference on onboard robot GPUs. The evaluation finds that offloading inference to edge or cloud GPUs can improve task success rates, allow larger models, and extend battery life, while a new toolset in the Physical AI Toolchain helps developers containerize and orchestrate distributed inference.
Why this matters
Physical AI — AI that acts in the real world — must operate in open, unpredictable environments, interact with people and other robots, and support diverse embodiments. Realizing robust physical AI requires progress in three areas: robot hardware, embodied AI models, and inference/training systems. Although hardware and model capabilities have advanced quickly, inference infrastructure for distributed, large-scale compute has received less attention but will be essential for deployed systems.
The prevailing assumption and its limits
The dominant approach today is to provision a GPU onboard the robot and run inference locally. While higher-level planning may use the cloud, task execution typically remains tied to the robot. Microsoft Research questions this assumption: as physical AI models grow in size and complexity, onboard GPUs impose constraints — they consume substantial power, reduce battery life, add cost and weight, and can lack the memory or compute to run the latest models.
The study: mobile manipulation workloads
To examine systems-level implications, the researchers performed the first systematic study on robotics workloads, focusing on mobile manipulation tasks such as “check for rubbish in the kitchen and put it in the trash.” Such tasks involve planning routes, perceiving the environment to find objects, navigating, picking up objects, and returning to dispose of them. The evaluation covered three core capabilities: semantic mapping and planning, navigation, and manipulation.
Measurements and main findings
Offloading inference from the robot to on-premise or cloud GPUs improved response time, accuracy, battery life, and cost efficiency. The team evaluated models across a range of onboard, edge, and cloud compute setups (detailed hardware is documented in the technical report).
Key quantitative observations:
-
Performance: Offloading significantly improved robot performance across mapping, planning, navigation and manipulation workloads. Some smaller GPUs could not accommodate the full mobile manipulation stack. On GPUs with limited capacity, mapping and planning ran up to 383% slower compared to an A100, limiting the robot’s abilities in dynamic spaces. Navigation experienced a 30% drop in timely obstacle detection on lighter GPUs. Very large (VLA) models did not always show dramatic slowdowns on smaller GPUs, but the observed latency increases were still enough to reduce their accuracies by 50%.
-
Battery life: Onboard GPUs substantially increased power draw. Replacing a robot’s onboard GPU with a Raspberry Pi 5 and offloading inference extended battery life significantly. In tests with larger onboard GPUs such as Jetson Thor, robot battery runtime was reduced by up to 160% (equivalent to several hours for the larger robots tested).
These results indicate that removing heavy GPU compute from the robot is important for operating in open-world settings with large models and long battery requirements. At the same time, offloading creates tradeoffs involving performance, network latency and bandwidth, and the availability of remote GPU resources.
Tooling for automatic offload
The researchers built a toolset that streamlines inference offloading and distribution between edge and cloud GPUs. Kubernetes serves as the platform to provide a uniform abstraction for distributing robotic AI across robot compute, edge GPUs, and cloud overflow. The toolset supports automatic containerization and offloading of robotics workloads via declarative specifications, distributes physical AI containers with policy-driven rules, and integrates with robotic simulators, LeRobot, and ROS2 to simplify development. A developer can specify what to offload; the system then creates separate GPU inference containers and deploys them appropriately.
Microsoft’s Physical AI Toolchain
Microsoft Research announced that the Physical AI Toolchain now includes an industry-first capability for offloaded physical AI inference. The Physical AI Toolchain is an open-source, production-ready framework that links Microsoft Azure cloud services with NVIDIA’s physical AI stack to accelerate robotics and physical AI development in areas such as perception, mobility, imitation learning, and reinforcement learning. The release includes example projects demonstrating offloading inference for SO-101 and UR10e robots.
As an illustration, the team demonstrates offloading inference of Microsoft’s Rho model — aimed at dual-arm robots — to a Jetson Thor GPU to control the Mobile Aloha robot in BusyBox interactions (e.g., pressing a blue button, turning a knob to a specific position).
Conclusions
The measurement study shows that offloading GPU inference can materially improve robot task success, enable larger and more capable models, and extend battery life. Implementing offload requires careful system design to balance compute placement against network latency and available remote resources. The researchers hope the measurements will guide the design of physical AI inference systems and encourage further infrastructure development. Microsoft Research published the findings and made the offload feature and example code available in the Physical AI Toolchain; they invite feedback from developers testing real-world deployments.
