The Human Engineering Research Laboratories (HERL) at the University of Pittsburgh is leading the Robotic Assistive Mobility and Manipulation Platform Providing Independence for People with Disabilities (RAMMP) project, backed by up to $41.5 million in funding from the Advanced Research Projects Agency for Health (ARPA-H). HERL is collaborating with ATDev and a national consortium that includes Kinova Robotics, LUCI Mobility, and academic partners at Carnegie Mellon, Cornell, Northeastern and Purdue.
Why RAMMP matters
About 5.5 million wheelchair users live in the United States; preserving mobility is essential for daily participation and opportunity. Existing assistive technologies often fail to reflect the complexity of real-world environments, which can undermine users’ confidence and safety. More than 100,000 wheelchair-related injuries are treated in U.S. emergency departments each year, frequently caused by trips and falls. These realities create urgent demand for smarter, safer, and more adaptable assistive systems.
Bringing machine vision to real environments
RAMMP seeks to remedy design shortcomings in current robotic mobility platforms by combining advanced robotics, new operating systems, and digital twin technology to create a virtual simulation environment for safe and scalable testing. The project heavily integrates artificial intelligence, including several of Meta’s open-source vision models such as the DINO family and the Segment Anything Model (SAM).
DINO (a self-supervised vision transformer) is effective at learning visual representations from unlabeled data, which is valuable where annotated datasets are scarce. SAM is intended to “segment anything,” able to identify and outline objects in images or video with minimal prompting, making it useful both for precise object recognition and for generating annotations.
Running DINOv3 and SAM on edge devices
For assistive-device users, reaction time matters: everyday environments are unpredictable — a child may dash across a sidewalk, a curb may appear unexpectedly, or keys may drop. Processing images and sensor data on-device (edge computing) lets robotic mobility platforms respond quickly without depending on network connectivity. Yet running high-capacity models like DINOv3 and SAM on limited, battery-powered hardware poses practical engineering constraints: battery life, heat dissipation, unreliable connectivity, and tight size and weight budgets.
RAMMP engineers optimize models for edge deployment by reducing memory footprint, using lower numeric precision where appropriate, and packaging models in formats suited for real-world hardware. Operating at practical resolutions and efficient batching keeps models fast and stable on compact, battery-powered robotic platforms and arms. In some cases the team intentionally trades a small amount of boundary precision or feature detail for the speed and dependability users need — a deliberate balance between precision and practicality.
"For assistive robotics, performance is not measured by benchmark accuracy alone, but by whether a system can operate reliably in the unpredictability of everyday life," said Sivashankar Sivakanthan, Chief of Staff to the RAMMP project. "Running models like DINOv3 and SAM on-device is what enables real-time perception that users can trust - without relying on connectivity or compromising safety."
Automated labeling and 360-degree awareness
RAMMP’s perception stack is built on RF-DETR, a lightweight detection model fine-tuned with DINOv2 embeddings. Training data is auto-labeled using SAM, allowing the team to quickly produce high-quality annotations across a wide range of angles, heights, backgrounds, and lighting conditions encountered in real-world usage. Data augmentations and multi-view strategies further enforce consistency across perspectives. The result is a system that delivers real-time, 360-degree environmental awareness and adaptive object detection.
By combining SAM’s labeling capabilities with DINOv2’s visual representations in a fine-tuned RF-DETR model, RAMMP achieves fast, adaptive perception for navigation and manipulation tasks.
User-centered development and partnerships
HERL and ATDev play complementary roles: HERL provides biomedical engineering expertise and user-centered research to set the project vision, while ATDev focuses on engineering to make the research work in real devices. HERL’s development approach is deeply collaborative, engaging wheelchair users, clinicians, and advocacy groups to ensure the technology meets real-world needs rather than hypothetical ones.
ARPA-H’s goal is to create a future where Americans with limited mobility can more easily live independently, work, and enjoy leisure through advanced robotics, said Mansoor Khan, ARPA-H Program Manager. Alicia Jackson, Ph.D., ARPA-H Director, added: "Through RAMMP, ARPA-H aims to create a future where Americans with limited mobility can more easily live independently... Partnering with world-class innovators like Meta to bring frontier AI into assistive robotics is how we make that happen faster than anyone thought possible."
Economic and social impact
Beyond improving individual mobility, RAMMP aims to catalyze workforce and manufacturing opportunities in Pittsburgh and across Pennsylvania by fostering advanced mobility solutions and domestic production. The initiative links technological progress with economic and social advancement in the region.
Next steps
The RAMMP team will continue integrating next-generation perception models, including SAM 3.1 and DINOv3, to enhance real-time environmental understanding and interaction. Future work will emphasize temporal consistency, robustness across diverse real-world conditions, and tighter integration between perception and control systems. The project intends to produce assistive systems that not only perceive environments accurately but also adapt over time to each user’s unique needs and behaviors, moving toward broader independence and participation for people with disabilities.
