Waymo, the self-driving car company spun out of Google and now part of Alphabet, faces particularly high stakes when deploying AI. Its models do more than generate text or automate back-office tasks: they enable vehicles to navigate unpredictable streets, react to human drivers and make split-second decisions in the physical world.
At VB Transform 2026, Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, described how the company trains, tests and deploys AI at scale. According to Waymo, the company has driven more than 220 million fully autonomous, or "rider-only," miles to date and recorded 17 times fewer serious crash injuries than human drivers over the same distance.
Eval-centric development: making evaluation integral to engineering
Joshi said Waymo follows an "eval-forced" or "eval-centric" development approach: evaluation is not a final gate before launch but a core part of engineering. Project maturity is partly inferred from the maturity of the tests around it.
In practice, Waymo judges a project’s readiness by the completeness and maturity of the surrounding tests. The same idea applies to enterprises building customer-service agents, coding assistants, financial systems or other AI applications: if an organization cannot reliably measure system performance, that system may not be fit for production.
Evaluation must continue after launch
Much of Waymo’s quality work has shifted toward evaluations conducted during model training, after training, and within both open-loop and closed-loop simulations. "Eval is not a one-time task to launch a model," Joshi said.
Waymo treats evaluation as a continuous process spanning on-road driving, simulation and validation. Its methodology combines first-party datasets, performance metrics and infrastructure designed to operate efficiently at scale. For enterprises, this implies that testing an agent before launch is insufficient: teams must keep evaluating systems as base models, business processes, user behavior and incoming data evolve. Those evaluations should tie to actual business outcomes instead of relying solely on generic industry benchmarks.
Evaluation data quality matters
Joshi warned that model-quality measurements are only as trustworthy as the evaluation data behind them. Waymo therefore pairs performance claims with details about the properties of the datasets used for testing.
Testing rare and dangerous scenarios
Waymo’s evaluation hierarchy is driven by a single overriding objective: safety. The company uses first-party driving logs, some third-party data and realistic simulations that expose systems to scenarios covering billions of synthetic miles. Task owners select specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones and other complex environments.
The same principle applies beyond autonomous driving: enterprises must test not only the routine requests their agents handle well, but also the uncommon situations where errors could cause financial, legal, security or reputational damage.
Human oversight in release decisions
Joshi emphasized that release decisions are not left entirely to automated systems. Waymo’s production-readiness reviews include extensive human oversight, and internal safety leaders approve software releases and service-area expansions. "This is not AI-driven and completely automated and zero human oversight," she said. "Human lives are at stake."
Efficiency without sacrificing reliability
Waymo also faces a familiar enterprise challenge: demand for compute, storage, memory and network capacity is growing faster than available resources. The company pursues efficiency across data extraction and storage, distributed model training, model distillation, simulation and evaluation. It places particular emphasis on "data efficiency," selecting the most useful training examples rather than assuming larger volume is inherently better.
Waymo began using transformers in 2017 and later expanded into large language models, vision-language models and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its foundation-model strategy. Technology is split between onboard systems in each vehicle and off-board infrastructure for model development, data processing and simulation, requiring optimization of both real-time inference and the supporting larger systems.
Agents need their own evaluations
Waymo also deploys AI agents internally as productivity tools for engineers. Agents help analyze data distributions, assess data efficiency and triage issues found in vehicle telemetry, training runs and failed evaluation jobs. The goal is to accelerate investigative work so engineers can focus on judgment and difficult technical problems. Waymo evaluates these agents too, to ensure they produce trustworthy, accurate results rather than sending employees down unproductive paths.
Takeaway for enterprise leaders
Joshi’s broader lesson is that agentic AI requires more than selecting a powerful model. Organizations need a clearly defined objective, representative evaluation data, continuous testing, infrastructure that can operate efficiently and named human decision-makers who remain accountable for deployment. "Earning trust is supremely important," she said.
(Information based on Manasi Joshi’s presentation at VB Transform 2026 and performance metrics reported by Waymo.)
