Xavi Amatriain, Expedia Group’s first chief AI and data officer, told the VB Transform 2026 audience in Menlo Park that “the new PRD are the evals.” He argued product behavior should be encoded through evaluation suites — including red teaming and other security‑oriented tests — and incorporated into the product design document before coding begins.
Amatriain went further, saying AI‑assisted or AI‑generated code will be the future and that most of the design thinking will be concentrated in the evals.
Background and experience
Before joining Expedia in December 2025, Amatriain was VP of AI and Compute Enablement at Google, working on platforms behind Gemini and Google Search. He has mentored talent who later founded Perplexity and Scale AI.
Evidence from enterprise surveys
VentureBeat’s VB Pulse research on the evaluation gap highlights the stakes: of 157 enterprises surveyed, 66 percent already permit some production deployment without human review or plan to do so within 12 months, yet only 5 percent fully trust the automated evaluations that would make that decision. Half of respondents reported shipping an agent that passed internal evals but then failed with a real customer.
Guardrails versus feedback
Amatriain warned that excessive guardrails and artificial business rules can harm systems by making them brittle and by biasing the user feedback loop. He called guardrails a “necessary evil” and said the objective should be to reduce their impact over time. Other speakers at Transform countered that the highest‑risk actions still require firm guardrails.
Three layers of governance: principles, processes, automation
Expedia governs AI with three layers. First come broadly communicated principles to guide distributed decision making. Next are the processes and tools that give those principles teeth. Automation sits on top of both.
In practice this is implemented as agent release toll gates: checkpoints calibrated to each agent’s risk level. Evaluation rounds, red teaming, and security reviews are tied to that risk level, and checks move from recommended to required as risk increases.
Specialized agents instead of a monolithic AGI
Amatriain does not believe in a single, unified AGI. He prefers composition: tools form skills, skills form sub‑agents, and sub‑agents are orchestrated into a full agentic system. Unified design principles — tone, user handling, context and memory passing — must be designed systemically rather than left to a single model.
Narrowly scoped agents are easier to secure because teams can evaluate and lock down individual agents before composing them into larger systems.
The user keeps the final click
Travel prices and availability change rapidly. Expedia blends retrieval‑augmented generation with direct API tool calls and selects the method based on latency needs. For example, a question about typical cost for a four‑star Chicago hotel in July should have an immediate, cacheable answer; niche queries (such as pet‑friendliness plus pool proximity) may tolerate longer reasoning windows.
Expedia cross‑references supplier self‑reports with its own review corpus; a generic chatbot would often only surface what a supplier claims.
A core design principle is that agents should not complete purchases on behalf of users: agents may recommend, suggest, and discuss, but the user must perform the final click. Amatriain framed this as a security decision as well — clear design constraints reduce the need for later guardrails.
Security by design and automated feedback loops
Security should be shifted left and embedded in design, Amatriain said. Monitoring signals should feed back into the eval suite so that the entire cycle can be nearly automated and fixes reported and implemented rapidly.
VentureBeat’s separate June Pulse survey of 107 enterprises underlines how narrow the margin is. Fifty‑four percent have experienced an agent security incident or near‑miss. Fifty‑nine percent plan to adopt, add, or replace agent security tooling within 12 months, and 29 percent plan to do so this quarter. Incident rates rise with organization size — 63 percent among enterprises with more than 1,000 employees versus 49 percent for companies with 101–1,000 employees. Sandbox isolation, the post‑breach control that limits damage, drops from 35 percent adoption at smaller companies to 20 percent at the largest.
Amatriain warned that future threats will increasingly come from other, external agentic AI systems. Detecting attacks is not enough; the time to fix becomes essential once something is detected.
Conclusion
Amatriain’s argument is that elevating evals to product requirement status, applying risk‑calibrated release gates, and building composable, narrowly scoped agent architectures — together with early security design and rapid feedback loops — will be necessary for enterprises to deploy agentic AI safely. VentureBeat’s survey data show, however, that many organizations already face security incidents and need faster tooling and processes to keep up with evolving threats.
