At VB Transform 2026 on Tuesday, Bryan Silverthorn, Director of AGI Autonomy at Amazon, argued that enterprises are held back from deploying AI agents by reliability issues rather than model capability. Citing Cisco data, he noted that 85% of enterprises are piloting AI agents but only 5% have shipped them to production.
Four dimensions of reliability
Silverthorn presented a framework — attributed in part to research from Princeton — that breaks reliability into four distinct dimensions: consistency, robustness, predictability, and safety. He said this separation helps untangle factors that are often conflated in internal evaluations.
Why agents pass internal tests but fail in the wild
The framework matters because agents commonly ace internal evaluations and then fail in real-world use. Silverthorn gave an example of a customer who deployed an agent for software QA to extract serial numbers from screens. The agent worked flawlessly for two months, then began intermittently reading incorrect numbers. The root cause was that the vision encoder behaved differently depending on where the serial number appeared on the screen; a software change imperceptible to humans triggered the failure.
According to Silverthorn, the lesson is about measurement as much as models. "The models have to be better," he acknowledged, "but teams need to identify their dimensions of variability and match measurement rigor to the stakes of the application." Proprietary VentureBeat research presented before the session supports this: half of surveyed companies shipped agents that passed internal evaluations but failed with real customers. Enterprises overwhelmingly track uptime while often neglecting accuracy — in other words, measuring the pulse without checking the diagnosis. Many organizations default to model makers' own evaluations and do little additional testing, leaving risk assessment somewhere between trusting the vendor and trusting nothing.
Amazon's 'intern' framing: cultural and operational advice
Silverthorn's most memorable prescription was cultural. Inside Amazon's AGI lab, researchers refer to agents as "interns" — a tongue-in-cheek label that encodes an operational philosophy. Agents can be powerful yet occasionally clueless, so managing them requires managerial judgment as much as software engineering.
He recommended asking ahead of time what could go wrong, building backups and undo capabilities, and explicitly deciding what risks are acceptable. The lab has accepted trade-offs in favor of research velocity — for example, allowing an agent to run experiments around the clock on its own high-level research plan, even if it sometimes executes the wrong experiment.
Practical guidance before scaling agents
Silverthorn was candid on current limits. He called "self-improving AI" a loaded term: Amazon continuously uses AI to improve models, but fully autonomous self-improvement remains distant. His lab continues to focus on computer use as a capability; he noted a commercial trucking customer already uses browser automation to stitch warranty claims across fragmented systems. He emphasized, however, that future agents will not rely on computer use alone — they will work alongside MCPs, APIs, and other tools to complete end-to-end workflows.
Techniques that use LLMs as judges are promising, but Silverthorn framed them as one of several strategies for aligning agent capability with acceptable risk.
His central recommendation for companies stuck in pilot purgatory: stop asking whether an agent can perform a single impressive task and start asking whether it can perform that task correctly a thousand times in a row. Firms that escape the 85% pilot plateau will not necessarily have the smartest agents; they will have the best management of agent risk and reliability.
Key takeaways
- 85% of enterprises pilot AI agents; 5% have shipped them to production (Cisco data).
- Reliability should be evaluated across consistency, robustness, predictability, and safety.
- VentureBeat research found half of companies shipped agents that passed internal evals but failed with real customers.
- Practical steps include measuring variability dimensions, building safeguards and rollback options, and emphasizing managerial oversight when scaling agents.
