At VB Transform 2026 on Tuesday, Bryan Silverthorn, Director of AGI Autonomy at Amazon, argued that the primary reason enterprises struggle to put AI agents into production is reliability rather than raw model capability. He cited Cisco data showing that 85% of enterprises are piloting AI agents, yet only 5% have shipped them to production.
A four‑dimensional reliability framework
Silverthorn—who joined Amazon via its acquisition of Adept AI and now leads multimodal agent training inside Amazon’s AGI lab—said reliability should be decomposed into four distinct dimensions: consistency, robustness, predictability, and safety. He credits Princeton research for the framework and said it helps separate factors that are often conflated in evaluations.
"It unpacks different factors that I see tangled together in almost every eval I've ever seen," he said.
Why agents pass internal evals but fail in production
The distinction matters because agents commonly pass internal tests and then fail in the wild. Silverthorn described a customer that deployed an agent for software QA to extract serial numbers from screens: it operated flawlessly for two months, then began intermittently returning incorrect numbers. The root cause was that the underlying vision encoder behaved differently depending on where the serial number appeared on the screen, and a software change imperceptible to humans triggered the failure.
His point was about measurement as much as models: teams must identify the dimensions of variability in their systems and match measurement rigor to the stakes of the application.
VentureBeat’s proprietary research, presented before the session, reinforced the claim: half of surveyed companies shipped agents that passed internal evaluations but failed real customers. The research also found that enterprises overwhelmingly track uptime while neglecting accuracy — checking whether a system is running without systematically validating whether its outputs are correct. Most companies default to model vendors’ evaluations and add little else, effectively leaving testing as a coin flip between trusting the vendor and trusting nothing.
Amazon’s "intern" approach to agent management
Silverthorn’s most memorable prescription was cultural. Inside Amazon’s AGI lab, researchers literally call their agents "interns"—as in, "I'll have my intern talk to your intern." The phrase embodies an operational philosophy: agents are powerful but occasionally clueless, capable of impressive results and spectacular derailments.
Managing agents, he argued, is more a management challenge than a pure software engineering problem: ask what could go wrong, add backups and undo capabilities, and consciously decide what level of risk is acceptable. "You can ask the intern, 'Hey, what might you do wrong here? How might you mitigate your negative outcomes?'" he said. The lab has accepted trade-offs in pursuit of research velocity, including one agent that ran experiments around the clock on its own high-level research plan.
Practical guidance for enterprises before scaling agents
Silverthorn was candid about current limits. He called "self-improving" AI a "loaded term": Amazon continually uses AI to improve models, but fully autonomous self-improvement is still distant. The lab continues to focus on computer use: a commercial trucking customer already uses browser automation to stitch together warranty claims across fragmented systems. Silverthorn stressed, however, that future agents will not rely on computer use alone; they will work alongside MCP, APIs, and other tools to complete end‑to‑end workflows.
LLM‑as‑judge techniques are promising, he said, but they are only one of several strategies for aligning agent capability with acceptable risk.
For enterprises stuck in pilot purgatory, Silverthorn’s recommendation is a mindset shift: stop asking whether an agent can do something impressive once, and start asking whether it can do the same thing correctly a thousand times in a row. In his view, companies that break out of the 85% pilot ceiling will succeed through better management, not merely smarter agents.
Takeaway
The conversation at VB Transform 2026 framed the barrier to enterprise agent adoption as a measurement and management problem. Addressing it will require decomposing reliability into specific dimensions, improving testing beyond vendor claims, and adopting managerial practices that reduce risk and ensure repeatable correctness.
