A June 2026 VB Pulse survey finds enterprise AI teams are granting agents increasing autonomy while confidence in automated evaluations is falling. The self-selected sample of 157 respondents from companies with 100 or more employees shows many organizations are deploying agents first and building control layers around them later.
What happened and who is affected
- Half of respondents reported that they had deployed an AI agent or LLM feature that passed internal evaluations yet still caused a customer-facing failure. One in four respondents reported this happening more than once.
- The sample consists of 157 self-selected qualified enterprise respondents, so results are directional rather than statistically representative.
Autonomy rising as trust falls
- 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months.
- Only 5% say they fully trust the automated evaluations that would make those release decisions.
- This gap between increasing autonomy and weak assurance is described as the "evaluation gap."
Why passing an evaluation is not the same as a working agent
Traditional software tests typically check whether a defined input yields an expected output. Agent testing is harder because the system may choose its own sequence of steps, call external tools, retrieve data, change state and behave differently across runs. Consequences include:
- An agent can make several individually plausible decisions and still produce the wrong business outcome (for example, retrieving the correct account but updating the wrong field),
- It may draft a valid refund request but send it without required approval,
- Earlier tool calls may succeed while a later step leaks sensitive information or leaves the workflow incomplete.
Why enterprises distrust automated evaluation
Respondents' top concerns were:
- 29%: poor alignment between tests and real-world outcomes,
- 21%: bias or inconsistency,
- 18%: lack of explainability,
- 17%: data leakage or privacy concerns. This hierarchy suggests companies believe scores often fail to predict what will happen when customers, employees or business processes meet an agent in production.
Standards bodies echo the point: NIST guidance
NIST's Generative AI Profile makes a similar argument: measurements taken in controlled environments may not transfer cleanly to deployment because behavior shifts with prompts, users, context and operating conditions. NIST recommends field testing, post-deployment monitoring and clear escalation processes for failures.
Capability is not the same as consistency
A single successful run demonstrates an agent can complete a task but not that it will do so reliably. Anthropic's guidance on agent evaluation differentiates between succeeding at least once across repeated attempts and succeeding every time — a crucial distinction for customer-facing and operational workflows. Recommended practices include:
- Treat repeatability as a first-class metric: run the same scenario multiple times, vary phrasing and context, test tool failures,
- Focus on whether the final business outcome remains correct even when the route changes,
- Make every production incident a permanent regression test: feed customer escalations, failed tool calls, incorrect approvals and data-handling mistakes back into the pre-deployment suite rather than leaving them as isolated support cases.
Autonomy should expand by risk, not ambition
The survey does not argue that every agent action must require a human. Human review cannot scale across millions of low-consequence decisions. But zero-human operation should be earned by demonstrated reliability and bounded by failure consequences.
- Low-risk actions (drafting internal summaries, categorizing documents) can tolerate broader autonomy.
- High-risk actions (financial transactions, customer communications, code deployment, access-control changes, data deletion) need stricter thresholds: repeated consistency tests, policy checks, rollback mechanisms and clear human escalation paths.
Differences by company size and market implications
Larger enterprises (2,500 or more employees) are moving fastest toward zero-human deployment: 70% of them permit or plan such deployments, compared with 64% of smaller firms. Larger organizations also report more agent deployments that later caused customer-facing failures (54% vs. 48%). This is a warning for enterprise leaders: removing the human from the loop does not remove uncertainty. Without stronger assurance, uncertainty becomes an automated production decision.
Conclusion: where budgets will shift next year
Market pressure will continue to push toward greater autonomy because the economic incentive is real. The organizations best positioned will not be those that remove people fastest; they will be those that treat repeatability and regression testing as seriously as deployment speed. The next year is likely to be a retrofit cycle: buyers will redirect budget toward systems that make agentic deployments governable and dependable, including identity, evaluation, cost, context and orchestration controls.
