During the first two years of enterprise generative AI adoption most companies focused on access, experimentation and deployment. As agentic systems move into production, different questions arise: how should leaders think about systems whose costs scale with usage and task complexity rather than user count; and what happens when information, software and code are increasingly generated on demand? Scaling an agent is increasingly a fast‑moving economics decision, not only a technical one.
To unpack these issues, McKinsey Senior Partner Lari Hämäläinen spoke with David Tepper, CEO and cofounder of Pay‑i, who has spent the last three years building measurement and economics infrastructure for enterprise agentic systems. Tepper previously spent 19 years at Microsoft, where he led aspects of Azure’s internal gen AI consumption strategy, and he holds an early gen AI patent from 2011.
Internal experience: when the economics diverged from cloud
Tepper describes that at the time ChatGPT launched he ran internal gen AI consumption for Azure. Teams were spending about $250,000 a day on AI and tracking it in Excel. After a set of small changes he realized he could save $300,000 per week in perpetuity — that crystallized for him how different the financial reality of gen AI is from cloud. Cloud has high upfront costs and then each new user adds marginal cost slowly. Gen AI is a consumption economy from the first call.
While building Pay‑i, he learned another lesson: even if token prices fall, enterprises continually increase what they ask the technology to do, so the overall bill does not necessarily decline. “Tokens are not value; tokens are the bill. The bill shows what you spent; it does not tell you whether you should have spent it.” Closing that gap between spend and business outcome is the core problem Pay‑i aims to address.
Why CFOs and CIOs come knocking today
Eighteen months ago engineers asked for instrumentation of cost. Today CFOs or CIOs ask how to justify AI spend. Organizations are confronting the fastest‑growing new cost center in the enterprise, and most have not built the measurement architecture to tell a credible story to a board. There is widespread “agent sprawl”: Copilot rollouts, customer bots, multiple coding assistants and other business‑unit initiatives often deployed without central IT awareness. Engineers sometimes post $100,000 token bills on LinkedIn as a badge of honor, but there is no line item showing what business outcome was achieved.
McKinsey’s experience highlights six drivers of agentic AI operating expenditures (detailed in the original interview). Crucially, many deployments do create real value — enterprises just lack instrumentation to observe it. This is not only a finance issue: engineering leaders who cannot prove an agent works cannot defend roadmaps or kill failing prototypes.
So the question has shifted from “Is AI worth it?” to “Which of my 47 agents are worth it, and how would I know?”.
Three categories that all get called “agents,” and why it matters
Tepper distinguishes three types often lumped together under “agent”: workflows, pipelines and true agentic systems.
- Workflow: traditional software with gen AI bolted into one or more steps. Cost is bounded per invocation.
- Pipeline: a predetermined series of steps that call an LLM at one or more points (most chatbots are pipelines). Cost is bounded by calls times a known number of calls.
- True agent: given a broad objective, tools and inputs, it decides at run time what to call, in what order and for how long. Coding assistants are the most common today.
This taxonomy matters because cost behavior and evaluation differ. A true agent is unbounded per task, and the same prompt can produce up to a 30× variance in outcomes. Moreover, evaluation methods that work for workflows and pipelines often fail for true agents.
Where “LLM‑as‑judge” evaluation breaks down
LLM‑as‑judge works where you have a canned prompt and reference answers (workflows and pipelines). It breaks for true agents because agents introduce randomness across many turns and internally construct prompts as they go. You cannot build a static database of acceptable inputs and outputs when inputs change each run.
The observed factor‑of‑30 variance in token consumption is instructive: the trajectory of calls is wide while the end point can be narrow. Most signal lives in the trajectory; a judge that only inspects the final output is grading the wrong artifact. When an agent fails it often failed several steps earlier, and the final step can cover up the error. Companies usually only use engineering traces to optimize code; few have systems to link agent decisions to business impact.
The fix is to anchor evaluation to business outcomes, not raw model outputs. For cases where the agent achieved a defined business outcome, extract core behaviors from traces of the entire use case, and build a curated corpus of those behaviors — and do the same for failed cases to enable counterfactual credit assignment. The corpus must be continuously curated and versioned (prompt, model, tool definitions), because the same agent and input will produce different traces over time and when new frontier models arrive.
A simple math rule for when an agent adds value
Tepper proposes a clean inequality: an agent adds value when the probability of success is greater than the verification time divided by the time it takes a human to do the work. Formally: P(success) > T(verify) / T(do).
Example: if a human task takes two hours to do (T(do)) and six minutes to verify (T(verify)), the ratio is ~5%. The agent needs only a 5% success probability to be net positive. That inequality is far more forgiving for many enterprise tasks than people assume.
The trap is when failures change the environment. A bad document that’s discarded is low cost; a customer service error that creates a refundable claim changes the environment and incurs recovery (rework) cost. Tepper calls this the ‘‘agency tax’’: verification cost plus rework cost. In production rework is rarely zero and the reliability bar rises sharply; most enterprises do not calculate this per use case.
This is why paying more for a frontier model can be rational: a more capable model often raises P(success) and lowers T(verify), so the productivity gains can outweigh higher per‑token spend. Tepper emphasizes that the cost of a gen AI use case should be viewed as a distribution with likelihoods and percentiles, not a single scalar.
Falling token prices, rising per‑task cost and hidden layers
Pay‑i’s tracking shows inference cost for a fixed model size has compounded down roughly 6.67% per month since 2022, amounting to about an 86% decline for models of similar size — if capability were held constant. The practical problem is that capability is not held constant and agents act as multipliers: a single agentic task can include dozens or hundreds of API calls, and the industry average inside an agent today is about 3.5 different models per task, often from multiple providers.
Microsoft and Stanford found agentic workflows consume roughly 1,000× more tokens than comparable code chat and code‑reasoning sessions, consistent with Pay‑i’s production telemetry. Context windows have grown roughly 250‑fold since GPT‑3, and cost is more a function of how full the window is than the headline rate. The majority of the bill is for what you ask the model to read (input tokens), not for the output. Cost per completed task has risen across workloads because enterprises continually ask the technology to handle more involved tasks.
Hidden cost layers compound: cached input tokens can be 75–90% cheaper, but only if prompts hit the cache; most prompts don’t. Internal reasoning tokens can dwarf visible output costs.
When capacity planning replaces token pricing as the conversation
Tepper’s rule‑of‑thumb: when annual spend concentrates at about $3 million with a single provider for a single model, shared infrastructure begins to break down and organizations must move to provisioned capacity. Below that, shared infrastructure typically suffices; above it, timeouts, rate limits and shortages hit production.
Provisioned capacity does not degrade gracefully: everything runs fine until one day it doesn’t, usually when a new agent or team is onboarded. The move to provisioned capacity is thus a graduation point: the application becomes mission critical and variable cost becomes effectively fixed cost, requiring disciplined planning.
Common failure mode: siloing. Defensive engineering teams provision separate pools per use case to avoid resource contention, but that destroys the economics of provisioning, which depends on diverse workloads sharing peaks. Many enterprises end up paying two or three times the capacity they need because of this mistake.
Also note: provider‑advertised TPM (tokens per minute) on reservations is a rough estimate, not a load specification. Real capacity depends on input–output mix, cache strategy and request‑per‑minute ceilings providers rarely publish. The honest way to size a reservation is to load‑test your actual workload.
Practical short list for the next 2–4 quarters
Tepper’s recommendations for CIOs and CFOs:
- Inventory agents and map them to use cases. Many firms cannot produce an accurate list of agents in production, the business owner for each, and the KPI each is supposed to move.
- Define quantifiable KPIs and intended business outcomes up front. Translate outcomes into hours saved, dollars generated or error‑rate reductions, and get the business sponsor’s sign‑off before launch.
- Instrument all business levers, not only the bill. Token spend by team is table stakes but insufficient. The most valuable instrumentation is at the workflow and agent‑step level where you can attribute cost, latency and outcomes.
- Decide who owns the measurement architecture. The most common organizational mistake is leaving ownership ambiguous. Assign engineering headcount for FinOps, a finance partner for the center of excellence, and an explicit business sponsor for the platform team.
- Treat capacity as a portfolio. Don’t silo provisioned pools per use case unless regulation requires it. Mix workloads with different traffic shapes across shared pools and use spillover deliberately.
- Run a quarterly model review. Frontier models move every 3–6 months; decisions early in a year may be obsolete by Q3.
- Stand up a joint operating group across finance and engineering and ensure a business sponsor for every material agent. Without this, the CFO defends spend without context, engineers defend models without business framing and business owners cannot measure value.
One year outlook: what CIOs and CFOs will argue about that they aren’t today
Tepper expects bifurcation in customer‑facing metrics as Mythos‑class models reach production: the gap between enterprises that can deploy them and those that cannot will compound. The boardroom question will shift from “Are we using AI?” to “Are we using the right AI?” Companies that bet on the wrong model tier in 2026 will feel it.
Observability and FinOps for agentic systems will become standard procurement items the way APM became ubiquitous in the 2010s. Agentic coding changes are a precursor to many other white‑collar tasks: where there is intelligence embedded in tokens, token‑maximizing behavior will spread as tools and awareness diffuse across employees.
Finally, Tepper argues the AI bubble narrative will fade: this is not an imminent bust but a phase in which 95% of enterprises are not measuring returns, inference revenue is compounding faster than any prior software category and capability improves on a 3–6 month cadence. Firms that conflate “we cannot see ROI” with “there is no ROI” risk ending up as the laggards did with cloud adoption in 2014.
Closing
David Tepper is CEO and cofounder of Pay‑i. Lari Hämäläinen is a senior partner in McKinsey’s Seattle office. The interview was edited for length and clarity. Comments and opinions expressed are those of the interviewees and do not represent McKinsey & Company’s positions.
