How to measure AI’s real return in software development
This article by James Shore was originally published on the Successful Software blog.
Software development has become one of the largest beneficiaries of AI automation. One CIO I spoke with, whose organization has 170 people, measured a 2.1× increase in productivity from AI. That gain came with significant cost: his top AI users each spent $21,000 per month on tokens. Another engineering leader reported token costs of $6,000–$8,000 per engineer per month.
These are not trivial sums: they must be accounted for, and they will likely be borne from personnel budgets. That has real consequences for people’s lives—and real business consequences for inaction.
If you are going to make choices about AI investments—and somebody will—you must make good choices. That requires knowing more than costs: you need to measure benefits too. Are you getting a return on your AI investment? Is there a different way to apply AI that would give a better return?
Anecdotes about AI effects are plentiful, but the reasoning behind many anecdotes is weak and some claims are outright false. Given the stakes, that is a problem.
This essay series shows how to assess AI’s impact on software development in your organization. It has four parts plus an epilogue:
- Why, and what not to do (this essay)
- Measuring delivery speed
- Measuring and modeling unintended consequences
- Modeling and forecasting business outcomes
- Epilogue: Notes for your CFO
Why you need to quantify impact
Engineering leaders everywhere feel pressure to do more with AI. But which approach is best? There are many strong opinions, little consensus, and even less hard data.
Even with hard data, there’s no one-size-fits-all solution. What works for an internal IT development shop may not be best for a startup, and what suits a new product may be wrong for a complex legacy system.
Faster is not always better. If you get twice as much done per person but your token costs force you to cut headcount in half, did you actually gain ROI? A more modest speed improvement that keeps costs down could be preferable.
You cannot make these trade-offs on gut feeling alone. In a well-known METR study, engineers believed AI had made them 49% faster than it actually had. You must measure impact to know what you are truly getting.
Don’t neglect costs
Costs are easier to measure than impact, but they have their own complications. The largest complication is subsidies. To gain market share, large AI firms are offering substantial discounts. One engineering leader told me some team members spent thousands of dollars in tokens while only paying a $200 subscription. Anthropic and OpenAI frequently offer discounts and promotions. At some point these subsidies may end and prices could rise.
Or they might fall. Open-weight models and local inference put downward pressure on token costs. Dedicated inference chips could lower token prices. Routers that direct prompts to the cheapest suitable model and models that require fewer tokens to accomplish the same tasks could reduce costs. When calculating ROI, don’t assume today’s costs are the right baseline. Make your assumptions explicit and model both optimistic and pessimistic scenarios.
One more wrinkle: adopting AI is not a switch you flip. People need time to learn how to use these tools. As with any major change, change management matters. Productivity often drops during a big transition before it rises, which adds cost and makes early measurements misleading.
Think long-term
Part 2 of this series shows how to measure delivery speed; that is only part of the picture. If you produce twice as many features but double your maintenance costs, productivity may spike and then collapse.
Engineer burnout is also a real concern. One engineering leader who uses spec-driven development said, “Every team is reporting burnout.” Burned-out people don’t produce high-quality work: they don’t push back when they should, they don’t report problems, and eventually they leave, taking valuable institutional knowledge with them.
Part 3 will show how to measure these unintended consequences of AI use and techniques to forecast future problems.
Ultimately, the goal is to deliver software value faster. Doubling bug-fix throughput is not the same as doubling the delivery rate of competitive new features. Delivering poorly thought-out features “just because we can” can actually worsen a product.
Part 4 will show how to model, forecast, and measure business outcomes.
Don’t measure lines of code
First, what not to do.
Don’t take the easy way out. The top item on the no-go list is lines of code. Engineering leaders have known for more than half a century that lines of code don’t correlate with productivity. All else equal, a feature implemented in 1,000 lines will be cheaper to maintain and less bug-prone than the same feature implemented in 10,000 lines.
AI is notorious for verbosity. If you measure lines of code, you measure bloat, not productivity. Don’t do it.
Don’t measure pull requests either
Pull requests (PRs) are nearly as bad. PRs are checkpoints where an engineer asks for code review, and they are completely arbitrary. An engineer (or AI) can create a 10-line PR or a 10,000-line PR; it can be a feature, a bug fix, or a tiny improvement. There is no inherent correlation between number of PRs and productivity.
It’s true that engineers usually produce consistent, logical PRs, so a stream of PRs can reflect a development pace and an increase in PRs can signal increased productivity. But AI upends that status quo by producing huge amounts of code quickly, which can overwhelm engineers’ ability to keep up. Without rigorous oversight, PRs no longer reflect a consistent pace, so PR counts stop being reliable metrics.
And, of course, when people believe PRs measure productivity, they will be tempted to game the system and produce more PRs than necessary.
Measure approaches, not people
This leads to Goodhart’s Law: “Once a measure becomes a target, it ceases to be a good measure.” My favorite book on this topic is Robert D. Austin’s Measuring and Managing Performance in Organizations:
“Gradually, measures fall (or, more accurately, are pushed) out of synchronization with true performance, as workers succumb to pressures to take shortcuts. Measured performance trends upward; true performance declines sharply. In this way, the measurement system becomes dysfunctional.”
Don’t start a performance-measurement program and leave it running. Use metrics to evaluate how well a particular AI approach works. Repeat the measurements periodically if needed, but do not use them to measure the performance of teams or individuals. Measure the approach, learn from it, and stop measuring.
Conclusion and what’s next
Enough about what not to do. In this series we will quantify AI’s impact—and do it correctly. It begins with measuring delivery speed, which I will cover next in Part 2.
