By James Shore
This article originally appeared on James Shore’s Successful Software blog and is republished here with the author’s permission.
Is AI making software development faster? Most people I speak with say yes. The key questions are: how much faster, and what value do you get for the money you spend? Are there alternative AI uses with better ROI?
This is the second part of a series on assessing AI’s impact on software development. Part 1 covered why impact assessment matters and common mistakes. Here we focus on measuring delivery speed. Part 3 will examine unintended consequences, and Part 4 will address business outcomes. An epilogue ties the advice together for CFOs.
What exactly are we measuring?
Delivery speed is not the same as productivity. It’s a component of productivity: given two equivalent teams building the same software, the team that finishes first appears more productive. But real productivity is about value produced divided by money spent. Faster delivery doesn’t guarantee you’re delivering valuable work.
For now we ask a narrower question: are teams completing assigned tasks faster, irrespective of the tasks’ ultimate usefulness?
The basic approach — and the pitfalls
There is no objective, universal measure of software development speed. Outputs are not standardized: features vary in size, bugs vary in difficulty, story points and velocity are arbitrary and driven more by what’s requested than how fast a team works. Lines of code and PR counts are also poor metrics.
A natural idea is to compare a team to itself. If a team used to take three times the estimate and now takes 1.5 times, that looks like a 2x speedup. But several snags complicate this:
- Estimates are not consistent. Actual-to-estimate ratios vary across tasks and people; plotting them often yields an S-shaped distribution rather than a single ratio.
- People work differently. Some overestimate, some underestimate; some spend time educating others, some work heads-down. Comparing everyone to everyone introduces noise; comparing individuals to their own baseline reduces it.
- Estimates change when people expect to use AI. If people anticipate AI will speed them up, they will give smaller estimates. To avoid this bias, collect estimates before people know whether AI will be used on a given task, then randomize.
- Process improvements confound attribution. Introducing AI often triggers workflow changes (streamlined request flows, closer contact between business experts and developers) that themselves increase effectiveness. If you simply compare pre-AI to post-AI periods, you can’t separate the effects of AI from parallel process changes.
How to control for those issues
A rigorous approach is a randomized controlled trial — specifically a within-subject repeated-measures design. It reduces the number of samples needed while keeping people doing their normal work.
Outline of the experiment:
- Define clearly a with-AI approach and a without-AI approach. Make sure participants have practiced both. If your with-AI flow includes process improvements, apply those same workflow changes to the without-AI approach so the only intended difference is the AI usage itself.
- Decide how you’ll attribute rework (code review changes, bug fixes) back to the originating task.
- Give each person or team a virtual bag of assignments split roughly half with-AI and half without-AI. For each task, have participants record their estimate first, then randomly draw whether the task will be done with or without AI.
- Collect the main data: estimated time, actual time, task type, and spending. Predefine whether you are tracking calendar time (including interruptions) or effort (excluding interruptions) and how you’ll handle parallel work. Consider collecting other metrics that will be useful later (bug counts, incidents, mental energy, excess lines of code) to model unintended consequences in Part 3.
You will need a statistician for detailed design and analysis. Broadly, each individual’s with-AI results are compared to their without-AI results, then aggregated across participants for the final estimate of effect.
Remember to translate measured speed changes into the bigger picture: if developers double coding speed but only spend half their time coding, the overall speed gain is smaller. For example, work that took 40 hours and now takes 30 hours yields 40 ÷ 30 = 1.33× improvement, not 2×.
Is this effort worth it?
This is more work than simple metrics like counting PRs and calling it a day. If this were an ordinary process improvement exercise, the cost of rigorous measurement might not be justified — it’s only practical when expected effect sizes are large enough. But AI adoption involves significant and variable spend: engineering leaders report order-of-magnitude differences in per-person expenditures. Should you spend thousands per person per month on agentic coding tools, or hundreds on analysis and smarter autocomplete? The right choice depends on what outcomes you actually get for the money.
Without measurement, you’re likely to be wrong: in a METR study, engineers overestimated AI’s speed benefits by nearly 50%. With a well-designed experiment and enough participants, you do not need many samples per person to detect meaningful effects. The teams can collect data as part of normal work; the main effort is precisely defining how AI is to be used — which is a valuable exercise in itself.
Delivery speed is only part of the story; sustainability and unintended consequences matter too. Those topics are the subject of the next article in the series.
All content is created by my human brain unless explicitly stated otherwise. If you found this useful, see my best writing and presentations and consider subscribing to updates. For consulting inquiries, email jshore@jamesshore.com.
