Several studies from late 2025 and early 2026 give a mixed picture of AI’s effect on software engineering productivity. A controlled experiment found a statistically significant slowdown for engineers using AI, while other internal and survey-based measurements report large productivity gains. Much of the tension stems from measurement issues: AI lowers the cost of starting work but does not automatically remove downstream shipping bottlenecks.
The uncomfortable METR finding
In early 2025 the research organization METR ran a controlled experiment with open-source developers and found that engineers using AI tools took 19% longer than those without AI (confidence interval: +2% to +39%). The slowdown was statistically robust. Paradoxically, participants believed they were around 20% faster, revealing a large perception–reality gap.
METR attempted a follow-up study beginning August 2025 but in February 2026 published an explanation for abandoning the experimental design. Too many developers refused to participate unless they could use AI for all tasks; among remaining participants 30%–50% reported selectively avoiding tasks they did not want to do without AI. The sample became biased toward developers and tasks least likely to showcase AI value.
Late-2025 data showed improving trends: for the subset of returning original developers the estimated effect shifted to an 18% speed improvement (confidence interval: -38% to +9%); newly recruited developers showed a 4% improvement (-15% to +9%). METR cautioned these figures are likely lower bounds because of self-selection. Their conclusion: AI tools have become more useful since early 2025, but selection effects make controlled measurement nearly impossible. That is not a methodological failure so much as a signal about how integrated AI has become.
Three more data points
- Anthropic: In late 2025 Anthropic surveyed 132 of its own engineers, conducted 53 interviews, and analyzed 200,000 Claude Code transcripts. Employees reported about a 50% productivity increase; pull requests per engineer per day rose 67%. Anthropic engineers use Claude in 60% of daily work and Claude autonomously performs more tasks over time.
- CircleCI: Analysis of 28 million CI workflows across thousands of teams found workflow throughput up 59%, but main branch throughput for the median team fell 7%. Build success rates dropped to 70.8%, a five-year low. More code is being produced but less of it reaches production; CI is becoming a chokepoint.
- Harvard Business School: Researchers studied 78 workers using AI to do tasks outside their expertise. AI helped everyone brainstorm equally well, yet on execution workers far from the domain underperformed domain experts by 13%.
METR’s May 2026 survey of 349 technical workers (conducted after their experimental design collapsed) found self-reported productivity gains of 1.4x–2x from AI tools. Notably, METR’s own researchers—who are most calibrated to the perception bias they previously documented—reported the lowest gains of any subgroup.
What this looks like in practice
A familiar scenario: activity metrics look great — more pull requests, more commits, higher velocity — and both leadership and engineers feel more productive. Then product items marked “in progress” six weeks ago are still in progress. The realization: timelines haven’t meaningfully changed. AI dramatically reduces the cost of starting work (first drafts, boilerplate, scaffolding, test-writing, explanatory notes), but production-ready polish remains hard.
The real bottlenecks were rarely the tasks made cheaper by AI. They were product decisions, design reviews, QA, compliance, infrastructure and release processes. When coding speeds up, more work-in-progress items pile up against the same downstream chokepoints. The CircleCI data on 28 million workflows illustrates this: massive activity in feature branches with flat or declining main-branch throughput.
This pattern is visible in high-AI teams too. Fiona Fung, engineering director for Claude Code at Anthropic, said in a June 2026 talk that writing code, tests and refactoring hardly slow her team anymore, but verification, code review and security became the bottlenecks. As teams generate more code, build systems and CI pipelines can struggle to keep up — the same constraint CircleCI’s data shows.
Anthropic’s statistic that 27% of AI-assisted work wouldn’t have happened otherwise is double-edged: some of that work is valuable (prototypes, documentation), some is low-priority work that now consumes review cycles and CI resources because it became cheap to create while remaining costly to review and maintain.
The competence–confidence gap
The Harvard study describes a mechanism: AI closes the confidence gap between novices and experts by giving everyone access to plans and first drafts, but it does not close the competence gap. A backend engineer assisted by AI can produce a frontend feature that looks correct on the surface but contains hidden incorrect assumptions and untested edge cases.
METR’s early result suggests this effect can even affect experienced practitioners within their domain: AI does not make them incompetent, but it can make them feel more capable than their output warrants. As METR’s follow-up study showed, once developers integrate AI deeply they may be unable or unwilling to work without it as a baseline—an instance of automation bias.
This is particularly concerning for engineering leaders: you cannot fix what you cannot see. If every engineer believes they are 50% more productive while ship dates remain unchanged, a real problem exists that no one recognizes.
Making AI-native development sustainable
- Make code review more rigorous, not faster: AI-generated code easily passes surface checks, so review should start from skepticism—ask what the model assumed, which edge cases are untested, and what won’t be visible in a diff. This is slower but necessary; pair it with automation for low-judgment tasks.
- Pair automation with human judgment: let AI handle style, linting, basic bug-catching and test generation as a first pass, but route security-sensitive code, trust boundaries and legal-risk areas to domain experts. The distinction is: AI handles surface correctness; humans own consequential judgment.
- Adapt CI to new failure modes: AI-generated code tends to be locally correct but architecturally inconsistent, passing unit tests and failing integration tests. Integration, contract tests and architecture fitness functions in the pipeline will catch more of these issues than linters or type checkers.
- Ship behind feature flags and monitor aggressively: accept that not everything is caught premerge. Deploy to a small percentage of users, observe dashboards, and rollback quickly when needed. This enforces investment in observability as well.
- Require human-written tests for AI-assisted code (for now): tests, particularly for edge cases and boundary conditions, force the developer to think through behavior. If an engineer cannot write the test, they likely do not understand the code well enough to ship it.
- Protect deliberate knowledge-sharing time: Anthropic found mentorship eroding as Claude replaced engineer conversations. Architecture decision records, rotating walkthroughs and pairing remain essential to maintain shared understanding.
The measurement problem
This is not an argument to stop using AI. Use it aggressively where it helps with tedious tasks, prototyping and verifiable work. But PR count and self-reported velocity are poor instruments to judge whether AI is helping teams ship. The studies combined indicate not just measurement noise but that the feedback loops leaders rely on have changed.
Harder-to-measure—but potentially more telling—metrics include cycle time from feature conception to delivery, the fraction of merged code that reaches production without rollback, the gap between planned and actual scope at sprint end, or company revenue growth correlated with AI investments (tooling, infrastructure, OpEx). None of these are easy to instrument, but the right question for teams is: “What would we need to measure to know?”
Note: The research described in this article was done in a personal capacity. The views expressed are the authors’ own and do not reflect those of their employers.
