This article is by James Shore, originally published on the Successful Software blog and republished here with the author’s permission. It is part 3 of Shore’s series assessing AI’s impact on software development; earlier parts covered why impact assessment matters and how to measure AI’s effect on delivery speed. Part 4 will examine business outcomes.
What the article examines
AI can make teams faster, but Shore asks whether that speed is sustainable. He explores unintended consequences of heavy AI use in software development — reduced maintainability, higher maintenance load, token costs, vendor lock‑in and developer burnout — and shows how to measure and model those effects.
Why maintainability matters economically
Shore emphasizes that concerns like maintainability, burnout and lock‑in have real economic impact. Hard‑to‑understand code makes feature work and bug fixes slower; burnt‑out staff reduce productivity and eventually leave, taking institutional knowledge with them; and rising token costs could force organizations to choose between big bills and scaling back development.
A unifying concept: cost of change
Many unintended consequences are well captured by the “cost of change” concept: a kind of tax on making changes to software. When cost of change is low, complex changes remain achievable; when it’s high, even small changes can take weeks.
Practical indicators to watch
- Value‑add percentage: the share of development effort spent on business‑value work (new features) versus non‑value‑add work such as bug fixes, incident response and manual release processes. Shore borrows the term "muda" (waste) from Lean; some muda is unavoidable, but rising nondiscretionary muda reduces time for value‑add work.
- DORA metrics: change lead time, deployment frequency, change fail rate, etc. Changes in these metrics are worth monitoring.
- Token usage: tokens are a literal cost when using LLMs. Token consumption varies with user adoption, prompts, models and harnesses, so it’s an advisory metric that should trigger further investigation rather than stand alone.
Shore trusts the value‑add percentage most because it measures where time is going. He recommends lightweight tracking (no detailed timesheets) such as categorizing multi‑day increments as value‑add or muda, with managers checking in a couple of times per week. The metric is noisy: trends often take 3–6 months to appear, and it may take a year to be confident about long‑term changes.
Use models and scenarios to supplement noisy measurements
Because indicators are noisy and lagging, Shore advises building models and scenario plans to explore risks. Models don’t predict the future, but they reveal possible outcomes.
Useful models Shore describes:
-
Maintainability model: treat maintenance as a mandatory “tax.” For each month of value‑add work, estimate maintenance in the following year and lower ongoing maintenance in later years. Model monthly engineering days, subtract maintenance, and the remainder is available for value‑add. Apply measured development speed improvements, estimate AI’s effect on maintenance costs and headcount, and explore scenarios with varying AI maintenance costs. Shore gives an example: a 50% AI speed increase offset by a 15% headcount reduction and doubled maintenance can be net zero.
-
Code churn / lines‑of‑code model: while lines of code aren’t a good productivity metric, they can approximate cost: more lines generally imply higher maintenance cost. Use traceability to label commits as value‑add or maintenance and measure gross churn. For each value‑add line, estimate maintenance churn over following years and model monthly maintenance needs. Real commit data (e.g., tmux) can help refine estimates.
-
Lock‑in model: if token prices rise, consider responses (cut headcount, reduce AI use, switch models). Update the maintainability model to simulate reduced engineering capacity and lowered AI performance.
-
Burnout and turnover: mental exhaustion likely reduces productivity and increases turnover. Shore says precise modeling is difficult, but recommends including mental exhaustion measures in development speed studies, running short periodic surveys about mental energy, monitoring turnover versus historical baselines, and checking exit interviews for AI‑related complaints.
The danger of gaming metrics
Shore warns that value‑add percentage is easy to game. If teams are judged by how they categorize work, they may reclassify work or defer maintenance to improve the metric. This isn’t always fraud — leaders might intentionally postpone maintenance to hit short‑term targets — but it distorts the indicator. He advises using these metrics to evaluate approaches and system behavior, not to judge individuals.
Conclusion
Development speed gains from AI and the unintended consequences of AI are two sides of the same coin: both determine how much and what quality of software gets delivered now and in the future. Shore recommends measuring indicators such as value‑add percentage, DORA metrics and token usage, and supplementing them with models for maintainability, code churn, lock‑in and burnout. Because measurements will be noisy and lagging, use models and judgment together, and treat metrics as tools for assessing approaches rather than people.
(Shore also references detailed explanations and spreadsheets for the suggested models and thanks Christophe Stjernlöf for tmux commit data used as an example.)
