AI Coding: How To Spot Waste And Measure What Works

🔍 Read the full analysis: AI Coding: How To Spot Waste And Measure What Works on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Engineering teams surveyed by The Pragmatic Engineer spend between $150 and about $3,000 per developer per month on AI coding tools, but many cannot say what that spending has delivered. The October 20 report draws on interviews at 27 workplaces and vendor data, finding that output measures range from flat to sharply higher and that companies need to judge results by accepted work and rework, not tool usage alone.

The spending range does not map neatly to company size. One publicly traded company with 1,500 engineers caps monthly use at $150 per developer; a two-person European .NET agency, Ix Technologies, spends about $3,000 per developer. Other firms described caps of $1,000 to $4,000, with typical monthly spending around $1,000. The report also describes a quant trading firm whose AI bill rose from $500,000 in January to $8 million in September. At that firm, traders—not engineers—were the biggest spenders, and desk-level profit and loss is used to judge the expense.

Evidence on productivity is mixed. CodeRabbit, a vendor whose platform is used by the teams in its data, reported that weekly pull requests rose 113% year over year. A Microsoft study of tens of thousands of engineers during an early-2026 rollout found adopters merged about 24% more pull requests than they otherwise would have over four months. Microsoft’s authors cautioned that a merged pull request does not by itself show the value delivered. Among the workplaces interviewed, some reported no change in PR frequency, while one said PRs became three times larger.

Controlled research offers a less settled picture. METR’s early-2025 study found experienced open-source developers took 19% longer with AI. A February follow-up estimated an 18% speedup for returning developers and 4% for new developers, but both confidence intervals included zero; METR said the results were unreliable, in part because participants held back some tasks they did not want to attempt without AI. The article also reports a Veracode test in which 44% of AI coding tasks produced a known vulnerability, underscoring that speed measures need to be weighed against security and review costs.

Budgets Need Outcome Measures

The immediate issue for companies setting 2027 budgets is not simply whether developers use AI, but whether the resulting work is useful enough to justify its cost. Pull-request volume can rise without a matching gain in delivered value if changes are larger, need more correction, or add to review and security workloads. A growing bill may also reflect token consumption rather than a useful business outcome.

The report points to places where teams described practical gains: customer-issue triage, code archaeology, prototypes, and backlog cleanup, especially in simple, consistent software environments. One SaaS company with about 600 engineers saw little change in merged-PR velocity, but a staff engineer said customer issue triage that could take a day was reduced to minutes of agent work plus up to an hour of human checking. Finding the history behind a code decision reportedly fell from a day or two to minutes.

Those examples do not establish a return across the industry. They show why organizations may need to track cost per accepted change, rework, change failure rates, and stories closed versus stories arriving, alongside spending. Those measures can help distinguish more activity from more valuable output.

Why AI Bills Are Under Review

The report’s timing reflects changing costs as well as uncertain results. The Pragmatic Engineer’s Gergely Orosz said API-plan spending peaked in June and July and had roughly halved by the time of his reporting, attributing the change to Opus 5.5 and a move to open models. Companies also described spending caps that were easy to get around and discounted subscriptions nearing their end; some said losing those discounts could raise bills five- to tenfold.

Billing models are shifting too. GitHub said that on June 1 it moved every Copilot plan to usage-based billing, charging tokens at each model’s API rate and removing the fallback to a cheaper model. The company said the change reflected rising compute demand from agentic use. The Pragmatic Engineer article’s findings are based on its freely readable material, interviews at 27 workplaces and aggregate data from CodeRabbit and Gitar; the article says some of its material is paywalled.

Reported gains also depend on what is measured and which developers are included. Microsoft’s PR results covered adopters during a specific rollout, while CodeRabbit’s numbers describe teams on its own platform. Neither metric alone establishes broad productivity gains or business value across all engineering teams.

“A merged PR is not the same as the value it delivers.”

— Microsoft study authors

Productivity Gains Remain Uneven

The reported results do not establish a single productivity gain that applies across companies. The interviews include workplaces reporting flat PR frequency and others reporting much higher output; vendor measurements and company-specific studies also cover different users and methods. The Microsoft result measures merged pull requests, not downstream business value, and the CodeRabbit figure comes from users of its platform.

METR’s estimates are especially uncertain: both speedup estimates in its February follow-up had confidence intervals that included zero, and the organization described its data as unreliable because some participants withheld tasks. The report’s claim that AI coding often pays in particular workflows is based on company accounts, not a common controlled comparison. It remains unclear how much of the reported time savings persist after review, maintenance, security checks and correction.

It is also unclear how spending will change as discounted plans expire and usage-based billing spreads. The cited Veracode result is a test finding, not a measure of the vulnerability rate in all AI-generated code in production.

Track Results Through 2027

Companies setting next year’s budgets can compare spending with accepted changes, rework, failure rates and completed work relative to incoming demand, rather than relying on token use or PR counts alone. They can also separate results by task type and software environment: the report’s examples suggest triage and backlog work on uniform stacks may perform differently from changes to complex legacy systems.

As discounts end and billing shifts toward usage, engineering and finance leaders will have more direct cost signals, but those bills will not by themselves answer whether the tools pay off. The next useful evidence will come from teams that publish comparable measurements over time, including quality and follow-up work. The report does not provide a shared industry benchmark or a definitive ROI figure, so companies will need to judge their own results while that evidence develops.

Key Questions

How much are companies spending on AI coding tools?

The workplaces in The Pragmatic Engineer report described spending from $150 to about $3,000 per developer per month. The figures are a range from 27 workplaces, not an industry-wide average.

Does AI coding increase developer output?

Some measures show increases, including Microsoft’s finding of about 24% more merged pull requests among adopters over four months and CodeRabbit’s report of 113% year-over-year growth in weekly PRs on its platform. Other companies reported flat PR frequency, and PR volume does not by itself measure delivered value.

What should companies measure to judge return on AI coding?

The report recommends tracking measures such as cost per accepted change, rework, change failure rate, and stories closed compared with stories coming in. Teams can also track review and security costs to avoid mistaking higher activity for better outcomes.

Where did teams report the clearest benefits?

Accounts in the report point to customer-issue triage, understanding code history, prototyping and clearing backlogs as promising uses, particularly on simple, uniform software stacks. These are reported company experiences, not proof that the same gains will occur elsewhere.

Is there a reliable industry-wide ROI figure?

No shared figure is established in the report. Its sources use different measures and populations, and METR said its follow-up estimates were unreliable. Many companies interviewed could not yet connect AI coding costs to a clear business return.

Source: ThorstenMeyerAI.com