Skip to content
BetaAI and tech news.Edition of Tue 27 Oct 2026 
AIAnalysis

AI coding agents write more code. Do they ship more software?

Controlled trials and repository studies agree that AI tools speed up typing code. Whether teams deliver more working software is far less settled.

By the Machine Notes desk5 min read
Share

What the studies found

Signs of gains

  • GitHub 2022: 55% faster
  • Google 2024: about 21% less time
  • DORA 2025: higher throughput

Signs of cost

  • METR 2025: 19% slower
  • Cursor repos: warnings up about 30%
  • Cursor repos: complexity up about 42%
  • Agent PRs: 1.62 times the odds of a fix
  • DORA 2025: lower stability
FIG. 1Source: Published studies and surveys

Not reliably. AI coding tools help people write code faster on contained tasks, but the evidence that they help teams ship more working software is thinner, and some of the best-designed studies point the other way.

The difference matters because "more code" is easy to measure and "more software" is not. Software that ships has been reviewed, tested, merged and kept working. Each of those steps takes time that a faster keyboard does not remove.

What the studies measured

The studies below are the ones we could read in full or in abstract on primary sources. They measure different things, which is part of why their results disagree.

StudyWho and whatFindingCaveat
GitHub, 202295 professional developers, randomly split, asked to write an HTTP server in JavaScriptCopilot group finished 55% faster (1 h 11 min vs 2 h 41 min); 78% vs 70% completed the taskOne small, self-contained task; confidence interval 21% to 89%; run by the tool's maker
Paradis et al., 202496 full-time Google engineers in a randomised lab study using internal AI toolsAbout 21% less time on a complex enterprise-style taskLab setting; wide confidence interval; internal tools from summer 2024
Becker et al. (METR), 202516 experienced open-source maintainers, 246 real tasks on their own projects, randomly allowed or barred from AI (mainly Cursor with Claude 3.5/3.7 Sonnet)Tasks with AI took 19% longerSmall group of developers; the authors say experimental artefacts cannot be fully ruled out
DORA (Google Cloud), 2025Survey of nearly 5,000 technology professionalsAI adoption now goes with higher delivery throughput but still with lower delivery stabilityCorrelation from a survey, not a controlled test
He et al., 2025-26806 open-source repositories that adopted Cursor, compared with 1,380 similar ones that did notLines added rose 281% in the first month, then the gain faded after about two months; static analysis warnings rose about 30% and code complexity about 42%, and stayed upAdoption inferred from Cursor config files; actual usage unknown; preprint
Takerngsaksiri et al., 20266,774 merged pull requests from five coding agents in popular open-source repositories, compared with 5,044 human onesAgent merges had 1.62 times the odds of needing a verified follow-up fixPreprint; limited to repositories with 500 or more stars

The case for real gains

The two randomised trials run by tool makers both found speed-ups. GitHub's 2022 experiment found developers with Copilot finished a set task 55% faster and were more likely to finish at all. Google's 2024 trial with 96 of its own engineers found a smaller effect, about 21% less time on task, on work designed to look more like real enterprise coding.

Google Cloud's DORA report on software delivery, published on 24 September 2025, found that 90% of the professionals surveyed use AI at work and more than 80% believe it has made them more productive. Unlike the previous year's report, it also found a positive link between AI adoption and delivery throughput, meaning how quickly changes reach production.

The case for caution

METR's study is the one most often cited by sceptics, and for good reason: it tested experienced developers on real issues in large projects they already knew well. Before starting, those developers expected AI to cut their time by 24%. Afterwards they still believed it had saved them about 20%. The measured result was a 19% slowdown. The gap between what people felt and what the clock showed is as important as the slowdown itself.

The same DORA report that found higher throughput also found that AI adoption still goes with less stable delivery. Its summary is that AI amplifies what a team already has: teams with strong testing and fast feedback benefit, while teams without them get more change volume and more instability.

The repository studies show what that looks like in code. In the Cursor study, the jump in output was large but lasted about two months. The rise in warnings and complexity did not fade, and the authors' models suggest that this added complexity is a major reason velocity later slowed. The September 2026 study of agent pull requests found that merged agent work needed follow-up fixes more often than human work in the same repositories, although agents wrote most of those fixes themselves.

Our take

The honest answer to the title is "sometimes, and less than the code volume suggests". The trials of short, bounded tasks found gains, while the one trial on experienced developers' own real projects found a slowdown. The studies that look past the first merge, at stability, complexity and rework, find costs that eat into it. Neither side has shown a clean, controlled result on whole teams shipping whole products, which is the number buyers actually want.

Two patterns stand out. First, self-reports are unreliable: METR's developers felt faster while being slower, and DORA's adoption and productivity figures are survey answers. Second, the bottleneck moves rather than disappears. If an agent can produce a pull request in minutes but a person still has to read, test and later fix it, review becomes the constraint. Teams that invested in tests and fast feedback before agents arrived seem best placed to turn more code into more software.

How to read the next productivity claim

  • Check what was measured. Lines of code, commits and "time to complete a task" are output. Shipped features, failed deployments and time spent fixing are outcomes.
  • Check the time window. A gain in the first month can disappear by the third.
  • Check who ran it. Studies by tool makers are not wrong by default, but independent replication matters.
  • Check whether it is self-reported. Feeling faster and being faster were different things in the strongest trial so far.

Sources

  1. GitHub Blog: Research: quantifying GitHub Copilot's impact on developer productivity and happiness (7 September 2022)github.blog
  2. Paradis et al., How much does AI impact development speed? An enterprise-based randomized controlled trial (arXiv, 2024)arxiv.org
  3. Becker et al., Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv, 2025)arxiv.org
  4. Google Cloud: Announcing the 2025 DORA Report (24 September 2025)cloud.google.com
  5. He et al., Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects (arXiv, v3 January 2026)arxiv.org
  6. Takerngsaksiri et al., Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests (arXiv, September 2026)arxiv.org