Back to Blog

How to Measure AI Coding Agent Productivity Without Falling Into the Speed Trap

In this article

• Quick Answer
• Key Takeaways
• Why Throughput Metrics Mislead With Coding Agents
• A Three Layer Framework for Measuring Coding Agents
• Layer One, the DORA Metrics
• How Do Your DORA Metrics Compare?
• Layer Two, Collaboration Quality
• What Published Benchmarks Say About Agent Written Code
• Layer Three, Cost Per Accepted Change
• Reduce Review Burden With Feedback Sensors
• Measurement Anti Patterns to Avoid
• Where the Data Comes From
• A 30 Day Plan to Start Measuring
• How Unico Connect Helps
• Frequently Asked Questions
• Sources

In early 2026 some technology companies started ranking engineers by how many AI tokens they consumed, a practice that became known as tokenmaxxing. In June 2026, the DORA research team at Google Cloud published a direct response, arguing that raw token counts are a vanity metric and that the history of software is littered with activity measures that promised a shortcut to productivity.

Two months earlier, the Thoughtworks Technology Radar warned that most coding agent metrics still focus too heavily on throughput, such as lines generated and tasks completed.

Both point to the same answer. Measure what reaches production and how much human effort it took to get there. This guide sets out a practical framework for measuring coding agent output, the metrics that matter, how to collect them, and the traps to avoid. Sources were checked on 22 September 2026.

Why Throughput Metrics Mislead With Coding Agents

Coding agents make activity cheap. An agent can produce thousands of lines, open many pull requests, and consume millions of tokens in a day, and none of that says whether the product improved. Thoughtworks describes the risk as the speed trap, where teams judge agents by time to first output, lines of code generated, and tasks completed.

Thoughtworks goes further in a separate April 2026 entry, placing coding throughput as a measure of productivity in its Caution ring. It warns that when lines of code or pull request counts are used in isolation, the result is often a flood of poorly aligned code that slows reviews, and it notes that lower first pass acceptance rates tend to increase change failure rates while repeated iteration cycles extend lead time for changes.

The research backs the caution. The DORA 2025 State of AI assisted Software Development report, based on a survey of nearly 5,000 technology professionals run between 13 June and 21 July 2025, found that 90 percent use AI in their work and more than 80 percent believe it increased their productivity. It also found that 30 percent report little or no trust in AI generated code, and that while AI adoption now improves delivery throughput, it still increases delivery instability. DORA summarizes the finding as AI being an amplifier, magnifying the strengths of high performing organizations and the dysfunctions of struggling ones.

Perceived speed is not proof either. In a randomized controlled trial by METR, using data from February to June 2025, 16 experienced open source developers took 19 percent longer to finish tasks when AI tools were allowed, yet afterwards still believed AI had sped them up by 20 percent. METR said in February 2026 that newer tools likely help more, but that self reported speedups can be quite unreliable. That is why this framework relies on delivery data rather than impressions.

DORA explains the tokenmaxxing problem with a principle named after the economist Charles Goodhart, that when a measure becomes a target it ceases to be a good measure. Once token counts become a target, they stop measuring real work, and DORA reports developers running agents on meaningless projects simply to stay above a company average.

A Three Layer Framework for Measuring Coding Agents

The layers work together. Faster lead time with rising rework means the agent is shipping problems faster. Higher first pass acceptance with flat lead time means the bottleneck is elsewhere, often in review, testing, or deployment.

Layer One, the DORA Metrics

Thoughtworks keeps DORA metrics in its Adopt ring and says they matter more than ever with AI assisted development. The set now has five measures.

1. Change lead time. Time from commit to running in production.
2. Deployment frequency. How often changes reach production.
3. Failed deployment recovery time. How long it takes to restore service after a deployment fails and needs immediate intervention. DORA renamed this measure from time to restore service in 2023.
4. Change fail rate. The share of deployments that need immediate intervention, such as a rollback or hotfix.
5. Deployment rework rate. The share of deployments that were unplanned and made to fix a user facing bug, added by DORA as the fifth metric in 2024. Thoughtworks describes the same signal as how much of the delivery pipeline is consumed by unplanned rework.

Thoughtworks is blunt about the logic. If lead times do not fall and deployment frequency does not rise, faster code generation does not translate into better outcomes.

How Do Your DORA Metrics Compare?

The DORA 2025 survey published how respondents answered each delivery question. Use it for context, but DORA notes that the most useful comparison is the same application or service over time.

If you do not have delivery tooling yet, the DORA Quick Check, updated in April 2026, scores all five metrics on a 10 point scale against benchmarks from the 2025 research.

Layer Two, Collaboration Quality

These are the metrics Thoughtworks highlights in its April 2026 entry on measuring collaboration quality with coding agents, which sits in the Assess ring.

Thoughtworks notes that teams using Claude Code can use the /insights command to generate reports on agent sessions, and that its own teams have tracked first pass acceptance of a customized review command. The signal it looks for is simple. Shorter feedback cycles and fewer failed builds indicate more effective interaction with coding agents.

What Published Benchmarks Say About Agent Written Code

Your own baseline matters more than any published figure, but research gives you a first reference point while you build one. The figures below all come from studies that name their population and their definition.

The largest dataset is AIDev, which covers 456,000 pull requests by five coding agents across 61,000 repositories and 47,000 developers (Li, Zhang and Hassan, July 2025). It reports the share of agent pull requests that get merged at all, and the median time from opening a pull request to closing it.

Across those five agents the average merge rate is 51.3 percent, against 76.8 percent for human authored pull requests in the same dataset. Treat the spread as the real finding. The gap between the best and worst agent is 27 points, wider than the gap between the average agent and a human, so a number from one tool says little about another.

Merged is not the same as accepted as written. A study of 567 Claude Code pull requests across 157 open source projects found that 83.8 percent were merged, and that 54.9 percent of the merged ones went in with no further modification (Watanabe and colleagues, September 2025). That works out to about 46 percent of submitted pull requests landing without a change, which is the closest published figure to a first pass acceptance rate.

Two more reference points are worth holding loosely, because each measures something narrower than it sounds.

• Rework is rising, and the size of the rise depends on the definition. GitClear, analyzing 623 million changed lines, reports that code churn, meaning lines reverted or rewritten within two weeks of being authored, is up 15 percent against its 2023 baseline (GitClear, June 2026). Other published churn figures use different denominators and are not comparable, so compare yourself against your own baseline rather than against a headline.
• Cost per attempted task is roughly a dollar fifty. Faros ran 197 common tasks from its own engineering work and recorded a typical cost of $1.51 a task for Claude Code with Opus 4.8 and $1.43 for Codex with GPT-5.5, with 95 percent of tasks at or below $6.84 and $3.21 (Faros Route Index, study dated 25 June 2026). That is cost per attempt, not cost per accepted change, so the number you care about is higher by whatever share of attempts you throw away.

Use these as a sanity check, not a target. If your merge rate sits far below 50 percent, the problem is usually instructions, context, or task selection rather than the model.

Layer Three, Cost Per Accepted Change

Token spend is real money, but it is an input. DORA suggests pairing the core metrics with efficiency and risk indicators such as cost per accepted change and code rework rate, and it agrees with an analysis by Adnan Masood that high token spend yields diminishing returns, with the teams on the biggest budgets often gaining only a marginal increase in throughput at around ten times the cost.

The practical version is to divide model and tool spend for a period by the number of agent assisted changes that were accepted and stayed in production. DORA also describes Shopify replacing an AI leaderboard with a usage dashboard and adding circuit breakers that catch runaway autonomous agents and flag abnormal daily spend. If you track tokens at all, DORA suggests tracking them by team, by application, or by use case rather than company wide or per person.

Reduce Review Burden With Feedback Sensors

The fastest way to improve collaboration metrics is to stop humans from being the first line of quality control. Thoughtworks places feedback sensors for coding agents in its Trial ring. The idea is to wire deterministic checks such as compilers, linters, structural tests, and test suites directly into the agent workflow, so failures trigger self correction before a person ever sees the change. Thoughtworks recommends that these checks run during the coding session and report clean results before a commit, rather than relying on post commit checks, and notes that coding agents make it cheaper to build custom linters and structural tests.

Alongside this, Thoughtworks now treats sandboxed execution for coding agents as a sensible default rather than an optional extra, running agents in isolated environments with restricted file system access, controlled network connectivity, and bounded resources.

Measurement Anti Patterns to Avoid

• Token leaderboards. DORA warns they encourage productivity theater, knowledge hoarding, and burnout.
• Lines of code and pull request counts. Thoughtworks places coding throughput as a measure of productivity in its Caution ring and calls measuring productivity by lines of AI generated code misleading.
• Individual scores. Both DORA and Thoughtworks recommend team level measurement used for reflection.
• No baseline. Without delivery metrics from before agent adoption, you cannot tell whether anything changed.
• Dashboards instead of conversations. Thoughtworks notes that simple check ins during retrospectives often improve capabilities more than detailed tracking tools.

Where the Data Comes From

Vendor acceptance rates count suggestions accepted in the editor or the coding session, not changes accepted in review, so they do not replace first pass acceptance rate.

A 30 Day Plan to Start Measuring

1. Week one, set the baseline. Pull the last quarter of lead time, deployment frequency, change fail rate, recovery time, and rework from your version control, CI, and incident tools, or run the DORA Quick Check with the team if that data is not instrumented yet.
2. Week two, tag agent work. Label pull requests and commits that were agent assisted, so every metric can be split by agent and non agent work.
3. Week three, add collaboration metrics. Start tracking first pass acceptance, iteration cycles, failed builds, and review time per change for agent assisted work.
4. Week four, add cost and review. Compute cost per accepted change, set spend alerts, and review all three layers together in a team retrospective.

After the first month, review monthly. Look for rising rework or review burden as the earliest signs that agent use is outrunning your quality controls.

How Unico Connect Helps

We help engineering teams adopt coding agents in a way that shows up in delivery metrics rather than only in activity. That includes setting up DORA and collaboration quality measurement, wiring feedback sensors into agent workflows, and sandboxing agent execution. Our Claude Code for engineering teams service covers rollout and measurement, our cloud and DevOps services cover the pipelines that produce DORA data, and our AI development services cover custom agent tooling. Teams that want experienced engineers who already work this way can hire AI engineers through us. We run the same approach internally, with standards aware review on every change, as described in our engineering enablement case study. For more, see AI coding tools vs AI agents, our analysis of AI generated code at scale, or talk to our team.

Frequently Asked Questions

Sources

Radar positions and descriptions come from the Thoughtworks Technology Radar entries for measuring collaboration quality with coding agents, DORA metrics, coding throughput as a measure of productivity, feedback sensors for coding agents, and sandboxed execution for coding agents. Survey findings come from the DORA 2025 State of AI assisted Software Development report, and the tokenmaxxing analysis from the DORA insight Finding balance in the era of tokenmaxxing, published on 2 June 2026. The METR findings come from its early 2025 developer productivity study and its February 2026 update. DORA metric names follow the DORA history of its software delivery metrics, and tool data follows the GitHub Copilot usage metrics and Claude Code analytics documentation. The benchmark figures for agent written pull requests come from AIDev, published 20 July 2025, and the empirical study of agentic coding pull requests, published 18 September 2025. Code churn comes from GitClear, June 2026, and the cost per task figures from the Faros Route Index, June 2026. All were read on 22 and 23 September 2026.