How to Measure AI Coding Agent Productivity Without Falling Into the Speed Trap
Saurav Jagdale
Technical Lead, Unico Connect
In this article
- Quick Answer
- Key Takeaways
- Why Throughput Metrics Mislead With Coding Agents
- A Three Layer Framework for Measuring Coding Agents
- Layer One, the DORA Metrics
- How Do Your DORA Metrics Compare?
- Layer Two, Collaboration Quality
- What Published Benchmarks Say About Agent Written Code
- Layer Three, Cost Per Accepted Change
- Reduce Review Burden With Feedback Sensors
- Measurement Anti Patterns to Avoid
- Where the Data Comes From
- A 30 Day Plan to Start Measuring
- How Unico Connect Helps
- Frequently Asked Questions
- Sources
In early 2026 some technology companies started ranking engineers by how many AI tokens they consumed, a practice that became known as tokenmaxxing. In June 2026, the DORA research team at Google Cloud published a direct response, arguing that raw token counts are a vanity metric and that the history of software is littered with activity measures that promised a shortcut to productivity.
Two months earlier, the Thoughtworks Technology Radar warned that most coding agent metrics still focus too heavily on throughput, such as lines generated and tasks completed.
Both point to the same answer, which is to measure what reaches production and how much human effort it took to get it there. The framework below does that in three layers (delivery outcomes, collaboration quality and cost), then shows where each number comes from and which measurement habits backfire. We checked the sources on 22 and 23 September 2026.
Quick Answer
Measure AI coding agents on outcomes, not activity. Track the five DORA metrics (change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate) to see whether faster code reaches users safely. Add the collaboration metrics Thoughtworks recommends, namely first pass acceptance rate, iteration cycles, failed builds and review burden. Swap token counts for cost per accepted change and report every metric at team level. Leave lines of code out entirely.
Key Takeaways
- DORA found in 2025 that AI adoption now improves software delivery throughput but still increases delivery instability, so read throughput and stability together.
- Watch rework rate closely. Thoughtworks calls a rise in rework rate an early warning sign of blind spots, technical debt and the risks of unchecked AI assisted development.
- Collaboration quality deserves as much weight as speed. First pass acceptance, iteration cycles, failed builds and review burden show whether agents save effort or move it onto your reviewers.
- Token counts are an input, so the number to report is cost per accepted change. DORA calls company wide token aggregates meaningless and individual token scores harmful, and recommends the cost measure instead.
- Keep the numbers at team level and discuss them in retrospectives. Both DORA and Thoughtworks recommend team level tracking used for reflection instead of ranking.
Why Throughput Metrics Mislead With Coding Agents
Coding agents make activity cheap. An agent can produce thousands of lines, open many pull requests, and consume millions of tokens in a day, and none of that says whether the product improved. Thoughtworks describes the risk as the speed trap, where teams judge agents by time to first output, lines of code generated, and tasks completed.
Thoughtworks goes further in a separate April 2026 entry, placing coding throughput as a measure of productivity in its Caution ring. It warns that when lines of code or pull request counts are used in isolation, the result is often a flood of poorly aligned code that slows reviews, and it notes that lower first pass acceptance rates tend to increase change failure rates while repeated iteration cycles extend lead time for changes.
The DORA 2025 State of AI assisted Software Development report backs the caution. Based on a survey of nearly 5,000 technology professionals run between 13 June and 21 July 2025, it found that 90 percent use AI in their work and more than 80 percent believe it increased their productivity. It also found that 30 percent report little or no trust in AI generated code, and that while AI adoption now improves delivery throughput, it still increases delivery instability. DORA summarizes the finding as AI being an amplifier, magnifying the strengths of high performing organizations and the dysfunctions of struggling ones.
Self reported speed is a weak signal as well. In a randomized controlled trial by METR, using data from February to June 2025, 16 experienced open source developers took 19 percent longer to finish tasks when AI tools were allowed, yet afterwards still believed AI had sped them up by 20 percent. METR said in February 2026 that newer tools likely help more, but that self reported speedups can be quite unreliable. That is why the framework below relies on delivery data instead of impressions.
DORA explains the tokenmaxxing problem with a principle named after the economist Charles Goodhart, that when a measure becomes a target it ceases to be a good measure. Once token counts become a target, they stop measuring real work, and DORA reports developers running agents on meaningless projects only to stay above a company average.
A Three Layer Framework for Measuring Coding Agents
| Layer | What it answers | Metrics |
|---|---|---|
| Delivery outcomes | Is faster code reaching users safely? | Change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate |
| Collaboration quality | Are agents saving human effort or moving it? | First pass acceptance rate, iteration cycles per task, post merge rework, failed builds, review burden |
| Cost and efficiency | Is the spend producing accepted work? | Cost per accepted change, token spend by team, application, or use case, runaway agent incidents |
Read the layers together. If lead time falls while rework rises, the agent is shipping problems faster. Higher first pass acceptance with flat lead time means the bottleneck sits somewhere else, often in review, testing or deployment.
Layer One, the DORA Metrics
Thoughtworks keeps DORA metrics in its Adopt ring and says they matter more than ever with AI assisted development. The set now has five measures.
- Change lead time. Time from commit to running in production.
- Deployment frequency. How often changes reach production.
- Failed deployment recovery time. How long it takes to restore service after a deployment fails and needs immediate intervention. DORA renamed this measure from time to restore service in 2023.
- Change fail rate. The share of deployments that need immediate intervention, such as a rollback or hotfix.
- Deployment rework rate. The share of deployments that were unplanned and made to fix a user facing bug, added by DORA as the fifth metric in 2024. Thoughtworks describes the same signal as how much of the delivery pipeline is consumed by unplanned rework.
Thoughtworks is blunt about the logic. If lead times do not fall and deployment frequency does not rise, faster code generation does not translate into better outcomes.
How Do Your DORA Metrics Compare?
The DORA 2025 survey published how respondents answered each delivery question. Use it for context, but DORA notes that the most useful comparison is the same application or service over time.
| Metric | DORA 2025 survey result |
|---|---|
| Change lead time | 24.4 percent ship a committed change to production in less than one day, and 9.4 percent in less than one hour |
| Deployment frequency | 16.2 percent deploy on demand, several times a day, and 44.6 percent deploy at least once a week |
| Failed deployment recovery time | 56.5 percent restore service in less than one day, and 21.3 percent in less than one hour |
| Change fail rate | 62.2 percent say 16 percent or less of changes need remediation, and 16.7 percent say 4 percent or less |
| Deployment rework rate | 52.6 percent say 16 percent or less of deployments were unplanned bug fixes, and 7.3 percent say more than 64 percent |
If you do not have delivery tooling yet, the DORA Quick Check, updated in April 2026, scores all five metrics on a 10 point scale against benchmarks from the 2025 research.
Layer Two, Collaboration Quality
These are the metrics Thoughtworks highlights in its April 2026 entry on measuring collaboration quality with coding agents, which sits in the Assess ring.
| Metric | How to measure it | What good looks like |
|---|---|---|
| First pass acceptance rate | Share of agent generated changes accepted in review without substantial rewrites | Rising over time as instructions and feedback improve |
| Iteration cycles per task | Number of prompt and correction rounds before a task is done | Falling, with fewer long back and forth sessions |
| Post merge rework | Follow up changes needed to fix agent code after it merged | Low, and not rising as agent use grows |
| Failed builds | Builds broken by agent changes, in CI or locally | Low, ideally caught before commit |
| Review burden | Reviewer time per change and comments per pull request | Stable or falling as volume rises |
Thoughtworks notes that teams on Claude Code can run the /insights command to generate reports on agent sessions, and that its own teams have tracked first pass acceptance of a customized review command. The signal it looks for is shorter feedback cycles and fewer failed builds, which indicate more effective interaction with coding agents.
What Published Benchmarks Say About Agent Written Code
Your own baseline matters more than any published figure, but research gives you a first reference point while you build one. The figures below all come from studies that name their population and their definition.
The largest pull request study cited here is AIDev, whose first release covered 456,000 pull requests by five coding agents across 61,000 repositories and 47,000 developers (Li, Zhang and Hassan, July 2025). For 7,122 of those pull requests, from repositories with at least 500 GitHub stars, it reports the share that get merged at all, and the median time from opening an accepted pull request to closing it.
| Agent | Pull requests merged | Median turnaround on accepted pull requests |
|---|---|---|
| OpenAI Codex | 65.3% | 0.3 hours |
| Claude Code | 52.5% | 6.9 hours |
| Cursor | 51.4% | 2.4 hours |
| Devin | 48.9% | 2.2 hours |
| GitHub Copilot | 38.2% | 17.2 hours |
| Human pull requests, for comparison | 76.8% | 3.9 hours |
Across those five agents the average merge rate is 51.3 percent, against 76.8 percent for human authored pull requests from the same repositories. The spread tells you more than the average does. The best and worst agents sit 27 points apart, a wider gap than the one between the average agent and a human, so a number from one tool says little about another.
Merge rate also overstates first pass acceptance, because many merged pull requests are edited on the way in. A study of 567 Claude Code pull requests across 157 open source projects found that 83.8 percent were merged, and that 54.9 percent of the merged ones went in with no further modification (Watanabe and colleagues, September 2025). That works out to about 46 percent of submitted pull requests landing without a change, which is the closest published figure to a first pass acceptance rate.
Two more reference points are worth holding loosely, because each measures something narrower than it sounds.
- Rework is rising, though how much depends on the definition. GitClear analyzed 623 million changed lines and reports that code churn, meaning lines reverted or rewritten within two weeks of being authored, is up 15 percent against its 2023 baseline (GitClear, June 2026). Other published churn figures use different denominators and cannot be compared with this one, so measure yourself against your own baseline.
- An attempted task costs roughly a dollar fifty. Faros ran 197 common tasks from its own engineering work and recorded a typical cost of $1.51 a task for Claude Code with Opus 4.8 and $1.43 for Codex with GPT-5.5, with 95 percent of tasks at or below $6.84 and $3.21 (Faros Route Index, study dated 25 June 2026). Those figures are per attempt, so your cost per accepted change is higher by whatever share of attempts you throw away.
Use these figures as a sanity check and do not set them as targets. If your merge rate sits far below 50 percent, look at instructions, context or task selection before the model, because that is usually where the problem lies.
Layer Three, Cost Per Accepted Change
Token spend is real money, and on its own it tells you only what went in. DORA suggests pairing the core metrics with efficiency and risk indicators such as cost per accepted change and code rework rate, and it agrees with an analysis by Adnan Masood that high token spend yields diminishing returns, with the teams on the biggest budgets often gaining only a marginal increase in throughput at around ten times the cost.
To calculate cost per accepted change, divide model and tool spend for a period by the number of agent assisted changes that were accepted and stayed in production. DORA also describes Shopify replacing an AI leaderboard with a usage dashboard and adding circuit breakers that catch runaway autonomous agents and flag abnormal daily spend. If you track tokens at all, DORA suggests breaking them down by team, by application or by use case, and avoiding company wide or per person totals.
Reduce Review Burden With Feedback Sensors
The fastest way to improve collaboration metrics is to stop humans from being the first line of quality control. Thoughtworks places feedback sensors for coding agents in its Trial ring. The idea is to wire deterministic checks such as compilers, linters, structural tests, and test suites directly into the agent workflow, so failures trigger self correction before a person ever sees the change. Thoughtworks recommends that these checks run during the coding session and report clean results before anything is committed, instead of relying on post commit checks. It also notes that coding agents make it cheaper to build custom linters and structural tests.
Thoughtworks now treats sandboxed execution for coding agents as a sensible default too, which means running agents in isolated environments with restricted file system access, controlled network connectivity and bounded resources.
Measurement Anti Patterns to Avoid
- Token leaderboards, which DORA warns encourage productivity theater, knowledge hoarding and burnout.
- Lines of code and pull request counts. Thoughtworks places coding throughput as a measure of productivity in its Caution ring and calls measuring productivity by lines of AI generated code misleading.
- Scoring individuals goes against both DORA and Thoughtworks, which recommend team level measurement used for reflection.
- Starting without a baseline. If you have no delivery metrics from before agent adoption, you cannot tell whether anything changed.
- Relying on dashboards and skipping the conversation. Thoughtworks notes that simple check ins during retrospectives often improve capabilities more than detailed tracking tools.
Where the Data Comes From
| Source | What it gives you | Watch out for |
|---|---|---|
| Version control and CI | Lead time, deployment frequency, failed builds, review time per pull request | Label agent assisted pull requests so every metric can be split |
| Incident tooling | Failed deployment recovery time, change fail rate, unplanned fix deployments | Count only failures with real user impact |
| GitHub Copilot usage metrics | Active users, suggestion acceptance rate, lines added, pull requests created by Copilot, median time to merge, adoption cohorts | Editor acceptance is not review acceptance |
| Claude Code analytics for Team and Enterprise plans | Suggestion accept rate, lines accepted, merged pull requests with Claude Code (public beta), and per user token counts and cost estimates through OpenTelemetry export | Lines accepted do not track later deletions |
| DORA Quick Check | A score out of 10 for all five DORA metrics against DORA 2025 benchmarks | Self reported, best used as a team conversation |
Vendor acceptance rates count suggestions accepted in the editor or the coding session. Acceptance in review is a separate step that comes later, so these rates cannot stand in for first pass acceptance rate.
A 30 Day Plan to Start Measuring
- Week one, set the baseline. Pull the last quarter of lead time, deployment frequency, change fail rate, recovery time, and rework from your version control, CI, and incident tools, or run the DORA Quick Check with the team if that data is not instrumented yet.
- Week two, tag agent work. Label pull requests and commits that were agent assisted, so every metric can be split by agent and non agent work.
- Week three, add collaboration metrics. Start tracking first pass acceptance, iteration cycles, failed builds, and review time per change for agent assisted work.
- Week four, add cost and review. Compute cost per accepted change, set spend alerts, and review all three layers together in a team retrospective.
After the first month, review monthly. Look for rising rework or review burden as the earliest signs that agent use is outrunning your quality controls.
How Unico Connect Helps
We help engineering teams adopt coding agents so the results show up in their delivery metrics. That work includes setting up DORA and collaboration quality measurement, wiring feedback sensors into agent workflows and sandboxing agent execution. Rollout and measurement sit under our Claude Code for engineering teams service, the pipelines that produce DORA data come from our cloud and DevOps services, and custom agent tooling is part of our AI development services. Teams that want experienced engineers who already work this way can hire AI engineers through us. We run the same approach internally, with standards aware review on every change, as described in our engineering enablement case study. For more, see AI coding tools vs AI agents, our analysis of AI generated code at scale, or talk to our team.
Frequently Asked Questions
How do you measure AI coding agent productivity?
Measure outcomes, meaning what agent work delivers to production and the human effort and spend it took to get there. Track the five DORA metrics to see whether changes reach production faster and more safely, add collaboration quality metrics such as first pass acceptance rate, iteration cycles, failed builds, and review burden, and track cost per accepted change. Measure at team level and compare against a baseline from before agent adoption.
What is rework rate in DORA metrics?
Rework rate, which DORA calls deployment rework rate, is the fifth DORA metric, added in 2024. It is the share of deployments that were unplanned and made to fix a user facing bug in production. In the DORA 2025 survey, 52.6 percent of respondents put it at 16 percent or less. Thoughtworks calls a rise in rework rate an early warning sign of blind spots, technical debt and the risks of unchecked AI assisted development.
What is tokenmaxxing?
Tokenmaxxing is the practice, which emerged in early 2026, of tracking and rewarding the engineers who consume the most AI tokens, often on internal leaderboards. DORA argues it is a gameable vanity metric that encourages productivity theater, wasted spend, technical debt, and burnout.
Should we track AI token usage at all?
Yes, track it as a cost signal and keep it out of performance targets. DORA suggests tracking usage by team, application, or use case rather than company wide or by individual, pairing it with cost per accepted change, and using spend alerts or circuit breakers to catch runaway agents.
What is first pass acceptance rate?
It is the share of agent generated changes that are accepted in review without substantial rewrites. Thoughtworks lists it among the collaboration quality metrics that give more meaningful signals than speed alone.
What is a good first pass acceptance rate for a coding agent?
There is no industry standard yet, so compare against your own baseline first. For a reference point, a study of 567 Claude Code pull requests found 83.8 percent were merged and 54.9 percent of those went in without further modification, which is about 46 percent of submitted pull requests landing as written. Across 7,122 pull requests from popular repositories in the AIDev dataset, the five agents merged between 38.2 and 65.3 percent of their pull requests, against 76.8 percent for human authored ones. A rate far below 50 percent usually points at instructions, context, or task selection rather than the model.
How do AI coding tools impact developer productivity?
It depends on the system around them. DORA 2025 found that AI adoption is linked to higher individual effectiveness, code quality, and software delivery throughput, but also to higher delivery instability, with no relationship to friction or burnout. A METR trial using early 2025 tools found experienced open source developers took 19 percent longer with AI while believing they were faster. METR now says newer tools likely help more, but the gap between what developers felt and what was measured is the reason to track delivery outcomes.
Should coding agent metrics be tracked per developer?
No. Thoughtworks recommends tracking collaboration quality at team level alongside DORA metrics, and DORA calls individual token scores harmful. Per person tracking invites gaming and discourages people from sharing effective workflows.
Sources
Radar positions and descriptions come from the Thoughtworks Technology Radar entries for measuring collaboration quality with coding agents, DORA metrics, coding throughput as a measure of productivity, feedback sensors for coding agents, and sandboxed execution for coding agents. Survey findings come from the DORA 2025 State of AI assisted Software Development report, and the tokenmaxxing analysis from the DORA insight Finding balance in the era of tokenmaxxing, published on 2 June 2026. The METR findings come from its early 2025 developer productivity study and its February 2026 update. DORA metric names follow the DORA history of its software delivery metrics, and tool data follows the GitHub Copilot usage metrics and Claude Code analytics documentation. The benchmark figures for agent written pull requests come from AIDev, published 20 July 2025, and the empirical study of agentic coding pull requests, published 18 September 2025. Code churn comes from GitClear, June 2026, and the cost per task figures from the Faros Route Index, June 2026. All were read on 22 and 23 September 2026.
