How to Tell Whether AI Tooling Actually Made Your Team Faster

You're probably here because your throughput chart went up and you don't quite believe it. Good instinct. This is a method for finding out.

The short version

• Stop measuring the stage AI touched. Writing code got cheap. Reading it costs exactly what it did in 2019. Measure the whole interval from first commit to deploy, or you're measuring your tooling rather than your team.
• Split that interval into four: authoring, queue wait, review and rework, release. AI compresses the first one. If the total didn't fall, the other three tell you where the time went instead. Every timestamp you need is already in your pull request data.
• Put a stability number next to it, always. Across nearly 5,000 technology professionals, DORA's 2025 report found AI adoption has a positive relationship with delivery throughput and a negative one with delivery stability. One of those numbers without the other is half an answer.
• Baseline backwards. You didn't measure before you rolled out and it doesn't matter, because Git recorded everything before you decided anything. Two windows of 8 to 12 weeks, medians rather than means.
• Don't ask your engineers, and don't expect a causal number. Self-report is the one measurement here that has been tested against a clock and lost. You can't run your team twice, so what you get is a direction and a location. That is still the answer to the question you're asking.

Why your throughput went up no matter what happened

Your throughput went up because AI made the cheap half of the work cheaper, and that was going to land on your chart whatever else was true. Generation is close to free now. Verification costs what it always did, because it runs on senior attention, and senior attention doesn't ship a new version every six weeks. Measure the half that got free and you'll find a win every single time, including the times there wasn't one.

A change doesn't reach production by being written. It gets written, then it waits, then somebody reads it, then it argues with that person for a while, then it ships. Speed at the end of that line is set by the slowest stage in it. AI is extremely good at the first stage and holds no opinion whatsoever about the other four. So "do my engineers produce more" isn't the question. They do, and you already know they do. The question is whether any of that production reached your users, or whether it's stacked up in front of the one stage that runs on people.

There's a second reason to distrust the chart, and it predates all of this. Charles Goodhart wrote it down in 1975, in a paper on UK monetary policy: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The punchier version everyone quotes, that when a measure becomes a target it ceases to be a good measure, is actually Marilyn Strathern's, from 1997. Goodhart's own wording is the one you want, because "pressure placed upon it for control purposes" describes exactly what your board did to your engineering metrics this year.

Somebody has now run that experiment at scale, though not deliberately. In a paper posted this month, Hao He and five co-authors tracked 802 developers and 196,212 pull requests between January 2024 and April 2026, at a company that had committed to doubling merged pull requests per engineer. The mandate worked. Per-capita throughput reached 2.09 times the pre-mandate baseline by April 2026, which the authors call "among the largest gains reported from a field deployment of AI coding tools to our knowledge."

Now read the clause that arrives fourteen words later in the same abstract: "per-reviewer load roughly doubled and automated review overtook human review, while merge and revert rates held steady."

The target was merged pull requests per engineer. They got merged pull requests per engineer. What they also got was double the reading, most of which stopped being done by humans. The authors gave the paper the bluntest title in this literature: AI Writes Faster Than Humans Can Review. Their throughput chart isn't lying, exactly. It's answering a question nobody wanted answered, which is whether a number moves when you push on it.

DORA's 2025 report shows the same shape across nearly 5,000 technology professionals rather than at one company: AI adoption has a positive relationship with software delivery throughput and a negative relationship with software delivery stability. Two directions at once, out of one variable. That's why the method below asks two questions instead of one.

Which of your metrics survive AI, and which ones invert

Most of your metrics survive AI unharmed. Three of them invert, meaning they now move in the opposite direction to the thing you wanted them to track, and a fourth has been falsified outright. Sort them before you measure anything, because a dashboard carrying an inverted metric is worse than no dashboard. It produces confident decisions pointing the wrong way.

Change lead time is the row to get right, because it's the one that misleads you while looking healthy. A team whose authoring interval halved and whose review queue doubled shows a flat lead time, and flat reads as "nothing happened." Something enormous happened. The work moved from a stage that scales with tooling to a stage that scales with headcount, and your chart is hiding that behind an average. Hence the split, and hence the split being the deliverable rather than a dashboard.

Self-report is the row to cut first, and the one everyone reaches for, because a survey takes ten minutes and a query takes an afternoon. I'll show you how badly it fails further down. If you want the individual-level discipline your engineers should be running before their output ever reaches your queue, the validation playbook is written for them. This post is the layer above it.

The method, on data you already have

To find out whether AI made your team faster, split your change lead time into four intervals, rebuild a baseline out of Git history you already own, and put your change fail rate next to the result. It takes an afternoon. It needs no new tooling, no vendor, and no budget line, because every number in it is already sitting in your pull request data and your deploy log.

Step 0, check that anything happened at all

Compute the time from a branch's first commit to its pull request opening, for the last three months and for a quarter well before you rolled anything out. If that interval didn't fall, stop here. You don't have a measurement problem. You have an adoption problem, and nothing downstream will mean anything until it's fixed.

This step exists because your engineers may report heavy AI use while the data flatly disagrees with them, and DX's caveat cuts both ways: overcounting "when developers treat AI use as a performance signal," which is a careful way of saying people tell their manager what pays, and undercounting from workflows so automated nobody registers them as AI use any more. You can't fix that by asking harder. You fix it by looking at the clock on the branch.

Step 1, rebuild the baseline backwards

You didn't baseline before you rolled out. Almost nobody did, and every guide that opens with "establish a baseline first" is worthless to you now. It doesn't matter. Git recorded everything before you decided anything, so your baseline already exists and you just have to go and collect it.

Pick two windows of 8 to 12 weeks. One well before adoption, one well after it settled. Leave a gap between them, deliberately: the transition period is the noisiest data you own, and you can't sharpen its boundary anyway, because the only record of when your team started using AI is people's memory of when they started using AI.

Then check what else was living in those windows before you trust either one. Did headcount change? Was there a migration, a reorg, an incident that ate a month, or a December? Did the definition of "done" quietly move? If any of that sits inside a window, move the window. This is the step people skip, and it's the step that decides whether the rest is arithmetic or astrology.

Step 2, split the lead time into four intervals

Five timestamps, and you already have all five. GitHub's pull request API exposes the last four, GitLab has equivalents, and the first is in git log.

A plus B plus C plus D is change lead time, which DORA defines as "the amount of time it takes for a change to go from committed to version control to deployed in production." So this is the standard metric, cut at its joints, because the composite is precisely where the finding hides. Four intervals tell you where your time is. One number tells you that you have some.

Step 3, put the stability number next to it

DORA sorts its metrics into throughput and instability, and instability is the half you'll want to skip, because it's the half that can only bring bad news. Change fail rate is "the ratio of deployments that require immediate intervention following a deployment." Deployment rework rate is "the ratio of deployments that are unplanned but happen as a result of an incident in production." If you track neither, revert commits and hotfix deploys are in your Git history and they're close enough to start with.

Compute both over the same two windows as the intervals. Not a different window, and not last quarter because it's tidier. The same two.

Step 4, the rules that keep this honest

Use medians, never means. Pull request durations are heavily skewed, and one PR that sat open for three weeks over Christmas will drag a mean somewhere interesting and false.

If a window holds fewer than about 20 merged pull requests, widen it or don't run the method. Below that, the medians are noise wearing a suit, and you'll read a story into four data points.

Never run this per individual. The SPACE framework, from Nicole Forsgren and five co-authors in ACM Queue in 2021, opens by saying that "developer productivity is about more than an individual's activity levels or the efficiency of the engineering systems relied on to ship software, and it cannot be measured by a single metric or dimension." These intervals are one dimension of one team. Point them at a person and you've built a surveillance tool that measures how big their tickets were.

Never run it against another team. That's DORA's own guidance, and they're firm: "The goal is to improve your team's performance over time, not to compete against other teams or organizations." Your comparison is your own team's past. It's the only control group you'll ever get.

Here's the whole thing on one page.

AI SPEED CHECK - worksheet

WINDOWS
  Baseline    8-12 weeks, ending at least a month before AI adoption started
  Current     8-12 weeks, starting at least a month after adoption settled
  Confounds in either window?  headcount change / migration / reorg /
                               incident / December / "done" redefined
                               -> if yes, move the window
  Fewer than ~20 merged PRs in a window?  -> widen it, or don't run this

TIMESTAMPS  (all five are already in your PR data, git log, and deploy log)
  t0  first commit on the branch
  t1  pull request opened
  t2  first review submitted
  t3  pull request approved
  t4  pull request merged
  t5  deployed

INTERVALS  (medians, not means, for each window)
  A   t0 -> t1   authoring            <- the only stage AI touches
  B   t1 -> t2   queue wait
  C   t2 -> t3   review and rework
  D   t3 -> t5   release friction
  TOTAL  t0 -> t5  = DORA change lead time

STABILITY  (same two windows, no exceptions)
  change fail rate      deploys needing immediate intervention
  rework / revert rate  unplanned deploys caused by a production incident

THE THREE QUESTIONS
  0.  Did A fall?              No -> adoption problem. Stop here.
  1.  Did TOTAL fall?
  2.  Did stability get worse?

  Team level only. Never per person. Never against another team.

Reading your result

Two questions, four answers, and every one is an observation you can make rather than a conclusion you have to argue for. Find the row your numbers landed in.

It worked means the compression in authoring survived all the way to deploy and nothing broke more often. It happens, it's rarer than the case studies suggest, and you should read it once and stop.

The bottleneck moved is the modal result, and the one I'd bet on for most teams reading this. It looks like failure and it isn't. Your engineers did get faster. Your system didn't, because the time they saved is sitting in interval B or C waiting on somebody senior. That's what the 802-developer study watched happen: throughput at 2.09x, per-reviewer load doubled, the reading quietly reassigned to machines. Check whether the growth in B plus C roughly matches the fall in A. If it does, you haven't lost the time. You've relocated it, onto the people who were already your constraint, and nobody decided that should happen.

You got looser is the uncomfortable one. Lead time fell and things break more often. The likely mechanism is that interval C shrank while the volume arriving at it went up, and there aren't many innocent explanations for that pair. The review work didn't disappear. It got skipped. Your speed is real and you borrowed it against an incident that hasn't happened yet.

AI isn't your variable is what a rising total usually means. Something else changed in that window and it's louder than your tooling. Go and find it. Don't credit or blame AI for a number it isn't driving, which is the throughput chart's error pointing the other way.

One honest note on the grid. "Flat" needs a threshold and I'm not going to invent one, because it depends on your volume and your variance and I can't see either. A workable rule: if the change is smaller than the spread between your best and worst month inside the baseline window, treat it as flat. If you can't compute that spread, your window is too small, and step 4 already told you so.

What this method can't tell you

This method gives you a direction and a location. It won't give you a causal number, and nothing else will either, because you cannot run your team twice.

That reads like a hedge. It's the most useful thing in this post, and here's the evidence behind it.

In July 2025, METR published a randomised controlled trial. Sixteen experienced open-source developers, 246 real issues, on repositories those developers already knew well, with AI access randomly assigned. The developers expected AI to make them 24% faster. They came out 19% slower. Afterwards, having just been measured going slower, they estimated AI had sped them up by 20%.

That result went everywhere. It's the number people reach for when they want to argue AI doesn't work, and you've read it quoted this year.

Now go and open the study page. There's a banner across the top, put there by METR: "These results are out of date. We have released results that are current as of early 2026, in a continuation of this study."

The continuation estimates, for the developers who came back, a speedup of -18% with a confidence interval running from -38% to +9%. For newly recruited developers, -4%, from -15% to +9%. Both intervals cross zero. And METR is harder on their own new data than any critic has been: developers were declining tasks they didn't want to attempt without AI, a selection effect that biases the estimate downward, so METR calls it "only very weak evidence." Their current position, from talking to participants, is that "it is likely that developers are more sped up from AI tools now - in early 2026 - compared to our estimates from early 2025."

I'm not telling you this to rescue AI's reputation. I'm telling you because of what it does to yours.

The best-designed study anybody has run on this question, with random assignment, real repositories, and a research organisation that has nothing to sell you, could not pin the effect size, and marked its own headline out of date inside eight months. If that's the ceiling with randomisation, the ceiling for two Git windows and no control group sits a long way underneath it. Anyone who tells you their dashboard proved AI made their team 30% faster is telling you about their dashboard.

So don't ask this method for a causal estimate. Ask it the three things it can answer. Did the total interval move? If it didn't, which interval ate the difference? Did anything break more often? Those are observations, and nobody has to accept your model of why, because you don't have one and you don't need one. The time moved from here to there, and that's on the record.

One finding survives both METR studies completely intact, and it's why step 0 exists. In early 2025 those developers' sense of their own speed pointed the opposite way to the clock. The effect size fell apart under scrutiny. The perception gap didn't. That's why nothing in the method above asks anyone how it's going.

What to do about the answer you got

Three of the four results route to something free, and I'd rather say that here than bury it at the end where it would look like modesty.

If it worked, stop. Close the query, don't build a dashboard, spend the afternoon on something else. A measurement that confirms a decision you already made is worth one run and no more.

If the bottleneck moved, you have a routing problem, and it's the cheapest fix available to you. Your seniors are spending attention uniformly across a queue that isn't uniform. Sort changes by what they can reach rather than by who or what wrote them, and a large share of that queue stops needing a senior at all. That costs a written standard and about a week.

If you got looser, the gate is the problem rather than the queue, and the fix starts with one named human accountable for every shipped change. Also free, also a document.

If AI isn't your variable, go and find what is. It isn't in this post.

And if what you're really deciding is whether to spend money on training, rather than what to do after you've spent it, that's a different question with five real answers, one of which is "don't buy anything."

Here's where a method stops being useful. Measurement tells you where the time went. It can't make the number move. If your four intervals say the time relocated into review, and you then write out the list of people who can actually sign off a change to your auth path, and that list has one name on it, then you have a measurement that worked perfectly and a problem no measurement will ever touch.

That's a capability question with two honest answers. Grow the second name internally, which is cheaper and sticks better and spends a lot of the senior bandwidth you have just finished proving you don't have. Or bring a standard in from outside, in a window you can actually measure.

Only the second one involves buying anything, and my reason for it is smaller than you'd expect from someone selling. Your AI rollout never had a start date. That's why step 1 was hard: you had to guess when it happened, and your team's memory was the only record. An engagement has a start date and an end date, which are the two things step 1 couldn't give you, so the windows stop being a judgment call. MentorCruise runs a 90-day sprint at mentorcruise.com/teams/ai, and we also run leadership workshops. I'm not going to describe either one here. Go and read the pages. This post has spent several thousand words arguing that you should look at a source yourself rather than take somebody's summary of it, and I'm not breaking that in the last paragraph to sell you something.

And if your grid came back "it worked," none of this is for you. Run the method again next year.

Questions you'll get when you show someone the result

Is measuring developer productivity even possible when AI writes half the code?

Yes, but not as one number, and not the way you were doing it. Measuring developer productivity never worked as a single score, and the SPACE framework said so in 2021: it "cannot be measured by a single metric or dimension." What AI changed is that the shortcuts people used, lines of code and story points and merged PR counts, went from weak to actively inverted, because AI made the thing they counted cheap to produce. Measure intervals and stability, at team level, over time, against your own past, and it works fine.

Do DORA metrics still work when AI writes the code?

They work, and change lead time only works if you split it. DORA's throughput metrics and instability metrics both still measure real things, and DORA's 2025 finding is that AI pushes those two groups in opposite directions at once. The trap is reading change lead time as a single number: authoring shrinks, review grows, the total sits flat, and you conclude nothing happened while the shape of your team's work has completely changed underneath it. Split the interval and DORA's definitions do the rest.

We never baselined before rolling out AI. Can we still measure anything?

Yes. Your baseline is in Git and it has been there the whole time. Version control recorded every commit, pull request, and merge from before anyone made a decision about AI, so you can compute the same intervals on a window from last year and compare. The one thing you can't recover is a clean adoption date, because the only record of that is people's memory. Work around it by leaving a wide gap between your two windows and ignoring the transition entirely. The binding constraint on how far back or forward you reach is volume rather than time: you need roughly 20 merged pull requests in a window before the medians mean anything, so a four-person team reaches wider than a twenty-person team.

Should we measure AI's impact per engineer?

No, and this is the one place I'd push back hardest. These intervals are properties of a team's system, and pointed at an individual they mostly measure how big that person's tickets were. SPACE's opening line is that developer productivity is about more than an individual's activity levels. There's a practical reason stacked on top of the principled one: the moment your engineers know their intervals are read individually, Goodhart arrives and the numbers stop describing anything. You'd be destroying your own instrument to answer a question it can't answer.

Is the METR "19% slower" study still valid?

No, and METR say so themselves. The July 2025 page carries their own banner: "These results are out of date. We have released results that are current as of early 2026, in a continuation of this study." The continuation estimates -18% for returning developers and -4% for newly recruited ones, with confidence intervals that both cross zero, and METR flags a selection effect biasing even those downward. It's still quoted as current across the web. If you're about to put it in a deck, either don't, or cite it accurately including the fact that it was superseded, which is a more interesting point about measurement than the original number ever was.