Professional AI Productivity Workshops for Your Team

Why you cannot tell whether your AI rollout worked

Developers in METR's randomized trial believed AI had made them 20% faster while they were measurably running 19% slower. That is a gap of roughly 39 points between what the people doing the work reported and what the clock recorded. The instrument most engineering leaders reach for when the CFO asks whether the tools paid off - asking the team - is the one instrument with a documented error that large.

The study was narrow, and its limits matter more than its headline. Sixteen experienced open-source developers worked 246 real issues on mature repositories they had known for around five years, using Cursor Pro. That is the hardest available condition for an AI coding assistant, and METR, an independent non-profit AI evaluations organization rather than a vendor, has since said the result is out of date.

What survives is the gap itself, and it deserves a name - the perception gap. Around 90% of software professionals now use AI and more than 80% believe it has increased their productivity, while 30% report little or no trust in the code it produces (DORA, 2025). Most engineering teams are running on belief and reporting it as measurement.

TL;DR

• Stop reporting every metric whose numerator is code produced - lines, commits, PR count, story points, acceptance rate. AI inflates all five.
• Don't survey the team and call the answer evidence. METR found people overestimate AI's effect on task time by 40 percentage points.
• Expect no general answer. Findings run from 26.08% more tasks completed to 19% more time per issue, and METR now flags its 2025 headline as out of date.
• Report a stability counter-metric beside every speed metric. DORA's 2025 research associates AI adoption with higher software delivery throughput and lower stability.
• Capture a baseline before rollout. If you already rolled out, same-engineer comparison against git history is the strongest move left.

Which developer productivity metrics survive contact with AI

Every metric whose numerator is "code produced" dies the day an AI coding assistant lands, because AI inflates output counts without anyone deciding to cheat. What holds is change failure rate, delivery stability, review cycle time, percentage of time on new capabilities, and same-engineer throughput against a pre-AI baseline. Sorting a dashboard into the two piles costs nothing.

Here is the split.

Report the metrics in the bottom half and retire the ones in the top, because a number that rises on its own is not evidence. Measuring developer productivity was never a single-metric job - the SPACE framework made that argument before AI arrived: never measure productivity with a single metric, and never with activity alone (Forsgren, Storey et al., ACM Queue, 2021).

DORA's four keys and DX Core 4 both build on it, and all three balance speed against quality rather than reporting either on its own.

Any metric whose numerator is code produced dies the day the tool lands

Goodhart's law explains the whole split. When a measure becomes a target, it ceases to be a good measure. The usual worry is gaming, where an engineer notices that commits are being counted and starts committing more.

AI removes the decision from the equation. Nobody has to choose to inflate lines of code when a model produces them by the hundred, and commit counts, PR counts, and story points all rise as a side effect of switching the tool on. They still move, so they still look like evidence.

Change failure rate holds because you cannot game it by producing more code

Change failure rate is the one number an AI coding assistant cannot inflate, which is precisely why it belongs on the CFO slide next to any speed claim. You cannot improve your failure rate by writing more code. If anything, more code is how you make it worse.

That is not a hypothetical. GitClear's analysis of 211 million changed lines found cloned lines rising from 8.3% of changed code in 2021 to 12.3% in 2024, while refactored lines fell from 25% to under 10%. GitClear is a commercial code-analytics firm, and 2024 was the first year on record where copy-paste exceeded moved code.

That correlation is industry-wide rather than a controlled attribution, so it does not prove AI caused the shift. It does tell you what to watch. If velocity is up and code churn is rising while refactoring collapses, part of the "gain" is technical debt borrowed against next year's maintenance budget, and a leader who reports velocity without duplication is handing their CFO half a P\&L.

What the research actually says about AI and developer productivity

The research does not agree with itself, and the disagreement is not noise. Published findings run from 55.8% less time on a greenfield task, to 26.08% more tasks completed across 4,867 developers, to roughly 21% less time at Google, to 19% more time per issue on mature codebases. Those studies measure different constructs with different seniority mixes, which is why they can all be right at once.

That last column is what turns a list of numbers into a method for reading numbers.

The studies disagree because they are not measuring the same thing

The studies disagree because they measure different constructs, on different codebases, with different seniority mixes. Peng measured time on one greenfield JavaScript task. METR measured time per issue on mature repositories the developers had known for about five years.

Cui measured tasks completed, a throughput construct rather than a speed one. Ask which of those most resembles your team's work and the range stops being confusing. Greenfield work in a familiar language behaves like Peng; a decade-old service behaves like METR.

Seniority is the finding most rollouts ignore. Gains concentrated in the less experienced engineers, who both adopted the tool more and got more from it (Cui et al., 2025, Management Science). Which makes "the seniors will work it out themselves" the least supported assumption in a typical enterprise rollout.

The largest measured effect in this research is not a productivity effect

The largest measured effect in this body of research is a perception effect, and nobody in this market has named it. METR's developers believed AI had made them 20% faster while measurably running 19% slower - a gap of roughly 39 points, in the direction of flattery.

It gets worse when you ask people directly. A METR survey of 349 technical workers returned a median self-reported speed gain of 3x, and METR warn that survey results are "not necessarily grounded in reality," citing their own finding that people overestimated AI's effect on their task time by 40 percentage points on average.

So the instrument most leaders reach for is the one instrument with a documented 40-point error. That does not make surveys worthless, because developer experience is real and worth measuring. It makes them the wrong instrument for the one question the CFO is actually asking.

The best-funded independent evaluator in the field cannot currently give a clean answer

METR's own February 2026 update states plainly that the 2025 results are out of date, which matters because that -19% is still quoted across this market as a live fact. Their newer data puts returning developers at -18% (confidence interval -38% to +9%) and newly recruited developers at -4% (CI -15% to +9%), and METR judges this an unreliable signal of the true current effect.

The reason is the most honest sentence in software engineering right now. Developers increasingly refuse to participate in a study that requires them to work without AI, so METR is changing its experiment design. The best-resourced independent evaluator in the field cannot cleanly run the study, because the subjects won't give up the tool.

If a randomized trial run by specialists cannot answer this from the outside, a VP Engineering will not answer it from a throughput chart. The answer has to be constructed locally, on purpose, by someone who has done it before.

How to baseline an AI rollout, and what to do if you already missed it

To baseline an AI rollout, decide what "faster" means before you look at any data - then capture the pre-period, write down what would falsify your claim, and keep a comparison cohort. Measuring developer productivity under AI is a design problem before it is a tooling problem, and four of these five steps become impossible the moment the tool goes live.

1. Decide what "faster" means for this team before you look at any data. A metric chosen after the fact is a metric chosen to flatter a decision that was already made.
2. Capture four to six weeks of one speed metric and one stability counter-metric. Review cycle time or lead time for changes, paired with change failure rate, is the default.
3. Write down what result would have falsified the claim. If nothing could have, you are not running a measurement.
4. Roll out to one team and hold a comparison cohort back. Staggered adoption gives you a control group for free.
5. Compare each engineer against their own pre-AI baseline rather than the team average. DX, which sells measurement tooling, calls this same-engineer analysis, and the method is genuinely good.

Step one determines everything downstream, and no dashboard makes it for you. Choosing between review cycle time and deploy frequency is a judgment about what your organization is actually short of, and getting it wrong wastes two quarters. It is worth an hour with someone who has designed this measurement before - an internal staff engineer with the scars, or an agentic coding mentor who has run the rollout elsewhere.

Measured gains are achievable, which is worth saying because the honest picture above can read as nihilism. A 12-month deployment across 300 engineers recorded a 31.8% reduction in PR review cycle time (Kumar et al., 2025). It was observational rather than randomized and the authors were evaluating their own platform, so treat it as an existence proof, not a forecast.

If you already rolled out and captured nothing, two moves are still open to you

Two retrospective moves are still open, and most engineering leaders need them, because most already rolled out and captured nothing. The first is same-engineer retrospective comparison. Source control is a partial baseline you already own, so cycle time, review latency, and change failure rate are all reconstructable from git history and the incident log.

The second is cohort comparison. Teams that adopted late are a usable control group against teams that adopted early, provided you can argue the two groups were comparable to begin with.

Then the honest limit. The perceptual baseline is gone for good, because you cannot ask someone in 2026 how they felt about their workflow in 2025. METR's research is the reason that is the measurement you will most wish you had taken.

What a defensible AI training ROI claim looks like to a CFO

A defensible AI ROI claim has five parts, and a number that survives challenge is worth more in that meeting than a big number that doesn't. Most engineering leaders walk in with a chart. What a CFO is actually testing is whether the chart would have looked different if the investment had failed.

Build the claim like this.

• Name one speed metric and commit to it in advance. Review cycle time and PR throughput are the two most defensible, because both are reconstructable from systems you already run.
• State the pre-period explicitly. "The eight weeks before rollout" is a claim. "Before AI" is a vibe.
• State the comparison method. Same-engineer, cohort, or before-and-after, and say which confounds each one leaves standing.
• Report a stability counter-metric next to the speed number. Change failure rate beside deploy frequency, every single time, without being asked.
• Write down what would have falsified the claim before you present it. A number nobody could have disproven is decoration.

Then calibrate the expectation before the meeting rather than during it. Most organizations should expect single digits to low teens on PR throughput. DX, which sells measurement tooling, reports a median 7.76% PR throughput increase across more than 400 companies where AI usage rose 65%, and a typical range of 5% to 15%.

That is vendor first-party data about the category DX sells into, not neutral research. It is still the best calibration anchor available, because time savings are the easiest thing to overstate and the hardest to prove.

Do the arithmetic honestly and it holds up fine. A 10% throughput gain across a large engineering org is real money, and it is defensible. A 10x gain is not on offer, and claiming one in front of a CFO is how an engineering leader loses the room for a year.

The cost side is where an AI training ROI case usually falls apart, and for a strange reason. Not one of the platforms offering to bring rigor to your ROI will tell you what they cost without a sales call. MentorCruise publishes its prices - from $250 for a two-hour session - which makes the line item easy to defend against a six-figure annual seat bill.

If the real question turns out to be org design rather than tooling, that is a job for an engineering management mentor or an AI strategy workshop for leaders.

Why training changes the number, and why "they'll figure it out" is a bet

"They'll figure the tools out themselves" is a bet, and DORA's 2025 research tells you exactly what you are betting on. The finding is worth quoting in full: "AI doesn't fix a team; it amplifies what's already there. Strong teams use AI to become even better and more efficient. Struggling teams will find that AI only highlights and intensifies their existing problems."

An amplifier is not a neutral object. Point it at strong review practice and fast feedback loops and you get one outcome; point it at weak testing and slow reviews and you get a louder version of that. Doing nothing is still a decision about where the amplifier points, made by default rather than on purpose.

The seniority evidence sharpens it. Gains concentrated in the less experienced developers across Cui et al.'s 4,867-person sample, which means the group most rollouts assume will self-serve is the group getting least from the tool.

Here is what MentorCruise will not claim. A MentorCruise workshop does not produce a specific productivity lift, because nobody has evidence for a claim like that, including MentorCruise - which is rather the point of everything above. The defensible claim is narrower - an expert who has designed this measurement before will help an engineering team pick the metric that matters, build a baseline that survives challenge, and read the result honestly.

Which is why the person in the room matters more than the format. MentorCruise accepts 8% of applicants as workshop hosts. Ask any provider for their acceptance rate - most cannot give you one, which tells you the screening is a conversation rather than a standard.

And the 97% satisfaction rate quoted on this page measures the session, not the delivery outcome. That distinction is the argument of this entire page, and it applies to MentorCruise too.

Match the format to the job honestly.

• A two-hour Fundamentals session is enough to design the measurement and pick the metrics.
• Building the baseline, the cohort split, and the rollout comparison is a Deep Dive's worth of work.
• A Bootcamp is where a larger team also works through what the measurement exposes.

If the measurement says the tools are fine and the practice around them is not, the onward path runs through a Cursor workshop for teams or a Claude Code workshop for teams. If the gap turns out to be systemic, it runs through AI transformation for teams.

Frequently asked questions

Should we just ask the team whether AI made them faster?

No, not as your only instrument. Self-report is a good instrument for developer experience and a documented failure for speed. METR's May 2026 survey of 349 technical workers found a median self-reported 3x speed gain, and METR warn the results are "not necessarily grounded in reality." Use surveys for how the work feels, never for how fast it is.

We already rolled out AI tools and never captured a baseline. Can we still measure the impact?

Partly, yes. Your repository already holds the record - review latency, cycle time, and change failure rate can all be rebuilt from version control and the incident log. Compare each person against their own pre-AI history rather than a team average. What cannot be rebuilt is the perceptual baseline: nobody in 2026 can reliably tell you how their 2025 workflow felt.

How do I justify AI tooling spend to my CFO?

Set the range before the meeting rather than during it. A number that survives challenge beats a big number, and the honest expectation is single digits to low teens on PR throughput. Ten percent across a large engineering org is a number a CFO can bank, and it will hold up under questioning. Walking in implying 10x will not, and the credibility you spend there is expensive to earn back.

Is it worth training the team, or will they figure the tools out themselves?

It depends on who you mean. The assumption that senior engineers will self-serve is the least supported assumption in a typical rollout, because Cui et al. (2025) found the gains concentrated in less experienced developers. Which makes the seniors the weaker bet. MentorCruise screens workshop hosts to an 8% acceptance rate, which is worth checking against any provider you consider.

Do we need an engineering intelligence platform to measure AI's impact?

No, not to start. You need a named metric, a stated pre-period, a comparison method, and a stability counter-metric reported alongside the speed number. A platform automates collection, and that is worth paying for at scale. It does not judge what is worth collecting, and it cannot reconstruct a baseline you never captured. Worth noting that every commercial answer to this question comes from someone selling the instrument.