In a Hacker News thread this October about whether anyone is producing good code with coding agents, two replies in the same thread said almost opposite things. One engineer wrote that "the LLM-generated tests are pretty bad, even the frontier models on xhigh thinking". Another recommended the reverse: Claude is "really really good at adding tests", so use it to get near 100% test coverage.
I think both are right, and the gap between them is where a lot of damage actually happens! Agents are good at producing tests. They are much less reliable at producing tests that would catch an actual mistake, so a suite can be big, fast and green and still tell you very little about whether the code does what you needed.
Most senior engineers already know the textbook answers here, so I will keep those short. What I want to describe are three places where I found agent written tests failing in ways the textbook does not really warn you about, because they only show up when the same system writes the code, the tests and the test data.
The agent agrees with itself
You ask an agent to implement a feature and add tests. It reads your request, forms an understanding of it, writes the code from that understanding, and then writes the tests from the same understanding.
If the understanding is right, fine. If it misread something, say it decided an empty list should return zero when you needed an error, then the code returns zero and the test checks that it returns zero. They agree with each other, both are wrong, and the suite is green. The tests were written to describe the code, not your requirement.
A person who writes code first and tests second can do the same thing, so this is not new. What is new is the speed and the volume, and the fact that the agent does it with a confident summary at the end.
I spent over six years at PTC on Vuforia Engine working on software verification, the Benchmarking SDK test suite. Before that, my PhD at UCL was in medical imaging and computer vision, where statistics and experimental design were a big part of the work. Both taught me to ask, when a result looks good, whether the system actually improved or whether the way we measured it changed. When the test comes from the same belief as the code, it is the second case, and nothing in the green suite tells you so.
Keep the generator away from the fixtures
This is the one that took me longest to see in my own work, and I think it is the most important.
Agents are very helpful at producing test data. Ask for sample inputs, expected outputs, or a list of good and bad cases, and you get a neat file in seconds. The problem starts when the tests then use those same examples as fixtures. Whatever the agent believes about the problem is now baked into the thing being tested and into the thing doing the testing. The suite passes, and it would keep passing on code that anyone looking at real data would call broken.
It is easy to miss because it looks like good practice. There are fixtures, edge cases, a nice table of expected values. But every row came from the same source as the code, so the table cannot disagree with the code about anything that matters.
Soo, the rule I now follow is that examples a generator produced may never be the fixtures its output is judged against. Fixtures come from real cases. For example, a bug report, a production log, a real input a user sent, or cases a person wrote by hand from the requirement. The agent can help format them and add the boring variations. It should not be the one deciding what the right answer is.
I gave a talk at TestIstanbul this October called "What If We Tested the Tester?", and this was really the heart of it. Every check you add is itself a claim that can be wrong, so it should be tested against examples where you already know the right answer from somewhere else.
The agent edits the test to make it pass
The second failure is more mundane, common, and it is easy to miss in a long session, so I now check for it every time. You write a test first, it fails, and you ask the agent to make it pass. Somewhere in a long session it decides the fastest route to green is to adjust the test!! Wait, what? Loosen an assertion, change an expected value, mark something as skipped, or mock the part that was failing. The summary then says all tests pass, which is true. assert True==True. How funny is this. It is also True :)
The fix that works for me is to keep the two jobs in separate steps. In the first step, the tests are written from the requirement and I see them fail. In the second step, the agent implements, and I tell it the test files are read-only. Afterwards I do not take its word for that. I look at the diff of the test files, which on most repositories is one git diff --stat on the tests folder. If a test file changed during implementation, that change gets reviewed as carefully as the feature, because it usually is the feature in disguise.
Break it on purpose
Tests first need to fail. The red step proves a test can fail once. It does not show the suite would catch the next mistake. For code that matters, I do a hand-made version of mutation testing: in a separate session, I ask the agent to plant a small, plausible bug, like flipping a comparison, dropping an edge case or returning early, and then I run the suite. If it stays green, the suite does not cover that behaviour.
This is also the clearest way I know to read a coverage number. Coverage tells you which lines ran during the tests, not which behaviour was checked. An agent can get very high coverage with tests that call every function and assert very little, so "near 100% coverage" and "pretty bad tests" can describe the same suite. Planting three bugs and seeing which ones survive tells you more than the percentage does.
Every bug gets a test before it gets a fix
When something does slip through, the temptation is to paste the error into the agent and ask it to fix it. It will, usually quickly, and nothing stops the same kind of bug from coming back next month. So my first instruction is to write a test that reproduces the bug and show me it failing, and only then to fix it. Over a few months this builds a suite made of your real failures, which is far more useful than one made of the agent's guesses about what might go wrong. It also produces exactly the kind of real fixtures the earlier section asks for.
Where your time goes as the senior person
With all of this in place, the agent still writes most of the test code! What changes is the part you own. Guess what?
Exactly, turning the requirement into checks, choosing which edge cases are real, and deciding where the fixtures come from. That is exactly the step where an agent's misunderstanding gets in, so I think it is the best use of a senior engineer's attention.
A rough checklist I use:
- Fixtures come from real cases or from a person, never from examples the same agent generated.
- Tests are written and seen failing in one step; implementation happens in another, with the tests read-only.
- After implementation, the diff of the test files is checked, and any change is reviewed like a feature.
- Every red is read, and only a failure for the right reason counts.
- For important code, a planted bug turns the suite red.
- Every bug gets a reproducing test before a fix.
If you want to set this up on your own code
On MentorCruise I help engineers and tech leads bring Claude Code and Cursor into real codebases with test-first workflows and output you can actually review. If you bring one feature from your own repository to the first call, we can plant a few bugs in your current suite and see which ones it misses, and then set up the red-first loop on that feature together, so you leave with something you can keep using on your own.
You can find me here: Dr. Marcel Tella Amo on MentorCruise.