A 26-condition coding-agent test found that naming a technique rarely fixes the tests
Telling a coding agent to use test-driven development, fuzzing, Lean 4, or a property-testing library sounds like a sensible way to improve a risky implementation. A recent evaluation suggests the instruction alone often does not supply the judgment that makes those methods useful.
Dan Luu asked agents to implement Zstandard in Rust, then varied the added testing instruction. The experiment covered 26 conditions, including TDD, fuzzing, differential testing, property-based testing, QuickCheck, Proptest, Kani, Lean 4, TLA+, SMT solvers, and a default condition with no extra instruction. It also tried four skills. Each condition and effort level was averaged over 80 runs, and success meant passing every hidden test. The full evaluation reports what the agents actually did, not only a leaderboard.
No condition clearly ran away from the pack. The default condition performed well above average. At xhigh effort, fuzzing and property-based-testing related conditions did a little better on average than formal methods, while results at medium effort were more mixed. The recommended testing skills underperformed in this setup, and a short custom skill did reasonably well. That is evidence about one evaluation, not a verdict on TDD, formal verification, or any individual model.
The label and the practice drifted apart
Zstandard is a useful target for this sort of work. Its format includes entropy coding, bit-level operations, frames, and edge conditions. The Zstandard project describes it as a compression algorithm intended to combine high ratios with fast decompression. A correct implementation has plenty of places where a plausible-looking test can miss a real error.
In the evaluation, agents often followed the name of a method without applying the judgment that gives that method value. A property-based test can explore generated inputs and assert a relationship that must hold. Instead, many agents generated fully random bytes that travelled through a small set of invalid-input or rejection paths. Some checked a trivial property and then returned to conventional examples.
Formal methods had a related failure. Agents frequently proved facts that did not constrain the implementation's bug-prone parts. Differential testing rarely produced two independent implementations to compare. One error described in the report involved a byte-order reversal hidden by a palindromic test input: reversing the input did not change the test case, so the test passed for the wrong implementation.
In many cases, the tool ran, emitted output, and left consequential behavior unexamined. A green check can describe a successful command rather than a successful test strategy.
More tests did not mean more coverage of danger
The condition that explicitly asked for Rust's built-in test framework made agents write more tests: about twice the normal number at medium effort and 25% more at xhigh, according to the evaluation. It did not produce better correctness. The added tests often skipped important behavior or encoded incorrect behavior themselves.
That fits a familiar review problem. A test suite can be large, tidy, and fast while leaving the risky boundary untouched. With agents, the risk is amplified because they can produce that appearance cheaply. A patch may arrive with dozens of tests, a named framework, and an impressive activity log. None of those tell a reviewer whether the test distinguishes the correct program from a nearby wrong one.
Property testing had one partial bright spot. The report says Proptest's shrinking sometimes helped: when a generated case failed, the library could reduce it to a simpler failing input. But the overall use was still limited by weak generators and weak properties. Proptest's documentation explains the distinction: it generates inputs and shrinks failures, while the test author still needs to define what should remain true.
For a parser, useful properties might include round trips on structured valid inputs, agreement with a known decoder, or expected behavior around an exact length boundary. Random malformed byte strings rarely establish any of those.
Ask for discriminating checks
A better prompt asks the agent to expose a particular way the implementation could be wrong. It makes the expected contrast concrete.
For example, instead of "use property-based testing," give it the contract and the asymmetry:
For each risky parser field, write at least one valid structured input on each
side of its length or value boundary. Make sure changing byte order, field
position, or one boundary bit changes the expected result. Do not use the
production parser to calculate expected outputs. Report which bad
implementation each test would reject.
That prompt does not guarantee a good test. It does make a reviewer-facing claim that can be checked. The agent must name the alternative mistake, and the test must separate it from the intended behavior.
The experiment's custom skill moved in this direction. It asked agents to identify likely subtle bugs, choose asymmetric boundary examples, use structured random inputs, and independently re-derive high-risk logic. It achieved the highest score in that evaluation, though the author stresses that agents rarely followed the fresh-context step and that the skill needs more testing before anyone should treat it as a finished recipe.
Put the verifier outside the agent's favorite path
Keep expected-output generation separate from production helpers. Use a reference implementation when one exists. Feed structured cases through a public API rather than testing a private helper that shares the same assumptions. For distributed or concurrent work, assert observable invariants after failure and recovery, not only that a local function returned.
Formal tools can help when they are connected to executable code and a relevant property. Kani is a Rust verifier that checks assertions under bounded exploration. The evaluation found one real Kani use that caught a non-trivial bug, alongside many superficial uses. A tool can be valuable while an autonomous agent remains unreliable at selecting and framing the proof obligation.
This is where an AI coding workflow benefits from a review loop. Ask the agent to state the three likeliest semantic mistakes before implementation. After it writes a test, ask a separate pass to propose an implementation that would pass the test but violate the contract. If it can do that, the test needs work. A small, explicit set of rejection cases is more informative than a long list of test names.
Treat technique prompts as hypotheses
The evaluation should not be stretched into a universal ranking. It used a Rust Zstandard implementation task, specific agents and effort settings, hidden tests, and a limited collection of prompts and skills. Its results do not predict every web application, security review, or agent harness.
It does support one disciplined change in practice: a request such as "use TDD" is an input to the process, not evidence that TDD happened well. The evidence is the test's ability to reject a realistic wrong behavior. Require agents to show that evidence, keep independent checks in the pipeline, and use a human or a separate verifier where the cost of a mistaken green result is high.
