Coding Agents Need Experiments. But Should We Call Them Tests?

Coding agents write tests. Lots of them.

And that's not necessarily a bad thing.

They need ways to execute code, experiment with different approaches, challenge assumptions, reproduce unexpected behavior, and validate their own work.

That's exactly what we want them to do.

But are all those files really tests?

Sometimes an agent creates a PHPUnit test simply because it needs to understand how something behaves.

Maybe it's investigating an edge case, checking an assumption, or figuring out why an implementation doesn't work.

The file might deliberately fail. It might contain an incorrect assumption. It might only exist to answer a single question.

That's perfectly reasonable during development.

But somewhere along the way, we start treating these experiments as permanent regression tests.

They end up in our test suites, run in CI, and become maintenance obligations.

Maybe the problem isn't that coding agents write too many tests. Maybe we're calling too many things tests in the first place.

But Should We Just Delete Them?

Not necessarily.

These files can also tell us something interesting about how an agent worked.

What did it actually check? Which assumptions did it challenge? What behavior did it observe? What did it miss?

And why did it choose one implementation over another?

Of course, an executable file alone doesn't explain every decision. But together with its execution results and context, it might provide valuable evidence.

Even when it doesn't deserve to become a permanent regression test.

An independent reviewer, or another coding agent working on the same problem later, might benefit from that information.

So perhaps we should distinguish between:

• Experiments that help agents explore and understand behavior.
• Regression tests that protect established requirements and invariants.
• Evidence that preserves useful observations, including failures and unresolved questions.

These aren't necessarily different frameworks or even different file formats.

They're different purposes.

And they probably shouldn't all have the same lifetime.

What Should We Actually Call These Files?

Evidence files? Experiments? Validation files? Executable investigations?

I'm not convinced we've found the right terminology yet.

And I think the terminology matters more than it might seem.

If we tell a coding agent to write tests, it will probably create something that looks like a test and treat it accordingly.

But what if we explicitly encouraged agents to experiment, investigate, execute, observe, and learn, without implying that every resulting file belongs in our permanent test suite?

Could that change how agents approach development?

Could preserving some of these experiments help future agents understand previous decisions, avoid repeating mistakes, or challenge assumptions we didn't even realize we had?

And how would we distinguish valuable evidence from disposable experiments without creating another mountain of artifacts nobody maintains?

I don't think we need a new framework to explore this.

But perhaps we need a better name and a clearer understanding of what these files are actually for.

So what should we call them, and how do we teach future coding agents to use them correctly?