Test the Harness, Not the Agent
Explore how to design and test harness controls for coding agents to ensure boundaries and verification gates function correctly. Learn to invert traditional testing by treating the agent as a fixture and focusing on the harness components. Understand methods to create reproducible, deterministic tests that run without live models, improving reliability across sessions. This lesson helps you build confidence in your testing framework to catch errors effectively and maintain system integrity.
We'll cover the following...
- Why didn’t anyone notice a dead gate?
- What is actually under test here?
- What does one harness test look like?
- Why assert the code and not just the refusal?
- Where do the fixtures come from?
- Which tests have to stay green?
- What should a harness test never do?
- How do you find the gates that no test protects?
- Can you find the dead gates in Wrenfold?
- What can you add to your own harness today?
- What’s next?
Wrenfold’s harness had grown into real machinery. A Scope Contract with a diff check, a feature gate with typed refusals, an authority policy, a claim-to-proof matrix, and a verification route that replayed the concurrent booking race.
Then session forty-one shipped an out-of-scope rewrite of the patient route inside a CLIN-547 waitlist ticket. The diff check had run. It had printed SURFACE OK.
The check ran git diff --name-only with no revision argument. Every edit the agent made was already staged, so the command returned nothing, so the comparison found no path outside the expected surface. It had returned SURFACE OK for eleven consecutive sessions, including four that drifted.
Nobody noticed, because a gate that never fires looks exactly like a gate with nothing to catch. Both print green.
Why didn’t anyone notice a dead gate?
Because green is two different messages wearing the same color.
“I checked, and the work was clean” and “I am incapable of finding anything” produce identical output. The only way to tell them apart is to hand the gate something it should reject and confirm it does.
The booking code has 14 unit tests and an end-to-end route. The gates that judge it had none. They were written once, watched by hand, and trusted for eleven sessions.
This is the reliability side of the course’s line pointed at the harness itself. The model was capable, the contracts were correct on paper, and one line of shell in one gate quietly removed the whole scope boundary.
What is actually under test here?
Invert the usual arrangement. In a harness test, the agent is not the subject. It is part of the fixture.
The system under test is the control: diff check, transition guard, coverage rule, authority function, or exit ...