A Run Only Proves What It Exercised
Learn to identify why tests may miss concurrency bugs by understanding that a test run only proves the behavior it exercises. Explore how to design verification routes that cover crucial system boundaries—including concurrency, state, timing, and responses—ensuring AI agent contracts are reliably tested. This lesson guides you in building faithful test harnesses that expose defects hidden by narrow or mock-based tests and manage the complexity of concurrent interactions in multi-session agent tasks.
We'll cover the following...
- Why did the unit suite miss the collision?
- Does “full run” mean running the whole repository?
- What determines whether a run matches the claim?
- What should the verification route record?
- Why must the requests run concurrently?
- When can a dependency remain fake?
- Is a failing full run a failed harness?
- What changes when you build the route yourself?
- What can you add to your harness?
- How do you test the verification route?
- What’s next?
Wrenfold’s recurring-appointment ticket, CLIN-482, had survived every local check. All 14 availability unit tests passed. The route test returned 201. The request created six weekly appointments when executed without concurrent requests.
Then two receptionists submitted booking requests for the same 3:00 PM slot with Dr. Reyes within milliseconds of each other. Both requests returned 201 Created. The database stored both appointments.
The clinic’s core scheduling invariant had failed: no two appointments for the same provider may overlap.
The test report was accurate. Each check had run and passed. None reproduced the concurrent execution pattern that violated the invariant.
Why did the unit suite miss the collision?
The unit suite called overlaps() with a fixed list of existing appointments. It proved that the function recognized an overlap already present in its input.
The production path performed a longer operation:
Accept an HTTP request.
Convert clinic-local time to Coordinated Universal Time (UTC).
Query the shared database for existing appointments.
Decide that the slot is free.
Insert the new appointment.
Return the response.
Two requests can interleave between steps 3 and 5. Request A reads an empty slot. Request B reads the same empty slot before A writes. Both decisions are correct for the snapshots they saw, and both inserts violate the invariant together.
The bug lives in timing and shared state. A ...