Build the bug repro from the real failing case
A ticket comes in. You hand it to the agent, it writes a failing test, changes the code, the test goes green, and the PR merges. A week later the ticket gets reopened with the same screenshot.
When I’ve traced one of these back, the agent had made up the test inputs. It read the ticket, guessed at the bug, and wrote a test for its guess. The test went red, then green, but it was testing a case that wasn’t the one in the ticket.
So I make the agent do four things, in order, before it’s allowed to call a bug diagnosed. One made-up example runs through all of them.
The made-up example
A booking system takes group bookings and sometimes gives a group discount (the numbers are made up). A customer writes in: “We cancelled our whole group booking. We were charged 35.20 and refunded 35.19.” Single bookings refund fine. Groups without a discount refund fine. Hand that ticket to an agent cold and it’ll often write something like this:
it('refunds a discounted group booking in full', async () => {
const booking = await makeGroupBooking({ people: 4, pricePerPerson: 20.0, discount: 0.1 });
const refund = await refundBooking(booking.id);
expect(refund.amount).toBe(72.0);
});
Four people at 20.00 with 10% off is 72.00 whichever way you round it. That test can’t catch a rounding bug. If it’s red, it’s red for some other reason (a fixture the agent got wrong, a field it guessed), and whatever the agent changes to make it green won’t touch the cent the customer lost.
Step 1: take the signals from the ticket
First the agent pulls out what the ticket actually gives it: error text, status codes, ids, and the steps the reporter took. Here that’s a booking id, the two amounts, and “cancelled the whole group”.
The id matters most, because it’s the key to the real record. If the ticket has none, the agent should say so out loud. Without a concrete case the diagnosis stays a guess, and I’d rather see that written down than papered over.
Anything the ticket says about the cause (“I think the discount is applied twice”) is a hypothesis to check. My default hypothesis is that the system works as configured. Sometimes it does, and the right answer is “this is a setting”, with no branch at all.
Step 2: pull the real case
Now the agent goes read-only into the database and the logs. From the database: the booking row, the payments against it, and anything else hanging off it, with explicit columns and a limit. From the logs: one line where this path failed and one line where the same path worked.
Both log lines matter. The failing one alone tells you something broke. Put a passing one next to it and you can see what’s different. In the example, the failing line is a refund for a group of 3 at 12.35 each with 5% off, and the passing line is a group of 3 at 12.00 with no discount. Same path, same group size. The failing one has a discount and a price that isn’t a round number, the passing one has neither.
Those two rows make a better fixture than anything the agent would make up.
Step 3: name the root cause as a file and a mechanism
“Rounding issue in refunds” is too vague to check. The agent has to point at a line and say how it produces the wrong number:
refunds/calculate.ts:88takes each person’s share after the discount, rounds it, then adds the shares up. The charge rounded once, on the discounted total. 3 × 12.35 × 0.95 is 35.1975, which rounds to 35.20. Each share is 11.7325, which rounds to 11.73, and three of those make 35.19.
It also has to say why that line is the source. Here, with no discount every share is a whole number of cents, so nothing rounds, which is why only discounted groups break. If the explanation can’t account for why the passing log line passed, I’d keep digging.
Then the blast radius: which files a correct fix must touch, and which it must leave alone. One way to get this wrong is to fix the charge side instead. A refund bug gets “fixed” by changing how charges round, and now every new group booking costs a cent more. So the diagnosis names the charge path as off limits.
Step 4: write the test from that case, and make it fail for the right reason
The test goes into the existing spec next to the code, runs against a real test database (a local Postgres in Docker is fine), and is seeded with the exact data from step 2: the same prices, discount and party size. Swap names and emails for fakes. The values that drive the bug stay as they are.
it('refunds the charged amount for a discounted group of 3', async () => {
// seeded from the failing case: 3 x 12.35, 5% group discount, charged 35.20
const booking = await seedBooking(db, {
people: 3, pricePerPerson: 12.35, discount: 0.05, charged: 35.2,
});
const refund = await refundBooking(booking.id);
expect(refund.amount).toBe(35.2);
});
It asserts the correct behavior, the 35.20 the customer expected. Then you run it on the unfixed code and read the failure:
AssertionError: expected 35.19 to equal 35.2
That’s the line you want to see. The test failed, and the actual value is the 35.19 the customer saw. So this is the customer’s bug. A red test on its own doesn’t tell you that.
The other outcomes each mean something too:
- It passes. The guess about the mechanism is wrong, or the seed is missing something. Stop and say so. Don’t open a branch.
- It fails with a different value, say 35.18. You found a bug, probably a different one. Go back to step 3.
- It fails on setup. A missing fixture or a wrong column. That’s a problem in the test itself, and it tells you nothing about the bug yet.
When to call it confirmed
I only let the agent say “confirmed” when 3 things line up: the test is red for exactly the predicted reason, the root cause is one location, and the production evidence shows the same signature. If there are two possible causes, or the logs don’t cover that path, I’d call it medium confidence. With no real case, it’s low.
Medium or low gets a short drill: one more query or a deeper read, a refined test, a rerun. I cap it at 3 rounds. If it’s still short after that, the agent stops, writes down what’s ambiguous and what evidence is missing, and hands over the partial diagnosis anyway. A half-finished diagnosis is more use to whoever picks it up than a fix for the wrong bug.
The diagnosis ends there too. The red test stays uncommitted, and the fix is a separate step that starts from it.
I built this rule into falsify, a Claude Code plugin that takes a ticket to a PR. Its probe step (skills/task/SKILL.md) writes one test asserting the reporter’s expected value and halts with ASSUMPTION_UNPROVEN unless the actual value equals the reported one. The branch only exists after that.
The rule also fits in a few lines of any bug-fix prompt:
Before any fix:
1. List the ids, error text and steps in the ticket. No id → say so.
2. Pull the record, plus one failing and one passing log line for it.
3. Name the root cause as file:line + mechanism, and what the fix must not touch.
4. Add one test seeded with that record, asserting the expected value.
Run it. It must fail, and the actual value must equal what the
reporter saw. Otherwise stop and report what's missing.