Evidence or nothing: an AI bug sweep your team won't mute
An AI bug finder that files a pile of maybes every night gets muted fast. Someone writes a filter rule, the tickets land in a folder no one opens, and the tool keeps running anyway.
The sweep I’d build runs on one rule. It opens a ticket only when it can show the bug happening, with a screenshot the agent actually opened, a log line, or a row from the database. A finding without one of those goes into the morning report as unproven and never becomes a ticket. And the night is capped at 5 tickets, so even a bad night can’t flood the board.
Everything below, names and numbers in the examples included, is made up unless I say it’s a setting.
Why a nightly sweep at all
Features ship a little unfinished. The PR gets reviewed, it merges, and nothing looks at that feature again until a user trips over it. A sweep that runs after the working day is a second look at what changed, plus a check on what broke in production since yesterday.
It runs as a command file the agent follows step by step, in its own checkout, so it never touches anyone’s working tree. The scheduler passes it a deadline. If there’s none, it gives itself 2 hours.
Each night covers the last 24 hours. On Monday it looks back 73, which is the weekend plus the extra hour the October clock change adds. That sounds fussy, but without it an hour of errors falls between two windows once a year.
Where it looks
I’d give it three places, ordered by how likely each is to produce something real.
Recently merged features. The sweep picks up to 3 features merged in the last 14 days and walks each one in the browser like a user would. Book something, cancel it, toggle a setting, check the screen afterwards. It picks the least recently walked first. A walk that finishes gets marked as done. A walk that halts doesn’t, so it stays at the front of the line for tomorrow.
New error patterns in production. Most error logs repeat. A new one is interesting. So the sweep groups errors by a fingerprint (the handler and the status code) and compares the last day against the 7 days before it.
// new-errors.js (example)
const known = new Set(baseline.map((r) => r.fingerprint));
const fresh = today.filter((r) => {
if (!r.fingerprint || known.has(r.fingerprint)) return false;
const is5xx = / - 5\d\d /.test(r.fingerprint);
return r.count >= (is5xx ? 3 : 10);
});
// "OrdersController.refund - 500 Internal" with 4 hits today and none last week -> a finding
A 5xx needs 3 hits. A 4xx needs 10, because most 4xx come from bad input. And if the baseline query comes back empty, the sweep skips this whole source. An empty baseline makes every fingerprint look new, and the sweep would file tickets about errors you’ve had for a year.
Some failures never show up as an error response, like a call to a payment provider or an email service that fails in the background. For those, the sweep reads the code merged in the last 14 days and pulls out every new logger.error( and logger.warn( message, up to the first interpolated value.
added: this.logger.error(`Refund webhook rejected for order ${id}: ${err}`)
probe: "Refund webhook rejected for order "
It keeps a probe only if it’s at least 12 characters, takes at most 20, and counts each one in the logs. Three hits or more and it pulls 3 sample lines as evidence.
Review of yesterday’s merges. One sub-agent reads the diff. It gets a narrow brief: for each correctness bug you can name, return the file and line, a concrete failure scenario, and the query that would show it happening in production. No style points, no maybes.
Then the sweep runs those queries itself. A row that shows the failure makes it a finding. No row, and it’s unproven. I’d expect this source to be the noisiest without the rule, because a model reading a diff can always find something that looks wrong.
What counts as evidence
The evidence rule needs a sharp edge or it erodes. Three things count.
- A screenshot the agent opened and looked at in this run. A path to an image it never read doesn’t count.
- A log line, with the query that found it.
- A database row, from a read-only query, with the query.
Generic log searches don’t count. A search for %error% or %failed% matches lines from every part of the system, and an agent that wants a finding will happily take one of them as proof. I’d write that into the command in so many words, because it’s the easiest shortcut the model has.
Each finding gets stored in the same shape, so the filing step doesn’t have to guess.
{
"source": "prod-error",
"title": "[sweep] prod-error: OrdersController.refund - 500 Internal",
"symptom": "refund requests fail with a 500 since yesterday",
"evidence": [
"count today: 4, last 7 days: 0",
"2026-10-08T02:14:09Z Response ERROR [500]: OrdersController.refund - 500 Internal ..."
],
"expected": "refund goes through",
"observed": "500, no refund recorded"
}
The cap and the order
Before filing, the sweep lists open tickets from the last 30 days and drops any finding whose title matches one. Those go in the report as already filed.
Then it sorts. Walk findings go first, since a broken screen is the thing a user hits. Then new 5xx patterns, then background failures, then new 4xx, then review findings. It files the first 5. The rest go in the report as over cap, where you can read them in the morning and file one by hand if it’s real.
Every ticket gets a title prefix and one label, so the team can filter the board to just the sweep’s tickets, or out of them. Its first line says which night and which source found it. I’d set no priority. The sweep doesn’t know what matters to the business, and a ticket that claims to be urgent loses trust faster than one that just shows the bug.
Touching production safely
Walking a feature means clicking through it, and that writes data. So the rules for production go into the command as hard limits.
- Writes happen only on one demo account, behind a guard that refuses any other account.
- Every walk cleans up what it created. If cleanup didn’t finish, that’s the first line of the report.
- Database access is read-only. No raw SQL writes, ever.
A dry run files nothing and marks nothing, but it still walks, so it still books and cancels things on the demo account. Say that in the file, so a teammate reading it knows a dry run still writes.
Running out of time
The deadline matters because the scheduler kills the run when it’s up. A sweep that dies mid-review loses everything, including the findings it already had.
So the walks stop when there’s less than 30 minutes left, and each skipped feature goes in the report as not walked. If the deadline has passed by the time it reaches the review, it skips the review and goes straight to filing and the report. The work that’s done gets kept.
Nothing else stops the run. If one source fails, the report says so and the sweep moves on. If a step needs a fact it doesn’t have, it writes MISSING_FACT: <name> and moves on without guessing.
The morning report
The report is the other half of the deal. Tickets are for proven bugs, and the report holds everything else.
## sweep 2026-10-08 (live) 01:00 -> 02:41 deadline 03:00
Cleanup: clean
Walked: checkout-coupons 1 gap | gift-cards 0 gaps | not walked: deadline
New prod errors: 1 fingerprint · 0 background probes hit Review: 1 proven · 2 unproven
Filed: [sweep] walk checkout-coupons: total ignores second coupon
[sweep] prod-error: OrdersController.refund - 500 Internal
Already filed: none Over cap: none
Unproven: totals may round wrong for 3-item carts (no row matched the query)
retry loop may double-send emails (no log line)
The unproven list is where I’d look when I have 10 minutes. Some of those are real and just need a better query. But they’re never tickets on their own, and that’s the part that keeps the team reading the ones that are.