falsify
A Claude Code plugin that takes a ticket to a pull request and refuses to skip the proof at any step.
The problem
Your coding agent fixes a bug, the tests pass, and the bug is still there. The test was written against the agent’s theory of the bug, not against the symptom the reporter saw.
Speed isn’t the bottleneck with agents any more. Plausibility is: a wrong answer looks exactly like a right one, and both come with passing tests. Agents also edit tests to make them pass. ImpossibleBench (2025) found GPT-5 doing it on 76% of tasks whose tests contradicted the spec.
falsify is built on three premises:
- A diagnosis that hasn’t been executed is a guess.
- A test written after the theory tests the theory, not the bug.
- “The tests pass” means nothing if the agent chose the tests.
The name comes from Karl Popper: a claim counts only if it could have failed and didn’t. Every step of a run is set up so it can fail.
Core concepts
The default hypothesis
For a bug report, falsify starts from the assumption that the system works as configured. The run has to clear that bar with evidence; it doesn’t assume a bug and go looking for one. A root cause written in the ticket, a file:line or “the AI said it’s X” is treated the same way: as a hypothesis to disprove, not a spec.
Evidence comes first. The agent pulls identifiers out of the ticket, queries the real systems read-only (logs, database, third-party APIs, docs), and checks whether a feature was even enabled before asking why it misbehaves.
The verdict
Before any branch exists, the run commits to exactly one verdict and explains the mechanism behind it with the real values it found.
| Verdict | Means | Then |
|---|---|---|
CONFIRMED-BUG |
Evidence reproduces the defect, and the code contradicts the docs and the configuration. | Write the probe. |
WORKING-AS-DESIGNED |
The mechanism produces the reported value, correctly. | Explain and stop. No branch. |
CONFIG |
A setting produces this, not the code. | Name the setting and stop. |
NOT-BUILT |
The code path has no logic for what the ticket asks. | Stop. A missing setting isn’t a defect. |
UNCONFIRMED |
Evidence is insufficient or contradicts the reported cause. | Ask one question. Never branch. |
The probe
A confirmed defect gets a probe: one test, in the spec that already owns the code, asserting the value the reporter expected. It runs on the base branch and has to fail, and the actual value has to equal the one the reporter saw. That equality is the proof. Only then does a branch exist.
If the probe fails with a different value, it’s a different bug. If it passes, there’s no bug. Either way the run stops. The probe is never rewritten afterwards; it becomes the first red test the fix turns green.
The worklist and the coders
The design is a worklist: one change per item, each with the test that proves it. When two designs both pass, the one that touches fewer files wins. Coders, up to 4 at a time, each get their slice of the worklist and nothing else, never the raw ticket, and each runs on its own test database.
The gate battery
Each gate detects one class of defect, has a published source showing the class is real, and can’t pass while the property is false. The checkable gates are Node scripts with no dependencies. Each prints one machine-readable result line, and “could not measure” never counts as a pass.
A failing gate doesn’t stop the run. It starts a repair round: fix, re-run that gate and every gate the fix touched, then a full sweep at the end. Anything dismissed (a surviving mutant, a fan-out flag, an accepted test change) needs a written reason, and every reason goes into the pull request.
A reviewer that didn’t write the code
After the battery, a fresh agent reviews the change. It gets the reasoning behind each design decision and one instruction: prove it broken. A finding without evidence is dropped. It’s a separate agent because language models grade their own output favorably (Panickssery, Bowman and Feng, NeurIPS 2024).
Measured or inherited
Every result carries a grade. [A] means this run executed the measurement. [B] means it inherited or proxied it, for example when CI will run the full suite instead. A missing tool is never quietly downgraded: the run says which command failed and asks.
How a run goes
You start a run with /falsify:task <ticket>. It moves through 11 steps:
- Intake. Fetch the ticket and the base branch, decide bug or feature, and ask one question only if that’s ambiguous.
- Investigate. Gather read-only evidence and reach a verdict. A non-defect verdict ends here with an explanation.
- Probe. For a confirmed bug, write the failing test on base.
- Grill. Ask the design questions that would change the code, each with a recommended answer.
- Design. Write the worklist and a QA procedure.
- Build. Branch, then dispatch the coders on isolated test databases.
- Battery. Run the gates; repair until they pass.
- QA. The fresh reviewer tries to break the change; blocking findings go back to the battery.
- Score. Compute the score. Only HIGH ends the loop by itself.
- Commit and pull request. The PR carries the red and green test pairs, the score block verbatim, and every dismissal with its reason.
- Handoff. Comment on the ticket or the PR, then clean up.
A run ends on HIGH, on your explicit decision, or on a named halt code. Never on its own. Below HIGH, no pull request opens without your word.
The gates
Hard gates zero the score if they fail, because defects don’t average. The others are weighted.
| Gate | Catches | Rule | Evidence |
|---|---|---|---|
| Size (hard) | Review blindness on large diffs | Over 400 changed lines halts the run, unless it carries a written reason. | Rigby and Bird, FSE 2013; SmartBear, 2006 |
| Types (hard) | Type errors, including in files the diff didn’t touch | The branch’s error set must be a subset of the base’s. | Gao, Bird and Barr, ICSE 2017: 15% of public JavaScript bugs are detectable by TypeScript |
| Fan-out | The file you forgot | A file that changed together with a changed file in half its history, and is missing from the diff, is edited or dismissed with a reason. | Zimmermann et al., ICSE 2004: co-change mining puts a correct site in the top 3 over 70% of the time |
| Behaviors (hard) | A fix that doesn’t do what the ticket says | One test per behavior, red then green, zero failures. | Fucci et al., TSE 2017 |
| Tamper (hard) | Tests weakened to pass | A removed test, an added skip or a removed assertion fails the run. | ImpossibleBench, 2025: GPT-5 passes 76% of spec-contradicting tests by editing them |
| Regression (hard) | Broken existing behavior | Every spec whose imports reach a changed file runs whole. | Rothermel and Harrold, 1997; Ekstazi, ISSTA 2015 |
| Complexity | Code the next reader can’t hold | Cyclomatic over 10 or cognitive over 15 on a new or grown function. | McCabe 1976 and NIST 500-235; SonarQube’s default of 15 (a tool default, not a study) |
| Mutation | Tests that run the code and assert nothing | One mutant per added line; each survivor is killed or marked equivalent with a reason. | Just et al., FSE 2014; Google, ICSE 2018 and 2021 |
| Real-world (hard) | Works in the test, fails in the app | A screenshot, a rolled-back migration proof, or the deepest test run. | Zeller, Why Programs Fail; Ambler and Sadalage, 2006 |
| Cleanliness | Junk that taxes reviewers | The project’s formatter and linter run on the diff; a change re-runs types, behaviors and complexity. | Bacchelli and Bird, ICSE 2013 |
| QA (hard) | Defects the author’s tests couldn’t imagine | A reviewer agent reports zero blocking findings. | Panickssery, Bowman and Feng, NeurIPS 2024; Zheng et al., NeurIPS 2023 |
Every threshold states where it comes from, and the ones that are tool defaults say so. Full citations are in the repository.
How the score is computed
The score measures one thing: how much of the claimed confidence was executed in this run. It says nothing about correctness. A script computes it from the gates’ recorded results, so the model can’t edit it.
- Hard gates first. Any failed hard gate, or any blocking QA finding, makes the score 0 and the verdict NOT SHIPPABLE.
- Weighted evidence. Otherwise the score is 100 × the sum of weight × grade × pass, divided by the weights of the rows that apply. Grade
[A]counts 1.0 and[B]counts 0.5. - One override. If the dimension the ticket is about was only inherited (
[B]), the verdict is capped at MEDIUM.
| Row | Weight |
|---|---|
| Diagnosis (bugs only) | 20 |
| Behaviors | 20 |
| Regression | 15 |
| Mutation | 15 |
| Real-world | 10 |
| Fan-out | 10 |
| QA | 10 |
| Complexity | 5 |
| Cleanliness | 5 |
HIGH is 90 or more, MEDIUM 75 to 89, LOW below 75. In practice, one inherited row such as Regression still leaves a bug fix at 93.2, HIGH. Two inherited rows on important dimensions (Regression and Fan-out) bring it to 88.6, MEDIUM. That boundary is deliberate: two inherited rows on important dimensions is where pull requests go wrong.
The weights are declared priors, marked “unmeasured” in the config and printed on every score line. They get fitted once outcomes are tracked.
What it doesn’t do
- No coverage percentage. Coverage says a line ran, not that a test would notice if it changed, and an agent can raise it without asserting anything. Mutation testing does that job instead.
- No “code quality 1 to 10” from a model. Nobody can say what a 7 measures, and the model that wrote the code is the worst-placed grader of it.
- No outcome tracking yet. Whether HIGH predicts fewer reverts isn’t measured. The run record is kept so the weights can be fitted when it is.
- Claude Code only, for now. The gates are plain Node and run anywhere; the orchestration uses Claude Code’s skill format. The portable contract describes the same procedure for any harness.
These patterns come from one production codebase. They’re a hypothesis with a lot of practice behind it, not a study, and the repository lists its open questions.
Install
In Claude Code:
/plugin marketplace add maxymlyskov/falsify
/plugin install falsify@falsify
Then, once per repository:
/falsify:setup
Setup asks where tickets live (GitHub Issues, Linear, Jira or local files), what the test command is and how it reports a pass, which branch is the base, whether there’s a UI it can screenshot, and where the handoff goes. It runs each answer before accepting it and writes .claude/falsify.config.json. Requires Node 18 or later.
Where it’s used
Every fix on the open-source contributions page was found, proven and sent with falsify.