When an AI coding agent edits a file and claims a fix because the test goes green, that green test only tells you one thing: the command passes now, according to Livecristen in their post "Why I stopped trusting fixed from my coding agent". It does not tell you if it failed before, or if it would fail again without the change. To stop accepting unverified code, you can run a structured ten-minute check on your project using verification rules inspired by the FixProof CLI (v0.3.0, python 3.9+ standard library only).
Step 1: Record the Failure First
Before letting an agent touch any code, make sure the failing state is actually recorded. According to the source, initialize a baseline where your test command actively fails with an exit status of 1. If your reproduction command does not fail before the change, any subsequent green test is completely meaningless. Run your reproduction command explicitly and confirm it exits with 1.
Step 2: Test the Revert Condition
The most critical part of verifying an agent's work is checking whether the test would fail without the fix. According to the source, a robust verification flow temporarily removes the fix, runs the reproduction command again to ensure it fails, and then puts the file back. If your test still passes even when the fix is removed, your test is not actually targeting the bug. In that case, you end up with an unverified status instead of a verified one.
Step 3: Inspect the Five Verification Statuses
Categorize your agent's pull requests or edits into strict outcome buckets:
- VERIFIED: failed before, passes after on every rerun, fails again when reverted, and nothing else broke.
- FAILED: reproduction still fails, or something that passed before now fails.
- PARTIALLY VERIFIED: reproduction passes but something is missing, such as no suite or no revert test.
- UNVERIFIED: bug never reproduced, test passes even without the fix, or reruns are flaky.
- BLOCKED: missing tool, timeout, or not a git repository.
Step 4: Watch Out for Agent Blind Spots
Keep in mind what automated verification scripts cannot catch on their own. According to the source, if the agent picks the wrong reproduction command that does not actually hit the bug, a verified status just means the wrong command was verified. Additionally, local logs can be deleted if not managed carefully, and fixes living inside test files cannot always be cleanly reverted. Double-check the agent's work with these limitations in mind.