Three confirmed cases
--collect-only, so 23 failures were disguised as green → sealed the execution gate behind an env varFix
Green is a necessary condition. Verify live separately on real devices, real browsers, and instrumentation. Recompute verdicts on read.
The reverse trap: a regression contract freezes a bug value
In a contract test that compares re-runs against a captured baseline, if the capture happened while the bug was live, the bug gets frozen. Measured: baseline recorded images: 1 (a gallery bug) → the fix took it 1 → 11 → the contract went red.
An intended fix has to update the baseline with the re-extracted value.
The lazy-import trap
try: from PIL import Image / except: return None is the canonical pattern for collect-only CI safety, but when the package is missing it silently disables instead of raising. Any third-party import wrapped in a lazy import gets a requirements check + an explicit contract. "The tests pass so it's there" is not evidence: the tests run in the same container.
Fourth confirmed case: the mock checks a path production never takes
An external API error was fixed and reported as "reproduced, then passing" with a mock. But the mock was patched onto the direct-call leg, and production sends the same call through a relay. The branch is chosen by a single env var, and the sandbox default is the quiet side (direct), so it went green. With the relay switched on and the same code run again: direct leg called 0 times, relay called once. Production never once walks the path the test checked. The call site is a single gateway, so reading the code does not tell you which leg runs.
"I reproduced it" only holds once you say which code you reproduced. Green on the wrong path is worse than no green: it makes you believe it is fixed.
The mirror of the green trap: the exoneration trap
An icon looked broken in a captured screenshot. The agent ruled: "capture artifact. A file-based render just can't fetch the static-asset path; it's fine in the app." The mechanism was plausible and the conclusion was wrong: the same breakage turned up in a screenshot taken live.
An artifact verdict is a verdict that ends the investigation. Once a spot is filed under "not our problem," nobody looks at it again, so when the verdict is wrong the defect stays live. What went unseen gets seen eventually; what got exonerated never gets searched for. And a correct mechanism is not evidence that it caused this breakage. The burden sits on the expensive side, and a live check was replaced with an explanation.
★ "Not our problem" is the most expensive conclusion, so it demands the most expensive evidence. The green trap makes you skip verification; the exoneration trap makes you stop investigating. The more a verdict halts investigation, the heavier its evidence has to be. Even without a repro, the dependency-removing repair (inlining) is still available.