Test Agent Patches With an Oracle the Diff Cannot Touch

An agent patch is only as trustworthy as the checks it cannot rewrite. If properties, fixtures, and flake policy live in the same tree as src/ , the diff can weaken the proof. Move the oracle out of the writable tree and run it as a control loop with hysteresis, not as a skip list. Co-located tests fail this requirement in a predictable way. The agent adds an assertion that matches the new code. A fixture grows a default that hides a broken parser. A flaky case becomes skip . The suite stays green. Production still drifts. This article proposes a sidecar oracle: human-owned properties, sealed fixtures, and a two-threshold flake freeze. The design is a workflow, not a production case study. Treat the code as a proposed runner you can execute locally, not as a claim about a live fleet. What the loop decides The loop answers three questions on every candidate patch: Do independent properties still hold on generated inputs? Did the patch mutate a sealed fixture or depend on an unsealed one? Is a failing test a regression, or does it belong in a measured freeze? A skip list answers none of those. It only records that someone got tired of a red job. Layout: oracle beside the repo, not inside the diff Keep the application repo writable for the agent. Keep the oracle in a second directory that the agent cannot include in its patch. app/ # agent may write src/, not oracle paths src/ pyproject.toml oracle/ # human-owned; hashed before every gate properties/ test_invariants.py fixtures/ manifest.json http_empty_body.json flake_ledger.json path_deny.txt run_gate.py path_deny.txt is the first control, not the last. If the patch touches oracle files, tests the agent authored, or lockfiles it did not need, the gate fails before pytest starts. # oracle/path_deny.txt oracle/ **/test_*.py **/*_test.py **/conftest.py **/__snapshots__/ The deny list is deliberately blunt. Agent-authored tests can still exist as scratch. They do not count as evidence. Step 1 — Hash the oracle before the