标签：#Testing

A green test suite is evidence. It is not independent evidence. The current release of badBANANA Threat Observatory passes all 108 automated tests in its own development and CI environments. That tells me the implementation satisfies the assertions I wrote against the conditions I expected. It does not tell me whether an independent developer can check out the same commit in a clean environment and obtain the same result. That distinction matters more than the number 108. The dangerous failure is a believable one The Observatory presents source-backed threat-intelligence records, freshness information, and material-change events. In that kind of interface, an obvious crash is not necessarily the worst outcome. A more dangerous failure is one that looks healthy: An expired cached snapshot presented as current An invalid expiry value treated as usable A failed upstream source displayed as a successful zero-result response Demo or fallback data appearing without explicit disclosure A disabled or offline state silently normalized into success Those failures do not merely inconvenience the user. They change what the interface appears to know. For v1.2.2, the intended behavior is deliberately fail\ closed: Condition Required behavior Cached snapshot has expired Report it as stale Expiry value is invalid Fail closed to stale Source is offline, disabled, or failed Preserve that state Source data is missing or unavailable Do not present a successful zero result state Ingestion requests overlap Enforce the runtime concurrency limit deterministically Feed credentials are configured Keep them server side and absent from client output The test suite exercises these boundaries. The remaining question is whether the release reproduces cleanly outside the environment in which it was built. Passing tests and independent verification are different claims When the source, tests, build assumptions, and execution environment all come from the same maintainer, a successful run demonstrat

Agents fail quietly when they fill omitted tool arguments instead of refusing, and fluency-based evals often reward that invention. A compact negative golden set, scored by argument-diff rather than prose quality, catches those silent substitutions before they reach production traces. This article treats that failure as a testable contract, not as a prompt-tuning anecdote, and it stays useful without any vendor product. Recent developer discussion around agent workflows keeps returning to one operational surprise that chat logs tend to hide. Models do not only choose the wrong tool; they complete incomplete requests by guessing identifiers, dates, and scopes that nobody supplied. That behavior looks like initiative in a chat log, yet it resembles a clerk forging a zip code to stamp the form complete. The package then leaves the dock with valid-looking paperwork and the wrong city printed on the label. A conventional golden-answer harness scores the final sentence, which is the wrong surface for tool-using agents. The dangerous artifact is the tool payload, because downstream systems will execute invented primary keys with perfect syntax. If your eval suite only checks that a transfer looks helpful, it will greenlight a call that moved the wrong account. The pattern below is a proposal you can run locally, and it does not claim production metrics. It also does not depend on a particular model family or on a hosted evaluation service. You should treat every numeric threshold in the grader as a starting point rather than a published benchmark. Negative goldens assert a hole, not a pretty answer A positive golden case says the model should produce a known good action given a complete request. A negative golden case says the opposite: given a hole in the input, the model must not paper over that hole. The assertion is closer to a check constraint than to a writing rubric, because the failure is an illegal completion. Fluency still matters for users, but it is a poor prox

I was asked whether the system was sending patient data to an external body while the integration was half-built. I went and read the logs of every run. They all died early: some with a 415 because the content type wasn't what the other end expected, others with a 500. Not one showed an outbound call. I answered that nothing was going out. The first run that got past the 500 sent two requests carrying real clinical data . My answer had been false from the start, and the worst part is that it was false in a way that felt rigorous: I had looked. I had evidence. The evidence was logs of real executions, not assumptions. A negative says nothing on its own The mistake wasn't misreading the logs. It was not noticing what produced that silence. The runs died before reaching the code that sends. The log didn't say "I didn't send"; it said "I never got to the part that sends". Those are two different statements and they produce exactly the same output: nothing. That's the general shape of the problem, and it turns up everywhere once you look for it: A counter at zero can mean "it didn't happen" or "the counter was never incremented". A "not found" can mean "it doesn't exist" or "I looked in the wrong place". A green test can mean "it passed" or "it skipped itself". An exit 0 can mean "it worked" or "the command was strangled by a pipe that swallowed the exit code". A silent dashboard can mean "everything is fine" or "the process feeding it has been dead for three weeks". In all five, the evidence is identical. And in all five, the optimistic reading is the reassuring one, so it's the one chosen without thinking. The positive control The fix isn't to be more suspicious. It's to demand one specific thing before accepting any negative: Find something the log MUST show if the path was actually taken. If the system had reached the part that sends, something would have to appear in the log: the "preparing request" line, the batch identifier, the connection attempt. Any signal that

On 2026-09-04 I pointed a scanner at langchain-ai/langchain . Shallow clone of the default branch, HEAD 79cab2d , read only. It walked 2581 files and printed zero sites. Its own control had passed immediately before the run, with two positive fixtures seen and four negative fixtures clean, so the zero was a measurement rather than a crash. Then I opened one file by hand. libs/langchain_v1/langchain/agents/middleware/human_in_the_loop.py , line 403: def _should_interrupt ( self , tool_call , config , state , runtime ) -> bool : """ Return False if the `when` predicate rejects this tool call, True otherwise. """ when = config . get ( " when " ) if when is None : return True ... return when ( req ) when is supplied by the caller. It is declared NotRequired[Callable[[ToolCallRequest], bool]] on line 195 and documented as returning True to interrupt or False to auto-approve. Its result is handed back unchanged. A predicate that falls off a branch returns None , and the caller on line 436 reads: if not self . _should_interrupt ( tool_call , config , state , runtime ): continue None is falsy. The interrupt is skipped and the tool call proceeds with nobody looking at it. The annotation says bool ; nothing at runtime makes that true. Why the machine stayed quiet I took the failure apart instead of guessing at it. Three causes, each sufficient on its own: Vocabulary. 22 lines in that file matched the approval vocabulary the scanner looks for. Not one of them put line 403 inside its window. The nearest match was 26 lines away and sat in a comment. This project calls the decision interrupt , not approval. Window. The -> bool annotation is on line 378. The return is on 403. That is 25 lines apart, and the window was 12. Signals. Widened to 55 lines, the three behaviour signals still matched nothing on that line. The file walk was innocent. The file is .py , 18256 bytes, and no skip rule matched it. It was read. What I got wrong The window of 12 lines had no measurement behind it

Every AI code review metric you track measures the hour before merge, and that is precisely the hour when the least information exists. Acceptance rate, test pass rate, and review approval all describe how a patch looked in isolation, not how it behaves under real traffic. Revert rate is the only signal that arrives after the system has voted, which makes it the least gameable number in your pipeline. This article argues that you should stop celebrating AI patch acceptance and start measuring how many of those patches come back. Why the pre-merge metrics lie A green test run proves that a patch fits the expectations you encoded last quarter, not the behavior your users will hit tomorrow. Reviewers approve diffs under time pressure, and a cleanly formatted AI patch reads as competence even when its logic is wrong. The merge is where the real evaluation begins, and the revert is the only verdict that carries operational weight. Nobody plans a revert, so the metric cannot be gamed by prompt tweaks or review theater. The argument is not that pre-merge review is useless; it is that pre-merge signals saturate quickly. Once your review gate catches the obvious failures, the remaining defects are exactly the ones that look fine in review. Those defects surface as incidents, hotfixes, and reverts, which means your post-merge telemetry is the only source of new information. Treating acceptance as a quality metric is like judging a deployment by how well the rollout script ran. The artifact: a revert attribution watch The workflow below attributes every revert commit to the patch that caused it and computes a per-source revert rate. It requires only a git history, which makes it reproducible on any repository that has survived a few incidents. Run it on a local clone first, because a read-only analysis should never touch shared state. Step 1: List every revert commit in your window. git log --all --since = "90 days ago" --grep = "^Revert " --format = "%H %s" Step 2: Extract th

AI agents are easy to demo and surprisingly hard to evaluate. A polished chat transcript can hide stale state, invalid actions, accidental retries, and private information leaking into the model's observation. I built WagerCall as a bounded environment for studying those problems. Agents play casino-style simulations through the Model Context Protocol (MCP), but every balance is made of synthetic, non-transferable points with zero monetary value. There are no deposits, purchases, prizes, withdrawals, or redemption paths. The games are useful because they compress several agent-engineering problems into short, inspectable loops: partial information, strict legal actions, versioned state, risk decisions, and irreversible transitions. Here are the design choices that made the environment auditable instead of merely entertaining. 1. Bound the world before evaluating the agent An evaluation environment should say exactly what an agent can observe and change. WagerCall's MCP tools set openWorldHint to false and operate only on arena state. The agent cannot call a generic SQL, admin, execute, or debug tool. That boundary matters. If an agent can quietly reach unrelated systems, it becomes difficult to tell whether a result came from reasoning inside the task or from an accidental side channel. The same rule applies to the economy. Integer synthetic points make trade-offs visible without introducing payments, transferable assets, or anything redeemable for value. 2. Let pure game logic propose; let the database decide The game engine is deterministic and side-effect free. Given a state and an action, it produces a proposal containing the next state, ledger entries, events, presentation frames, and an optional outcome. A proposal is not yet a fact. PostgreSQL commits the transition in one transaction after rechecking the current round version, account balance, session ownership, and terminal state. It either writes the action, balance change, new round state, and audit event

Two instructions went into an Auditor my agent collaborators and I built to surface conflicts in...

A system has to run a chunk of code you don't control —a plugin, a dependency, something generated— and you want to guarantee that code cannot touch the file system or spawn processes. Not that it "shouldn't": that it can't , mechanically. That's capability confinement, and it's one of the central problems of runtime security. Designing it and demonstrating it are two different things, and confusing them is expensive. A design is a claim You can write an impeccable document: "access to fs and to spawning processes is controlled like this, with these mechanisms, under this threat model". It's real and necessary work. But it's a claim . And the failure mode of a security boundary is that it looks like it holds until it doesn't —silent, invisible in tests, visible only when someone crosses it—. In security, an unverified claim has exactly the shape of a beautiful, wrong architecture. The design can assume that a module-loader hook fires at a point where it actually doesn't, and all the reasoning hanging off that is correct and worth nothing. The mechanisms exist, and there are several In Node, to name a concrete runtime, there are at least three layers, and they aren't interchangeable: The native permission model ( --permission , --allow-fs-read …), which cuts access to fs and to spawning processes at the whole-process level. SES / Hardened JavaScript (Compartments, lockdown() ), which confines what each module can import within the process. Module-loader interception , which controls what resolves when the code asks for something. Choosing well among them is the design. But choosing well doesn't prove the choice holds against the real dependency tree you're going to run. Demonstrate instead of claim The alternative to signing off a design is delivering a confinement harness : untrusted code that tries to reach the dangerous capability —open a file, spawn a process— against the real runtime and its real dependency tree, and a log that shows each attempt was blocked . O

Refactoring Safely: A Step-by-Step Guide We all know that feeling: a function that's 200 lines long, a class that does too many things, or a variable named data2 . Refactoring is the cure, but doing it recklessly can break your app and your confidence. Here's how I approach refactoring safely, step by step. 1. Start with a Safety Net Before touching any code, make sure you have tests. If your project lacks tests, write a few key ones first. Focus on the behavior you're about to change. The goal is to have a safety net that tells you when you've broken something. # example test for a function we'll refactor import unittest from mymodule import calculate_total class TestCalculateTotal ( unittest . TestCase ): def test_with_discount ( self ): self . assertEqual ( calculate_total ( 100 , discount = 0.1 ), 90 ) If tests aren't feasible, at least have a manual checklist. But automated tests are worth the effort. 2. Make Small, Atomic Changes Don't try to refactor everything at once. Pick one logical change. For instance, extract a method or rename a variable. Each change should be small enough that if it breaks, you know exactly what caused it. // before function processOrder ( order ) { const total = order . items . reduce (( sum , item ) => sum + item . price , 0 ); const tax = total * 0.08 ; const final = total + tax ; return final ; } // after step 1: extract tax calculation function processOrder ( order ) { const total = order . items . reduce (( sum , item ) => sum + item . price , 0 ); const final = total + calculateTax ( total ); return final ; } function calculateTax ( amount ) { return amount * 0.08 ; } Run your tests after each tiny step. If they pass, move on. If they fail, you know the last change caused it. 3. Use Your IDE's Refactoring Tools Modern IDEs can rename variables, extract methods, and change signatures safely. They update all references automatically. This reduces human error. For example, in VS Code, right-click a function and choose "Extract to

Your first AI patch should fail closed today. Do not ship a feature on day one. Prove one target file can revert cleanly now. You joined a messy repo this morning. The assistant wants a wide rewrite. Your job is a tiny reversible cut only. This drill gives you a blast-radius file first. You fill it before any model writes code. Then a short script checks the revert path. Why day-one AI diffs explode Cheap code is not cheap to unwind. One extra import can touch auth. One extra migration can lock deploys. You will not know the architecture yet. You also should not pretend otherwise. A blast-radius file makes unknowns explicit fast. If the file cannot name a revert, stop. You do not prompt for more code. You shrink the change until revert is boring. What you will build today You will add two artifacts on your branch. Keep both files in the first PR. blast_radius.py — the contract for this change. scripts/check_blast_radius.py — the fail-closed proof. The contract is the source of truth. The checker is the only merge gate. No green checker means no review yet. Step 1: Freeze one target file Pick one production file you can read. Do not pick a whole folder. Do not pick generated vendor code. git ls-files '*.py' '*.ts' '*.go' | head -n 40 TARGET = src/billing/invoice.py wc -l " $TARGET " git log -n 5 --oneline -- " $TARGET " Read the last five commits on that file. Write two plain sentences in notes. Pick a smaller file if you cannot yet. You now have a hard fence. Everything outside that fence is forbidden. Your assistant may not cross it. Step 2: Write the blast-radius contract Create blast_radius.py at the repo root. Keep the dict small. Fill every field with your own hands. # blast_radius.py # Day-one contract. Humans edit this. Models do not. BLAST = { " change_id " : " day-one-001 " , " intent " : " Add a fail-closed guard on invoice totals. " , " target_files " : [ " src/billing/invoice.py " , " tests/billing/test_invoice_flag_off.py " , " blast_radius.py " , " scr

A green required check is not a merge decision. It is a signal that one job graph finished without a red X. If a pre-push hook was skipped, or a snapshot fixture was regenerated without a reason, you can still ship a lie. This article walks through a merge packet: a small JSON artifact your CI publishes next to the check. The packet records hook results, fixture drift, and required-job status. A model may write the eight-line brief. It does not get a vote. Why green still lies CI dashboards collapse many facts into one glyph. You see green. You click merge. You miss three common failures. First, someone pushed with --no-verify and skipped the hook that keeps fixture hashes honest. Second, a test helper rewrote golden files because a serializer added a field. Third, a retry job went green on the second attempt and nobody recorded that the first attempt failed. You do not need a platform rewrite to catch this. You need a contract the merge button cannot ignore. Cheap code generation makes the second failure more common. When it is easy to regenerate tests, it is easy to regenerate the fixtures those tests pin. The pin becomes a moving target. Treat unexplained fixture diffs as merge blockers, the same way you treat a failed unit job. What the merge packet contains Keep the packet boring. One file. One schema. Commit it as a CI artifact, not as a comment that can be edited after the fact. { "commit" : "REPLACE_WITH_SHA" , "generated_at" : "2026-09-03T00:00:00Z" , "hooks" : { "pre_push_fixture_guard" : "passed" }, "fixtures" : { "manifest_path" : "tests/fixtures.sha256" , "changed_paths" : [], "unexplained_paths" : [] }, "required_jobs" : [ { "name" : "unit" , "conclusion" : "success" }, { "name" : "contract" , "conclusion" : "success" } ], "merge_ready" : false , "brief" : null } merge_ready is computed by a script you own. Not by a prompt. The brief is optional prose for humans who will not open the JSON. Step 1: Pin fixtures with a manifest Pick a directory you alrea

There is a category of defect that cannot appear in your acceptance criteria. Not because nobody thought of it, but because the shape of a requirement has no room for it. A requirement describes a state and a rule. A customer can apply a valid promo code at checkout. State: the code is valid. Rule: it is accepted. Both are evaluated at a single instant, because a sentence has one tense. Real systems do not have one instant. They have two, and sometimes a lot more. The gap between checking and using Take that promo code. The system validates it when the customer types it into the basket. The system commits it when the customer pays. Between those two events sits an unbounded amount of time — thirty seconds if they have their card handy, three days if they leave the tab open on a laptop lid. If the code expires in that gap, what happens? The requirement cannot tell you. It never contemplated a gap, because it was written as one sentence about one moment. And a test written by hand almost certainly cannot tell you either, because a person writing a test naturally writes it the way they would perform it: enter code, assert accepted, pay, assert charged. Three lines, one instant, no gap. This is time-of-check to time-of-use. Most developers first meet it as a security problem — access() then open() , and a symlink swapped in between. The same shape appears at business timescale, and there it is far more common and far less discussed: Stock is reserved at basket, decremented at dispatch. Someone else buys the last one. A permission is checked when the page loads, enforced when the action fires. The role changed. A price is quoted at quote time, charged at renewal. The tariff moved. A rate limit is checked at admission, consumed at execution. The window rolled over. A feature flag is read at session start, branched on at submit. Someone flipped it. A token is validated at the gateway, used by a downstream call. It expired in flight. Every one of those is a real defect clas

Playwright test data is the set of database rows or API records your application needs to already contain before a browser test runs against it — a logged-in user, their orders, the products those orders reference — generated deterministically so the same run produces the same data every time. Unlike unit tests, a Playwright (or Cypress) spec drives a real browser against a real, running app, which means the backend behind it needs real rows to serve, not an intercepted network response. Getting that data right, and getting it there before the first test starts, is most of what makes a browser E2E suite fast and non-flaky instead of slow and order-dependent. Why is E2E test data hard to manage? Three patterns keep showing up, and each causes a different failure mode: Tests create their own data through the UI. A test that needs an order to exist first signs up a user, logs in, adds a product to a cart, and checks out — all before the actual assertion it cares about. That's slow multiplied across every spec that needs similar setup, and it means the thing under test (the UI) is also the thing doing the setup, so a bug in signup breaks fifty unrelated tests. A shared, mutable test database. If every spec reads and writes the same rows, test order starts to matter: a test that deletes a user breaks a later test that assumed that user still exists. This is one of the most common sources of a suite that passes locally, one file at a time, and fails intermittently in CI when specs run in parallel or in a different order. Hand-maintained fixture SQL or JSON. A fixtures.sql file or a static users.json works until the schema changes — a column gets renamed, a new required field is added — and the fixture silently stops matching what the app expects, or starts failing inserts with no clear signal about which of forty rows is the problem. The fix for all three is the same shape: generate the data the suite needs from a definition (a template), with a fixed seed, right before t

An AI coding agent working inside Claude Desktop or Cursor can read your code, write new files, and run your test suite — but it can't open a browser, log into a dashboard, and click "generate" to get a batch of realistic test data. It has no hands for a UI. AI agent test data generation only works if there's something the agent can call : a tool with a defined schema it can invoke mid-session, the same way it calls a file-write or a shell command. That's exactly what the Model Context Protocol (MCP) is for, and it's why we shipped @jsonfabrica/mcp-server on npm. What AI agent test data generation requires over MCP MCP lets an AI client — Claude Desktop, Cursor, or anything else that speaks the protocol — launch a small local server over stdio and treat its exposed functions as tools it can call during a conversation. The agent decides when to call jsonfabrica_generate_from_template the same way it decides when to call read_file . For that to work, three things have to exist: a server process the client can start, a set of tool definitions with typed inputs and outputs, and — underneath all of it — some actual operation the tool call triggers. MCP server test data generation is that last piece: the tool call has to result in real, schema-conformant data coming back, not a stub. @jsonfabrica/mcp-server , concretely We published @jsonfabrica/mcp-server v0.1.1 as a local MCP server: the AI client launches it itself over stdio, no separate process to manage, no port to open. It exposes the JsonFabrica gateway as a set of MCP tools — jsonfabrica_create_template , jsonfabrica_generate_from_template , jsonfabrica_generate_adhoc , jsonfabrica_create_batch , jsonfabrica_create_sequence , and more. Mid-session, an agent can create a template matching the shape of your User or Order model, generate a batch of realistic records against it, and drop the result straight into a fixture file or a seed script — without you leaving the editor to go configure anything by hand. Why thi

If you're generating test data today, you've probably landed on one of three approaches: click through a UI like Mockaroo, pull in a library like Faker.js and write generation code yourself, or call a hosted API like JsonFabrica. Comparing these test data generation tools side by side, the real differences aren't about which one produces "better" fake data — Faker.js, Mockaroo, and JsonFabrica are all capable of that. The differences are about where the tool lives, how it handles relationships between records, and who's responsible for running it. Three test data generation tools compared, shape by shape Mockaroo is a browser-based UI: you define columns and types through a web form, preview rows, and export a file — or hit its API directly, which is available even on the free tier (paid tiers raise the volume ceiling rather than gate API access itself). Faker.js is a JavaScript library: you import it into your own code and call functions like faker.person.fullName() or faker.internet.email() to build up objects yourself, one field at a time. JsonFabrica is an API-first hosted service: you send a schema (or use a template) to an endpoint and get structured, schema-conformant JSON back, with no UI step and no library to install in your own codebase. That distinction matters more than it sounds. A UI tool is something a person operates by hand. A library is something a developer owns and maintains inside their own project — you write the loops, the relationships, the edge cases. An API-first tool is infrastructure: something your CI pipeline, your seed script, or an AI coding agent can call directly, without a human in the loop or generation logic living in your repo. UI vs. library vs. API, in practice Mockaroo's UI is genuinely fast for a one-off task — sketch a schema, click generate, download a CSV or JSON file. What it isn't built for is wiring generation into an automated pipeline where nobody is clicking anything. Its API can cover that, but at free-tier volume

Plenty of test data tools are built as a UI first and an API second, if there's an API at all. You open a dashboard, configure some fields, click "generate," and download a file. That works fine for a one-off demo. It falls apart the moment test data generation needs to be part of your actual engineering workflow — running in CI, seeding a database on every branch, or producing ten thousand records instead of ten. That's the case for a test data generation API over a click-driven dashboard: the primary interface is a request you can make from code, and everything else — a UI, a CLI — is built on top of that same API. Automation and CI integration A UI is something a person operates. CI doesn't have a person sitting at it. If test data generation only exists behind a login screen and a click, it can't run as a step in your pipeline — someone has to generate the data ahead of time, commit it, and hope it doesn't drift from what the tests actually need. An API-first tool is just another HTTP call your pipeline makes: fetch fresh, schema-conformant data as part of the build, every run, with no manual step in between. Scriptability — no clicking required Generating test data through a UI means clicking through the same sequence of dropdowns and fields every time you need a new batch. That's tedious for one dataset and untenable for the dozens of shapes a real test suite needs — different entity types, different edge cases, different volumes. An API call is a script. Write it once, parametrize it, and reuse it for every collection you need, without a human repeating the same clicks. Wiring a test data generation API into pipelines and seed scripts Seed scripts are code that runs at a specific point in a workflow — before a test suite, on container startup, in a migration. They need a function call or an HTTP request they can invoke programmatically, not a browser tab. With a test data generation API, "seed the dev database with realistic orders" is a line in a setup scrip

An agent patch can pass every unit test and still write outside the workspace, call an undeclared tool, or read an env key the task never named. Gate the blast radius first. Score the prose later. This article is a method, not a field report. It proposes a fail-closed envelope around filesystem roots, tool names, environment keys, and network hosts. Side-effect violations never freeze. Only a dual-runner disagreement on a non-envelope property may freeze, and only with a hashed evidence bundle. The conclusion in one rule Treat an agent patch as a capability change. If the run touches anything outside a declared envelope, the gate fails closed. Flakes in ranking, wording, or latency do not override that rule. Cheap generation does not make side effects cheap to reverse. A green suite that never watched /tmp , os.environ , or outbound sockets is not a verification result. It is a missing observer. What this gate is not It is not a golden-file of model text. It is not a mutation score. It is not a full-suite rerun after every hunk. It answers four questions only: Did the run write or delete outside allowed roots? Did it invoke a tool name that is not on the allowlist? Did it read an environment key that is not on the allowlist? Did it open a network host that is not on the allowlist? If any answer is yes, fail. Do not freeze. Do not retry for luck. Artifact: a locked envelope and an observer log Pin the envelope as a fixture. Hash it. Refuse to run if the hash drifts without a review note. { "envelope_id" : "agent-patch-envelope-v3" , "allowed_roots" : [ "/work/repo" , "/tmp/agent-scratch" ], "allowed_tools" : [ "read_file" , "apply_patch" , "run_tests" ], "allowed_env" : [ "CI" , "RUN_ID" , "ENVELOPE_HASH" ], "allowed_hosts" : [], "network" : "deny" } sha256sum envelope.json > envelope.json.sha256 # CI must compare this digest before the agent process starts. Label the next block as a proposed harness, not a production sandbox. User-space tracing will miss kernel-leve

Have you ever watched an LLM endpoint return a clean answer that belonged to a different prompt entirely? I spent forty-eight hours blaming sampling noise, temperature, and a free model that would not sit still. The request logs looked honest enough, and the health check on the box stayed green the whole time. The bug was quieter than that: a cache key that hashed the user message and ignored everything else that actually changes a completion. I was trying to keep a small eval loop cheap, which is a very ordinary instinct. Free-model access is useful when you want overnight volume without treating every call as precious. I parked a thin HTTP wrapper on a free server, hashed each prompt, and stored the JSON body on disk so retries would not hammer the model. Does that sound reasonable? It did, until two different system prompts started colliding on the same key and I spent a day chasing "nondeterminism" that was just a hash. I ran that wrapper against MonkeyCode's free model access on the free server option because I wanted a boring place to reproduce the cache bug, not a production SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing below depends on a named model, a quota, or a hardware claim. The lesson is the key function, and it still applies if you delete the product name from the stack. What I walked into The wrapper looked like every weekend cache I have written under time pressure. Incoming POST bodies were reduced to user_message , run through hashlib.sha256 , and written under ./cache/<hex>.json . A hit returned the file. A miss called the model, then wrote the file. I even logged X-Cache: HIT so future-me would feel scientific. That design has one attractive property and one fatal one. The attractive property is that identical user text becomes free after the first call. The fatal property is that user text is not the request. System prompt, temperature, stop sequences, tool schemas, and even a date injected into th

A recorded expected value is a leak. An agent that can read assert f(x) == y can patch f until that line is green and leave every unlisted input broken. A metamorphic relation does not publish y . It only publishes a constraint the output must keep under a known transform. That is the gate worth automating. Fixtures still matter, but only as seeds. Flaky tests still need a freeze, but the freeze must not cover the relation itself. This article is a proposed layout, not a production case study. No runtime metrics are claimed. The commands and modules below are labeled so they can be copied into a scratch repo and executed against your own function under test. Why snapshots fail as a merge gate Golden files encode one transcript. An agent patch is a search over many transcripts. If the search can see the answer key, the cheapest passing program is a lookup table for the keys in tree. That program is green. It is also wrong on the next customer file. Property-style checks reduce that leak because they do not ship the answer. They still need a seed corpus, a replay runner that the patch cannot edit, and a quarantine file that expires. Mix those three and you get a gate that fails closed when the agent rewrites tests, when a fixture drifts, or when a flake is used to hide a broken invariant. Three relation classes worth encoding first Start with relations you can state in one line. If you cannot state the line, you do not have a gate. You have a recorder. Idempotence. f(f(x)) == f(x) for normalizers, formatters, and canonicalizers. Round-trip. parse(serialize(x)) equals x on the fields you actually guarantee, not on whitespace you do not. Oracle-free comparison. f(t(x)) relates to t(f(x)) for a transform t you control: shuffle independent rows, rename equivalent keys, NFC vs NFD unicode, scale a quantity and its unit together. These are not universal laws. They are hypotheses about your function. Write them down as code. Keep the seed inputs boring. The relation, not the

An agent patch is only as trustworthy as the checks it cannot rewrite. If properties, fixtures, and flake policy live in the same tree as src/ , the diff can weaken the proof. Move the oracle out of the writable tree and run it as a control loop with hysteresis, not as a skip list. Co-located tests fail this requirement in a predictable way. The agent adds an assertion that matches the new code. A fixture grows a default that hides a broken parser. A flaky case becomes skip . The suite stays green. Production still drifts. This article proposes a sidecar oracle: human-owned properties, sealed fixtures, and a two-threshold flake freeze. The design is a workflow, not a production case study. Treat the code as a proposed runner you can execute locally, not as a claim about a live fleet. What the loop decides The loop answers three questions on every candidate patch: Do independent properties still hold on generated inputs? Did the patch mutate a sealed fixture or depend on an unsealed one? Is a failing test a regression, or does it belong in a measured freeze? A skip list answers none of those. It only records that someone got tired of a red job. Layout: oracle beside the repo, not inside the diff Keep the application repo writable for the agent. Keep the oracle in a second directory that the agent cannot include in its patch. app/ # agent may write src/, not oracle paths src/ pyproject.toml oracle/ # human-owned; hashed before every gate properties/ test_invariants.py fixtures/ manifest.json http_empty_body.json flake_ledger.json path_deny.txt run_gate.py path_deny.txt is the first control, not the last. If the patch touches oracle files, tests the agent authored, or lockfiles it did not need, the gate fails before pytest starts. # oracle/path_deny.txt oracle/ **/test_*.py **/*_test.py **/conftest.py **/__snapshots__/ The deny list is deliberately blunt. Agent-authored tests can still exist as scratch. They do not count as evidence. Step 1 — Hash the oracle before the