标签：#testing

My harness flagged the model for sending the wrong arguments. It compared what the model actually...

Run this against the real, published image and watch it fail: docker run --rm --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --version Exit code 127. Not a crash inside trelix-mcp, not a stack trace, not a permissions error — 127 is the shell's own way of saying the binary you asked for does not exist. And it didn't. The console script trelix-mcp is supposed to install as part of every trelix package was simply absent from the image, on both the slim tag and the -local tag, for the entire life of the 3.2.1 release. Every unit test in the suite was green. Every line of source that builds trelix-mcp was correct. The thing a user would actually get from docker pull did not have the binary its own --version flag implies exists. This article covers four releases — v3.2.2, v3.2.3, v3.2.4, and v3.2.5 — spanning 173 commits and 88 changed files since v3.2.1, which is where the last article in this series left off. That one was about tests that pass without exercising the code they claim to cover: a MagicMock standing in for a real embedder, an all-ones attention mask that makes masked and unmasked math identical, a unit test that asserted a bug as its own specification. This one, on the heels of the mutation-testing push that closed out that arc, is about a different and in some ways more uncomfortable failure mode: tests that pass while exercising the wrong artifact entirely. A green pytest run against src/ says nothing about whether the wheel on PyPI, the image on GHCR, or the binary on the GitHub Releases page actually does what it claims. Those are three separate build products, built by three separate pipelines, and none of trelix's 4,353 collected unit tests had ever touched any of them directly. v3.2.2 through v3.2.4 is the story of finding that gap and closing it with an actual gate, not a promise to be more careful next time. v3.2.5 is a short postscript proving the discipline stuck. The Docker image that shipped without its own server The 127 above wasn't

In December 2024, Google quietly updated its closed testing rules for personal developer Console accounts. For months, indie developers had to recruit at least 20 testers to keep their app opted in for 14 consecutive days before applying for production access. Under the revised guidelines, that threshold dropped from 20 to 12 testers. Understanding the nuances of the Google Play 20 testers vs 12 testers shift helps you plan your release schedule accurately without running into unexpected delays during Google Play Console verification. While lowering the number by eight testers sounds like a major relief, the core requirements behind closed testing have not changed. Google still enforces a strict 14 consecutive day duration, and the Play Console continues to monitor tester retention and engagement. A lower numerical requirement means less logistical hassle, but maintaining a stable group of committed testers remains the primary hurdle for independent developers. The Policy Shift: From 20 to 12 Testers Google originally introduced mandatory closed testing in November 2023 to improve app quality and curb low-effort submissions on the Play Store. Initially, all new personal accounts registered on or after November 13, 2023, were required to run a closed test with at least 20 opted-in testers for 14 days without interruption. After roughly a year of developer feedback regarding how difficult it was for solo creators to find 20 reliable participants, Google reduced the requirement to 12 testers in December 2024. It is crucial to understand who this rule applies to. The requirement exclusively targets personal developer accounts created on or after November 13, 2023. If you operate an organization or business developer account, or if your personal account was registered before November 13, 2023, you are currently exempt from this mandatory closed testing gate. However, if you fall under the new personal account category, reaching 12 continuous opt-ins is a strict prerequis

Struct tags are just strings — a typo'd checker name, a wrong-typed field, or a renamed cross-field target all compile fine and fail silently at runtime. checkerlint catches all three before you ship. Struct tags are string literals. The Go compiler checks that your struct compiles — it has no idea what checkers:"eq-field:Passwrd" means, so a typo in a field name, a checker applied to a field of the wrong type, or a renamed field that a cross-field rule still points at all compile fine. They fail later, at runtime, sometimes silently, sometimes as a panic in the middle of handling a request. type Registration struct { Password string `checkers:"trim required"` ConfirmPassword string `checkers:"required eq-field:Passwrd"` // typo: no such field Age int `checkers:"email"` // email is string-only } Nothing here trips go build , go vet , or a normal linter — they all treat checkers:"..." as an opaque string. The first bug only surfaces the moment someone submits a registration form and eq-field can't find a field called Passwrd . The second is worse: email assumes a string under the hood, so calling it on an int field panics at validation time instead of returning a normal error. checkerlint is a go/analysis -based static analyzer, shipped as its own module in the Checker repo, that reads these tags at build/lint time and catches exactly this class of bug before it ships: ./registration.go:3:2: checkerlint: eq-field references field "Passwrd", which doesn't exist on this struct ./registration.go:4:2: checkerlint: email requires a string, but the field's type is int What it actually checks Three things, all specific to how checkers / validate tags can go wrong: Unknown checker names. Every token in the tag has to be a registered checker, normalizer, field-relative checker, omitempty , or a name your own code registered via RegisterMaker / RegisterFieldMaker with a string literal. Typo requird instead of required and checkerlint flags it — nothing else in your toolchain w

My puzzle app, Keyhole, carries 296 dark stories, each with an illustrated card. A dark story is a situation that looks impossible until you drop one false assumption you did not know you were making, and the illustration must show the situation and never the reveal. Draw the aeroplane over the desert and story one is over before the player has read it. In August I ruled that the app does not ship while any card is still flagged by the judge. "End of story," I wrote in the decision, and then spent two days learning what that sentence cost. Two things get judged, the text and the art, and one design is shared by both. The judge is a model, run blind: it sees the finished card and the story the player sees, and neither the finding that triggered the redraw nor the old card. That is the whole trick. A judge that knows what was wrong last time grades the fix. A judge that knows nothing grades the card. Blindness is what makes a pass mean something, and it is why the judge is a separate call from the writer and from the illustrator, never the same conversation. The text pass first. A rubric written for the genre, with one test at its centre, "name the one assumption the solver will make that is false", and four semantic questions after it: does the reveal explain everything the situation promised, does the situation give the reveal away, is there a contradiction, can the answer be reached by yes/no questions without knowledge nobody has. Over all 296 stories it flagged 27: five unanswered, nine spoilers, ten sense breaks, three unsolvable. The fix lane rewrites only what a finding names, the deterministic gate must still pass, and the blind judge reads the result cold before it is written back. A fact-check over the rewrites then cleared them, or left a truth note where no honest fix existed. The art pass is where the numbers live. Each open card was redrawn from a scene brief and judged blind, in waves. The judge wrote a note on every failure, and the lever changed from

1. Código de infraestrutura também quebra Nos dois artigos anteriores desta série, vimos o OpenTofu como alternativa para provisionar infraestrutura e o Ansible para configurá-la depois de criada. Mas há uma pergunta que fica no ar em qualquer um desses fluxos: como saber, antes de rodar apply em produção, que um módulo Terraform não vai abrir uma porta que não deveria, destruir um recurso por engano, ou simplesmente ter um erro de sintaxe? Testar infraestrutura como código é tão importante quanto testar qualquer outro software — só que, diferente de uma função pura, os "efeitos colaterais" de um teste malfeito aqui podem ser uma conta de nuvem inesperada ou um serviço em produção fora do ar. Este artigo fecha a série cobrindo três camadas complementares de teste: análise estática com tflint , verificação de segurança e compliance com checkov , e testes de integração de verdade com Terratest . 2. As camadas de teste em IaC Vale pensar nessas ferramentas como camadas que rodam em momentos diferentes do ciclo de vida do código, da mais rápida/barata para a mais lenta/cara: Lint e análise estática (tflint): roda em segundos, sem precisar de credenciais de nuvem nem de rodar terraform plan . Pega erros de sintaxe, más práticas e problemas específicos de cada provider. Análise de segurança e compliance (checkov): também estática, mas focada em identificar configurações inseguras (bucket público, criptografia desabilitada, security group aberto para 0.0.0.0/0 ) comparando o código contra um catálogo de políticas. Testes de integração (Terratest): a camada mais próxima da realidade — de fato roda terraform apply num ambiente isolado, valida o resultado, e depois roda terraform destroy . Mais lento e mais caro (usa recursos reais de nuvem), mas é o único jeito de garantir que o módulo realmente funciona de ponta a ponta. Um pipeline de CI/CD maduro roda as três, nessa ordem, falhando rápido nas camadas mais baratas antes de chegar nas mais caras. 3. tflint na prática O tfli

A green test run is not a behavior spec. An agent patch can keep every existing assertion passing and still change encodings, error types, empty-input handling, or the bytes written to stdout. Shadow-compare the candidate against a frozen baseline on the same corpus. Merge only after every divergence is classified in an accepted-delta ledger. This article is a testing workflow, not a model bake-off. The harness below is labeled as a proposed, runnable pattern. It does not claim production timings, model names, or pass rates. Why green CI misses the patch Agent patches optimize for the tests they can see. Hidden behavior lives in branches the suite never names: trailing newlines, NaN keys, timezone-naive stamps, None versus [] . Those are cheap to alter. They are expensive to notice after merge. A dual-run gate treats the old artifact as the oracle for unspecified behavior. Specified behavior still belongs in ordinary tests. The ledger exists for the remainder: diffs you accept on purpose, and diffs you refuse. Do not use this as a substitute for code review. Use it as a filter that review should not have to do by hand. Artifact: baseline, candidate, ledger Three files define the contract. baseline/ — a pinned checkout, wheel, or container digest. Not main at HEAD. candidate/ — the agent patch, applied on top of the same pin. delta_ledger.yaml — every previously classified output divergence, keyed by fixture id. Proposed layout: shadow/ corpus/ # deterministic fixtures only 001_empty.json 002_unicode.json 003_nested_null.json delta_ledger.yaml canonicalize.py shadow_compare.py The corpus must be I/O-free. No clocks. No DNS. No home-directory probes. If a fixture needs time, inject it. If it needs a filesystem, pass a temp root the harness owns. Step 1 — Freeze the baseline as an artifact Record the exact bytes you will rerun. A git SHA is enough when the tree is hermetic. Prefer a built artifact when native extensions or generated code are in play. git rev-parse HEAD

A green test suite is evidence. It is not independent evidence. The current release of badBANANA Threat Observatory passes all 108 automated tests in its own development and CI environments. That tells me the implementation satisfies the assertions I wrote against the conditions I expected. It does not tell me whether an independent developer can check out the same commit in a clean environment and obtain the same result. That distinction matters more than the number 108. The dangerous failure is a believable one The Observatory presents source-backed threat-intelligence records, freshness information, and material-change events. In that kind of interface, an obvious crash is not necessarily the worst outcome. A more dangerous failure is one that looks healthy: An expired cached snapshot presented as current An invalid expiry value treated as usable A failed upstream source displayed as a successful zero-result response Demo or fallback data appearing without explicit disclosure A disabled or offline state silently normalized into success Those failures do not merely inconvenience the user. They change what the interface appears to know. For v1.2.2, the intended behavior is deliberately fail\ closed: Condition Required behavior Cached snapshot has expired Report it as stale Expiry value is invalid Fail closed to stale Source is offline, disabled, or failed Preserve that state Source data is missing or unavailable Do not present a successful zero result state Ingestion requests overlap Enforce the runtime concurrency limit deterministically Feed credentials are configured Keep them server side and absent from client output The test suite exercises these boundaries. The remaining question is whether the release reproduces cleanly outside the environment in which it was built. Passing tests and independent verification are different claims When the source, tests, build assumptions, and execution environment all come from the same maintainer, a successful run demonstrat

Agents fail quietly when they fill omitted tool arguments instead of refusing, and fluency-based evals often reward that invention. A compact negative golden set, scored by argument-diff rather than prose quality, catches those silent substitutions before they reach production traces. This article treats that failure as a testable contract, not as a prompt-tuning anecdote, and it stays useful without any vendor product. Recent developer discussion around agent workflows keeps returning to one operational surprise that chat logs tend to hide. Models do not only choose the wrong tool; they complete incomplete requests by guessing identifiers, dates, and scopes that nobody supplied. That behavior looks like initiative in a chat log, yet it resembles a clerk forging a zip code to stamp the form complete. The package then leaves the dock with valid-looking paperwork and the wrong city printed on the label. A conventional golden-answer harness scores the final sentence, which is the wrong surface for tool-using agents. The dangerous artifact is the tool payload, because downstream systems will execute invented primary keys with perfect syntax. If your eval suite only checks that a transfer looks helpful, it will greenlight a call that moved the wrong account. The pattern below is a proposal you can run locally, and it does not claim production metrics. It also does not depend on a particular model family or on a hosted evaluation service. You should treat every numeric threshold in the grader as a starting point rather than a published benchmark. Negative goldens assert a hole, not a pretty answer A positive golden case says the model should produce a known good action given a complete request. A negative golden case says the opposite: given a hole in the input, the model must not paper over that hole. The assertion is closer to a check constraint than to a writing rubric, because the failure is an illegal completion. Fluency still matters for users, but it is a poor prox

I was asked whether the system was sending patient data to an external body while the integration was half-built. I went and read the logs of every run. They all died early: some with a 415 because the content type wasn't what the other end expected, others with a 500. Not one showed an outbound call. I answered that nothing was going out. The first run that got past the 500 sent two requests carrying real clinical data . My answer had been false from the start, and the worst part is that it was false in a way that felt rigorous: I had looked. I had evidence. The evidence was logs of real executions, not assumptions. A negative says nothing on its own The mistake wasn't misreading the logs. It was not noticing what produced that silence. The runs died before reaching the code that sends. The log didn't say "I didn't send"; it said "I never got to the part that sends". Those are two different statements and they produce exactly the same output: nothing. That's the general shape of the problem, and it turns up everywhere once you look for it: A counter at zero can mean "it didn't happen" or "the counter was never incremented". A "not found" can mean "it doesn't exist" or "I looked in the wrong place". A green test can mean "it passed" or "it skipped itself". An exit 0 can mean "it worked" or "the command was strangled by a pipe that swallowed the exit code". A silent dashboard can mean "everything is fine" or "the process feeding it has been dead for three weeks". In all five, the evidence is identical. And in all five, the optimistic reading is the reassuring one, so it's the one chosen without thinking. The positive control The fix isn't to be more suspicious. It's to demand one specific thing before accepting any negative: Find something the log MUST show if the path was actually taken. If the system had reached the part that sends, something would have to appear in the log: the "preparing request" line, the batch identifier, the connection attempt. Any signal that

On 2026-09-04 I pointed a scanner at langchain-ai/langchain . Shallow clone of the default branch, HEAD 79cab2d , read only. It walked 2581 files and printed zero sites. Its own control had passed immediately before the run, with two positive fixtures seen and four negative fixtures clean, so the zero was a measurement rather than a crash. Then I opened one file by hand. libs/langchain_v1/langchain/agents/middleware/human_in_the_loop.py , line 403: def _should_interrupt ( self , tool_call , config , state , runtime ) -> bool : """ Return False if the `when` predicate rejects this tool call, True otherwise. """ when = config . get ( " when " ) if when is None : return True ... return when ( req ) when is supplied by the caller. It is declared NotRequired[Callable[[ToolCallRequest], bool]] on line 195 and documented as returning True to interrupt or False to auto-approve. Its result is handed back unchanged. A predicate that falls off a branch returns None , and the caller on line 436 reads: if not self . _should_interrupt ( tool_call , config , state , runtime ): continue None is falsy. The interrupt is skipped and the tool call proceeds with nobody looking at it. The annotation says bool ; nothing at runtime makes that true. Why the machine stayed quiet I took the failure apart instead of guessing at it. Three causes, each sufficient on its own: Vocabulary. 22 lines in that file matched the approval vocabulary the scanner looks for. Not one of them put line 403 inside its window. The nearest match was 26 lines away and sat in a comment. This project calls the decision interrupt , not approval. Window. The -> bool annotation is on line 378. The return is on 403. That is 25 lines apart, and the window was 12. Signals. Widened to 55 lines, the three behaviour signals still matched nothing on that line. The file walk was innocent. The file is .py , 18256 bytes, and no skip rule matched it. It was read. What I got wrong The window of 12 lines had no measurement behind it

Every AI code review metric you track measures the hour before merge, and that is precisely the hour when the least information exists. Acceptance rate, test pass rate, and review approval all describe how a patch looked in isolation, not how it behaves under real traffic. Revert rate is the only signal that arrives after the system has voted, which makes it the least gameable number in your pipeline. This article argues that you should stop celebrating AI patch acceptance and start measuring how many of those patches come back. Why the pre-merge metrics lie A green test run proves that a patch fits the expectations you encoded last quarter, not the behavior your users will hit tomorrow. Reviewers approve diffs under time pressure, and a cleanly formatted AI patch reads as competence even when its logic is wrong. The merge is where the real evaluation begins, and the revert is the only verdict that carries operational weight. Nobody plans a revert, so the metric cannot be gamed by prompt tweaks or review theater. The argument is not that pre-merge review is useless; it is that pre-merge signals saturate quickly. Once your review gate catches the obvious failures, the remaining defects are exactly the ones that look fine in review. Those defects surface as incidents, hotfixes, and reverts, which means your post-merge telemetry is the only source of new information. Treating acceptance as a quality metric is like judging a deployment by how well the rollout script ran. The artifact: a revert attribution watch The workflow below attributes every revert commit to the patch that caused it and computes a per-source revert rate. It requires only a git history, which makes it reproducible on any repository that has survived a few incidents. Run it on a local clone first, because a read-only analysis should never touch shared state. Step 1: List every revert commit in your window. git log --all --since = "90 days ago" --grep = "^Revert " --format = "%H %s" Step 2: Extract th

AI agents are easy to demo and surprisingly hard to evaluate. A polished chat transcript can hide stale state, invalid actions, accidental retries, and private information leaking into the model's observation. I built WagerCall as a bounded environment for studying those problems. Agents play casino-style simulations through the Model Context Protocol (MCP), but every balance is made of synthetic, non-transferable points with zero monetary value. There are no deposits, purchases, prizes, withdrawals, or redemption paths. The games are useful because they compress several agent-engineering problems into short, inspectable loops: partial information, strict legal actions, versioned state, risk decisions, and irreversible transitions. Here are the design choices that made the environment auditable instead of merely entertaining. 1. Bound the world before evaluating the agent An evaluation environment should say exactly what an agent can observe and change. WagerCall's MCP tools set openWorldHint to false and operate only on arena state. The agent cannot call a generic SQL, admin, execute, or debug tool. That boundary matters. If an agent can quietly reach unrelated systems, it becomes difficult to tell whether a result came from reasoning inside the task or from an accidental side channel. The same rule applies to the economy. Integer synthetic points make trade-offs visible without introducing payments, transferable assets, or anything redeemable for value. 2. Let pure game logic propose; let the database decide The game engine is deterministic and side-effect free. Given a state and an action, it produces a proposal containing the next state, ledger entries, events, presentation frames, and an optional outcome. A proposal is not yet a fact. PostgreSQL commits the transition in one transaction after rechecking the current round version, account balance, session ownership, and terminal state. It either writes the action, balance change, new round state, and audit event

Two instructions went into an Auditor my agent collaborators and I built to surface conflicts in...

A system has to run a chunk of code you don't control —a plugin, a dependency, something generated— and you want to guarantee that code cannot touch the file system or spawn processes. Not that it "shouldn't": that it can't , mechanically. That's capability confinement, and it's one of the central problems of runtime security. Designing it and demonstrating it are two different things, and confusing them is expensive. A design is a claim You can write an impeccable document: "access to fs and to spawning processes is controlled like this, with these mechanisms, under this threat model". It's real and necessary work. But it's a claim . And the failure mode of a security boundary is that it looks like it holds until it doesn't —silent, invisible in tests, visible only when someone crosses it—. In security, an unverified claim has exactly the shape of a beautiful, wrong architecture. The design can assume that a module-loader hook fires at a point where it actually doesn't, and all the reasoning hanging off that is correct and worth nothing. The mechanisms exist, and there are several In Node, to name a concrete runtime, there are at least three layers, and they aren't interchangeable: The native permission model ( --permission , --allow-fs-read …), which cuts access to fs and to spawning processes at the whole-process level. SES / Hardened JavaScript (Compartments, lockdown() ), which confines what each module can import within the process. Module-loader interception , which controls what resolves when the code asks for something. Choosing well among them is the design. But choosing well doesn't prove the choice holds against the real dependency tree you're going to run. Demonstrate instead of claim The alternative to signing off a design is delivering a confinement harness : untrusted code that tries to reach the dangerous capability —open a file, spawn a process— against the real runtime and its real dependency tree, and a log that shows each attempt was blocked . O

Refactoring Safely: A Step-by-Step Guide We all know that feeling: a function that's 200 lines long, a class that does too many things, or a variable named data2 . Refactoring is the cure, but doing it recklessly can break your app and your confidence. Here's how I approach refactoring safely, step by step. 1. Start with a Safety Net Before touching any code, make sure you have tests. If your project lacks tests, write a few key ones first. Focus on the behavior you're about to change. The goal is to have a safety net that tells you when you've broken something. # example test for a function we'll refactor import unittest from mymodule import calculate_total class TestCalculateTotal ( unittest . TestCase ): def test_with_discount ( self ): self . assertEqual ( calculate_total ( 100 , discount = 0.1 ), 90 ) If tests aren't feasible, at least have a manual checklist. But automated tests are worth the effort. 2. Make Small, Atomic Changes Don't try to refactor everything at once. Pick one logical change. For instance, extract a method or rename a variable. Each change should be small enough that if it breaks, you know exactly what caused it. // before function processOrder ( order ) { const total = order . items . reduce (( sum , item ) => sum + item . price , 0 ); const tax = total * 0.08 ; const final = total + tax ; return final ; } // after step 1: extract tax calculation function processOrder ( order ) { const total = order . items . reduce (( sum , item ) => sum + item . price , 0 ); const final = total + calculateTax ( total ); return final ; } function calculateTax ( amount ) { return amount * 0.08 ; } Run your tests after each tiny step. If they pass, move on. If they fail, you know the last change caused it. 3. Use Your IDE's Refactoring Tools Modern IDEs can rename variables, extract methods, and change signatures safely. They update all references automatically. This reduces human error. For example, in VS Code, right-click a function and choose "Extract to

Your first AI patch should fail closed today. Do not ship a feature on day one. Prove one target file can revert cleanly now. You joined a messy repo this morning. The assistant wants a wide rewrite. Your job is a tiny reversible cut only. This drill gives you a blast-radius file first. You fill it before any model writes code. Then a short script checks the revert path. Why day-one AI diffs explode Cheap code is not cheap to unwind. One extra import can touch auth. One extra migration can lock deploys. You will not know the architecture yet. You also should not pretend otherwise. A blast-radius file makes unknowns explicit fast. If the file cannot name a revert, stop. You do not prompt for more code. You shrink the change until revert is boring. What you will build today You will add two artifacts on your branch. Keep both files in the first PR. blast_radius.py — the contract for this change. scripts/check_blast_radius.py — the fail-closed proof. The contract is the source of truth. The checker is the only merge gate. No green checker means no review yet. Step 1: Freeze one target file Pick one production file you can read. Do not pick a whole folder. Do not pick generated vendor code. git ls-files '*.py' '*.ts' '*.go' | head -n 40 TARGET = src/billing/invoice.py wc -l " $TARGET " git log -n 5 --oneline -- " $TARGET " Read the last five commits on that file. Write two plain sentences in notes. Pick a smaller file if you cannot yet. You now have a hard fence. Everything outside that fence is forbidden. Your assistant may not cross it. Step 2: Write the blast-radius contract Create blast_radius.py at the repo root. Keep the dict small. Fill every field with your own hands. # blast_radius.py # Day-one contract. Humans edit this. Models do not. BLAST = { " change_id " : " day-one-001 " , " intent " : " Add a fail-closed guard on invoice totals. " , " target_files " : [ " src/billing/invoice.py " , " tests/billing/test_invoice_flag_off.py " , " blast_radius.py " , " scr

A green required check is not a merge decision. It is a signal that one job graph finished without a red X. If a pre-push hook was skipped, or a snapshot fixture was regenerated without a reason, you can still ship a lie. This article walks through a merge packet: a small JSON artifact your CI publishes next to the check. The packet records hook results, fixture drift, and required-job status. A model may write the eight-line brief. It does not get a vote. Why green still lies CI dashboards collapse many facts into one glyph. You see green. You click merge. You miss three common failures. First, someone pushed with --no-verify and skipped the hook that keeps fixture hashes honest. Second, a test helper rewrote golden files because a serializer added a field. Third, a retry job went green on the second attempt and nobody recorded that the first attempt failed. You do not need a platform rewrite to catch this. You need a contract the merge button cannot ignore. Cheap code generation makes the second failure more common. When it is easy to regenerate tests, it is easy to regenerate the fixtures those tests pin. The pin becomes a moving target. Treat unexplained fixture diffs as merge blockers, the same way you treat a failed unit job. What the merge packet contains Keep the packet boring. One file. One schema. Commit it as a CI artifact, not as a comment that can be edited after the fact. { "commit" : "REPLACE_WITH_SHA" , "generated_at" : "2026-09-03T00:00:00Z" , "hooks" : { "pre_push_fixture_guard" : "passed" }, "fixtures" : { "manifest_path" : "tests/fixtures.sha256" , "changed_paths" : [], "unexplained_paths" : [] }, "required_jobs" : [ { "name" : "unit" , "conclusion" : "success" }, { "name" : "contract" , "conclusion" : "success" } ], "merge_ready" : false , "brief" : null } merge_ready is computed by a script you own. Not by a prompt. The brief is optional prose for humans who will not open the JSON. Step 1: Pin fixtures with a manifest Pick a directory you alrea

There is a category of defect that cannot appear in your acceptance criteria. Not because nobody thought of it, but because the shape of a requirement has no room for it. A requirement describes a state and a rule. A customer can apply a valid promo code at checkout. State: the code is valid. Rule: it is accepted. Both are evaluated at a single instant, because a sentence has one tense. Real systems do not have one instant. They have two, and sometimes a lot more. The gap between checking and using Take that promo code. The system validates it when the customer types it into the basket. The system commits it when the customer pays. Between those two events sits an unbounded amount of time — thirty seconds if they have their card handy, three days if they leave the tab open on a laptop lid. If the code expires in that gap, what happens? The requirement cannot tell you. It never contemplated a gap, because it was written as one sentence about one moment. And a test written by hand almost certainly cannot tell you either, because a person writing a test naturally writes it the way they would perform it: enter code, assert accepted, pay, assert charged. Three lines, one instant, no gap. This is time-of-check to time-of-use. Most developers first meet it as a security problem — access() then open() , and a symlink swapped in between. The same shape appears at business timescale, and there it is far more common and far less discussed: Stock is reserved at basket, decremented at dispatch. Someone else buys the last one. A permission is checked when the page loads, enforced when the action fires. The role changed. A price is quoted at quote time, charged at renewal. The tariff moved. A rate limit is checked at admission, consumed at execution. The window rolled over. A feature flag is read at session start, branched on at submit. Someone flipped it. A token is validated at the gateway, used by a downstream call. It expired in flight. Every one of those is a real defect clas

Playwright test data is the set of database rows or API records your application needs to already contain before a browser test runs against it — a logged-in user, their orders, the products those orders reference — generated deterministically so the same run produces the same data every time. Unlike unit tests, a Playwright (or Cypress) spec drives a real browser against a real, running app, which means the backend behind it needs real rows to serve, not an intercepted network response. Getting that data right, and getting it there before the first test starts, is most of what makes a browser E2E suite fast and non-flaky instead of slow and order-dependent. Why is E2E test data hard to manage? Three patterns keep showing up, and each causes a different failure mode: Tests create their own data through the UI. A test that needs an order to exist first signs up a user, logs in, adds a product to a cart, and checks out — all before the actual assertion it cares about. That's slow multiplied across every spec that needs similar setup, and it means the thing under test (the UI) is also the thing doing the setup, so a bug in signup breaks fifty unrelated tests. A shared, mutable test database. If every spec reads and writes the same rows, test order starts to matter: a test that deletes a user breaks a later test that assumed that user still exists. This is one of the most common sources of a suite that passes locally, one file at a time, and fails intermittently in CI when specs run in parallel or in a different order. Hand-maintained fixture SQL or JSON. A fixtures.sql file or a static users.json works until the schema changes — a column gets renamed, a new required field is added — and the fixture silently stops matching what the app expects, or starts failing inserts with no clear signal about which of forty rows is the problem. The fix for all three is the same shape: generate the data the suite needs from a definition (a template), with a fixed seed, right before t