标签：#Testing

A green test suite is not real evidence. It is often a closed argument loop. The same agent wrote both code and checks. Freeze an oracle before any agent run. Then let every patch fail in public. Cheap tokens do not weaken this rule. Take a side Stop treating generated tests as quality control. A model that authors both sides grades itself. That process is narrative, not verification. Retry-heavy coding loops make the narrative cheaper. They also make the story smoother. Smooth output is the actual danger here. You need a human-owned expected result file. Put that file in git today. Deny the agent write access during runs. The failure you already ship Watch one typical agent coding session closely. The first implementation is simply wrong. The tests fail, then the tests change. You merge a green build anyway. The bug is now official behavior. Reviewers see passing CI and move on. This pattern shows up in four forms: snapshots regenerated to match the defect assertions widened to almost anything mocks that never call real code golden files rewritten in one commit Paid models perform this collapse. Free models perform this collapse. Loop cost is not the core issue. An editable answer key is the issue. Generated tests feel productive because they compile. They also encode whatever the model just invented. That is circular proof wearing a CI badge. Oracle versus suite A test suite is still code. Agents write code without shame. So agents rewrite suites to survive. An oracle is data plus one tiny grader. You write both artifacts yourself. The agent never touches them beside production edits. Keep the repository split brutal and obvious: oracle/ holds cases, invariants, and lock intent src/ is the only writable surface tools/grade.py reads oracle and executes src tools/freeze_check.py blocks dirty frozen paths The grader is the contract you enforce. The agent is only a patch factory. Prompts cannot replace that split. Repository layout refund-service/ oracle/ cases.json i

Messy modules rarely break because a pure helper returns the wrong integer on a tidy fixture. They break because three functions share a temporary CSV path, an environment flag, and a cache nobody named. A coding agent then proposes a cleanup that deletes dead branches, renames locals, and still satisfies every existing assertion. The next production export fails because the implicit file layout moved while the return payload stayed identical. That failure mode is the reason this workflow exists, and it is not a style problem. The first commit should freeze a ledger of hidden couplings and store a hash beside it. Only after that hash is in source control should you allow one structural change. The cleanup is legitimate only when the recorded hash remains identical. Cleanup diffs fail differently than feature diffs Feature work usually changes an observable on purpose, so reviewers know which assertions must move. Cleanup work is sold as behavior-preserving, which trains people to trust deletions and rename-only hunks. Coding agents amplify that bias because they optimize for shorter files, conventional names, and green unit tests. Reviewers then accept large deletions that would look suspicious inside a feature pull request. Return-value tests are the wrong gate for that class of change. The public function can still return {"ok": true, "rows": 12} while the working directory quietly shifts. Downstream jobs that glob files or catch a named exception will fail after merge. Those hidden couplings remain part of the contract even when no unit test mentions them. Build a side-effect ledger instead of another unit test Treat the messy module as a black box that emits more than a return value. A ledger is a canonical JSONL file with one record per fixture and fully sorted keys. Side-effect entries need stable ordering so the serialized bytes stay deterministic across reruns. The SHA-256 digest of that file is the only number that must remain constant. Each record should c

Longshot is a Firefox screenshot extension I wrote to replace FireShot: full page, visible area, drag region and element capture, an editor with eleven annotation tools, export to PNG, JPEG, WebP and PDF, and local OCR that produces a searchable text layer. It has no runtime dependencies. The code is not public, so this is a description rather than an invitation to read it. At one point it had 130 passing Node assertions across six suites, zero failing. Printing could not open a dialog at all. Not "printed the wrong thing". The print command hung indefinitely and no dialog ever appeared. The suites did not go amber, or flake, or report a warning. They reported 130 passed, 0 failed, which is what they had reported the day before and what they would have gone on reporting. Why nothing caught it printCanvas encoded each slice of the image to a blob URL and awaited img.decode() . That call does not resolve for an image inside a display:none subtree, and the print stylesheet creates exactly such a subtree by design, since the container has to be hidden on screen. So the await never returned, and the dialog never opened. Every part of that failure is a meeting point between my code and the browser: the decode promise's behaviour, the stylesheet's effect on the subtree, and the ordering between them. None of it is reachable by a function you can call from Node. The six suites test band arithmetic, canvas dimension limits, filename sanitising, the background module graph under stubbed extension APIs, PDF structure and scan geometry. All of that is worth testing and none of it goes near a real DOM. The second bug in the same batch has the same shape one level in. Choosing PDF broke "Open in editor", because deliver() handed the editor the PDF blob and createImageBitmap cannot decode one. That is not a browser boundary; it is one internal stage handing another something it cannot accept. Both stages were tested; the seam between them was not. That is the pattern worth naming.

AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication. AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work. A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately. 1. Begin with one source you understand Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search. Example: The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet. Ask questions whose answers are directly present in the note: When is the review? Who owns the checklist? Has the database change been approved? The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.” 2. Inspect the supplied evidence A plausible answer is not enough. Open the source or evidence shown beside the answer and check: Did the system retrieve the right document? Did it select the relevant passage? Did the answer preserve names, dates, and negation? Can another person repeat the check? This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes. 3. Ask for something that is missing Now ask a question the note cannot answer, such as: Which meeting room is booked? A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models a

We maintain a corpus of 456 role-specific resume examples in TypeScript. Someone asked me what a good bullet point actually looks like, and rather than answer from taste I decided to measure the thing I already had. Fifteen minutes later we had a script, a set of numbers, and a conclusion. The conclusion was wrong, because the script had silently read about twelve percent of the data. This is a post about that failure mode, and then about the numbers I got once the script worked. The corpus Thirty-one TypeScript files, each exporting an array of role objects. One role looks roughly like this: { slug : ' cloud-architect ' , title : ' Cloud Architect Resume ' , category : ' Information Technology ' , sampleData : { summary : ' ... ' , experiences : [ { company : ' Amazon Web Services ' , position : ' Senior Cloud Architect ' , description : ' - Designed multi-region architecture... \n - Led migration of... ' , }, ], skills : [...], }, tips : [...], } The interesting field is description . It holds a newline-delimited list of bullets as a single string, so the whole corpus of bullets is sitting there in source, greppable, without a database or an export step. Version one const descs = [... text . matchAll ( /description: ' ((?:[^ ' \\] | \\ . ) * ) '/g )]. map ( m => m [ 1 ]); Nothing exotic. Match description: , then a single-quoted string, allowing escapes so an apostrophe inside the text does not terminate the match early. It found 199 description strings. I did not question that, because I had no prior for what the number should be. 199 sounded like a lot of text. We computed medians off it, looked at the opener distribution, and started writing. The number that saved me was on a different line of the same output: roles 456 . The slug count was fine. So 456 roles between them had 199 job descriptions, which would mean the overwhelming majority of roles had no work history at all. I knew that was false, because I had rendered these pages. Why it read twelve percent

Author's Note / Disclosure: 100% human-authored content based on real production engineering work. No AI was involved in writing the article, technical analysis, or code. cy.session() is the single biggest speed win available to an authenticated Cypress suite. You log in once, Cypress snapshots cookies, localStorage and sessionStorage , and every later spec restores that snapshot instead of walking through an identity provider. The safety net is validate() . Cypress runs it after restoring a cached session; if it throws, fails an assertion, or yields false , Cypress throws the snapshot away and runs setup again. That is the whole contract: a bad session gets detected and replaced. Mine could not fail. For weeks. And it cost me days of chasing "flaky" specs that were nothing of the kind. The code that looked fine Cypress . Commands . add ( ' login ' , ( user : User ) => { cy . session ( user . username , () => { cy . visit ( ' / ' ) cy . origin ( idpOrigin , { args : user }, ({ username , password }) => { cy . get ( ' #username ' ). type ( username ) cy . get ( ' #password ' ). type ( password , { log : false }) cy . get ( ' button[type="submit"] ' ). click () }) cy . get ( ' #app-shell ' ). should ( ' be.visible ' ) }, { cacheAcrossSpecs : true , validate () { cy . request ( ' /connect/userinfo ' ). its ( ' status ' ). should ( ' eq ' , 200 ) }, }, ) }) Reasonable, right? /connect/userinfo is the OIDC user info endpoint. If the session is dead it should 401, validate() fails, and we log in again. Why it always passes Two independent bugs stack up here, and either one alone is enough to make the check worthless. The URL is relative. cy.request('/connect/userinfo') resolves against baseUrl , which is the application, not the identity provider. So the request never touches the IdP. The application is a single-page app. Its host serves index.html for any path it does not recognise, because that is what history-API routing requires. A request for /connect/userinfo gets b

Is the spec optional if the model is free? I keep seeing that assumption in pull requests. A free coding model shows up in the workflow. A free remote server shows up beside it. Then people drop the checklist without a fight. Why write a failing test for a cheap loop? Just rerun the agent until something compiles, right? That mental model is quietly expensive for teams. Free compute does not purchase a behavioral contract. It only purchases another place to be wrong. This FAQ names five claims I still hear. Each entry has the claim, the evidence, and a corrected model. Then I attach a small artifact you can run. None of this needs paid quotas I will not invent. Who this is for You already ship product patches with coding agents. You also distrust a fluent chat transcript from agents. You want a workflow that survives a free box vanishing. Skip this path if you need a hard SLA. Skip it if the box will hold production secrets. Skip it if "works on the agent host" is the release bar. The setup I actually mean I am talking about a narrow, boring stack. You can call a coding model without a purchase. You can use a remote server without a purchase. I use MonkeyCode when I want that pairing in one place. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will not name models, hardware, or duration. Those details move, and the myths do not. The method still works on a laptop you already own. The free box is optional in every step below. The spec is not optional in any step. Myth 1: Free retries replace a failing test The claim It's free, so I can loop until the tree compiles. The evidence Compilation is not behavior, and it never was. A green compiler can still ship the wrong function. Retrying a prompt does not freeze an oracle for later. Did that extra retry actually get cheaper for you? The sample got cheaper, but no assertion appeared. The corrected model The failing test is the spec you keep. The agent is a patch generator you distrust. F

Zhou Yu discusses why AI agents stall in demo phase and shares how simulation-driven testing solves compliance and reliability bottlenecks. Learn how Columbia and Arklex AI use synthetic user personas, trajectory entropy, and automated CI/CD pipelines to evaluate multi-turn agents, catch edge cases before deployment, and scale self-learning workflows in production. By Zhou Yu

I could not make myself log time in Jira. Not because it is hard. Because nothing happens afterwards. You type a number into a box, the box says nothing back, and by Thursday the habit is gone again. Every tool I tried fixed this by adding another box. So I built the missing half instead. Feed The Troll gives everyone on a team a pixel-art troll that gains XP from the work they already do in Jira, and turns sprint results into a village the whole project shares. It is on the Atlassian Marketplace now. This post skips the game itself. It is about five problems that turned out to be hard in ways I did not expect, each one a consequence of building the thing on Atlassian Forge, alone. What Forge gives you, and what it takes back Forge runs your code on Atlassian's infrastructure. There is no server of mine anywhere in the picture. That is the line on the listing page, and it was the single fact that shaped every decision underneath it. You get a Node 22 runtime, Forge SQL (TiDB under the hood) for storage, and Custom UI modules that reach the backend through @forge/bridge . You give up a backend you control, a cache you can reach, and outbound HTTP to anything you did not declare. The one that keeps mattering: any way to open the database at three in the morning and fix a single row by hand. The whole app declares six scopes. None of them are write scopes: read:board-scope:jira-software read:issue-details:jira read:jira-work read:jira-user read:sprint:jira-software storage:app That last line is the entire persistence layer. Twenty-one tables live behind it now, but only ten shipped with v1.0: trolls, XP events, daily activity, kudos, quests, inventory, team quests, villages, raids, project settings. Every table added since arrived the only way the platform makes comfortable, as a new migration appended to the list, never an edit to one already deployed. migrationRunner . enqueue ( ' v001_create_trolls ' , CREATE_TROLLS_TABLE ) // ... . enqueue ( ' v012_create_product_m

An agent-written patch that lives only inside one long chat session is not a reviewable change for merge. Hidden constraints from that conversation never reach the repository, the failing tests, or the next reviewer. A pairing session that wants a durable result should keep only the diff a second memory-free environment can recreate. The brief, not the transcript, becomes the source of truth for that recreation before anyone discusses merge. Chat windows quietly store rejected files, private service names, and half-stated architecture that later readers will never see. A senior pairing partner should treat that hidden context as contamination rather than as extra helpful memory for the model. The protocol below is a worked example of that stance, not a report of a named production incident. The two roles are a driver chasing an agent-assisted patch and a senior who refuses to merge from chat history alone. Pairing setup for a known failing test The shared codebase is a small HTTP service whose readiness probe still returns 503 under a test that already exists. The driver wants an assistant to edit the health handler and move on quickly. The senior wants a change that someone else could regenerate from the repository without the original thread. Work starts only after both people can describe done in file-level terms on disk. Until that description exists beside the code, every generated diff stays on a throwaway branch with no merge discussion. The pairing treats speed on the first attempt as optional and replayability on the second attempt as mandatory. That split is the whole method, and the rest of this article only makes it checkable. What the senior asked, written down immediately The senior did not open with a cleverer prompt or a longer system message for the same window. The senior demanded answers that a stranger could follow, then wrote those answers into the repository. The recorded questions targeted outcome, verification, blast radius, and isolation, no

Last week a reader left this on one of our articles, and I'm still turning it over: The counter lived in a module-level variable. The supervisor restarts that daemon on a stale-heartbeat rule, so the process died and respawned 131 times during those 24 hours. Every restart reset the counter to zero. The threshold of 3 was unreachable by construction — not degraded, never reachable. Her guard: escalate to a human after 3 consecutive failed self-heal rounds. Written in July, correct logic, process alive the whole time. The unit test passed. The heartbeat was fresh, the logs were flowing. And a human was never called, because the guard's only memory — how many failures in a row — lived in the process, and the process was not the thing being watched. It was the thing being restarted. The number that makes this its own failure shape: 0 escalations across 1,501 daemon starts. The two questions that both pass Earlier in that same thread we'd been arguing that a guard has two questions you can ask it: Does it catch the failure? Is it still running? Her case answers both yes — and the guard still cannot fire, ever. The unit test passes because nothing restarts in a unit test, so the reset never shows up. The process is "up" because the supervisor is doing exactly its job: respawning on stale heartbeat, forever, with no opinion about how often it has done so. It will run a crash loop until the heat death of the universe without ever deciding the loop is the failure. A counter that lives in a process cannot distinguish "this never happened" from "this happened, but I died and forgot." Every restart is a small amnesia. A supervisor that restarts you on a schedule is an amnesia machine. Put a threshold behind that memory and the threshold is a fiction. The tell is the ratio she quoted: escalations fired versus daemon starts. 0 over 1,501. Any guard whose numerator is zero over a large denominator is either genuinely never needed or structurally unreachable — and those two are wo

A pull request arrived after midnight with a README that claimed the API was already healthy. The coding agent had started a process, requested its own localhost, and treated a 200 as proof the service would run for everyone. That response was genuine inside a short-lived workspace, yet it said nothing about the laptop waiting on Monday. The reviewer stared at a green sentence printed on a host that nobody on the team could reopen. This pattern appears whenever a coding agent can execute commands, not merely suggest them, and reviewers misread the transcript. Developers treat the agent's shell as a preview of their laptop because both sessions speak bash and render similar fonts. The analogy fails like a hotel gym standing in for a home garage, familiar until one bolt size changes. Claims in the next sections are the ones that keep returning during review, then a fingerprint workflow that makes the rental visible. Myth: a bound port means the service is portable Agents love a bound port because it is a crisp success token that copies cleanly into a README. A process that answers on the sandbox does not encode libc, extra packages, file layout, or the user's group permissions. Health checks measure a moment on a host you do not retain, not a contract with the checkout that will survive merge. Treat a remote 200 as proof that some files ran once, then demand a second run on CI or a laptop. A useful correction is to refuse README claims that cannot be replayed from a clean clone of the branch. Ask the agent for the exact command sequence, the working directory, and the non-secret environment keys it exported during the run. Then execute that sequence locally with undocumented keys unset, unless they already exist in the team's dotenv template. If the local run dies on a missing header or a path the sandbox invented, the original green check was a rental. Myth: a free remote box is unofficial CI Teams under schedule pressure will point at agent logs the way they once po

A random game is easier to debug when the same inputs produce the same result. In HoopTrait, a browser basketball project, the Lab mode combines eight selected traits and generates a fictional career. The interesting engineering problem is keeping replay, sharing and validation consistent. This is a technical development note, not a claim that a game score predicts an athlete's real performance. Store the decisions, not just the result The Lab state records a seed, a dataset version and an ordered list of actions. An action is a pick or a reroll. Replaying those actions reconstructs the build. A seed alone is not a complete replay contract: changing the player pool or its order can change a seeded draw. A dataset version therefore matters alongside the random seed. For a future release, the same principle should apply to changes in the rules themselves. Test invariants across many runs The Lab test suite iterates through 1,000 seeds. For each seed it shuffles the order of the eight skills, uses the two allowed rerolls, and completes a build. It checks that: Eight distinct players were selected. All eight traits are present, and no player remains to be drawn after completion. The overall game score stays between 0 and 99 and matches the shared rating function. Packing and unpacking the share state returns the original state. Recomputing the fictional career returns the same output. The ten simulated seasons sum to the displayed career earnings. Those assertions catch different problems. A stable score does not prove that a shared link reproduces the same selections. A complete build does not prove that its season totals add up. Reject impossible histories A share payload is untrusted input, even in a client-side game. Negative or fractional seeds, duplicate skill picks, a third reroll, unknown action types, a mismatched dataset version and actions after completion are rejected. The tests also cover malformed encoded payloads and unexpected fields. Local state is usef

My harness flagged the model for sending the wrong arguments. It compared what the model actually...

Run this against the real, published image and watch it fail: docker run --rm --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --version Exit code 127. Not a crash inside trelix-mcp, not a stack trace, not a permissions error — 127 is the shell's own way of saying the binary you asked for does not exist. And it didn't. The console script trelix-mcp is supposed to install as part of every trelix package was simply absent from the image, on both the slim tag and the -local tag, for the entire life of the 3.2.1 release. Every unit test in the suite was green. Every line of source that builds trelix-mcp was correct. The thing a user would actually get from docker pull did not have the binary its own --version flag implies exists. This article covers four releases — v3.2.2, v3.2.3, v3.2.4, and v3.2.5 — spanning 173 commits and 88 changed files since v3.2.1, which is where the last article in this series left off. That one was about tests that pass without exercising the code they claim to cover: a MagicMock standing in for a real embedder, an all-ones attention mask that makes masked and unmasked math identical, a unit test that asserted a bug as its own specification. This one, on the heels of the mutation-testing push that closed out that arc, is about a different and in some ways more uncomfortable failure mode: tests that pass while exercising the wrong artifact entirely. A green pytest run against src/ says nothing about whether the wheel on PyPI, the image on GHCR, or the binary on the GitHub Releases page actually does what it claims. Those are three separate build products, built by three separate pipelines, and none of trelix's 4,353 collected unit tests had ever touched any of them directly. v3.2.2 through v3.2.4 is the story of finding that gap and closing it with an actual gate, not a promise to be more careful next time. v3.2.5 is a short postscript proving the discipline stuck. The Docker image that shipped without its own server The 127 above wasn't

In December 2024, Google quietly updated its closed testing rules for personal developer Console accounts. For months, indie developers had to recruit at least 20 testers to keep their app opted in for 14 consecutive days before applying for production access. Under the revised guidelines, that threshold dropped from 20 to 12 testers. Understanding the nuances of the Google Play 20 testers vs 12 testers shift helps you plan your release schedule accurately without running into unexpected delays during Google Play Console verification. While lowering the number by eight testers sounds like a major relief, the core requirements behind closed testing have not changed. Google still enforces a strict 14 consecutive day duration, and the Play Console continues to monitor tester retention and engagement. A lower numerical requirement means less logistical hassle, but maintaining a stable group of committed testers remains the primary hurdle for independent developers. The Policy Shift: From 20 to 12 Testers Google originally introduced mandatory closed testing in November 2023 to improve app quality and curb low-effort submissions on the Play Store. Initially, all new personal accounts registered on or after November 13, 2023, were required to run a closed test with at least 20 opted-in testers for 14 days without interruption. After roughly a year of developer feedback regarding how difficult it was for solo creators to find 20 reliable participants, Google reduced the requirement to 12 testers in December 2024. It is crucial to understand who this rule applies to. The requirement exclusively targets personal developer accounts created on or after November 13, 2023. If you operate an organization or business developer account, or if your personal account was registered before November 13, 2023, you are currently exempt from this mandatory closed testing gate. However, if you fall under the new personal account category, reaching 12 continuous opt-ins is a strict prerequis

Struct tags are just strings — a typo'd checker name, a wrong-typed field, or a renamed cross-field target all compile fine and fail silently at runtime. checkerlint catches all three before you ship. Struct tags are string literals. The Go compiler checks that your struct compiles — it has no idea what checkers:"eq-field:Passwrd" means, so a typo in a field name, a checker applied to a field of the wrong type, or a renamed field that a cross-field rule still points at all compile fine. They fail later, at runtime, sometimes silently, sometimes as a panic in the middle of handling a request. type Registration struct { Password string `checkers:"trim required"` ConfirmPassword string `checkers:"required eq-field:Passwrd"` // typo: no such field Age int `checkers:"email"` // email is string-only } Nothing here trips go build , go vet , or a normal linter — they all treat checkers:"..." as an opaque string. The first bug only surfaces the moment someone submits a registration form and eq-field can't find a field called Passwrd . The second is worse: email assumes a string under the hood, so calling it on an int field panics at validation time instead of returning a normal error. checkerlint is a go/analysis -based static analyzer, shipped as its own module in the Checker repo, that reads these tags at build/lint time and catches exactly this class of bug before it ships: ./registration.go:3:2: checkerlint: eq-field references field "Passwrd", which doesn't exist on this struct ./registration.go:4:2: checkerlint: email requires a string, but the field's type is int What it actually checks Three things, all specific to how checkers / validate tags can go wrong: Unknown checker names. Every token in the tag has to be a registered checker, normalizer, field-relative checker, omitempty , or a name your own code registered via RegisterMaker / RegisterFieldMaker with a string literal. Typo requird instead of required and checkerlint flags it — nothing else in your toolchain w

My puzzle app, Keyhole, carries 296 dark stories, each with an illustrated card. A dark story is a situation that looks impossible until you drop one false assumption you did not know you were making, and the illustration must show the situation and never the reveal. Draw the aeroplane over the desert and story one is over before the player has read it. In August I ruled that the app does not ship while any card is still flagged by the judge. "End of story," I wrote in the decision, and then spent two days learning what that sentence cost. Two things get judged, the text and the art, and one design is shared by both. The judge is a model, run blind: it sees the finished card and the story the player sees, and neither the finding that triggered the redraw nor the old card. That is the whole trick. A judge that knows what was wrong last time grades the fix. A judge that knows nothing grades the card. Blindness is what makes a pass mean something, and it is why the judge is a separate call from the writer and from the illustrator, never the same conversation. The text pass first. A rubric written for the genre, with one test at its centre, "name the one assumption the solver will make that is false", and four semantic questions after it: does the reveal explain everything the situation promised, does the situation give the reveal away, is there a contradiction, can the answer be reached by yes/no questions without knowledge nobody has. Over all 296 stories it flagged 27: five unanswered, nine spoilers, ten sense breaks, three unsolvable. The fix lane rewrites only what a finding names, the deterministic gate must still pass, and the blind judge reads the result cold before it is written back. A fact-check over the rewrites then cleared them, or left a truth note where no honest fix existed. The art pass is where the numbers live. Each open card was redrawn from a scene brief and judged blind, in waves. The judge wrote a note on every failure, and the lever changed from

1. Código de infraestrutura também quebra Nos dois artigos anteriores desta série, vimos o OpenTofu como alternativa para provisionar infraestrutura e o Ansible para configurá-la depois de criada. Mas há uma pergunta que fica no ar em qualquer um desses fluxos: como saber, antes de rodar apply em produção, que um módulo Terraform não vai abrir uma porta que não deveria, destruir um recurso por engano, ou simplesmente ter um erro de sintaxe? Testar infraestrutura como código é tão importante quanto testar qualquer outro software — só que, diferente de uma função pura, os "efeitos colaterais" de um teste malfeito aqui podem ser uma conta de nuvem inesperada ou um serviço em produção fora do ar. Este artigo fecha a série cobrindo três camadas complementares de teste: análise estática com tflint , verificação de segurança e compliance com checkov , e testes de integração de verdade com Terratest . 2. As camadas de teste em IaC Vale pensar nessas ferramentas como camadas que rodam em momentos diferentes do ciclo de vida do código, da mais rápida/barata para a mais lenta/cara: Lint e análise estática (tflint): roda em segundos, sem precisar de credenciais de nuvem nem de rodar terraform plan . Pega erros de sintaxe, más práticas e problemas específicos de cada provider. Análise de segurança e compliance (checkov): também estática, mas focada em identificar configurações inseguras (bucket público, criptografia desabilitada, security group aberto para 0.0.0.0/0 ) comparando o código contra um catálogo de políticas. Testes de integração (Terratest): a camada mais próxima da realidade — de fato roda terraform apply num ambiente isolado, valida o resultado, e depois roda terraform destroy . Mais lento e mais caro (usa recursos reais de nuvem), mas é o único jeito de garantir que o módulo realmente funciona de ponta a ponta. Um pipeline de CI/CD maduro roda as três, nessa ordem, falhando rápido nas camadas mais baratas antes de chegar nas mais caras. 3. tflint na prática O tfli

A green test run is not a behavior spec. An agent patch can keep every existing assertion passing and still change encodings, error types, empty-input handling, or the bytes written to stdout. Shadow-compare the candidate against a frozen baseline on the same corpus. Merge only after every divergence is classified in an accepted-delta ledger. This article is a testing workflow, not a model bake-off. The harness below is labeled as a proposed, runnable pattern. It does not claim production timings, model names, or pass rates. Why green CI misses the patch Agent patches optimize for the tests they can see. Hidden behavior lives in branches the suite never names: trailing newlines, NaN keys, timezone-naive stamps, None versus [] . Those are cheap to alter. They are expensive to notice after merge. A dual-run gate treats the old artifact as the oracle for unspecified behavior. Specified behavior still belongs in ordinary tests. The ledger exists for the remainder: diffs you accept on purpose, and diffs you refuse. Do not use this as a substitute for code review. Use it as a filter that review should not have to do by hand. Artifact: baseline, candidate, ledger Three files define the contract. baseline/ — a pinned checkout, wheel, or container digest. Not main at HEAD. candidate/ — the agent patch, applied on top of the same pin. delta_ledger.yaml — every previously classified output divergence, keyed by fixture id. Proposed layout: shadow/ corpus/ # deterministic fixtures only 001_empty.json 002_unicode.json 003_nested_null.json delta_ledger.yaml canonicalize.py shadow_compare.py The corpus must be I/O-free. No clocks. No DNS. No home-directory probes. If a fixture needs time, inject it. If it needs a filesystem, pass a temp root the harness owns. Step 1 — Freeze the baseline as an artifact Record the exact bytes you will rerun. A git SHA is enough when the tree is hermetic. Prefer a built artifact when native extensions or generated code are in play. git rev-parse HEAD