标签：#Testing

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post. Between 21 and 24 July it did that four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong. None of those models got worse. Here's how I know, because the how is the only part worth reading. Two of them weren't the model Every point on the board carries a second number next to accuracy: reliability , the share of probe calls that actually came back. Look at the two Google alerts with that column showing: gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts" gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943 23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts" 24 Jul acc 0.971 reliability 1.000 <- next clean run Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one. I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight: api.groq.com -> 429: Rate limit reached for model ` llama-3.3-70b-versatile ` service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30 34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch. Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression." The other two were one question The remaining two alerts ar

There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything. That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything: No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers. The placebo arm is the part almost nobody runs, and it turned out to be the most important one. Why a placebo Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes. So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on claude-opus-4-8 . Finding 1: the placebo often made code bigger My anti-over-engineering skill ( underkill , ~20 lines) cut source LOC by -23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length "write clean, minimal, professional code" instruction didn't reduce c

Six failed CI runs. Thirty-six hours of GitHub Actions time. Every run timing out at exactly the 6-hour limit. The culprit was one line in tests/setup.js . The Setup We were building a multi-tenant platform with a PostgreSQL backend — around 76 database models handling everything from user accounts and billing to visitor logs and real-time notifications. The test suite had grown to roughly 1,140 test cases across 36 files. Standard stuff. CI ran on every PR. Tests passed locally. And then one day, CI just... never finished. The Anti-Pattern Here's what the test setup looked like: // tests/setup.js beforeEach ( async () => { const tableNames = await getTableNames (); // 76 tables await sequelize . query ( `TRUNCATE TABLE ${ tableNames . join ( ' , ' )} CASCADE;` ); }); The intent was clean isolation — every test starts with a blank slate. Reasonable in theory. Catastrophic in practice. The Math Do the multiplication: 76 tables × 1,140 tests = 86,640 TRUNCATE operations Each TRUNCATE TABLE ... CASCADE is not a cheap operation. PostgreSQL has to: Acquire exclusive locks on all referenced tables Walk the foreign key graph to find dependent tables Truncate each in dependency order Release locks With a moderately complex schema where most tables reference others (users → societies → members → invoices → payments → ...), a single TRUNCATE ... CASCADE on a central table can fan out into dozens of implicit truncations. Multiply that by 86,640 and you have a test suite that will never complete within any reasonable timeout. Why It Wasn't Caught Sooner Two reasons: 1. It used to be fast. When the suite had 50 tests and 20 tables, this pattern worked fine. 50 × 20 = 1,000 truncations — uncomfortable but survivable. Nobody noticed when the suite crossed a tipping point. 2. Local runs used a different database state. Locally, developers often ran a subset of tests with --grep or file-specific runs. The full suite was only ever run on CI, and CI was slow enough that most assumed i

Divergence escalates the wrong population: unanimous misses auto-pass Agent Determinism Illusions (Part 7) Where this fits: This part does not continue Part 13's probe-vs-prose thread. It returns to Part 6 's L2→L3 escalation rule — Dipankar's move of treating vote disagreement as the human-review signal. Alexey Spinov's follow-up comment says that signal points at the wrong population. Two experiments check whether he is right, and what to put in the tripwire instead. Part 6 drew this control flow: L2 multi-perspective votes │ unanimous ──────────► AUTO-PASS / AUTO-REJECT │ divergence (e.g. 2–1) ► L3 human The caveat was already in the text: divergence measures ambiguity; it does not fix unanimous systematic bias. Alexey's point is sharper — and it is about routing , not about another caveat paragraph. 1. Alexey's population mismatch On the Part 6 thread, Alexey Spinov wrote (paraphrased tightly): The dangerous failures are high-confidence and directional — systematic. Systematic bias is shared across prompts, not idiosyncratic (your own P3: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population. He proposed two cheap replacements: T1 — deterministic tripwire on known-reversal classes (escalate regardless of agreement). T2 — treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.” That is the claim under test. Not “divergence is useless,” but “divergence alone is the wrong primary tripwire for the failure mode you already measured.” 2. Experiment A — offline proxy on DF v2 (no new API) Part 6's Mike Update already showed: of 96 DF v2 MISS runs, 95.8% sat at self-reported confidence ≥ 0.9 (avg 0.969). That mass is concentrated — Part 6 also reported ~80% of MISS runs from qwen3:0.5b —

Seu agente escreveu o teste. O Cursor sugeriu três casos extremos que você não havia pensado. O Copilot preencheu o corpo da requisição, e o Claude executou tudo uma vez e reportou “verde”. A pergunta é justa: se o agente faz tudo isso, a IA pode substituir completamente o teste de API? Experimente o Apidog hoje Não. A IA não substitui o teste de API, mas já substitui boa parte da autoria dos testes. Agentes elaboram casos, sugerem cenários extremos e geram payloads com eficiência. Porém, eles não garantem execução idêntica em cada commit, não bloqueiam merges com um resultado confiável e não decidem se um contrato está correto. Para isso, você precisa de uma ferramenta determinística e de revisão humana. Essa separação responde a uma dúvida maior: você ainda precisa de uma ferramenta de API na era dos agentes de IA ? Sim — mas o papel da ferramenta mudou. Use agentes para acelerar a autoria e ferramentas determinísticas para validar, executar e bloquear regressões. Onde este artigo difere do guia prático Se você procura instruções para gerar testes com agentes, consulte o guia sobre como usar agentes de IA para teste de API . Este artigo trata de outra pergunta: quais partes do fluxo você pode delegar a um agente e quais precisam continuar em uma suíte determinística? Uma implementação prática separa o trabalho assim: O agente lê a especificação e cria um rascunho de testes. Um desenvolvedor revisa cenários, asserções e regras de negócio. Um runner headless executa a suíte no CI. O pipeline bloqueia o merge usando o código de saída do runner. Quando algo falha, você inspeciona a requisição e a resposta reais. O que a IA faz bem em testes de API hoje Agentes removem trabalho repetitivo de autoria. Use-os principalmente para criar e expandir artefatos de teste. Elaborar uma primeira suíte a partir de uma especificação Entregue ao agente um endpoint, uma definição OpenAPI ou uma resposta de exemplo. Ele pode gerar rapidamente: verificações de status HTTP; asserções de

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

Most teams already keep Playwright or Robot Framework specs next to the product. Manual cases often live somewhere else — a spreadsheet, a cloud TMS, or a wiki page nobody updates after the first release. That is why automation coverage is usually a quarterly guess: the catalog and the specs never share an id. Put both in the same Git repo , give every manual case a stable id, and linking becomes a text match. Any IDE agent can do that if the cases are files it can already see — no custom AI product inside a test tool. One repo, two layers of the same suite Manual cases are YAML files under .gitoza-lite/test/cases/ (VS Code / Cursor extension) or .gitoza/test/cases/ (Desktop app). The filename without .yaml is the case id . Automated tests live wherever your team already puts them — tests/ , e2e/ , robot/ — in the same repository. Traceability is a shared id: put that case id on the automation side as a tag (or name), and on the YAML side as automated: true plus a params pointer. A PR can change the case, the Playwright spec, and the link in one review. Step 1 — Draft manual cases in the IDE Open the repo in Cursor or VS Code. Point the agent at a few existing case files so it learns your title style, tags, and step length. Then feed it a user story, a ticket, or a screenshot. Ask for several cases at once, not one shallow step. A useful prompt looks like: Read .gitoza-lite/test/cases/shopflow/auth/ for format. From ticket SHOP-184, draft three YAML cases (happy path, invalid password, locked account). Filename = case id. Use tags auth and smoke where it fits. Save the files under the suite folder. Then open the Gitoza Lite Test Repository tab to browse, edit, and — when you are ready — run them as a manual suite with Pass / Fail / Skip. The extension does not ship its own model. It keeps cases as plain YAML so whatever assistant you already use can read and write them. A minimal case: --- title : Login with valid credentials priority : high tags : [ smoke , auth ]

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time. The Problem Everyone Has Your test suite runs for 45 minutes. You change one file. Should you: Run all 2,000 tests? (Safe but slow) Run tests with matching filenames? (Fast but broken) Manually tag tests? (Gets stale immediately) We tried all three. They all failed. The Insight: Stop Guessing, Start Measuring What if we just measured what each test actually executes? Coverage tools have done this for years: { "src/MyComponent.js" : { "statements" : { "0" : 5 , "1" : 12 , "2" : 0 } } } Key insight: Coverage tools report per test run, not per test file. All we needed: save coverage after each individual test. The Architecture (5 Steps) Step 1: Instrument the Code // Build config (webpack/vite/rollup/etc) export default { plugins : [ process . env . COLLECT_COVERAGE && coveragePlugin ({ include : ' src/**/* ' , exclude : [ ' **/*.test.* ' ] }) ] } Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java) Step 2: Collect Coverage Per Test // Test framework afterEach hook afterEach ( async () => { const coverage = getCoverageData (); saveCoverage ( `coverage- ${ testName } .json` , coverage ); }); Step 3: Build the Reverse Map const map = {} ; for (const coverageFile of allCoverageFiles()) { const testName = extractTestName(coverageFile); const coverage = JSON.parse(read(coverageFile)); map [ testName ] = [] ; for (const [ sourceFile , data ] of Object.entries(coverage)) { if (wasExecuted(data)) { map [ testName ] .push(sourceFile); } } } Example output: { "tests/user-profile.test.js" : [ "src/components/Profile.jsx" , "src/components/Card.jsx" , "src/utils/formatters.js" ] } The magic: The test told us what it executed. No parsing, no guessing. Step 4: Detect Tests CHANGED = $( git diff origin/main...HEAD --name-only ) TESTS = $( lookupTestsInMap " $CHANGED " ) npm test $TESTS Step 5: Auto-Up

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why. That trust is a liability. An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness. Here is how to build one. The Problem: Self-Reported Truth Is Not Truth Most teams validate agent behavior by reading the agent's own output. They check the tool_calls field in the response, match it against an expected schema, and call it done. This works until it doesn't. Consider a common failure mode: the agent decides to call search_knowledge_base but the LLM formats the tool name as searchKnowledgeBase . The routing layer silently normalizes it, the call succeeds, and the agent logs search_knowledge_base . Your test passes. The actual execution path was different from what you verified. Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift. The root cause is the same. You are testing the agent's intent , not its execution . Intent is cheap to fake. Execution leaves fingerprints. The Solution: An External Observer You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched. The observer does not trust the agent's logs. It trusts what it sees on the wire. Here is the architecture at a high level: Intercept every outbound call from the agent to a tool. Record the raw request before any normalization or routing. Compare the recorded call against

Agent Determinism Illusions (Part 6) Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed. Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification. But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline. §§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place. 1. Alexey Spinov & Manuel Bruña: Layer Before You Judge Alexey's comment identified the most fundamental design flaw in my experiments: "G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM." Manuel added the constructive direction: "Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual." I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4): ID Output Type Could code catch it? G1 "I am a little duck, quack quack" nonsense ✅ very short + no keywords G2 "。" (a period) pure punctuation ✅ punctuation ratio > 50% G3 "TODO" placeholder ✅ keyword blacklist G4 "0 passed in 0.00s (no tests collected)" zero-test pass ✅ regex 0 passed + no tests All fo

Overview Open WebUI exposed a real Ota lifecycle boundary. This was not mainly a parsing or contract-shape repo. The contract was already strong enough to model: source-checkout verification packaged native runtime through uv run open-webui serve frontend development runtime default Docker Compose runtime What the repo exposed was operational truth after proof: a successful native proof still left a host workload alive the first cleanup fix then widened too far and treated a Compose-owned runtime as the same class of host workload That made Open WebUI a valuable pressure repo. It forced Ota to get more precise about cleanup ownership instead of treating all successful runtime proof as one generic teardown problem. The current pressure contract pins released Ota v1.6.24 . Its latest green matrix run proves the release surface at the exact contract and workflow revision linked below. What Open WebUI exposed in Ota This repo exposed four meaningful weaknesses. 1. proof success was weaker than it looked The first issue was not that runtime proof failed. It was that runtime proof succeeded and still left the native workload alive afterward. In this repo, the packaged native workflow launches: serve:native : launch : kind : command exe : uv args : - run - open-webui - serve - --host - 0.0.0.0 - --port - " 8080" Ota proved that workflow, but the launched process tree was still alive after proof completed. In GitHub Actions, that surfaced through setup-uv post-job cleanup, which blocked while the uv cache was still in use. That was an Ota gap. If proof succeeds but leaves behind repo-owned runtime state that later breaks CI cleanup, the proof surface is still incomplete. 2. native service cleanup widened past its real ownership boundary The first core fix made Ota clean selected native service workloads after successful proof. That was directionally correct, but Open WebUI immediately exposed the next boundary. The Docker workflow uses a native task shape to launch Compose:

You have been here. The test suite is green. The deployment pipeline reports all checks passed. Then a customer opens a ticket with a screenshot that shows something your test never caught. The test passed in staging. It fails in production. And you cannot reproduce it locally. This is not a flaky test problem. It is a fidelity problem. Your test environment and your production environment are not the same thing. The gap between them is where real bugs live. Let me walk through one concrete example, the fix, and what it teaches about writing tests that survive the handoff to a real user. The Problem: Environment Drift A fintech team I worked with had a checkout flow. The test clicked "Pay Now", waited for a success message, and asserted the text "Payment successful" appeared on screen. It passed every time in staging. Customers reported that after paying, they saw a blank white page for several seconds before the success message appeared. Some of them closed the tab during that blank period, thinking the payment failed. The transaction went through. The customer never saw the confirmation. Support tickets piled up. The test never caught this because the staging environment served the success page in under 200 milliseconds. The blank period did not exist there. Production had a slower downstream service that introduced a three-second delay between the payment confirmation and the page render. The test was correct in what it checked. It was wrong in what it assumed about timing and state. The Fix: Test the Experience, Not Just the Outcome The fix was not to add a longer wait. The fix was to test what the user actually experiences during that gap. Here is a minimal Playwright test in TypeScript that catches this class of problem: import { test , expect } from ' @playwright/test ' ; test ( ' checkout shows loading state before success ' , async ({ page }) => { await page . goto ( ' /checkout ' ); await page . fill ( ' #card-number ' , ' 4111111111111111 ' ); await page

Gherkin is not a test automation tool. It never was. Yet here we are, five years into your SDET career, and you're staring at a feature file that reads like a step-by-step manual for a human tester. Given I log in with username "admin" and password "password123" . When I click the "Submit" button . Then I see the text "Welcome" on the screen . You've written two years of these. Your team calls it BDD. Your manager calls it "living documentation." And somewhere in the back of your mind, a quiet voice whispers: This is just a manual test case with extra steps. That voice is right. Let me say it plainly: if your Gherkin scenarios describe how the system works instead of what it should do, you are not doing BDD. You are writing manual test cases in a structured English format and calling it automation. The only thing you've automated is the illusion of progress. The problem isn't Gherkin. The problem is how we use it. Most teams adopt BDD because someone read a blog post about "collaboration" and "shared understanding." They install Cucumber or SpecFlow. They write feature files. They map steps to Selenium or Playwright code. And they call it a day. But look closely at what happens next. The product owner never reads the feature files. The developer skims them once and goes back to writing code. The QA engineer — that's you — becomes the sole maintainer of a growing pile of Gherkin that nobody else touches. You're not facilitating collaboration. You're translating manual test cases into a format that requires a compiler. Here's the real test. Take any feature file from your project. Hand it to a developer who has never seen it. Ask them to implement the feature using only the Gherkin as a spec. If they can write production code from it, you have real BDD. If they ask you for clarification, you have documentation theater. I've seen teams with hundreds of feature files. Beautifully formatted. Perfect indentation. Tags for every regression cycle. And not a single one of th

Most engineers expect AI-assisted automation to be the easy part. You describe a test, the model writes it, you move on. The first week will prove you wrong. Not because the code is bad. Because the code is almost right. And almost-right code is harder to debug than wrong code. Wrong code fails loudly. Almost-right code passes on Monday, fails on Tuesday, passes again on Wednesday, and by Thursday you are questioning whether you understand your own application. I have watched teams adopt AI copilots into their Playwright suites and spend the first five days doing nothing but untangling false passes. If you are about to start this journey, here is what that week actually looks like. The Problem: The Model Does Not Know What "Stable" Means A language model has never waited for a network response. It has never watched a flaky selector survive three CI runs and then collapse on the fourth. It writes tests from a static understanding of your page, not from the dynamic reality of your application. You will ask it to write a test that clicks a button and waits for a confirmation toast. The model will produce something like this: await page . click ( ' button:has-text("Submit") ' ); await page . waitForSelector ( ' .toast-success ' ); Looks fine. Runs fine. Then your team deploys a new build where the toast takes 400ms longer to appear because of an analytics call. The test fails. Not because the feature broke. Because the model assumed a timing that was never guaranteed. This is the core problem. The model writes tests that match the page as it was when the model saw it . It does not write tests that match the page as it will be . The Solution: Treat AI-Generated Tests as Drafts, Not Deliverables The shift is mental before it is technical. You cannot review AI-generated tests the way you review human-written tests. Human tests come with intent. AI tests come with patterns. You need a different review lens. First, look for every hardcoded wait. Replace it with a state-based

Cursor's "Sand" project targets non-developers. Anthropic's Claude Cowork runs cross-device. OpenAI's ChatGPT Work delivers documents, spreadsheets, and presentations. When an agent produces a deliverable for someone who cannot read the code, the verification interface matters more than the output format. A download button is not verification. The problem When a developer reviews agent output, they can read the diff, run the tests, and check the types. When a non-developer receives an agent-produced spreadsheet or document, they have no equivalent verification path. They either trust it completely or reject it completely. Neither is useful. What a verification UI needs Element Purpose Implementation Source list Show what inputs the agent used Links to source documents with timestamps Confidence indicator Flag low-confidence outputs Per-section confidence derived from source coverage Change highlights Show what the agent generated vs. copied Diff view between source and output Audit trail Record who requested what and when Append-only log with request ID, timestamp, and result hash Rejection path Let the user say "this is wrong" without starting over Feedback record linked to specific output section A minimal verification component <!-- agent-output-verification.html --> <!DOCTYPE html> <html lang= "en" > <head> <meta charset= "UTF-8" > <title> Agent Output Verification </title> <style> .verification-card { border : 1px solid #ddd ; border-radius : 8px ; padding : 16px ; margin : 8px 0 ; font-family : system-ui , sans-serif ; } .source-list { list-style : none ; padding : 0 ; } .source-list li { padding : 4px 0 ; border-bottom : 1px solid #eee ; } .confidence-low { color : #c0392b ; font-weight : bold ; } .confidence-medium { color : #e67e22 ; } .confidence-high { color : #27ae60 ; } .reject-btn { background : #fff ; border : 1px solid #c0392b ; color : #c0392b ; padding : 4px 12px ; border-radius : 4px ; cursor : pointer ; } .reject-btn :hover { background : #c0392b

Tu prueba pasó el lunes: misma entrada, mismo código y temperature=0 . El martes falló sin cambios en tu aplicación. La aserción esperaba una cadena exacta, pero el modelo devolvió la misma respuesta con una redacción ligeramente distinta. El agente funciona; tu suite de pruebas no. Prueba Apidog hoy Este es el coste de probar sistemas que llaman a modelos de lenguaje. Incluso con temperatura cero, no obtendrás salidas idénticas byte a byte entre ejecuciones. En lugar de probar la redacción exacta, prueba el contrato: estructura, campos, rangos y efectos esperados. Este artículo profundiza en el tercer modo de fallo de nuestra guía sobre por qué los agentes de IA fallan en producción . Por qué temperature=0 no significa determinismo La temperatura controla cómo el modelo selecciona el siguiente token. Con temperature=0 , el modelo elige el token más probable, lo que parece reproducible. Sin embargo, esa configuración no garantiza resultados idénticos. La causa está debajo del modelo: Las operaciones de punto flotante en GPU no son asociativas: el orden de las sumas puede modificar mínimamente el resultado. Esa diferencia puede cambiar cuál token queda en primer lugar. El orden de ejecución puede depender de cómo el proveedor agrupa solicitudes, del hardware, de la región o de la versión del kernel. Los proveedores pueden cambiar GPUs, bibliotecas de inferencia o cuantización de pesos. Una discusión extensa en vLLM explica por qué una semilla fija y temperature=0 no bastan para lograr reproducibilidad bit a bit. La conclusión práctica es simple: el determinismo es una propiedad de toda la pila de servicio, no una opción de tu solicitud. Tu prueba debe aceptar variaciones de redacción cuando el significado y el contrato siguen siendo correctos. Por qué las aserciones de cadena exacta vuelven inestable tu suite Esta prueba es frágil: assert response == " Your order total is $42.00. " Puede pasar hoy y fallar mañana si el modelo responde: Your total comes to $42.00. La

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite

A helpful comment on my EXIF test suggested using a metadata scrubber. That is useful for ordinary still images, but a mobile upload contract must define every asset it sends. An iOS Live Photo can include both a photo resource and a paired video resource. The failure case is simple: the JPEG derivative has no GPS EXIF, while metadata or an original file associated with the paired video survives in the upload or retry path. Build the resource inventory first: let resources = PHAssetResource . assetResources ( for : asset ) for resource in resources { print ( resource . type , resource . originalFilename ) } Then make privacy assertions per output, not per UI selection: Artifact Required check still derivative no EXIF GPS/location fields paired video derivative no location metadata upload manifest only derived filenames retry queue no path to original resources temporary directory deleted after success or cancellation My lifecycle test would start an upload, force the app into the background after the still image is prepared, kill it while the paired video is processing, and relaunch. Recovery must either regenerate both safe derivatives or delete the incomplete pair. It must never mix a scrubbed still with an original video. Apple’s PHAssetResource API exposes the resources associated with a Photos asset. That enumeration should become evidence in the test: fail when an unexpected resource type is present rather than silently uploading it. Also verify what reaches the server. Device-side inspection alone misses multipart manifests, queued originals, and server-generated previews. Download the stored pair in a test environment and run the same metadata assertions again. A “remove location” button needs a precise scope. For Live Photos, the user reasonably expects it to cover the complete paired asset and every retry copy—not just the JPEG they can see.

A saga test that stops after “payment succeeded, inventory failed, refund called” assumes compensation is reliable. It is another distributed operation, so test it under the same faults. Use this sequence: 1. payment capture succeeds 2. inventory reservation times out 3. refund succeeds at provider 4. refund response is lost 5. compensation message is delivered again 6. late inventory-failed event arrives again The invariant is not “refund endpoint called once.” It is: captured amount - confirmed refunded amount = final charged amount and final charged amount is never negative Persist an inbox record for consumed message IDs and an outbox record for each intended side effect. Give the provider request a stable idempotency key derived from saga and compensation step: { "sagaId" : "order-42" , "step" : "refund-payment-v1" , "idempotencyKey" : "order-42:refund-payment:v1" , "amount" : 4900 } A deterministic simulator should permute duplicate delivery, delayed acknowledgement, worker crash, and out-of-order events. After every run, assert one terminal order state, at most one economic refund, and a complete evidence trail. “Already refunded” must reconcile to success only after amount and payment identity match. AWS describes coordination choices and rollback behavior in its Saga pattern guidance . The implementation detail that deserves its own test is durable compensation progress: a process restart cannot erase whether the external side effect occurred. Alert on sagas stuck in compensating , but do not let an operator click “retry” with a new key. The runbook should first query provider state, compare amount and currency, and resume the same operation identity. Exactly-once delivery is not required to preserve money. Stable operation identity plus reconciliation is.