标签：#test

How to Get the Maximum Value Out of This Framework Having built and iterated on this prompt through multiple production edge cases, here are the exact execution strategies I recommend depending on your workflow: 1. The Human-in-the-Loop Workflow (Recommended for Chat UI) Run it in two separate chat threads: Don’t let long conversation history degrade your test accuracy. Run Phase 1 in Thread A to get your gap analysis and critical questions. Review the gaps, clarify what you can, and then update your original requirement text. Start Thread B for Phase 2: Open a fresh conversation, paste the updated requirements + this framework, and jump straight into generation. This completely eliminates context drift and keeps the LLM laser-focused on state mutation rules. 2. The 2-Pass Programmatic Auditor (For Automated CI/CD Pipelines) If you’re calling an LLM via API or integrating this into a pre-commit GitHub Action, split the execution into two isolated passes: Pass 1: Run Phase 1 & 2 to generate the initial test table. Pass 2 (The Audit Pass): Feed the generated table into an isolated, secondary prompt whose only job is to enforce the Verification Check (verifying exact boundary literals, API status codes, and non-mutation assertions). Separation produces drastically higher assertion reliability than asking a model to self-audit in a single turn. 3. How to Live-Demo or Teach This For Live Streams & YouTube: This framework makes for a high-signal live demo. Paste an intentionally ambiguous user story (e.g., a webhook handler or payment endpoint), watch Phase 1 halt at the gate live, discuss the surfaced edge cases on camera, reply PROCEED, and review the generated DEFERRED risk rows. It shifts the content focus from “Look at this cool AI tool” to “This is how Senior SDETs think about systems.” For Technical Writing & Post-Mortems: The progression from a naive “write me test cases” prompt to a strict 2-phase state-machine framework is a technical narrative in itself. Break

A few years ago, the hardest part of building a browser test framework was getting started. You had to choose a runner, configure browsers, create page objects, wire up reporting, add retries, manage secrets, connect it to CI, and convince someone else on the team to learn how the whole thing worked. Today, you can open an AI assistant and ask it to generate most of that before lunch. That sounds like a dramatic improvement. In some ways, it is. But it also moves the bottleneck. The question is no longer, “Can we create a framework?” The question is, “Can we operate what was created?” That distinction matters more than it appears. Generation cost is not ownership cost A generated framework feels cheap because the first version arrives quickly. The code compiles, a few tests pass, and the pull request looks more complete than anything you could have written in an afternoon. Then reality starts applying pressure. The application changes. Authentication behaves differently in staging. A shared helper starts hiding failures. Parallel workers collide over test data. Someone upgrades a dependency and three reporters stop agreeing with one another. The initial generation was fast. The ownership cost was merely deferred. This is the central problem described in what actually breaks when Claude generates a large Playwright framework . Large generated systems often fail in the seams: fixtures, abstractions, environment assumptions, test data, and conventions that were never explicitly agreed upon. The code may be readable line by line while the system remains difficult to reason about as a whole. That is a dangerous form of complexity because it looks productive. More code can hide less understanding Teams sometimes evaluate AI-generated automation by counting output: number of test files; number of scenarios; number of passing checks; number of prompts completed; number of lines added. Those numbers are easy to produce and easy to report. They are also weak proxies for confi

Everything you need to know to find your first vulnerability, get paid, and build a real reputation in cybersecurity — without breaking any laws. If you've typed "how to start bug bounty hunting" into Google recently, you're not alone. It's one of the fastest-growing searches in cybersecurity right now, and for good reason: it's one of the only paths in tech where a total beginner with no degree can find a real flaw, report it, and get paid the same week. This guide answers the questions people are actually searching in 2026 — what bug bounty hunting is, which bugs pay the most right now, how AI has changed the game, and how to land your first bounty. What Is Bug Bounty Hunting, Exactly? Companies invite independent researchers to test their websites, apps, and APIs for security flaws — legally. When you find a real vulnerability, you write a report explaining what it is, how to reproduce it, and what damage it could cause. If the company confirms it, they pay you based on severity. It's not hacking in the movie sense. It's structured, permitted testing within a defined scope — the specific domains, apps, or features the company has authorized you to test. Step outside that scope, and you've crossed from "bug bounty hunter" into "unauthorized access," which is a crime in nearly every country. The Best Platforms to Start On in 2026 Three platforms dominate the space: HackerOne — the largest and most beginner-friendly, with the widest range of programs Bugcrowd — strong onboarding and clear scope documentation Immunefi — the go-to platform if you're interested in web3 and smart contract security, which currently pays some of the highest bounties in the industry Start with Vulnerability Disclosure Programs (VDPs) — these often don't pay, but they let you build a track record, earn private invites, and practice on real targets without competing against thousands of other hunters for a bounty. What Bugs Are Actually Paying Right Now The vulnerability landscape has shifte

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

I wrote about the Electricity Planning Engine a little while back, including a timezone bug that made a correct price look "not found" after a database round trip. A few days later, Alex Shev left this comment: Timezone bugs are brutal in planning engines because the result can look mathematically correct while being operationally wrong. Energy workflows especially need tests around boundaries, not just averages. That is a genuinely sharp way to put it, and it is not just a comment about the bug I already wrote about. It is a comment about how I test the project in general, and I did not like how well it applied once I went and checked. The part that stung a little "Looks mathematically correct while being operationally wrong" is exactly what the original timezone bug was. PriceSeries::priceAt() threw a clean "price not found" error, which is arguably the good version of that failure mode: loud, easy to catch, hard to ship. A quieter version of the same class of mistake, off by one hour instead of missing entirely, would not throw anything. It would just return a plan that looks completely reasonable and is wrong the entire time it runs. Alex's second point, boundaries over averages, is the one I actually had to go check rather than just agree with in the abstract. So I opened tests/Unit/Domain/Contract/PricingStrategyTest.php and looked at every hour used in every peak/off-peak assertion: new DateTimeImmutable ( '2026-07-18 14:00:00' ) // peak new DateTimeImmutable ( '2026-07-18 23:00:00' ) // off-peak new DateTimeImmutable ( '2026-07-18 05:00:00' ) // off-peak 14:00, 23:00, 05:00. Every single one comfortably inside its window. None of them anywhere near the actual transition. The off-peak slot in the config is 22:00 to 06:00 , and the comparison behind that lives in TimeSlot::contains() : // wraparound slot, e.g. 22:00 -> 06:00 return $minuteOfDay >= $this -> startMinuteOfDay || $minuteOfDay < $this -> endMinuteOfDay ; That >= versus < is exactly the kind of one-

The CI job just hit 28 minutes. Again. You pull up the duration report expecting to blame a bloated integration test or a slow environment spin‑up. Instead the longest stage stares back at you: a collection of Cucumber feature files that haven’t caught a real bug in months. Maybe years. They run on every commit, green circle after green circle, while your team mutters about slow pipelines and nobody dares touch them. Most teams treat those scenarios like documentation. “They describe the system,” someone once said, as if a Gherkin file were a legal contract. Others cling to the sunk cost: a year ago a whole squad spent two sprints writing them, polishing the grammar, aligning step definitions. Deleting them would feel like admitting waste. Experienced engineers see it differently. They treat a scenario that never fails as a liability you’re paying for on every push. Not neutral. Liable. Compute cycles, developer attention, flake‑debugging time, and the quiet toll it takes on trust in the pipeline. The principle is blunt: if a test hasn’t failed in the last few sprints, you’re already paying its full cost and receiving nothing in return. That doesn’t mean you delete everything green. But it does mean you audit with the same seriousness you’d use for a memory leak. What the green wall actually costs The damage is not abstract. A pipeline bloated with stale scenarios hurts you in five concrete ways. First, feedback slows. Every extra minute between push and result stretches the loop that tells a developer they’re safe to merge. Multiply across a team and you’re losing hours per week to waiting. Second, flakiness increases. When you have many scenarios, a single unstable environment variable can produce a handful of failures that are not regressions at all. Engineers learn to retry, then to ignore. Third, confidence erodes. If half the suite is ceremonial, a genuine failure might be dismissed as “just another flaky test” until it reaches production. Fourth, maintenance

The manager opens your portfolio. Your resume says you have five years of automation experience. The README lists Selenium, Playwright, Appium, Jenkins, Docker, Kubernetes. He scrolls. There is no code. The browser tab closes. This is you. Not because you lack skill—you have it—but because your public proof reads like a shopping list. The tools you name say nothing about how you think when a flaky test fails at 2am, or how you convince a developer that a bug is real. If you’re serious about landing a role that demands more than record-and-playback, you need to stop treating your portfolio like a keyword bingo card. Here are three mistakes that kill your chances instantly, and exactly how to fix them. Mistake 1: Tool jockeying Listing every automation framework you’ve heard of is a reflex. A hiring manager sees "Proficient in Cypress, Playwright, Selenium, WebDriverIO" and assumes you ran npm init once in each and called it done. Most testers frontload tools because they’re scared of the empty space where code belongs. Experienced testers show one test, deliberately written, with a comment that explains a trade-off they chose. The difference is not volume. A single 30-line script that handles a login flow with a purposeful wait strategy teaches more about you than a six-tool résumé. I’ve deleted my own old projects after re-reading them and realizing they said nothing about why any assertion existed. That quiet cringe is the signal you’re ready to improve. What you ship in your portfolio must answer one question: "What did this person decide, and why?" Move your tool list to a footnote. Let a real test carry the message. Mistake 2: The perfect test trap A portfolio full of green builds is a trap. Every team knows that real automation breaks: the CI node runs slow, the third-party API throttles you, the DOM renders a fraction of a second late. Showing only passing tests hides how you handle the ugly parts of the job. Most testers polish every assertion until it’s spot

Most people think the SDET title means you are automatically more valuable than a manual tester. The SDET writes Playwright scripts. The SDET configures CI pipelines. The SDET talks about page objects and retry strategies. The manual tester clicks through screens and writes bug reports. Here is the truth I have watched play out across teams: the manual tester who can write a SQL join will consistently outperform the SDET who cannot. Not because SQL is magic. Because SQL is the shortest path to understanding what the system actually stores, not what the UI shows you. The problem with automation-first thinking I have seen SDETs spend three sprints building a test suite that validates every button, every dropdown, every error toast. The suite passes in CI. The suite passes in staging. The suite passes in production. And the bug still ships. Why? Because the test checked that the UI rendered correctly. It never checked that the database actually saved the right record. The SDET wrote assertions against DOM elements, not against data. The manual tester, meanwhile, ran a simple query. Saw the order status was "pending" when it should have been "confirmed." Filed a bug with the exact SQL that proved the issue. The developer fixed it in ten minutes. That is not a story about manual versus automated. That is a story about data literacy versus UI obsession. What a SQL join gives you that a locator never will A Playwright locator tells you something is on the screen. A SQL join tells you something is true. When you write page.getByText('Order confirmed') , you are testing that the frontend displays those words. You are not testing that the backend actually confirmed the order. You are not testing that the payment gateway returned success. You are not testing that the inventory decremented. A SQL join connects those dots. SELECT o . id , o . status , p . status AS payment_status , i . quantity AS remaining_stock FROM orders o JOIN payments p ON o . id = p . order_id JOIN invent

LLM applications fail silently at the semantic level. Standard unit tests verify that functions return values, but they cannot detect if the output is factually wrong, off-tone, or missing steps. Evals fix this by running a prompt, inspecting the output, and programmatically or judgmentally deciding its quality. Why LLM Testing is Different Unit tests break on non-determinism. Temperature-driven randomness means identical inputs produce varying valid outputs, which defeats static equality checks. Semantic equivalence ("Paris is the capital" vs. "Capital is Paris") makes string matching useless. Small model or prompt updates cause gradual quality drift that only surfaces at scale. Tone and accuracy require judgment that basic assertions cannot encode. The Three Eval Types 1. Heuristic Evals check measurable properties: JSON validity, word counts, PII presence. They are fast and objective — ideal for regression gates on every pull request. 2. LLM-as-Judge uses a second LLM to grade output against a rubric. Best for subjective content too complex for code but too vast for manual review. 3. Human Evals are the ground truth. Use them to build your golden dataset, calibrate the LLM judge, and finalize decisions before major releases. Building Your Eval Infrastructure Start minimal: a Python script, a JSON file of test cases, and a loop comparing LLM output against assertions or a rubric. Curate a golden dataset of 500–1000 production-representative requests including edge cases and adversarial prompts. Refresh it quarterly to match user behavior drift. Automate the harness in GitHub Actions. Block deployments when pass rates drop below your threshold (90% is a common starting point). Run fast heuristic tests on every PR and full LLM-judge evals nightly. Key Tools PromptFoo — Open-source, YAML-based test cases with a built-in CLI runner and diff reporting. Braintrust — Hosted platform for dataset management and experimental comparison. Inspect — Open-source framework by th

A test failure that takes 20 minutes to surface, buries the error in 3000 lines of log output, and gives no context about what changed is nearly useless. Good test reporting transforms raw pass/fail data into actionable signals. Failing fast — stopping the pipeline the moment you have enough information to make a decision — keeps feedback loops tight and respects developer time. These two concerns are deeply connected: you can only fail fast confidently when your reporting is good enough that a fast failure still gives you everything you need to fix the problem. What Good Test Reporting Looks Like Before discussing implementation, it's worth being precise about what "good" means here: Immediate visibility — failures are surfaced at the PR/commit level, not buried in logs Failure context — what failed, with what input, producing what output, and in which file/line Historical comparison — is this a new failure or a pre-existing one? Trend data — is this test getting flakier? Is the suite getting slower? Actionability — the report points to a fix, not just a symptom Most teams get #1 and stop. The teams that nail all five have fundamentally different debugging velocity. JUnit XML: The Universal Format JUnit XML is the lingua franca of CI test reporting. Almost every test framework can emit it, and almost every CI platform can ingest it. Understanding the format helps you produce better reports. <?xml version="1.0" encoding="UTF-8"?> <testsuites name= "My Test Suite" tests= "42" failures= "2" errors= "0" time= "8.432" > <testsuite name= "UserService" tests= "15" failures= "1" time= "2.1" > <testcase name= "should create user with valid email" classname= "UserService" time= "0.234" > <!-- Empty = passed --> </testcase> <testcase name= "should reject duplicate email" classname= "UserService" time= "0.089" > <failure message= "Expected 409, got 200" type= "AssertionError" > Expected status code 409 but received 200 Request: POST /api/users Body: {"email": "existing@example

A coding agent can produce a confident patch for the wrong problem when the input is only: The upload sometimes fails. Please fix it. That sentence does not identify the smallest failing input, exact error, environment, expected result, frequency, or even whether the reporter reproduced it personally. If the first instruction is “fix it”, the agent has room to turn a suspected cause into a fictional fact. The safer sequence is: preserve the observed failure; record the real environment; state expected versus actual behaviour; reduce the failing case; prove whether it repeats; diagnose and repair only after that evidence exists. 1. Preserve the observed failure Capture the exact error, status code, incorrect output, affected route or command, timestamp, and smallest known input. Remove secrets and personal data before putting logs into an agent context. If the report came from another person and you have not reproduced it, label it as second-hand rather than silently upgrading it to fact. Weak: CSV imports are broken. Useful: At 14:22 UTC, POST /imports returned HTTP 500 for minimal.csv. Response: "column index out of range". The same account can import one-column.csv successfully. 2. Record the environment you can prove Inspect rather than guess: repository and commit; runtime version; operating system or container image; package lockfile; relevant feature flags; local, test, staging, or production target. Do not infer production configuration from your laptop. Environment differences are often part of the bug. 3. Separate expected and actual behaviour Write two observable statements: Expected: POST /imports accepts the smallest valid two-column CSV and returns HTTP 201. Actual: The same fixture returns HTTP 500 with "column index out of range". Neither statement should include the suspected root cause. “Expected: parser handles the off-by-one bug” already assumes the diagnosis. You have not earned that conclusion yet. 4. Reduce the reproduction Start with the repor

Most ad-hoc timing code in Python looks like this: start = time . perf_counter () result = do_work () elapsed = time . perf_counter () - start stats [ name ]. append ( elapsed ) It works, until do_work() raises. Then the line that records the timing never runs, the exception propagates, and the one call that was probably slowest — the one that failed — is silently missing from your stats. If you're using timing data to find what's expensive, the failing case is exactly the one you can least afford to lose. timerx is a small, dependency-free Python timing library — a decorator, a context manager, and named stopwatches, all backed by one stats store. The one rule that shapes the whole implementation: a timing gets recorded whether or not the timed code raised. Decision 1: finally , everywhere, no exceptions to the rule @functools.wraps ( target ) def wrapper ( * args : Any , ** kwargs : Any ) -> Any : started = self . _clock () try : return target ( * args , ** kwargs ) finally : elapsed = self . _clock () - started with self . _lock : self . _record ( label , elapsed ) return wrapper The async wrapper is the identical shape with await added. The context manager ( _Lap ) does the same thing structurally, just split across __enter__ / __exit__ instead of try / finally : def __exit__ ( self , * exc_info : object ) -> bool : if self . _started is None : raise RuntimeError ( " timerx lap exited before it was entered " ) elapsed = self . _timer . _clock () - self . _started with self . _timer . _lock : self . _timer . _record ( self . _name , elapsed ) return False Note the return False — __exit__ deliberately never swallows the exception. It records the timing and lets the exception continue propagating unchanged, because a timing library has exactly one job here: observe, not intervene. A version that suppressed exceptions to "clean up" would be actively dangerous to drop into someone else's codebase. Three entry points — decorator, context manager, stopwatch — and all t

If you're writing Playwright API tests manually from an OpenAPI/Swagger spec, you're doing work that should be automated. Every endpoint in your spec already tells you: What the request looks like (path, method, parameters, body schema) What responses to expect (200, 401, 404, 422...) What fields are required What security is needed That's not documentation — it's a test plan. You're just not running it yet. What I built I got tired of the boilerplate loop: read spec → write happy path → add 401 test → add missing-field test → repeat for 40 endpoints. So I built a tool that does it for you. swagger-to-playwright.vercel.app takes your OpenAPI 3.x spec (YAML or JSON) and generates a ready-to-run Playwright .spec.ts file. For each endpoint, it produces four tests: 1. Happy path — calls the endpoint with valid data, asserts 2xx response and key fields in the body. 2. Auth check — if your spec declares a security scheme, it calls without a token and asserts 401. Only generated when the spec actually says authentication is required — no false positives. 3. Input validation — sends a request with missing required fields (or wrong types, invalid enums) and asserts 422. Reads directly from your schema's required array and field types. 4. Contract validation — if there's a path parameter, it calls with an invalid value and asserts 404. What the output looks like Here's what you get for a POST /users endpoint with email and password required: import { test , expect } from ' @playwright/test ' ; test . describe ( ' POST /users ' , () => { test ( ' happy path — creates user successfully ' , async ({ request }) => { const res = await request . post ( ' /users ' , { data : { email : ' test@example.com ' , password : ' password123 ' } }); expect ( res . status ()). toBe ( 201 ); const body = await res . json (); expect ( body ). toHaveProperty ( ' id ' ); }); test ( ' auth — 401 without token ' , async ({ request }) => { const res = await request . post ( ' /users ' , { headers : {

I gave an AI agent a calculator because I wanted one hard, inspectable point inside a probabilistic workflow. The model could interpret the request and explain the result. The calculator would perform the computation. It seemed like a clean division of labor. Then I changed one multiplication sign into addition. The calculator still passed 389 of the 390 tests in its Rust library harness. The sole failure compared its answer with NIST's certified results for the Longley regression dataset. That bothered me more than a completely broken build would have. I had treated deterministic computation as safer than asking a language model to improvise arithmetic. But deterministic does not mean trustworthy. A program can return the same wrong answer forever. “Source of truth” suddenly felt too comfortable. Before an AI agent delegates authority to a tool, that authority should be challenged—and remain revocable by evidence. The calculator is only the specimen. The larger idea is a way to place inspectable, replayable instruments inside probabilistic systems. The useful boundary is generation versus execution The interesting distinction is not model weights versus a “real CPU.” Model inference also runs on processors, and language models can learn genuine arithmetic procedures. The useful boundary is between generating an answer and executing a defined operation under a tested contract . Research on Program-Aided Language Models (PAL) makes a related split: the language model reads and decomposes a natural-language problem, while a runtime such as a Python interpreter executes the generated program. The model contributes flexible interpretation; the runtime contributes executable semantics. That is the division I want in an agent: At the semantic edge , the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result. At the computational edge , a narrow tool validates inputs, applies specified operations, enforces limits, and ret

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post. Between 21 and 24 July it did that four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong. None of those models got worse. Here's how I know, because the how is the only part worth reading. Two of them weren't the model Every point on the board carries a second number next to accuracy: reliability , the share of probe calls that actually came back. Look at the two Google alerts with that column showing: gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts" gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943 23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts" 24 Jul acc 0.971 reliability 1.000 <- next clean run Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one. I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight: api.groq.com -> 429: Rate limit reached for model ` llama-3.3-70b-versatile ` service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30 34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch. Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression." The other two were one question The remaining two alerts ar

There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything. That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything: No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers. The placebo arm is the part almost nobody runs, and it turned out to be the most important one. Why a placebo Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes. So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on claude-opus-4-8 . Finding 1: the placebo often made code bigger My anti-over-engineering skill ( underkill , ~20 lines) cut source LOC by -23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length "write clean, minimal, professional code" instruction didn't reduce c

Six failed CI runs. Thirty-six hours of GitHub Actions time. Every run timing out at exactly the 6-hour limit. The culprit was one line in tests/setup.js . The Setup We were building a multi-tenant platform with a PostgreSQL backend — around 76 database models handling everything from user accounts and billing to visitor logs and real-time notifications. The test suite had grown to roughly 1,140 test cases across 36 files. Standard stuff. CI ran on every PR. Tests passed locally. And then one day, CI just... never finished. The Anti-Pattern Here's what the test setup looked like: // tests/setup.js beforeEach ( async () => { const tableNames = await getTableNames (); // 76 tables await sequelize . query ( `TRUNCATE TABLE ${ tableNames . join ( ' , ' )} CASCADE;` ); }); The intent was clean isolation — every test starts with a blank slate. Reasonable in theory. Catastrophic in practice. The Math Do the multiplication: 76 tables × 1,140 tests = 86,640 TRUNCATE operations Each TRUNCATE TABLE ... CASCADE is not a cheap operation. PostgreSQL has to: Acquire exclusive locks on all referenced tables Walk the foreign key graph to find dependent tables Truncate each in dependency order Release locks With a moderately complex schema where most tables reference others (users → societies → members → invoices → payments → ...), a single TRUNCATE ... CASCADE on a central table can fan out into dozens of implicit truncations. Multiply that by 86,640 and you have a test suite that will never complete within any reasonable timeout. Why It Wasn't Caught Sooner Two reasons: 1. It used to be fast. When the suite had 50 tests and 20 tables, this pattern worked fine. 50 × 20 = 1,000 truncations — uncomfortable but survivable. Nobody noticed when the suite crossed a tipping point. 2. Local runs used a different database state. Locally, developers often ran a subset of tests with --grep or file-specific runs. The full suite was only ever run on CI, and CI was slow enough that most assumed i

Divergence escalates the wrong population: unanimous misses auto-pass Agent Determinism Illusions (Part 7) Where this fits: This part does not continue Part 13's probe-vs-prose thread. It returns to Part 6 's L2→L3 escalation rule — Dipankar's move of treating vote disagreement as the human-review signal. Alexey Spinov's follow-up comment says that signal points at the wrong population. Two experiments check whether he is right, and what to put in the tripwire instead. Part 6 drew this control flow: L2 multi-perspective votes │ unanimous ──────────► AUTO-PASS / AUTO-REJECT │ divergence (e.g. 2–1) ► L3 human The caveat was already in the text: divergence measures ambiguity; it does not fix unanimous systematic bias. Alexey's point is sharper — and it is about routing , not about another caveat paragraph. 1. Alexey's population mismatch On the Part 6 thread, Alexey Spinov wrote (paraphrased tightly): The dangerous failures are high-confidence and directional — systematic. Systematic bias is shared across prompts, not idiosyncratic (your own P3: majority voting doesn't fix it). So the three perspectives will tend to agree on exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population. He proposed two cheap replacements: T1 — deterministic tripwire on known-reversal classes (escalate regardless of agreement). T2 — treat unanimous + high-confidence on a historically reversal-prone class as escalate — the inverse of “high confidence, auto-pass.” That is the claim under test. Not “divergence is useless,” but “divergence alone is the wrong primary tripwire for the failure mode you already measured.” 2. Experiment A — offline proxy on DF v2 (no new API) Part 6's Mike Update already showed: of 96 DF v2 MISS runs, 95.8% sat at self-reported confidence ≥ 0.9 (avg 0.969). That mass is concentrated — Part 6 also reported ~80% of MISS runs from qwen3:0.5b —

Seu agente escreveu o teste. O Cursor sugeriu três casos extremos que você não havia pensado. O Copilot preencheu o corpo da requisição, e o Claude executou tudo uma vez e reportou “verde”. A pergunta é justa: se o agente faz tudo isso, a IA pode substituir completamente o teste de API? Experimente o Apidog hoje Não. A IA não substitui o teste de API, mas já substitui boa parte da autoria dos testes. Agentes elaboram casos, sugerem cenários extremos e geram payloads com eficiência. Porém, eles não garantem execução idêntica em cada commit, não bloqueiam merges com um resultado confiável e não decidem se um contrato está correto. Para isso, você precisa de uma ferramenta determinística e de revisão humana. Essa separação responde a uma dúvida maior: você ainda precisa de uma ferramenta de API na era dos agentes de IA ? Sim — mas o papel da ferramenta mudou. Use agentes para acelerar a autoria e ferramentas determinísticas para validar, executar e bloquear regressões. Onde este artigo difere do guia prático Se você procura instruções para gerar testes com agentes, consulte o guia sobre como usar agentes de IA para teste de API . Este artigo trata de outra pergunta: quais partes do fluxo você pode delegar a um agente e quais precisam continuar em uma suíte determinística? Uma implementação prática separa o trabalho assim: O agente lê a especificação e cria um rascunho de testes. Um desenvolvedor revisa cenários, asserções e regras de negócio. Um runner headless executa a suíte no CI. O pipeline bloqueia o merge usando o código de saída do runner. Quando algo falha, você inspeciona a requisição e a resposta reais. O que a IA faz bem em testes de API hoje Agentes removem trabalho repetitivo de autoria. Use-os principalmente para criar e expandir artefatos de teste. Elaborar uma primeira suíte a partir de uma especificação Entregue ao agente um endpoint, uma definição OpenAPI ou uma resposta de exemplo. Ele pode gerar rapidamente: verificações de status HTTP; asserções de