标签：#Testing

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade , the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks. If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now. The setup mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle. Round 1, on three real servers, cost about twelve cents and produced this: Server Static score Tool selection Args Refusal context7 (2 tools) 100 38% 100% 100% server-memory (9 tools) 81 93% 100% 100% server-slack (8 tools) 97 54% 100% 100% Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken. The eval was broken Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history . context7's get-library-docs needs a library ID that comes from resolve-library-id . These are pipelined tools : their required arguments are produced by other tools. My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong. The model wasn't confused. The model was right . The benchmark was grading correct multi-step reasoning as failure — and me

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

Every unsupervised AI agent we've reviewed that wrote Flutter code made the same seven mistakes. These aren't typos or stylistic differences. They're structural failures that compound—bad state management plus missing tests plus hardcoded colors means the codebase becomes expensive to theme, hard to test, and impossible to maintain at scale. Here's a small one to set the tone: a developer asked an agent to implement a GET request to an external service in a Dart project. The agent's solution was to shell out to curl via Process.run and parse the stdout. Not package:http . Not dio . Not even dart:io 's own HttpClient . A subprocess call to a CLI tool, inside a language that's had first-class HTTP clients since Dart 1.0. That one is worth sitting with, because it's not really a Flutter problem — it's the whole pattern in miniature. The agent wasn't "wrong" that curl can make a GET request. It optimized for "this pattern appears constantly in training data" over "this is the idiomatic way to do it in the language I'm currently writing." Bash and curl show up in approximately every tutorial, README, and Stack Overflow answer ever written. package:http shows up in Dart-specific docs. Given no other constraint, the agent reached for the statistically dominant pattern, not the contextually correct one. The seven gaps below are the same failure mode, just less obvious than "shells out to curl." Here's what we found, with real code examples and the fixes that work. 1. Recomputing Derived State The Problem: Agents recalculate the same values across multiple locations instead of maintaining one source of truth. Imagine a checkout flow where the cart total is computed three separate ways: In the checkout page: (items.sum + tax) - discount In the footer: items.sum - discount + tax In the order summary: (items.sum - discount) * (1 + taxRate) Different calculations. Same semantic meaning. One will break first. The Fix: Derive values once in the state layer using streams. Let all w

How to Get the Maximum Value Out of This Framework Having built and iterated on this prompt through multiple production edge cases, here are the exact execution strategies I recommend depending on your workflow: 1. The Human-in-the-Loop Workflow (Recommended for Chat UI) Run it in two separate chat threads: Don’t let long conversation history degrade your test accuracy. Run Phase 1 in Thread A to get your gap analysis and critical questions. Review the gaps, clarify what you can, and then update your original requirement text. Start Thread B for Phase 2: Open a fresh conversation, paste the updated requirements + this framework, and jump straight into generation. This completely eliminates context drift and keeps the LLM laser-focused on state mutation rules. 2. The 2-Pass Programmatic Auditor (For Automated CI/CD Pipelines) If you’re calling an LLM via API or integrating this into a pre-commit GitHub Action, split the execution into two isolated passes: Pass 1: Run Phase 1 & 2 to generate the initial test table. Pass 2 (The Audit Pass): Feed the generated table into an isolated, secondary prompt whose only job is to enforce the Verification Check (verifying exact boundary literals, API status codes, and non-mutation assertions). Separation produces drastically higher assertion reliability than asking a model to self-audit in a single turn. 3. How to Live-Demo or Teach This For Live Streams & YouTube: This framework makes for a high-signal live demo. Paste an intentionally ambiguous user story (e.g., a webhook handler or payment endpoint), watch Phase 1 halt at the gate live, discuss the surfaced edge cases on camera, reply PROCEED, and review the generated DEFERRED risk rows. It shifts the content focus from “Look at this cool AI tool” to “This is how Senior SDETs think about systems.” For Technical Writing & Post-Mortems: The progression from a naive “write me test cases” prompt to a strict 2-phase state-machine framework is a technical narrative in itself. Break

A few years ago, the hardest part of building a browser test framework was getting started. You had to choose a runner, configure browsers, create page objects, wire up reporting, add retries, manage secrets, connect it to CI, and convince someone else on the team to learn how the whole thing worked. Today, you can open an AI assistant and ask it to generate most of that before lunch. That sounds like a dramatic improvement. In some ways, it is. But it also moves the bottleneck. The question is no longer, “Can we create a framework?” The question is, “Can we operate what was created?” That distinction matters more than it appears. Generation cost is not ownership cost A generated framework feels cheap because the first version arrives quickly. The code compiles, a few tests pass, and the pull request looks more complete than anything you could have written in an afternoon. Then reality starts applying pressure. The application changes. Authentication behaves differently in staging. A shared helper starts hiding failures. Parallel workers collide over test data. Someone upgrades a dependency and three reporters stop agreeing with one another. The initial generation was fast. The ownership cost was merely deferred. This is the central problem described in what actually breaks when Claude generates a large Playwright framework . Large generated systems often fail in the seams: fixtures, abstractions, environment assumptions, test data, and conventions that were never explicitly agreed upon. The code may be readable line by line while the system remains difficult to reason about as a whole. That is a dangerous form of complexity because it looks productive. More code can hide less understanding Teams sometimes evaluate AI-generated automation by counting output: number of test files; number of scenarios; number of passing checks; number of prompts completed; number of lines added. Those numbers are easy to produce and easy to report. They are also weak proxies for confi

Everything you need to know to find your first vulnerability, get paid, and build a real reputation in cybersecurity — without breaking any laws. If you've typed "how to start bug bounty hunting" into Google recently, you're not alone. It's one of the fastest-growing searches in cybersecurity right now, and for good reason: it's one of the only paths in tech where a total beginner with no degree can find a real flaw, report it, and get paid the same week. This guide answers the questions people are actually searching in 2026 — what bug bounty hunting is, which bugs pay the most right now, how AI has changed the game, and how to land your first bounty. What Is Bug Bounty Hunting, Exactly? Companies invite independent researchers to test their websites, apps, and APIs for security flaws — legally. When you find a real vulnerability, you write a report explaining what it is, how to reproduce it, and what damage it could cause. If the company confirms it, they pay you based on severity. It's not hacking in the movie sense. It's structured, permitted testing within a defined scope — the specific domains, apps, or features the company has authorized you to test. Step outside that scope, and you've crossed from "bug bounty hunter" into "unauthorized access," which is a crime in nearly every country. The Best Platforms to Start On in 2026 Three platforms dominate the space: HackerOne — the largest and most beginner-friendly, with the widest range of programs Bugcrowd — strong onboarding and clear scope documentation Immunefi — the go-to platform if you're interested in web3 and smart contract security, which currently pays some of the highest bounties in the industry Start with Vulnerability Disclosure Programs (VDPs) — these often don't pay, but they let you build a track record, earn private invites, and practice on real targets without competing against thousands of other hunters for a bounty. What Bugs Are Actually Paying Right Now The vulnerability landscape has shifte

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

I wrote about the Electricity Planning Engine a little while back, including a timezone bug that made a correct price look "not found" after a database round trip. A few days later, Alex Shev left this comment: Timezone bugs are brutal in planning engines because the result can look mathematically correct while being operationally wrong. Energy workflows especially need tests around boundaries, not just averages. That is a genuinely sharp way to put it, and it is not just a comment about the bug I already wrote about. It is a comment about how I test the project in general, and I did not like how well it applied once I went and checked. The part that stung a little "Looks mathematically correct while being operationally wrong" is exactly what the original timezone bug was. PriceSeries::priceAt() threw a clean "price not found" error, which is arguably the good version of that failure mode: loud, easy to catch, hard to ship. A quieter version of the same class of mistake, off by one hour instead of missing entirely, would not throw anything. It would just return a plan that looks completely reasonable and is wrong the entire time it runs. Alex's second point, boundaries over averages, is the one I actually had to go check rather than just agree with in the abstract. So I opened tests/Unit/Domain/Contract/PricingStrategyTest.php and looked at every hour used in every peak/off-peak assertion: new DateTimeImmutable ( '2026-07-18 14:00:00' ) // peak new DateTimeImmutable ( '2026-07-18 23:00:00' ) // off-peak new DateTimeImmutable ( '2026-07-18 05:00:00' ) // off-peak 14:00, 23:00, 05:00. Every single one comfortably inside its window. None of them anywhere near the actual transition. The off-peak slot in the config is 22:00 to 06:00 , and the comparison behind that lives in TimeSlot::contains() : // wraparound slot, e.g. 22:00 -> 06:00 return $minuteOfDay >= $this -> startMinuteOfDay || $minuteOfDay < $this -> endMinuteOfDay ; That >= versus < is exactly the kind of one-

The CI job just hit 28 minutes. Again. You pull up the duration report expecting to blame a bloated integration test or a slow environment spin‑up. Instead the longest stage stares back at you: a collection of Cucumber feature files that haven’t caught a real bug in months. Maybe years. They run on every commit, green circle after green circle, while your team mutters about slow pipelines and nobody dares touch them. Most teams treat those scenarios like documentation. “They describe the system,” someone once said, as if a Gherkin file were a legal contract. Others cling to the sunk cost: a year ago a whole squad spent two sprints writing them, polishing the grammar, aligning step definitions. Deleting them would feel like admitting waste. Experienced engineers see it differently. They treat a scenario that never fails as a liability you’re paying for on every push. Not neutral. Liable. Compute cycles, developer attention, flake‑debugging time, and the quiet toll it takes on trust in the pipeline. The principle is blunt: if a test hasn’t failed in the last few sprints, you’re already paying its full cost and receiving nothing in return. That doesn’t mean you delete everything green. But it does mean you audit with the same seriousness you’d use for a memory leak. What the green wall actually costs The damage is not abstract. A pipeline bloated with stale scenarios hurts you in five concrete ways. First, feedback slows. Every extra minute between push and result stretches the loop that tells a developer they’re safe to merge. Multiply across a team and you’re losing hours per week to waiting. Second, flakiness increases. When you have many scenarios, a single unstable environment variable can produce a handful of failures that are not regressions at all. Engineers learn to retry, then to ignore. Third, confidence erodes. If half the suite is ceremonial, a genuine failure might be dismissed as “just another flaky test” until it reaches production. Fourth, maintenance

The manager opens your portfolio. Your resume says you have five years of automation experience. The README lists Selenium, Playwright, Appium, Jenkins, Docker, Kubernetes. He scrolls. There is no code. The browser tab closes. This is you. Not because you lack skill—you have it—but because your public proof reads like a shopping list. The tools you name say nothing about how you think when a flaky test fails at 2am, or how you convince a developer that a bug is real. If you’re serious about landing a role that demands more than record-and-playback, you need to stop treating your portfolio like a keyword bingo card. Here are three mistakes that kill your chances instantly, and exactly how to fix them. Mistake 1: Tool jockeying Listing every automation framework you’ve heard of is a reflex. A hiring manager sees "Proficient in Cypress, Playwright, Selenium, WebDriverIO" and assumes you ran npm init once in each and called it done. Most testers frontload tools because they’re scared of the empty space where code belongs. Experienced testers show one test, deliberately written, with a comment that explains a trade-off they chose. The difference is not volume. A single 30-line script that handles a login flow with a purposeful wait strategy teaches more about you than a six-tool résumé. I’ve deleted my own old projects after re-reading them and realizing they said nothing about why any assertion existed. That quiet cringe is the signal you’re ready to improve. What you ship in your portfolio must answer one question: "What did this person decide, and why?" Move your tool list to a footnote. Let a real test carry the message. Mistake 2: The perfect test trap A portfolio full of green builds is a trap. Every team knows that real automation breaks: the CI node runs slow, the third-party API throttles you, the DOM renders a fraction of a second late. Showing only passing tests hides how you handle the ugly parts of the job. Most testers polish every assertion until it’s spot

Most people think the SDET title means you are automatically more valuable than a manual tester. The SDET writes Playwright scripts. The SDET configures CI pipelines. The SDET talks about page objects and retry strategies. The manual tester clicks through screens and writes bug reports. Here is the truth I have watched play out across teams: the manual tester who can write a SQL join will consistently outperform the SDET who cannot. Not because SQL is magic. Because SQL is the shortest path to understanding what the system actually stores, not what the UI shows you. The problem with automation-first thinking I have seen SDETs spend three sprints building a test suite that validates every button, every dropdown, every error toast. The suite passes in CI. The suite passes in staging. The suite passes in production. And the bug still ships. Why? Because the test checked that the UI rendered correctly. It never checked that the database actually saved the right record. The SDET wrote assertions against DOM elements, not against data. The manual tester, meanwhile, ran a simple query. Saw the order status was "pending" when it should have been "confirmed." Filed a bug with the exact SQL that proved the issue. The developer fixed it in ten minutes. That is not a story about manual versus automated. That is a story about data literacy versus UI obsession. What a SQL join gives you that a locator never will A Playwright locator tells you something is on the screen. A SQL join tells you something is true. When you write page.getByText('Order confirmed') , you are testing that the frontend displays those words. You are not testing that the backend actually confirmed the order. You are not testing that the payment gateway returned success. You are not testing that the inventory decremented. A SQL join connects those dots. SELECT o . id , o . status , p . status AS payment_status , i . quantity AS remaining_stock FROM orders o JOIN payments p ON o . id = p . order_id JOIN invent

LLM applications fail silently at the semantic level. Standard unit tests verify that functions return values, but they cannot detect if the output is factually wrong, off-tone, or missing steps. Evals fix this by running a prompt, inspecting the output, and programmatically or judgmentally deciding its quality. Why LLM Testing is Different Unit tests break on non-determinism. Temperature-driven randomness means identical inputs produce varying valid outputs, which defeats static equality checks. Semantic equivalence ("Paris is the capital" vs. "Capital is Paris") makes string matching useless. Small model or prompt updates cause gradual quality drift that only surfaces at scale. Tone and accuracy require judgment that basic assertions cannot encode. The Three Eval Types 1. Heuristic Evals check measurable properties: JSON validity, word counts, PII presence. They are fast and objective — ideal for regression gates on every pull request. 2. LLM-as-Judge uses a second LLM to grade output against a rubric. Best for subjective content too complex for code but too vast for manual review. 3. Human Evals are the ground truth. Use them to build your golden dataset, calibrate the LLM judge, and finalize decisions before major releases. Building Your Eval Infrastructure Start minimal: a Python script, a JSON file of test cases, and a loop comparing LLM output against assertions or a rubric. Curate a golden dataset of 500–1000 production-representative requests including edge cases and adversarial prompts. Refresh it quarterly to match user behavior drift. Automate the harness in GitHub Actions. Block deployments when pass rates drop below your threshold (90% is a common starting point). Run fast heuristic tests on every PR and full LLM-judge evals nightly. Key Tools PromptFoo — Open-source, YAML-based test cases with a built-in CLI runner and diff reporting. Braintrust — Hosted platform for dataset management and experimental comparison. Inspect — Open-source framework by th

A test failure that takes 20 minutes to surface, buries the error in 3000 lines of log output, and gives no context about what changed is nearly useless. Good test reporting transforms raw pass/fail data into actionable signals. Failing fast — stopping the pipeline the moment you have enough information to make a decision — keeps feedback loops tight and respects developer time. These two concerns are deeply connected: you can only fail fast confidently when your reporting is good enough that a fast failure still gives you everything you need to fix the problem. What Good Test Reporting Looks Like Before discussing implementation, it's worth being precise about what "good" means here: Immediate visibility — failures are surfaced at the PR/commit level, not buried in logs Failure context — what failed, with what input, producing what output, and in which file/line Historical comparison — is this a new failure or a pre-existing one? Trend data — is this test getting flakier? Is the suite getting slower? Actionability — the report points to a fix, not just a symptom Most teams get #1 and stop. The teams that nail all five have fundamentally different debugging velocity. JUnit XML: The Universal Format JUnit XML is the lingua franca of CI test reporting. Almost every test framework can emit it, and almost every CI platform can ingest it. Understanding the format helps you produce better reports. <?xml version="1.0" encoding="UTF-8"?> <testsuites name= "My Test Suite" tests= "42" failures= "2" errors= "0" time= "8.432" > <testsuite name= "UserService" tests= "15" failures= "1" time= "2.1" > <testcase name= "should create user with valid email" classname= "UserService" time= "0.234" > <!-- Empty = passed --> </testcase> <testcase name= "should reject duplicate email" classname= "UserService" time= "0.089" > <failure message= "Expected 409, got 200" type= "AssertionError" > Expected status code 409 but received 200 Request: POST /api/users Body: {"email": "existing@example

A coding agent can produce a confident patch for the wrong problem when the input is only: The upload sometimes fails. Please fix it. That sentence does not identify the smallest failing input, exact error, environment, expected result, frequency, or even whether the reporter reproduced it personally. If the first instruction is “fix it”, the agent has room to turn a suspected cause into a fictional fact. The safer sequence is: preserve the observed failure; record the real environment; state expected versus actual behaviour; reduce the failing case; prove whether it repeats; diagnose and repair only after that evidence exists. 1. Preserve the observed failure Capture the exact error, status code, incorrect output, affected route or command, timestamp, and smallest known input. Remove secrets and personal data before putting logs into an agent context. If the report came from another person and you have not reproduced it, label it as second-hand rather than silently upgrading it to fact. Weak: CSV imports are broken. Useful: At 14:22 UTC, POST /imports returned HTTP 500 for minimal.csv. Response: "column index out of range". The same account can import one-column.csv successfully. 2. Record the environment you can prove Inspect rather than guess: repository and commit; runtime version; operating system or container image; package lockfile; relevant feature flags; local, test, staging, or production target. Do not infer production configuration from your laptop. Environment differences are often part of the bug. 3. Separate expected and actual behaviour Write two observable statements: Expected: POST /imports accepts the smallest valid two-column CSV and returns HTTP 201. Actual: The same fixture returns HTTP 500 with "column index out of range". Neither statement should include the suspected root cause. “Expected: parser handles the off-by-one bug” already assumes the diagnosis. You have not earned that conclusion yet. 4. Reduce the reproduction Start with the repor

Most ad-hoc timing code in Python looks like this: start = time . perf_counter () result = do_work () elapsed = time . perf_counter () - start stats [ name ]. append ( elapsed ) It works, until do_work() raises. Then the line that records the timing never runs, the exception propagates, and the one call that was probably slowest — the one that failed — is silently missing from your stats. If you're using timing data to find what's expensive, the failing case is exactly the one you can least afford to lose. timerx is a small, dependency-free Python timing library — a decorator, a context manager, and named stopwatches, all backed by one stats store. The one rule that shapes the whole implementation: a timing gets recorded whether or not the timed code raised. Decision 1: finally , everywhere, no exceptions to the rule @functools.wraps ( target ) def wrapper ( * args : Any , ** kwargs : Any ) -> Any : started = self . _clock () try : return target ( * args , ** kwargs ) finally : elapsed = self . _clock () - started with self . _lock : self . _record ( label , elapsed ) return wrapper The async wrapper is the identical shape with await added. The context manager ( _Lap ) does the same thing structurally, just split across __enter__ / __exit__ instead of try / finally : def __exit__ ( self , * exc_info : object ) -> bool : if self . _started is None : raise RuntimeError ( " timerx lap exited before it was entered " ) elapsed = self . _timer . _clock () - self . _started with self . _timer . _lock : self . _timer . _record ( self . _name , elapsed ) return False Note the return False — __exit__ deliberately never swallows the exception. It records the timing and lets the exception continue propagating unchanged, because a timing library has exactly one job here: observe, not intervene. A version that suppressed exceptions to "clean up" would be actively dangerous to drop into someone else's codebase. Three entry points — decorator, context manager, stopwatch — and all t

If you're writing Playwright API tests manually from an OpenAPI/Swagger spec, you're doing work that should be automated. Every endpoint in your spec already tells you: What the request looks like (path, method, parameters, body schema) What responses to expect (200, 401, 404, 422...) What fields are required What security is needed That's not documentation — it's a test plan. You're just not running it yet. What I built I got tired of the boilerplate loop: read spec → write happy path → add 401 test → add missing-field test → repeat for 40 endpoints. So I built a tool that does it for you. swagger-to-playwright.vercel.app takes your OpenAPI 3.x spec (YAML or JSON) and generates a ready-to-run Playwright .spec.ts file. For each endpoint, it produces four tests: 1. Happy path — calls the endpoint with valid data, asserts 2xx response and key fields in the body. 2. Auth check — if your spec declares a security scheme, it calls without a token and asserts 401. Only generated when the spec actually says authentication is required — no false positives. 3. Input validation — sends a request with missing required fields (or wrong types, invalid enums) and asserts 422. Reads directly from your schema's required array and field types. 4. Contract validation — if there's a path parameter, it calls with an invalid value and asserts 404. What the output looks like Here's what you get for a POST /users endpoint with email and password required: import { test , expect } from ' @playwright/test ' ; test . describe ( ' POST /users ' , () => { test ( ' happy path — creates user successfully ' , async ({ request }) => { const res = await request . post ( ' /users ' , { data : { email : ' test@example.com ' , password : ' password123 ' } }); expect ( res . status ()). toBe ( 201 ); const body = await res . json (); expect ( body ). toHaveProperty ( ' id ' ); }); test ( ' auth — 401 without token ' , async ({ request }) => { const res = await request . post ( ' /users ' , { headers : {

I gave an AI agent a calculator because I wanted one hard, inspectable point inside a probabilistic workflow. The model could interpret the request and explain the result. The calculator would perform the computation. It seemed like a clean division of labor. Then I changed one multiplication sign into addition. The calculator still passed 389 of the 390 tests in its Rust library harness. The sole failure compared its answer with NIST's certified results for the Longley regression dataset. That bothered me more than a completely broken build would have. I had treated deterministic computation as safer than asking a language model to improvise arithmetic. But deterministic does not mean trustworthy. A program can return the same wrong answer forever. “Source of truth” suddenly felt too comfortable. Before an AI agent delegates authority to a tool, that authority should be challenged—and remain revocable by evidence. The calculator is only the specimen. The larger idea is a way to place inspectable, replayable instruments inside probabilistic systems. The useful boundary is generation versus execution The interesting distinction is not model weights versus a “real CPU.” Model inference also runs on processors, and language models can learn genuine arithmetic procedures. The useful boundary is between generating an answer and executing a defined operation under a tested contract . Research on Program-Aided Language Models (PAL) makes a related split: the language model reads and decomposes a natural-language problem, while a runtime such as a Python interpreter executes the generated program. The model contributes flexible interpretation; the runtime contributes executable semantics. That is the division I want in an agent: At the semantic edge , the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result. At the computational edge , a narrow tool validates inputs, applies specified operations, enforces limits, and ret

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post. Between 21 and 24 July it did that four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong. None of those models got worse. Here's how I know, because the how is the only part worth reading. Two of them weren't the model Every point on the board carries a second number next to accuracy: reliability , the share of probe calls that actually came back. Look at the two Google alerts with that column showing: gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts" gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943 23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts" 24 Jul acc 0.971 reliability 1.000 <- next clean run Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one. I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight: api.groq.com -> 429: Rate limit reached for model ` llama-3.3-70b-versatile ` service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30 34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch. Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression." The other two were one question The remaining two alerts ar