标签：#Testing

In the last article, I wrote about a release story that was weaker than the engine underneath...

You have done everything right. You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you. Six months later, your engineers have started ignoring test failures. Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt. What is flakiness? A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services. All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it. The reason is not the technical difficulty. The scale The numbers are worth stating clearly, because they establish what is actually at stake here: At Google , approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression. At Microsoft , roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure. Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling. Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise. These are not team

Turning an iOS app into a React web app is no longer the hard part. Modern coding agents can generate a convincing first version quickly. The hard part is answering three less glamorous questions: Did we discover every important screen and state? Did the generated app preserve the source behavior and data? Is the result actually close to the native UI, or does it merely look plausible? I built Grounded iOS-to-Web Harness to make those questions auditable. 🔗 GitHub: https://github.com/tiezhu0415/grounded-ios-to-web-harness What it is Grounded iOS-to-Web Harness is an experimental, lightweight grounding + verification layer for migrating iOS apps into complete, interactive, mobile-sized WebApps. Claude Code remains the primary implementer. The Harness does not prescribe the React component tree, choose a state-management library, or replace the coding agent. Instead, it establishes source facts and verifies the result against evidence from the original app. iOS source + Assets + code graph + runtime states ↓ locked source facts ↓ per-screen implementation context ↓ agent builds the React WebApp ↓ coverage + truth + behavior + critical VRT ↓ bounded repair, then human review Why prompt-only migration is not enough A prompt such as “convert this iOS app to React” can produce a good demo. But on longer tasks, an agent may miss screens, implement only one state, invent data or assets, choose navigation that differs from iOS, forget earlier facts, or optimize a screenshot while breaking real interaction. This project treats the iOS source and Assets as the truth for content and behavior, while runtime screenshots provide evidence for what the result should look like. The pipeline 1. Discover and reconcile source facts Static source inspection, codebase-memory, and necessary iOS runtime exploration are combined into machine-readable facts for screens, UI states, actions, navigation outcomes, cross-screen flows, real data and asset origins, and source confidence. Facts are l

By using a sustainable testing strategy, you can skip unnecessary tests, ensure failing fast and early, and only run tests affected by code changes. Tracking energy use per test and using static code analysis can help spot inefficiencies and guide optimization efforts. By Ben Linders

The next version of the European accessibility standard is scheduled for citation on 30 November 2026. EN 301 549 V4.1.1 swaps WCAG 2.1 for WCAG 2.2, and six new success criteria arrive at levels A and AA. There is a small industry of readiness checklists for it already. So I measured what the current version looks like first. The answer is that the deadline people are preparing for is not the one they have missed. I scanned the most-visited websites on EU country domains and counted which clauses of EN 301 549 they fail today, under the version cited right now. Not the one arriving. The one in force since before the European Accessibility Act deadline passed in June 2025. Sixty-four per cent fail clause 9.4.1.2, Name, Role, Value. It is Level A, the lowest bar the standard has, and it has been in every version of WCAG since 2008. Here is the full picture, and then the reasons to distrust parts of it. What was measured Clause Criterion Level Sites failing 9.4.1.2 Name, Role, Value A 96 of 149 (64%) 9.1.4.3 Contrast (Minimum) AA 66 of 149 (44%) 9.2.4.4 Link Purpose (In Context) A 53 of 149 (36%) 9.2.5.8 Target Size (Minimum) AA 51 of 149 (34%) 9.1.1.1 Non-text Content A 35 of 149 (23%) 9.1.3.1 Info and Relationships A 27 of 149 (18%) Target size is the odd one out: it is a WCAG 2.2 criterion and not currently required. It is in the table because it is the only one of the six arriving in V4.1.1 that the rule engine used here has a check for, which is a point I will come back to. Thirty-two sites of the 149, about one in five, failed nothing that automated testing can detect. That is not the same as passing. Two of those rows are not independent. The rule that most often breaks Name, Role, Value is a link with no accessible name, and the same defect also fails Link Purpose. One missing label lands in two rows of that table. I am pointing this out because a table of six numbers implies six problems, and some of them are the same problem counted twice under different cla

In the porting guide I wrote that the exam is built to be stolen — follow five steps and it moves to any job. So I tried being the other person. Following only what the guide says, start to finish. Where to steal it to — my own tool, of all places For the second job I picked YouTube comment classification : scraping 20,000 comments and sorting each one into "a need," "chatter," or "a signal someone would pay." Every number in the 20,000-comments post came out of this classifier. Which makes this a double-edged experiment. It tests whether the exam ports — and at the same time it tests whether the tool that produced my own published numbers can pass an exam. The twist comes first — the tool wasn't an AI Before writing a single question, I opened the classifier's code to understand what I was about to test. The thing that sorted 20,000 comments was not an AI. It was a regex — word matching: "if the comment contains this keyword, it's this category." The second line of the actual data file was already an accident. My grad-school senior bet that nobody would bother replacing humanities majors because they don't pay. He was right. Social commentary. Not a need, and certainly not about errors. The classifier had filed it as a need in the "errors & debugging" category — because the Korean phrase for "doesn't pay" contains the same two characters as the error keyword "doesn't work." With 10,000 likes, it sat near the top of the ranking. The accident showed up before the exam even existed. Then I followed the five steps exactly Step 1 — write down the worst. These classifications feed decisions about what to build and what to sell. So the worst accident is "promoting chatter into a need and manufacturing fake demand." A product decision built on fake demand burns weeks. Step 2 — the grade table. Four grades: fatal, risky, missed, harmless. In the guide I had written "only the first line, FATAL, is redefined per project; the other three read the same everywhere." Porting it,

An agent patch that passes its own tests is a baseline, not a verdict. The same model wrote the code and the tests, so both share the same blind spots. Mutation testing scores the tests themselves: inject a fault, run the suite, and see whether it notices. In practice, the first mutant often survives. Previous rounds on this account established three gates before merge: property checks, fixtures, and a freeze on flaky tests. This round adds a fourth gate that runs after the suite is green. It answers a different question — not "does the patch work?" but "would the tests catch it if it didn't?" Why green tests from an agent are weak evidence Code coverage measures execution, not detection. A test can execute a line and still miss the bug on it. A suite that only checks is_even(2) and is_even(4) runs both lines, passes both assertions, and stays blind to a mutation that flips == to != . Agents produce this shape of test by default. They follow the happy path, mirror the implementation, and rarely probe boundaries. The result is a suite that is green, fast, and weak for regression. Mutation testing converts that intuition into a number. For each small fault, rebuild and rerun. If the tests fail, the mutant is killed. If they pass, it survived — and you found a hole in the suite, not in the code. A minimal harness The harness below applies one mutation at a time to the implementation file, compiles it together with an unchanged test file, runs the resulting binary, and records the outcome. It is deliberately small: regex-based, two files, no dependencies beyond a compiler. #!/usr/bin/env python3 # mutate.py — score a test binary against source mutations. import re import subprocess import sys import tempfile from pathlib import Path MUTATIONS = [ ( " eq_to_neq " , r " == " , " != " ), ( " lt_to_le " , r " < " , " <= " ), ( " add_to_sub " , r " \+ " , " - " ), ( " zero_to_one " , r " return 0; " , " return 1; " ), ] def mutate_once ( src : str , pattern : str , replaceme

The most common documentation failure is not a weak prompt or a lazy writer; it is the absence of a clear boundary between machine-draftable content and human-owned claims. A pipeline that drafts reference sections with free-tier model access and then verifies them with a symbol drift check turns docs into a testable artifact instead of a trust exercise. The model writes the inventory, and the human owns the promises. Why documentation rots inside a healthy CI pipeline Documentation bugs share a distinctive property: they are usually discovered by the people who consume the API, not by the pipeline that builds it. A function renamed in the last refactor stays documented under its old name until a user files an issue, and a newly added flag never appears in the docs at all. The root cause is structural, because nothing in the merge pipeline compares the documented surface against the actual code surface. A prompt cannot know what changed inside a pull request, so the fix has to live in the pipeline around the model. The workflow drafts reference material, validates that every documented symbol still exists, and routes the remaining claims to a human reviewer. That division of labor is the entire design, and each step has a concrete tool. The ownership boundary: what a model may draft The first step is to separate documentation into two classes by asking a single question: can this statement be verified against the codebase alone? If the answer is yes, a model may draft it, and if the answer is no, a human must own it. The table below applies that test to the statement types that appear in most API docs. The model may draft A human must own Function and class inventories Behavioral guarantees CLI flags and their defaults Security and authentication properties Config keys and their types Compatibility and support promises Error codes and exit statuses Deprecation timelines Compilable usage examples Performance or cost claims Parameter descriptions from signatures Ratio

The agent patch passed the gates I ran on it. Its unit tests were green, fixtures matched, nothing was flaky. Then I seeded four faults into the implementation, one at a time. Two survived. That gap is what this article is about. A green suite is a claim, not a measurement. Mutation testing turns it into a measurement: introduce a fault, run the suite, and see whether the suite notices. I now run this loop before merging any agent-written patch, and the whole thing costs a few rebuilds. Why green tests lie A passing test proves one thing only: the test and the implementation agree on the inputs the test exercised. When an agent writes both the patch and the tests, the tests inherit the patch's assumptions. If the implementation encodes a wrong assumption, the test encodes the same one. The suite is green because it is blind, not because the code is right. The patch in this article came from a free model on MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model wrote a bounded queue and a test file. The test file was not wrong. It was blind in exactly the place the implementation was wrong. The method: five steps Mutation testing is easy to describe and awkward to skip: Freeze flaky tests first. A flaky test fails at random, so it makes every mutation look like a kill. The signal is garbage. This is the flaky freeze from the gates post; without it, the numbers mean nothing. Select the functions the patch touched. Mutating untouched code measures someone else's tests. Generate mutations. Each mutation is one small fault: drop a modulo, flip a comparison, change an increment. Run the suite against each mutation. Rebuild, run, record. Gate on the kill rate. A surviving mutation means the suite cannot detect that fault class. Send the patch back with the survivor list as evidence. The artifact A minimal bounded queue, the agent's test, and a small Python driver. The queue: // bounded_queue.h #pragma once

I Reviewed 12 Free-Tier Integrations. The Same Six Myths Kept Appearing. Last month I reviewed twelve integrations that used free model servers. All twelve carried the same wrong assumptions. None of them tested those assumptions. That's the real problem. Not the free tier. The mental model. How many of these myths do you believe? I believed all of them. Here's what the code told me. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use their free server option in side projects. The probe below works with any OpenAI-compatible endpoint, including theirs. The Six Myths Myth 1: "Free tier is just a demo" Teams treat free servers like toy boxes. They build demos, then throw them away. Evidence: three of the twelve integrations were internal tools in daily use. The free tier was the production environment. Nobody planned for that. Corrected mental model: free tier is a constraint, not a demo. If the tool survives, the constraint becomes your architecture. Design for it from day one. Myth 2: "A 200 means it worked" The most dangerous assumption. A 200 only means the HTTP layer succeeded. It says nothing about the content. I found empty completions, truncated JSON, and repeated boilerplate. All returned 200. All broke the caller. Corrected mental model: validate the payload, not the status code. Check schema, length, and content markers. Myth 3: "Retries are free" When a request fails, developers retry immediately. Then again. Then again. That's a retry storm. It amplifies load exactly when the server struggles. I saw one integration fire eleven requests in four seconds. Corrected mental model: retries are a queue, not a hammer. Use exponential backoff with jitter. Add a circuit breaker. Myth 4: "The model is the same everywhere" Free and paid tiers often serve different models. Or the same name with different behavior. You cannot assume. Evidence: two integrations hard-coded model names that no longer existed. Responses came back, but from

The people who know your AI agent best may be the people least able to see all of its flaws. Not because they are bad engineers. Because they built it. Years ago, when I was taking art classes, my teacher told me something I've never forgotten: “Sara, you can't judge your own art.” I remember thinking, of course I can. 😂 Then she explained. After spending hours looking at the same piece, your eyes get filled with it. You stop seeing what is actually there. You see what you expect to see. I've used that lesson everywhere since. And I think AI agents have the same problem. You designed the requirements. You designed the system. You know why every decision was made. Then you design the evaluation and ask: “Does my agent actually work?” That's where the blind spot can appear. Your evaluation may end up testing the system according to the same assumptions that created it. The evaluator can inherit the system's assumptions Consider a simple requirement: “The agent should answer customer questions accurately.” Seems reasonable. So the team creates an evaluation set with questions that have clear intent and well-defined answers. The agent performs beautifully. 94%. Green dashboard. 🎉 But an external evaluator might ask a different question: What happens when the customer's request has two plausible interpretations? Now you have a different test: “Can I change my billing address?” Does the agent answer immediately? Does it ask which account or address the customer means? Does it make an assumption? The original evaluation may have been technically correct. It just never tested the ambiguity. That is the blind spot. Internal evaluation is still essential This isn't an argument that internal teams shouldn't evaluate their own systems. They absolutely should. The people who built the system understand its requirements, architecture, constraints, tools, and intended behavior better than anyone. That knowledge is extremely valuable when designing evaluations. But it can also crea

Measuring the impact of test automation goes beyond simple pass/fail ratios. To demonstrate real engineering excellence and business value, automation metrics must capture execution speed, suite stability, test coverage, maintenance cost, and CI/CD integration. Here is a comprehensive, unified KPI framework designed specifically for Playwright & JavaScript automation suites. 📊 Executive KPI Targets Category Metric Target Execution Speed Runtime Reduction 50% ↓ Efficiency Throughput +40% ↑ Stability Flaky Tests < 3% Reliability Retry Dependency < 5% Coverage Automation Coverage 80%+ Quality Defect Leakage 20–30% ↓ Productivity Script Dev Time 30% ↓ CI/CD Pipeline Time 40% ↓ ROI Automation ROI Positive (3–6 months) Cost Manual Effort Reduction 30–50% ↓ 1. Execution Efficiency & Speed Test Execution Time Reduction: Target 40–60% reduction vs legacy frameworks like Selenium. $$\text{Reduction \%} = \frac{\text{Old Time} - \text{New Time}}{\text{Old Time}} \times 100$$ Parallel Execution Efficiency: Measure tests executed per hour and parallel thread utilization. $$\text{Efficiency \%} = \frac{\text{Sequential Time} - \text{Parallel Time}}{\text{Sequential Time}} \times 100$$ Test Throughput: Maximize total test cases executed per CI window. CI/CD Pipeline Cycle Time: Aim for a 30–40% total reduction in build + test execution duration. 2. Stability & Reliability Flaky Test Rate: Keep flaky tests under 2–3% by leveraging Playwright's native auto-waiting and resilient locators. $$\text{Flakiness \%} = \frac{\text{Flaky Tests}}{\text{Total Tests}} \times 100$$ Retry Dependency Ratio: Track the percentage of tests passing only after retries to minimize false positives. Failure Root Cause Accuracy: Target >90% of test failures pointing directly to genuine application defects rather than script instability. 3. Coverage Metrics Automation Coverage: Maintain 80%+ regression coverage across all functional scenarios. Cross-Browser & Device Coverage: Measure test runs across Chromi

Tracking test automation metrics manually often leads to outdated figures and missed engineering gaps. To solve this, automated reporting directly from your test suites—such as Playwright and Cucumber—provides clear visibility into health, execution speed, and coverage. Below is a breakdown of how to structure an Automation KPI Dashboard to streamline test metrics, track trends, and establish actionable engineering goals. Executive Summary Dashboard KPI Metric Target Current Value Status Trend Total Test Cases 100% coverage 85% 🟡 Partial ↗️ Up Automated Test Coverage 90%+ 78% 🟡 Partial ↗️ Up Pass Rate (Last Run) 95%+ 92% 🟡 Partial ↔️ Stable Avg. Execution Time < 30 min 28 min 🟢 Good ↘️ Down Flaky Test Rate < 2% 1.5% 🟢 Good ↔️ Stable Defects Detected — 3 🟡 Review ↔️ Stable CI/CD Pipeline Success 100% 98% 🟡 Partial ↗️ Up Key Metric Breakdowns 1. Coverage & Execution Total Test Suite: 120 tests (94 Automated, 26 Manual). Latest Run (2026-05-29): 94 executed — 87 passed, 7 failed, 0 skipped. 2. Flakiness Tracking Flaky Tests (Last 10 Runs): 2 scenarios identified. Top Offenders: Scenario A: UI timeout issues. Scenario B: Data synchronization lag. 3. Defect Detection & CI/CD Performance Defect Lifecycle: 3 opened, 1 closed (Avg. resolution time: 2 days). Pipeline Health: 98% success rate, 12 min average build time. Primary Cause of Pipeline Failure: Dependency resolution errors. Execution & Pass Rate Trends (Last 6 Runs) Run Date Pass % Fail % Flaky % Duration (min) 2026-05-29 92% 8% 2% 28 2026-05-28 91% 9% 2% 29 2026-05-27 90% 10% 3% 30 2026-05-26 89% 11% 3% 31 2026-05-25 88% 12% 4% 32 2026-05-24 87% 13% 4% 33 Next Engineering Action Items Automation Expansion: Push total automated coverage past 90%. Flakiness Mitigation: Refactor explicit waits and isolation for UI timeout and data sync scenarios. Pipeline Stability: Resolve dependency caching errors to bring CI/CD success to 100%. Optimization: Lower execution suite duration below 25 minutes using parallel run setups.

More agents do not automatically produce better engineering. They usually add total tokens, duplicated context, handoff delay, and integration risk. Their defensible advantages are narrower: reduced elapsed time for independent work, isolated investigation, or specialist evidence that one agent might omit. The useful question is therefore not “Can this task use subagents?” It is: Does this task contain independent, bounded work whose value exceeds the coordination cost? Codex How To now includes a dependency-free benchmark for testing that question instead of answering it from intuition. Disclosure: I maintain Codex How To , the independent open-source project containing the benchmark, evaluator, and measurements used here. The minimum decision rule Use one agent when the change is small, the interface is unsettled, or several steps must edit the same central files. Consider bounded orchestration only when all of these are true: The task has at least two genuine ownership surfaces. Each writer can own exclusive paths. The interface between those paths is frozen before implementation. The controller retains integration, system checks, and final review. Every worker returns concise evidence rather than a narrative transcript. One external acceptance bar can evaluate every execution method. flowchart TD A["One task contract"] --> B{"Independent write surfaces?"} B -- "No" --> C["One agent or sequential work"] B -- "Yes" --> D{"Frozen interface and exclusive paths?"} D -- "No" --> C D -- "Yes" --> E["Bounded workers"] E --> F["Controller integrates and evaluates"] F --> G{"Coverage or elapsed-time value exceeds coordination cost?"} G -- "Unproven" --> H["Keep measuring"] G -- "Repeated evidence" --> I["Adopt for this task class"] Job titles are not ownership boundaries. “Backend agent,” “test agent,” and “review agent” may still collide on the same files or execute dependent stages. A useful boundary is concrete: one writer owns incident/** , another owns web/** , and n

Part 1 covered the mechanism: a fingerprint gate decides whether a change ships in minutes over-the-air or needs a full store release. But a gate that only checks "is this native-safe" says nothing about whether the change is good . If every merge to main can reach production within minutes, your safety net can't be a release train that gives everyone time to notice a problem before it ships — it has to be built into the pipeline itself, because there's no train to catch it on the way out. The PR gate Every pull request into main runs through the same automated gate before it's mergeable: a type check, a lint pass, an automated test suite, and end-to-end checks against a real device build. None of that is negotiable — it's the floor, not a nice-to-have. E2E is a big enough topic on its own — closing the loop between what a unit test can see and what actually happens on a phone in someone's hand — that it deserves its own dedicated post rather than a paragraph here. jobs : typecheck : run : npm run typecheck lint : run : npm run lint test : run : npm test e2e : run : npm run e2e Nothing exotic under the hood — ESLint for the lint pass, Husky for local pre-commit/pre-push hooks so the same checks catch you before CI even runs, Jest as the test runner, and React Native Testing Library for component-level tests. Popular, boring, well-documented tooling on purpose — the pipeline's value is in how these are wired together and gated, not in any one tool being clever. Feature flags are the real safety valve Here's the entry condition that makes OTA-from- main safe at all: shipping code and releasing a feature are two different actions. A merge can put new code on every user's device within minutes — that's deploy. Whether that code actually does anything visible is a separate switch, controlled by a remote feature flag, not by whether the code merged. That decoupling is what makes trunk-based development survivable. Nobody has to get the timing of a merge exactly right, bec

API documentation has a reliability problem. The code gets updated; the OpenAPI spec gets forgotten. The spec gets updated; the TypeScript client doesn't regenerate. By the time an enterprise client asks for your API contract, the document you hand them describes a system that no longer exists. Baklava, an open-source library by Iterators , solves this structurally: documentation is generated from the tests that verify your actual API behaviour, so it cannot drift. The problem Documentation drift is the default state of any API that lives long enough. The causes are well-understood: docs and code are maintained separately, documentation updates require extra discipline at every PR, and no automated check catches a route signature change that wasn't reflected in the OpenAPI file. The consequence is real. Clients building against a stale spec hit integration errors in production. Internal teams onboarding to a service spend hours reconciling the documented contract with actual behaviour. TypeScript front-ends break when an API response field changes without a corresponding client update. The problem compounds as the API grows. The solution Baklava integrates into your existing test suite. When routing tests run, baklava observes each request and response, infers the API surface, and generates documentation as a test output, not as a separate build step, not as a manually-maintained file. In baklava, the test is the documentation spec. Instead of a standard assertion block, each route is defined with path() , supports() , and onRequest() scenarios that both verify the API behaviour and describe it for documentation output: ​`// The test IS the documentation spec class UserApiSpec extends AnyFunSpec with BaklavaPekkoHttp[Unit, Unit, ScalatestAsExecution] with BaklavaScalatest[Route, ToEntityMarshaller, FromEntityUnmarshaller] { path("/users/{userId}")( supports( GET, pathParameters = p Long , summary = "Get user by ID" )( onRequest(pathParameters = 1L) .respondsWith Use

Free AI Tiers Bill You in Hours, Not Dollars Free model access looks like a bargain until you track the hours you spend feeding context back into a model with no memory. A zero-cost invoice hides the most expensive resource in your workflow: your own attention. My position is straightforward: treat a free tier like a metered service and measure the hidden costs before you adopt it. The token counter tells you almost nothing about the real price. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm using MonkeyCode's free model access and free server option as a concrete example; the measurement approach applies to any free tier. The dashboard shows tokens, not time Every free plan advertises a generous token allowance and a server that wakes up on demand. What the marketing page omits is the labor you spend reassembling context, waiting for cold starts, and double-checking output. Those costs do not appear on any invoice, but they consume your day in chunks. Four of them matter more than the token meter. Context reconstruction — Every new conversation starts from zero, so you re-explain your stack, your file layout, and your constraints. Those re-pasted tokens count against the same allowance you were trying to save. Cold-start waiting — A free server that sleeps after idle adds seconds to every call. Multiply that by a scheduled job that fires hourly and you have lost real time. Human verification — Confident output still needs a human to check it, and that check is the most expensive line item in the whole system. Attention fragmentation — A free allowance looks huge until you split it across codegen, debugging, and review. Small tasks nibble the budget faster than big ones. A ten-minute audit script The script below turns the argument into a reproducible measurement. It sends three representative prompts to any OpenAI-compatible endpoint, records wall-clock latency, and extracts token usage from the response. Run it several times du

Here's the conclusion up front: a free LLM endpoint is a moving target. You can't see the changes, but they're happening — model updates, quantization tweaks, server-side prompt rewrites. And your app will feel them, usually as a slow, invisible quality dip. I've spent weeks on this account probing free LLM servers, caching tokens, and building evaluation harnesses. The pattern I keep seeing: teams pick a free tier, wire it in, and then never look at it again. They treat it like a static API. It isn't. The fix is a nightly drift check. A small script that runs your most important prompts against the endpoint, compares the outputs to a baseline, and tells you when something changed. Not a benchmark. Not a one-time eval. A recurring alarm. This post walks through a 90-line harness you can run tonight. I'll use MonkeyCode's free server as the reference endpoint — it's an open-source project with free model access, a free server option, and, as advertised at the time of writing, a 10M token grant. The exact numbers may move, so check the repo's README before you depend on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Why drift is the silent killer of free-tier apps Let's be honest: free endpoints don't come with changelogs. The provider can swap the underlying model, adjust the temperature default, or add a safety filter without telling you. Your tests still pass. Your error rate stays flat. But the responses get a little shorter, a little more evasive, a little less useful. Users notice before you do. They don't file bugs for 'the bot got dumber.' They just stop using it. A drift check turns 'the bot got dumber' into a concrete signal: 'the pass rate on 12 core prompts dropped from 92% to 74% overnight.' That's something you can act on. Step 1: Define your core prompts Don't test everything. Pick 10-20 prompts that represent the actual workload your app handles. For each prompt, define what 'good' looks like. Prompt Expected beha

First post in a build-in-public series about verdict , an MCP server that gives coding agents structured, sandboxed test feedback. The problem Watch a coding agent work and you'll see it run pytest in your shell, unsandboxed, and then push 40,000 tokens of raw output through its context window to answer one question: did my change break anything? That's three problems in one command: Token waste. The agent needs ~10 lines of signal and pays for a wall of dots, warnings, and tracebacks. No sandbox. The tests run on your machine, in your environment, with your files writable. No memory. When a test fails, the agent can't tell whether it broke it or whether it was broken before it arrived - so it either "fixes" pre-existing failures nobody asked about, or ships regressions it assumes were already there. verdict is an MCP server that replaces the pytest shell-out with four tools: tool what it returns verify(scope?) impact-selected tests, run in an ephemeral container, as a ~400-token typed verdict explain_failure(check_id) the full traceback - only on demand history(fingerprint) first seen / last seen / times seen for a failure run_checks(["ruff","mypy"]) lint & type checks, same verdict shape ▶️ Watch the 30-second demo - Claude Code fixing a bug with verdict verifying in a container. The three ideas 1. Verdicts, not output. verify returns typed JSON: counts, per-failure message + location, and nothing else. Full tracebacks live behind explain_failure . The whole verdict for a real failing run is ~400 tokens - the raw pytest output it replaces was ~40k. The design rule in the repo is blunt: nothing bulky rides in the summary, ever. 2. Fingerprints give failures identity. Every failure is hashed from its normalized signature - volatile tokens (addresses, tmp paths, ids, durations) collapsed first. Same logical failure ⇒ same fingerprint, across runs and refactors. Fingerprints are what make the third idea possible: 3. History answers "was it me?" verdict keeps a small S

Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS. So I wrote the QA framework I wished I had, packaged it as ttsproof , and then ran it as a blind study against a production TTS service so the results would be more than an opinion. The two failures WER cannot see A TTS pipeline breaks in two different ways, and a single WER number blurs both. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform. Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names. ttsproof splits them apart and handles each one honestly: Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else. Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error. ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS. The study: 390 samples, and a blind human check I evaluated the method