标签：#test

Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality. So I built evalgate , a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment. The core idea The important design decision is what question CI is allowed to ask. "Is this prompt good?" is subjective and unwinnable in an automated gate. "Is this worse than it was on main?" is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness. The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done. How it works A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one: name : my-agent provider : mock # works with no API key threshold : 0.9 # mean score required to pass cases : - id : greeting input : prompt : | Reply with the standard greeting. exactly: Hi there! How can I help you today? expected : " Hi there! How ca

How do you actually test an AI agent? Not "does it respond," but: does it route to the right tool, chain calls correctly, recover from failure, resist prompt injection, and stay within cost/latency budget? I spent weeks working through this on a running agent, and open-sourced the entire methodology — framework-agnostic , so it applies regardless of your language, runtime, or toolset. What's inside •⁠ ⁠ 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena, AgentDojo, LongMemEval and more, categorized by what they actually measure •⁠ ⁠ 58 universal test blocks across 7 tiers (L1–L4, Error Recovery, Multi-Turn, Security). Each block = a tool-agnostic capability definition + a concrete reference implementation •⁠ ⁠ Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to 6 universal security test blocks •⁠ ⁠ Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k, trajectory vs end-state, observability (OpenTelemetry GenAI), automated red-teaming (garak, PyRIT, DeepTeam) •⁠ ⁠ Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001 How to use it Take Part II, replace the reference-implementation fields with your own agent's tool names and expected outputs. The universal capability definitions need no changes. Blank templates are included. PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real reference case study — but the methodology is the product, not the agent. No marketing narrative: ⁠ STORY.md ⁠ documents the real bugs, real test runs, and real corrections that shaped each version. Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome. 👉 https://github.com/trgysvc/AgentTestMethodology

I'm building an app with Claude Code right now. The setup is a little unusual. One "commander" session writes the instructions, and separate "worker" sessions implement them in parallel. The commander never touches the keyboard itself. Its whole job is to turn "what to do next" into a handoff prompt that a worker can read and just run with. And that handoff prompt is quietly the scariest thing in the whole loop. The moment I hand it over, the worker trusts it as the spec and starts sprinting. If the spec is wrong, the wrong thing gets built. Fast, and with confidence. Before handing it over, I did my usual ritual I've picked up a habit lately. Before I throw the instructions at a worker, I run them past a subagent whose entire job is to demolish them. It's defined to never approve, to hunt for holes, and to never, ever close with "this looks broadly reasonable." What was different this time: this thing didn't just read the prose of my prompt. It went and read the actual code. And the reply it came back with made my stomach drop a little. Objection 1: I wrote "make the test pass (go green)", but that test's green meant nothing In the instructions, I'd written this as a definition of done: "Get the XX test passing (green)." The demolition agent's answer: "That test goes green even when the thing it's testing fails. " I read it. It was true. The test only checks "did the process run all the way to the end." It never checks the one thing that matters: did it succeed? Fail, and as long as it "returned a failure result and finished running," green. On top of that, the batch path was swallowing exceptions, so no matter what blew up, still green. So even if a worker reported back "DoD met, tests green!", nothing was actually proven. The completion criterion I wrote myself was an empty pass. Objection 2: I wrote "just flip a flag", but that switch didn't exist One more. I'd written " flip a config flag and it swaps in the real component ", as if it were a feature that alread

Introduction Hello from Japan! 🇯🇵 I am tosane932 , a professional truck driver working in logistics while teaching myself Python. In my previous article, I tested a Docker multi-stage build and measured the actual change in image size. At the end of that article, I said that I would write next about pytest and CI/CD. This article was supposed to be the practical follow-up. However, while preparing for that work, I encountered an unexpected side issue. I only intended to introduce Flask-Migrate. Instead, the pip installation logs revealed that the version of a library in my local development environment had been changed without me noticing. The library was: google-genai From there, I went through the following process: Identify the version mismatch Restore the version that had already been tested locally Update requirements.txt Manually verify the Gemini API functionality Run pytest to check for regressions This article records that process without hiding the inconvenient parts. https://github.com/tosane932/sales_data_app Overview While installing Flask-Migrate, I noticed a mismatch between: The version of google-genai installed in my local development environment The version declared in requirements.txt The local environment had been using: google-genai 2.10.0 However, requirements.txt still specified: google-genai==2.4.0 When I ran: pip install -r requirements.txt pip followed the configuration file and replaced the newer local version with the older declared version. This article explains how I discovered the issue, synchronized the environments, and verified the application behavior with automated tests. 1. The Problem and Its Background I was preparing to introduce Flask-Migrate. During that work, I ran: pip install -r requirements.txt The installation log contained the following lines: Attempting uninstall: google-genai Found existing installation: google-genai 2.10.0 Uninstalling google-genai-2.10.0: That message caught my attention. After checking the environ

Every automated system receives a particular shape of the world. That shape is expressed through records, documents, events, exceptions, and missing values. If the designers have not identified those forms—and the ways they can become malformed—the machine inherits their ignorance and reproduces it at scale. The question is not simply whether the AI failed. The useful question is whether the human-built system knew what success meant, knew the shape of its data, and knew how to recognize when it was wrong. Start with the shape of the data Before selecting a model, draw the workflow as a sequence of data transformations. What enters each stage? In what form and from what source? Which values are valid, absent, duplicated, stale, delayed, or contradictory? How will each violation be detected? What must the workflow do next? Each data shape needs a corresponding failure model. An unknown here is not merely uncertainty for the machine; it is a measurement failure in the organization. The remedy is to collect the missing data or explicitly design for its absence. Otherwise, the system is being asked to operate in a world its designers have not described. Stabilize the deliverable A system cannot be stabilized around a target that continues to move. The deliverable must be more than an aspiration written in a prompt. It should be expressed as observable conditions and anchored to a representative corpus: examples that are acceptable; examples that are unacceptable; examples that are genuinely ambiguous. Human reviewers should first demonstrate that they can apply those distinctions consistently. If they cannot agree on what success looks like, the model is not being measured against a specification. It is being measured against human disagreement disguised as one. The model is not the system Only then does it become meaningful to place an AI model inside the workflow. The model is one transformation among many: Input → validation → retrieval → normalization → model infere

I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed. They did not — and the test I already had for exactly this had been green the whole time. The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text. The setup The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab. A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable. Here is what I actually got, same seed, same inputs: JVM browser first obstacle x, frame 60 405.426 304.426 still alive at frame 360 yes no final score 9 6 Not a rounding difference. A different game. The cause is boring. The test failure is not. GameEngine used java.util.Random . Its algorithm is specified down to the constants — you can read the exact linear congruential generator in the Javadoc. So a seed ought to name exactly one sequence. But my code was not running that algorithm. It was running whichever implementation the runtime supplied , and TeaVM's is not the JVM's. The specification describes what java.util.Random does; it does not force a foreign runtime's reimplementation to match. The fix took ten minutes: write the LCG out longhand so both builds execute the same arithmetic instead of trusting that they will. The interesting part is the test. The test that could not have caught it I had a test called theSameSeedProducesTheSameRun . It ran the engine twice, with the same seed, and asserted the results matched. It passed on every commit, including every commit during which the browser build was playing a different game. It had to pass. It runs the engine twice in the same runt

CompTIA PenTest+ / Ethical Hacking Certification Series Professional Reference Guide — GitHub Edition Covers: Passive Reconnaissance · OSINT · DNS · Social Media · Cryptographic Analysis · Shodan Table of Contents 3.0 Introduction 3.1 Performing Passive Reconnaissance 3.1.1 Overview 3.1.2 Active Reconnaissance vs. Passive Reconnaissance 3.1.3 The OSINT Methodology — How Professionals Think 3.1.4 OSINT Tools — The Complete Professional Arsenal 3.1.5 DNS Lookups — Deep Dive 3.1.6 DNS Reconnaissance — Advanced Techniques 3.1.7 Identification of Technical and Administrative Contacts 3.1.8 WHOIS Intelligence — Extracting Maximum Value 3.1.9 DNS Lookups — Lab-Level Practical Reference 3.1.10 Cloud vs. Self-Hosted Applications and Related Subdomains 3.1.11 Social Media Scraping 3.1.12 Employee Intelligence Gathering 3.1.13 Cryptographic Flaws 3.1.14 Finding Information from SSL Certificates 3.1.15 Company Reputation and Security Posture 3.1.16 File Metadata 3.1.17 Web Archiving, Caching, and Public Code Repositories 3.1.18 Finding Out About the Organization — Aggregation Techniques 3.1.19 Advanced Searches — Google Dorking and Beyond 3.1.20 Open-Source Intelligence (OSINT) Gathering — Frameworks and Automation 3.1.21 Shodan — The Search Engine for Everything Connected 3.1.22 Breach Data Intelligence — Leaked Credentials and Exposure Monitoring 3.0 Introduction Module Overview: Information Gathering and Vulnerability Scanning Module Objective: Perform information gathering and vulnerability scanning activities at a professional, senior-level standard. Before a single exploit is launched, before a single payload is crafted, every professional penetration tester invests significant time in a discipline that separates competent practitioners from exceptional ones: information gathering . The reconnaissance phase is the intelligence foundation upon which the entire attack strategy is built. The quality of your reconnaissance directly determines the quality of your attack. Why T

If you've built an MCP App — the HTML widget an MCP server hands a host to render inline — you may have hit this: the tool call succeeds, structuredContent comes back fine, the model announces that a widget rendered, and the user sees nothing. No error. No console output. Just a gap in the conversation. There's a long issue full of people with this exact symptom, all of them (me included) posting variations of "my server is spec-correct and nothing renders." That's a hard thing to act on. So I built a probe server designed to answer one question at a time and ran it 36 times. This post is mostly about method — how you experiment on a host whose source you can't read and whose renderer you can't attach a debugger to. The MCP specifics are the worked example. The most useful part is at the end, where my measurements lied to me twice. Everything behavioural here was measured on 31 July 2026 against claude.ai web. Host behaviour changes; treat the numbers as a snapshot, not a spec. The finding, up front The most-upvoted lead in that thread says claude.ai silently refuses to place the iframe unless your resource declares _meta.ui.domain , computed as sha256(<your endpoint URL>)[:32] + ".claudemcpcontent.com" . Here's what varying that one field actually does: _meta.ui.domain iframe mounted sandbox origin computed value 10/10 one stable origin, every render absent 10/10 host default — differs per conversation present but wrong 0/8 never created Omitting it doesn't stop anything. What it actually controls is origin stability , which is exactly what the SDK docs say it's for: a fixed origin your API server can allowlist for CORS. But a wrong value is fatal. And the easy way to produce one is hashing an endpoint string that differs slightly from the URL the client connected with — a trailing slash, a missing path segment, http vs https . So the advice inverts the risk: follow it imprecisely and you convert a working app into a broken one. The original comment wasn't wrong ab

Delete a Telegram bot and https://t.me/your_deleted_bot keeps returning HTTP 200 with a page that looks completely normal. Every link checker I know of — CI actions, directory scripts, monitoring cron jobs — reports it as healthy forever. If you maintain anything that lists Telegram bots, some fraction of your list is already dead and your checks are telling you it is fine. Reproducing it Pick a username that has never existed: curl -s -o /dev/null -w "%{http_code} \n " https://t.me/nonexistent_test_bot_77712 # 200 Two hundred. No redirect, no 404, no soft-404 marker in the body that a status check would catch. Where the truth is The status code is useless here, but the Open Graph title is not. I measured four usernames — two live bots, two that do not exist: URL og:title t.me/BookClassBot (live) BookClass t.me/instanavy_bot (live) StoryViewer - anonymous instagram story viewer tool t.me/nonexistent_test_bot_77712 Telegram: Contact @nonexistent_test_bot_77712 t.me/zzz_definitely_not_a_real_bot_9182 Telegram – a new era of messaging A live bot puts its own display name in og:title . A dead one gets one of two Telegram placeholders: Telegram: Contact @<username> , or — if the username is not even syntactically valid — the generic Telegram – a new era of messaging . That is the whole signal. curl -s https://t.me/some_bot | grep -o '<meta property="og:title" content="[^"]*"' The check Standard library only, no dependencies: import re import urllib.request UA = " Mozilla/5.0 (compatible; linkcheck/1.0) " DEAD_EXACT = { " Telegram – a new era of messaging " , " Telegram " } DEAD_PREFIX = " Telegram: Contact @ " def telegram_bot_exists ( url : str ) -> bool : """ True if the bot behind a t.me URL still exists. HTTP status is not usable here: Telegram serves 200 with a placeholder page for usernames that were deleted or never existed. The Open Graph title is what actually differs. """ req = urllib . request . Request ( url , headers = { " User-Agent " : UA }) with urllib .

Quick check: does your agent stack have a memory.md in it somewhere? An AGENTS.md ? A notes file the agent appends to when something seems worth keeping? Thought so. Mine did too. It's the pattern everyone converges on, it takes twenty minutes to build, and it genuinely works — right up until the day it hands a customer a fact that stopped being true in March. This post does three things: shows you exactly why the pattern rots (with a real-shaped sample file we'll dissect), gives you a small script to audit your own file tonight, and walks through the architecture change that actually fixes it. No vendor required for any of it. The pattern we all built Strip away the framework and every self-managed memory loop looks like this: MEMORY = Path ( " memory.md " ) def run_task ( task : str ) -> str : context = MEMORY . read_text () # 1. dump everything in result = llm ( SYSTEM + context + task ) # 2. do the actual job note = llm ( # 3. agent grades its own homework " What from this interaction is worth remembering? " " Reply with one line, or NONE. \n\n " + result ) if note . strip () != " NONE " : with MEMORY . open ( " a " ) as f : # 4. append forever f . write ( f " - { note . strip () } \n " ) return result Be fair to it first: this is human-readable, versionable, greppable, zero-infrastructure. For one agent, one job, small working set — it's honestly hard to beat. Now look at what it doesn't do. Step 4 is the entire lifecycle. Nothing in this loop ever updates, merges, expires, or questions a line once written. The file has exactly one behavior: it grows. Dissecting a six-month-old memory file Here's a condensed, realistic slice of what that loop produces by month six. Read it the way retrieval reads it — every line equally true: - Customer Acme runs their workload in us-east1 - Acme prefers Slack over email for escalations - Acme's staging env uses the legacy auth flow - Acme contact is Priya (prefers email) - The feature flag `beta_router` must stay ON for Acme -

Every recorded mobile test I have ever inherited died the same way: someone moved a button. The recording said "tap at (340, 712)". The redesign moved that button up by one row, and the test kept tapping — now on empty space, or whatever happened to land there instead. It didn't fail right away. Three sprints later, it started failing in confusing ways, and by then nobody trusted the suite anymore. The fix isn't a better recorder. It's recording a different thing: not where you tapped, but what you tapped. That needs an element tree, and for a while we didn't have one. tapflow is an open-source, self-hosted tool that streams iOS simulators and Android emulators into a browser, so a whole team can test builds without installing anything. Until now, everything it moved was pixels in one direction and taps in the other. This post is about getting an element tree out of a simulator with no window, on both platforms. What we do with that tree — replaying flows that survive a redesign — is the next post in this series. The automation axis this feeds — the flow runner and the MCP server — is experimental . The manual browser QA path is the mature one. The constraint: no WebDriverAgent, and no simulator window tapflow already injects touches into the iOS simulator without WebDriverAgent — it loads CoreSimulator.framework and pushes HID events through SimDeviceLegacyHIDClient (that story is ep.1 ). Streaming reads the framebuffer IOSurface directly. Neither path needs Simulator.app on screen, and that's deliberate: an agent Mac in a closet running four simulators shouldn't be babysitting four windows. So whatever we used for the tree had to follow the same rule. No WDA to install and keep in sync with Xcode. No simulator window on screen. Our first attempt ran into exactly that limitation. macOS exposes an accessibility API ( AXUIElement ), and Simulator.app publishes its content through it. We wrote a helper around it, and it worked perfectly on a developer's laptop. On the

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets. That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over. OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it. What's in the repo: A versioned JSON Schema spec for test cases, grader configs, and result records. TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets. A CLI with validate, convert, init, and summarize commands. Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite. The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered. Repo: https://github.com/adhabnr-ux/openeval Would love feedback from anyone who has hit this same portability problem while switching between eval tools.

1. Introduction Most public demonstrations of AI coding agents begin with an empty directory and end when a visible feature works. That is useful, but it measures only the first and easiest part of software engineering: generation. Real projects are stateful. They contain existing behavior, data that must survive, security boundaries, partially completed work, and verification steps that cannot be replaced by a convincing demo. This benchmark was designed as a small, practical record of that harder problem. It documents two Codex runs on the same full-stack task-manager project. Phase 1 was a greenfield build. Phase 2 returned to the existing application for an authentication and database-migration refactor. The second run also included a long, user-initiated interruption, making recovery part of the observed workflow. The narrow result is encouraging: the initial application was recorded as complete in approximately 34 minutes, and the later refactor took approximately 42 minutes of active execution, excluding the user-initiated pause. The Phase 2 run reported 16 backend tests, 1 legacy-migration test, and 25 frontend tests passing, along with lint, type, build, dependency, and browser-acceptance checks. Those numbers need boundaries. This is a single-environment case study, not an official benchmark, not a controlled comparison between providers, and not a production guarantee. The current documentation set does not retain the complete source tree, exact prompts, raw terminal output, migration artifact, database fixture, or browser trace. Results are therefore presented as supplied run records rather than independently reproduced evidence. 2. Why Simple Code Generation Tests Are Insufficient A prompt such as "build a task app" primarily tests synthesis. The agent chooses a stack, creates files, connects components, and produces a working path. It may reveal speed and basic tool use, but it says little about how the agent behaves when constraints collide. Maintenan

Download my Automation Testing Interview Questions from ⬇️ Apple AppStore - https://apps.apple.com/us/app/qa-automation-interview-prep/id6786760948 👈 ⬇️ Playstore - https://play.google.com/store/apps/details?id=com.app.seleniuminterviewquestions 👈 The standard advice for passing a QA Engineering interview is broken. If you ask a forum how to prepare, you will be told to "learn Playwright," "memorize XPath," or "know how to write a basic API GET request in Postman." That advice worked in 2021. Today, engineering teams do not want manual testers who learned basic syntax. They are hiring Software Engineers in Test (SDETs) who understand system architecture, CI/CD pipelines, and data state. If you are failing technical rounds, it is rarely because you forgot a WebDriver command. It is because you are testing the syntax instead of the system. Here are the two architectural concepts you are actually being judged on in a modern QA interview, and how to approach them. 1. The API Race Condition & Idempotency Trap In a technical round, a senior engineer will rarely ask you to "test a login endpoint." Instead, they will give you a scenario like this: "We have a microservice that processes payments. The user clicks 'Submit', but the network drops, so they click it again. How do you automate a test to ensure they aren't charged twice?" The Junior Answer: "I will write an automated script that clicks the button twice quickly and checks the database." The Senior Answer (What they want to hear): "I will write a test that validates the API's idempotency . I will intercept the first request, capture the unique idempotency key from the header, and fire a duplicate POST request with the exact same payload and key. The test must assert that the backend returns a 409 Conflict or a 200 OK with the original transaction ID, verifying the database state didn't duplicate the charge." If you do not understand idempotency, payload validation, and race conditions, your API automation is just che

Missing data is not the same thing as a convenient default. That sounds obvious, yet temporal software regularly converts an empty time field into midnight, noon, the current time, or the start of a day. The interface may look complete after that conversion, but the program has silently changed an unknown fact into a known one. This matters anywhere an hour can change the result: medical timelines, transport schedules, legal deadlines, astronomical calculations, historical records, and calendrical systems. I encountered the problem while working with a BaZi calculation pipeline. A BaZi chart can use year, month, day, and hour components. If the birth time is absent, the honest result is a three-component analysis with hour-dependent conclusions withheld. Inserting noon would make the output look richer while making its provenance weaker. The useful engineering question is not “Which fallback time should we choose?” It is “How do we keep uncertainty visible through every layer of the system?” The public calculation evidence repository provides the concrete calendar-domain fixtures referenced below. The rest of this article focuses on the reusable software boundary behind them. Model knowledge, not just a string A common input model makes absence too easy to erase: const birthTime = form . time || " 12:00 " ; After this line runs, downstream code cannot tell whether noon came from the user or the fallback. Validation, analytics, caching, and the result renderer all see the same string. The information loss happens before the calculation begins. A small discriminated union keeps the two states separate: /** * @typedef {{ kind: "known", localTime: string, source: "user" }} * KnownTime * @typedef {{ kind: "unknown" }} UnknownTime * @typedef {KnownTime | UnknownTime} BirthTime */ function parseBirthTime ( value ) { const normalized = value ?. trim (); return normalized ? { kind : " known " , localTime : normalized , source : " user " } : { kind : " unknown " }; } This typ

I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no. That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources. Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact. This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me. The setup I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like: "Best personal injury law firm in New York City?" "Top immigration lawyers in Mumbai?" For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose. First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap. Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer: Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself. If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal. The control The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade , the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks. If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now. The setup mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle. Round 1, on three real servers, cost about twelve cents and produced this: Server Static score Tool selection Args Refusal context7 (2 tools) 100 38% 100% 100% server-memory (9 tools) 81 93% 100% 100% server-slack (8 tools) 97 54% 100% 100% Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken. The eval was broken Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history . context7's get-library-docs needs a library ID that comes from resolve-library-id . These are pipelined tools : their required arguments are produced by other tools. My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong. The model wasn't confused. The model was right . The benchmark was grading correct multi-step reasoning as failure — and me

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

Every unsupervised AI agent we've reviewed that wrote Flutter code made the same seven mistakes. These aren't typos or stylistic differences. They're structural failures that compound—bad state management plus missing tests plus hardcoded colors means the codebase becomes expensive to theme, hard to test, and impossible to maintain at scale. Here's a small one to set the tone: a developer asked an agent to implement a GET request to an external service in a Dart project. The agent's solution was to shell out to curl via Process.run and parse the stdout. Not package:http . Not dio . Not even dart:io 's own HttpClient . A subprocess call to a CLI tool, inside a language that's had first-class HTTP clients since Dart 1.0. That one is worth sitting with, because it's not really a Flutter problem — it's the whole pattern in miniature. The agent wasn't "wrong" that curl can make a GET request. It optimized for "this pattern appears constantly in training data" over "this is the idiomatic way to do it in the language I'm currently writing." Bash and curl show up in approximately every tutorial, README, and Stack Overflow answer ever written. package:http shows up in Dart-specific docs. Given no other constraint, the agent reached for the statistically dominant pattern, not the contextually correct one. The seven gaps below are the same failure mode, just less obvious than "shells out to curl." Here's what we found, with real code examples and the fixes that work. 1. Recomputing Derived State The Problem: Agents recalculate the same values across multiple locations instead of maintaining one source of truth. Imagine a checkout flow where the cart total is computed three separate ways: In the checkout page: (items.sum + tax) - discount In the footer: items.sum - discount + tax In the order summary: (items.sum - discount) * (1 + taxRate) Different calculations. Same semantic meaning. One will break first. The Fix: Derive values once in the state layer using streams. Let all w