标签：#Testing

A Practical Guide to Faster, Deterministic Flutter & Dart Unit Testing If you’ve ever written unit tests for classic package:bloc applications using bloc_test , you know the drill: build your BLoC, dispatch an event in act , and assert state emissions in expect . Under the hood, classic BLoC processes state updates asynchronously via Dart microtask-queue Streams . While robust, testing asynchronous streams can introduce microtask timing headaches, race conditions, or the need to drain queues or use fakeAsync when testing complex side-effects. In BlocSignal , state updates propagate synchronously . Calling emit(newState) updates the underlying signal graph in the exact same call stack frame. This handbook is a practical, recipe-based guide to testing BlocSignal and CubitSignal applications using package:bloc_signals_test . Whether you’re coming from classic BLoC or brand new to Signals, this guide shows you how to test every scenario cleanly—and why it’s significantly easier than classic stream-based testing. 🤖 AI Assistant Tip : Working with an AI coding assistant (like Antigravity, Gemini CLI, or Cursor)? The official bloc-signals plugin includes a pre-built testing skill ( plugins/bloc-signals/skills/bloc-signals/ ) that automatically teaches your AI assistant these exact testing conventions, observer scoping rules, and declarative blocSignalTest patterns! 🛠️ Quick Reference: BLoC Streams vs. BlocSignal Testing Testing Task Classic BLoC ( package:bloc_test ) BlocSignal ( package:bloc_signals_test ) Why it’s easier in BlocSignal Execution Environment Often requires flutter test engine Pure dart test execution Blazing Speed : Business logic tests run in pure Dart CLI without booting Flutter UI engine. Simple State Assertions Requires async stream listener or blocTest Direct expect(cubit.state, 1) or blocSignalTest Synchronous : State updates on the next line of code without microtask delay. Failure Diagnostics Legacy Instance of 'CounterCubit' Built-in toString() :

You're Not Comparing Models. You're Comparing Contracts. Two teams publish scores on the same agent benchmark. One lands in the low sixties. The other clears seventy. A procurement team reads the spread and makes a call. What they do not see: both teams may be running the same model. They did not need to change the weights for the gap to appear. The spread can come from scaffold alone. One team wrapped the model in a harness with better retries. Different tool defaults. A planner step the other team had skipped. None of that appears on the leaderboard. The comparison that drove the decision was not between two agents. It was between two contracts. There Is No Benchmark The mistake hiding behind this story is a category error. People talk about agent benchmarks as if they measure a thing called “the model.” They do not. They measure a coupled system. The model is one component. The rest is a stack of protocol decisions that are almost never disclosed and almost always matter. The score is the output of that stack. Change any layer and you change what the number means. Recent research on agent evaluation has named those layers explicitly. There are at least seven. Deployment regime. Observation channel. Harness and scaffold. Metric and action. Configured evaluator. Grader protocol. Audit bundle. Each is a contract. Each is negotiable. And each can silently change the verdict while the headline looks the same. That is what a benchmark actually is. Not a measurement of a model. A measurement of an entire testing contract, of which the model is one slot. There is structural reason the seven layers are the seven layers. They cluster into three corners that show up in almost every published agent-evaluation failure. What the model is rewarded for. How that reward is optimised. And how the test contract differs from production. Once you hold those three corners in view, the seven-layer stack stops feeling like a checklist and starts behaving like the actual shape of what is

This article is about a poker-analysis framework, but the engineering problem is common to LLM tool use. The framework uses an LLM as an input and control layer. It reads a natural-language poker question, chooses a local calculator, and proposes typed fields. A Python program, not the LLM, performs the numerical calculation and returns a structured result with verification data. In this evaluation, a coordinator manually passed each calculator-eligible saved proposal to the command-line calculator; no automatic runtime bridge connected them. The design intent was to reduce manual arithmetic checking by sending numerical claims to deterministic, internally verified local software. The evaluation below tests whether those claims were checked and whether the handoff remained auditable. It did not measure time saved or the overall quality of the resulting poker analysis. The calculator catalog is poker-specific. This is not a poker strategy guide, and you do not need to know poker strategy to follow the failure. The question is whether a correct calculator can produce a verified result after the LLM chooses an interpretation without asking the user to confirm it. The workflow was tested with 25 hand-authored cases that were fixed before execution. They are labeled C01 through C25: C01–C23 tested the LLM's routing, proposed input, and boundary decisions; C24 and C25 repeated two accepted inputs to check non-volatile result semantics. The labels are test numbers, not poker terminology. The main example uses one small pot-odds model. In this model, the pot is the shared pool of chips the players are competing for. The calculator's inputs are: pot_before_bet : the amount already in the pot before the opponent's new bet; opponent_bet : the amount the opponent adds; call_cost : the amount the player must add to continue; expected_rake : an optional amount removed from the final pot. The calculation is: net final pot = pot_before_bet + opponent_bet + call_cost - expected_rake

You just installed Cursor, opened a TypeScript file, and asked the agent to fix a bug. Ten seconds later it handed you a type SomeType = any and a @ts-ignore above the line that wouldn't compile. This is the moment most developers discover that AI coding agents are powerful but undisciplined. The fix isn't a better model. It's rules. Most AI coding agent best practices boil down to a single idea: tell the agent what good looks like before it starts typing. Cursor lets you define rules files in .cursor/rules/ that load alongside your project context and tell the agent how to behave. Claude Code has its own rules system, Windsurf has a rules directory, Copilot reads .github/copilot-instructions.md . Learn to configure cursor rules properly and your agent starts behaving like a careful senior engineer instead of an eager intern. What cursor rules files are Cursor rules are markdown files with a .mdc extension stored in .cursor/rules/ at your project root. Each file is a set of instructions the agent reads before it starts working. When a rule's conditions match the file being edited, the instruction is injected into the model's context window. A cursor rules file has two parts: a YAML frontmatter block between --- markers, and a markdown body with the actual instructions. How to configure cursor rules: the frontmatter fields Three fields matter. description (required). A short summary of what the rule enforces. Cursor surfaces this when you toggle rules, so make it specific. globs (optional). File patterns the rule applies to. Without globs, the rule applies to everything, which wastes context and creates conflicts. alwaysApply (optional). Set to true for rules that should load in every session, regardless of the files involved. Leave it false for rules that only trigger when matching files are touched. Real example: --- description : Enforce strict TypeScript, no any, no ts-ignore globs : ** /*.{ts,tsx} alwaysApply : false --- # Strict TypeScript ## Context This codeb

Every AI coding agent I've used shares one habit: it writes the plausible thing. The code compiles. The tests pass. And a senior engineer reviewing it would reach for a red pen. any where a union belongs. Tests that assert on implementation so they survive any refactor. catch (e) {} blocks that quietly swallow production errors. The fix isn't a better model or a cleverer prompt. It's a set of rules at the repo level, written in a format the agent is guaranteed to read. What rules files are Cursor reads .cursor/rules/*.mdc . Claude Code reads CLAUDE.md and AGENTS.md . The .mdc format is plain markdown with YAML frontmatter. Here's the opening of the TypeScript rule file from the AgentForge sample pack: --- description: "Strict TypeScript discipline for production code" globs: "**/*.{ts,tsx}" alwaysApply: true --- Three fields carry the weight. description tells the agent in one sentence what the file is for. globs scopes it, so a TypeScript rule never fires on a Python file. alwaysApply: true loads it into every session. Agents skip a 2,000-line rules file. They read a 40-line one. Write a discipline contract, not a wish list Claude Code follows instructions frighteningly well, and that cuts both ways. Tell it "write good code" and it will be confidently, grammatically wrong. An AGENTS.md contract fixes that. Start with a baseline of rules: think before you act, take small verifiable steps, never claim what you haven't verified, no drive-by refactoring. Then add a verification ladder with six rungs. Does it compile? Does the changed behavior work? Does it break anything adjacent? Does it follow the codebase's conventions? Does it hold at the boundaries? Is it observable in production? The first three are mandatory for every change. Then the failure protocol, which is the line that pays for itself: First failure: fix and re-verify. Second failure: re-derive, your mental model is wrong, form at least two new hypotheses. Third failure: stop, revert to last known-good, d

Last week I ran a demo of the agent for a colleague who was deciding whether to bet a feature on it. The approval gate from Part 7 worked exactly as designed. The agent searched, added to cart, asked for the address, and stopped at the confirmation link. My colleague nodded and asked one question: "OK, but is it actually good?" I did not have an answer. I had 31 passing tests from Part 6, which prove the agent is bug-free. I had a state machine, which proves it cannot place an order without a human. Neither of those proves the agent answers customers well. A bug-free agent can still tell a customer the shop ships within two days when the shipping partner takes five. No unit test catches that, because no unit test reads the answer. That is the gap this part closes. Part 7 ended with a promise: the next part would build an evaluation harness, so "is it good" stops being a feeling and becomes a score. This is that part. I built an LLM-as-a-judge harness for the same e-commerce agent as Parts 1 through 7: same nine tools, same supervisor, same memory. It runs 40 real conversations from production logs against five metrics every night and prints a score for each one. The first run was uncomfortable, and that is exactly why it exists. I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI agents with Spring Boot and Spring AI for over a year. Everything below is the harness as I actually run it. The Difference Between Tested and Good Part 6 tested the agent without an LLM: 31 tests, zero model calls, asserting on tool calls, services, and the order state machine. That suite answers "did the agent call the right tool, in the right order, with the right arguments?" It cannot answer "was the answer right?" because you cannot write an assertion for an LLM's wording. Evaluation is a different layer. You cannot assert on the answer, but you can judge it, and you can use an LLM to do the judging. Spring AI documents this pattern in its LLM-as

AI coding tools have solved one problem remarkably well: They can produce code extremely quickly. That sounds obviously good. And most of the time, it is. But software development has never really been constrained by how fast we can type. The expensive part comes later. Understanding the code. Reviewing it. Debugging it. Changing it six months later when the person—or model—that wrote it has forgotten why it exists. Test automation is where this becomes especially interesting. Generating the Test Is the Cheap Part You can ask an AI coding assistant: Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages. And a few minutes later you might have hundreds or thousands of lines of test code. It feels like incredible leverage. Until the suite starts failing. That’s the argument behind looking at the hidden cost of AI-generated test code . Generation cost has collapsed. Maintenance cost hasn’t. In some cases, AI actually increases it because you now have more code than your team would have written manually. AI Pull Requests Need Different Review There’s another subtle problem. Humans tend to judge large AI-generated pull requests differently. When someone on your team writes 80 lines, you probably read them. When an AI assistant generates 1,800 lines? You skim. You look at the filenames. You check whether CI is green. Merge. That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months. There are good ideas in this guide to testing AI coding assistant pull requests , but the bigger principle is simple: AI-generated tests need validation just like AI-generated product code. “Generated successfully” does not mean “tests the right thing.” Agents Add Another Failure Mode Now we’re moving from AI that writes test code to AI that actually decides what actions to take. That introduces a new question: What if the model chooses the wrong tool? An agent m

Two pages on my site went live on July 22. On August 8 they had zero impressions in Google. Not low. Zero, across three weekly exports. URL Inspection didn't say "crawled, not indexed." It said Google could not recognise the URL. Referring sitemap: none detected. Referring pages: none detected. Last crawl: not applicable. Never discovered. Seventeen days. The pipeline was green the entire time My deploy is a small chain: rsync the file, import it into MySQL, restart the service, ping IndexNow. Every step returned success. The last step returned 200 on every URL, every deploy, for three weeks. Here's what I'd never examined: IndexNow doesn't feed Google. It's Bing, Yandex, Seznam, Naver. My green light was real — it was just about a different search engine than the one whose console I was reading. That's the whole bug, and it isn't an SEO bug. It's the generic one: system A returns 200 → I conclude something about system B → nothing in the response object ever objected If you've ever read a webhook 202 as "the downstream processed it," or a CDN purge 200 as "the edge is cold," it's the same shape. What actually broke Search Console's Sitemaps report: Submitted: 2026-07-22 Last read: 2026-07-22 ← seventeen days ago Discovered: 101 URLs ← the file has had 117 for weeks The two pages went live on July 22 — the same day as the only read. Google fetched the sitemap and moved on, within hours of the file changing. Then nothing brought it back, because a sitemap changing on your server notifies nobody. There is no push. It's a pull-only resource with no cache invalidation, and if the consumer doesn't happen to return, your new URLs live in a document no one is reading. Resubmitting took two minutes. Read immediately, 117 URLs. So I wrote the check. It doesn't catch the bug. This is the part worth more than the fix. I wrote a post-deploy verifier. It does two things: // 1. every published, non-redirected page appears in the live sitemap const missing = published.filter((p) =

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor

Last updated: August 2026 The first few orders of an online store rarely make anyone think seriously about infrastructure. When there are only a handful of orders, almost any problem can be handled manually: check inventory, correct a status, contact a supplier, or figure out why incorrect information is appearing on a product page. At 10 orders, this is still a perfectly workable model. At 100, manual tasks start consuming a noticeable amount of time. Once operations reach 1,000 and beyond, however, the nature of the problem changes: small inconveniences begin turning into systemic limitations. As we develop Droplox, we increasingly look at scaling from this perspective. Growth from 1 to 10, then 100, and eventually 1,000 orders may look like nothing more than an increase in a single number. For architecture, data, and operational processes, these are completely different operating conditions. 1–10 Orders: Almost Everything Can Be Fixed Manually At the earliest stage, manual work is not necessarily a problem. In fact, it can be useful. The team gets to observe real user scenarios and understand which processes are genuinely worth automating and which happen so rarely that building a dedicated system for them would be premature. An incorrect order status can be checked manually. Outdated product information can be corrected quickly. A supplier issue can be handled as an isolated case. The difficulty comes later. Temporary solutions have an unfortunate tendency to become permanent. A spreadsheet created “for a couple of weeks” is still being used months later. A manual check becomes a mandatory step. A field added for one specific scenario suddenly becomes involved in five more. As long as the number of operations remains small, the cost of these compromises is almost invisible. Around 100 Orders: Random Problems Start Repeating As order volume grows, it isn’t only the workload that increases. Situations that once seemed like isolated incidents begin appearing regula

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything? It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had. The benchmark was measuring the order I ran things in machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25% . That claim also shipped inside machin guide , which is what every coding agent reads to learn the language. When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output: for kernel in kernels : for lang in [ machin , rust , zig ]: for _ in range ( 5 ): # all 5 machin, THEN all 5 rust, THEN all 5 zig time ( binary ) It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last . Zig always went last. Zig always looked slowest. The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did: intsum 10^9 before (blocked) after (interleaved) machin 2832 ms 3079.7 ms rust 3764 ms 3223.8 ms zig 3556 ms 3189.7 ms "machin +20-25%" machin +3% = a TIE A 20-25% win became a tie. I deleted the claim from the README and from machin guide . The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start

The most consequential bug in this project had been there since the beginning, survived several full end-to-end runs, and was reported as a PASS every time. ✓ PASS slow-and-low detected within 30m (7.3m), never exceeding legit rate That line is true. The scorer flagged it correctly, well inside the bound. What the line doesn't say is that the attacker was served every single request it ever made . Zero non-allow decisions, across the entire scenario. Detected and never once stopped. This is one instance of a pattern that accounts for more real bugs in this project than every other cause combined: a component contributes nothing, no error is raised, and every surrounding number stays plausible. The bug The scorer computes windows at 1m, 5m and 1h, and publishes each result to a per-client key in Redis and OPA. Each result. To the same key. So the last writer won. And 1-minute windows close most often, so they always won. slow-and-low issues about two requests a minute. Its 1m windows fall below the minimum request count and score zero. Its 5m and 1h windows accumulate the miss ratio that earns a deny . Every one of those zeroes immediately overwrote the deny. The entire premise of a multi-scale pipeline — that different attacks are visible at different scales — was silently violated by the publication step. Any detection that only appeared at a coarser scale was discarded. The fix is a roll-up: publish the most severe verdict across window sizes within the freshness horizon the policy already uses. Afterwards, the same attacker is denied on 24–31 of its 44 requests. Why it survived so long Because the report could not express it. Detection latency was computed as the earlier of two very different facts: the scorer's first non-allow window, and the gateway's first non-allow decision. Printed under one heading — detected — a client that was noticed but never touched looked identical to one that was noticed and blocked. A report that averages over the distinction you ar

A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding models before you commit . That harness answers one question: which model should I use? It does not answer the harder follow-up: once a model generates a patch for my real codebase, when is it safe to merge? This week there was a great discussion on DEV about "understanding over origin" — the idea that it doesn't matter whether code came from a human or a model, only whether someone actually understands it. I agree with the principle, but principles don't survive contact with a busy afternoon. What survives is a checklist with teeth. So here is the pipeline I bolted onto my model harness: every AI-generated patch has to pass through a scripted review gate before I even read it, and the script produces a scorecard that tells me how carefully I need to read it. The problem with eyeballing diffs When a model produces a 40-line diff that looks idiomatic, my brain does a dangerous thing: it pattern-matches on style and skips semantics. The code reads like something I'd write, so I approve it like something I'd write. The failures I've actually shipped from AI-generated code were never syntax errors — the tests even passed. They were things like: A retry loop that retried on the wrong exception type, so real errors got swallowed. A query filter that was subtly wider than the one it replaced (tests passed because fixtures were too small to notice). A dependency added for a one-liner the standard library already covers. All three would have been caught by asking four boring questions before reading the code. So I scripted the questions. The review gate: a reproducible artifact The gate is a small shell script. It takes a patch file, applies it to a throwaway worktree, and runs four checks. It never touches my working branch, and it prints a one-line verdict at the end. #!/usr/bin/env bash # review-gate.sh <patch-file> <base-branch> set -euo pipefail PATCH = " $1 " BASE = " ${ 2 :

I had a podcast pipeline I was proud of. It took a transcript, turned it into a two-person conversation with text-to-speech, laid in the music, and produced an MP3 I could publish. I'd built it in one app, and it worked. I loved the output. So when I started a second app that needed the same flow, I didn't want to rebuild the pipeline. I already had one. I just wanted it over there. So I asked the AI to copy it. And it did. Here's the part that matters: I didn't just copy it and hope. I checked. I opened a fresh session (a clean one, no memory of the first) and told it to look at the new pipeline and make sure everything was right. It went and looked. It came back and told me everything was good. Everything looked good. Or so I was told. Then I loaded the first real transcript and ran it. It was wrong. Not a little wrong. The voices were wrong. The music didn't come in when it was supposed to. It didn't cut off when it was supposed to. It didn't fade. It just stopped. The words were all there, every one of them, in the right order. But everything that made the first pipeline good (the timing, the production, the feel) was gone. I walked away from my desk for a bit. It pissed me off, because I'd done what I was supposed to do. I'd asked. It had answered. The check was green. And the check was a lie. Here's what I think I actually got wrong, and it's not "I trusted the AI." It's subtler than that. When I asked a fresh session to "make sure everything's good," I got back a confident yes. But the session had no way of knowing what good sounded like. It never heard the first pipeline. It had no stake in whether the podcast was any good. It reported what it could see (the code looked reasonable) and what it could see was almost never the thing I actually cared about. That's the trap, and it isn't a beginner's trap. I have a whole process built to avoid exactly this: spec, adversarial review, a plan, a build, a code review. And I skipped it, on a task I decided was too sma

The last lesson was about validating what a model hands you. The story behind it: a set of prompts that had returned real, criteria-matched vendors for weeks came back in staging with placeholder junk, literally the words Vendor A, Vendor B, Vendor C. So I built the validation layer, and the last gate in it is a model checking a model. Then FromZeroToShip asked three questions in the comments, and all three were about the gate rather than the model. That's the harder thing to look at, and I hadn't written all of it down. Here's the long version. What was on the fail list that I hadn't already been burned by? More than the question assumes, and not because I got clever about imagining failures. The placeholder output changed what I do with a failure . I stopped fixing the instance and asked what class it belonged to, and that class is a lot wider than "the model emitted example data." It's a suggestion that looks fine and isn't usable. Two of those I had never hit went in on the back of it: A vendor that's wrong for the category. A vendor that's no longer in business. Neither has anything to do with placeholder text, and both would sail through a schema check looking like a perfectly real answer. They also changed the prompt that produces the suggestions, not just the gate. Fixing only the failure I actually met would have left both of them live. So the list isn't purely retrospective. It grows by generalizing from the one failure you hit to the class it sits in, and it keeps growing from what the running system actually throws at me rather than from what I remembered to imagine. Is it foolproof? No. What's left is the case worth worrying about: results that read as real, pass the schema, satisfy every criterion I gave, and are still wrong. You can't validate the truth of a guess from inside the system. You can only lower the cost of it being wrong. That means a human in the loop at the stage where being wrong is expensive, the confidence surfaced so the answer is ch

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed. That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something. A few places I keep finding the real cause hiding. Tests that quietly depend on each other Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47. The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually. The app is racing, not the test Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off. This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, acro

Lets start with a bit of back story. I am a full stack developer. Developer being the keyword here, not a QA developer. But in my current role, I was recently asked to come up with a testing suite for the web application and the Flutter app I was managing and maintaining. At that time, I didn’t have anything better to do and thought this would be a fun little project to work on for a couple of weeks. Boy o boy, I was wrong. People in QA are so opinionated. Everyone has their preferred framework, structure, naming convention, abstraction, folder structure and a very strong opinion about why your approach is wrong. Starting with the industry best practices I started by trying to follow the trends and best practices used in the industry. Page Object Models, reusable helpers, proper assertions and all the usual bits and bobs. For the web application, which was built with React, I chose Playwright. For the Flutter app, I went with integration_test . Sounded simple enough. The login test that took three hours The first test I tried to write was a simple login flow. Open the application Enter the username and password Press the login button Wait for the dashboard Easy, right? It took me ages. And by ages, I mean roughly three hours just to get the web test to pass reliably. The actual Playwright test ended up being around 300 lines once I included the boilerplate, setup, selectors, assertions, waits, Page Object Model structure and everything else needed around the actual journey. Then came the Flutter app. That one was worse. The app has its own custom way of starting different flavors, and both the web application and Flutter app are white-labelled products. That means there are a lot of variations to cover. Different branding, configurations, screens and sometimes slightly different user journeys. Before I could even test the login flow, I needed a pile of setup code just to launch the correct version of the app. The Flutter test eventually went beyond 500 lines, includ

I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. What Is an Agent Evaluation Harness? An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: Define success metrics — what does "good" mean? Create test suites — what scenarios will you test? Run evaluations — how does the agent perform across all scenarios? Measure regressions — did a change make the agent worse? Track improvements — is version 2 better than version 1? This is not optional. This is engineering 101 . Why Most Agent Evaluations Are Wrong I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: Mistake 1: Single-Task Testing What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. Mistake 2: No Edge Cases What they do: Test happy paths only. "Book a flight when everything works.

Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow. This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do? The evaluation trap Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like: "Add retry logic to this half-migrated HTTP client without breaking the old call sites." "Write tests for a function whose behavior depends on a config file three directories up." "Refactor this 200-line function, but the ORM calls must stay in the same transaction." These tasks share a trait: correctness is checkable, but only by you . Your test suite, your type checker, your lint rules. That's actually good news — it means evaluation can be automated against artifacts you already have. The artifact: a task-runner harness The core idea is dumb on purpose. Define a set of tasks as directories. Each task has a prompt, a snapshot of the relevant code, and a verification command. The harness applies a model's patch and runs the verifier. No scoring model, no LLM-as-judge — just your own build. eval/ ├── tasks/ │ ├── 001-retry-http-client/ │ │ ├── prompt.md │ │ ├── repo/ # snapshot of the relevant files │ │ └── verify.sh # exit 0 = pass │ ├── 002-test-config-loader/ │ └── 003-split-billing-fn/ └── run_eval.py Here's a minimal runner (Python 3.10+, stdlib only): #!/usr/bin/env python3 """ run_eval.py — apply a model-produced patch to each task and

If you've experimented with AI-driven testing, you've probably lived this cycle: you ask an AI agent to "test the checkout flow," and it does something — clicks around, declares success, and leaves you unsure what was actually verified. The next day you ask again and it does something different. The browser automation works; the testing discipline is missing. That gap is what Snagly is for. Rather than describe it, I pointed it at softwaretestingtrends.com — my own production site, nothing fixed beforehand — and recorded the whole thing. It found eleven issues, including a critical accessibility bug on my own signup page. One of its findings turned out to be wrong, and I'll come back to that, because it matters more than the ones it got right. 📺 Watch the full walkthrough — installed from an empty folder, run against production, ~20 minutes. What it is Snagly is a free, MIT-licensed set of 30 skills for AI coding agents — GitHub Copilot , Claude Code , Cursor, Codex and 70+ others — that turn "an AI that can drive a browser" into "an AI that tests like a QA professional." A skill, if you haven't met them yet, is a reusable instruction set that teaches the agent a specific working method — when to use it, what rigor it requires, what evidence to capture, and what it must never do. Each skill in Snagly has one job, and they hand off to each other the way a real testing practice does: start-testing is the front door — say "what can you test here?" and it routes you to the right skill, checking prerequisites before handing off. Discovery & strategy : scenario-mapper explores your site and produces a prioritized list of test scenarios; test-case-writer expands any of them into a reviewable spec; test-plan sets strategy, cadence, and release exit criteria; qa-onboarding writes the guide for your next hire. Execution : flow-runner drives real user journeys step by step, asserting outcomes (not just that clicks happened) and capturing evidence the moment anything fails. cru