标签：#test

Last week I ran a demo of the agent for a colleague who was deciding whether to bet a feature on it. The approval gate from Part 7 worked exactly as designed. The agent searched, added to cart, asked for the address, and stopped at the confirmation link. My colleague nodded and asked one question: "OK, but is it actually good?" I did not have an answer. I had 31 passing tests from Part 6, which prove the agent is bug-free. I had a state machine, which proves it cannot place an order without a human. Neither of those proves the agent answers customers well. A bug-free agent can still tell a customer the shop ships within two days when the shipping partner takes five. No unit test catches that, because no unit test reads the answer. That is the gap this part closes. Part 7 ended with a promise: the next part would build an evaluation harness, so "is it good" stops being a feeling and becomes a score. This is that part. I built an LLM-as-a-judge harness for the same e-commerce agent as Parts 1 through 7: same nine tools, same supervisor, same memory. It runs 40 real conversations from production logs against five metrics every night and prints a score for each one. The first run was uncomfortable, and that is exactly why it exists. I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI agents with Spring Boot and Spring AI for over a year. Everything below is the harness as I actually run it. The Difference Between Tested and Good Part 6 tested the agent without an LLM: 31 tests, zero model calls, asserting on tool calls, services, and the order state machine. That suite answers "did the agent call the right tool, in the right order, with the right arguments?" It cannot answer "was the answer right?" because you cannot write an assertion for an LLM's wording. Evaluation is a different layer. You cannot assert on the answer, but you can judge it, and you can use an LLM to do the judging. Spring AI documents this pattern in its LLM-as

AI coding tools have solved one problem remarkably well: They can produce code extremely quickly. That sounds obviously good. And most of the time, it is. But software development has never really been constrained by how fast we can type. The expensive part comes later. Understanding the code. Reviewing it. Debugging it. Changing it six months later when the person—or model—that wrote it has forgotten why it exists. Test automation is where this becomes especially interesting. Generating the Test Is the Cheap Part You can ask an AI coding assistant: Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages. And a few minutes later you might have hundreds or thousands of lines of test code. It feels like incredible leverage. Until the suite starts failing. That’s the argument behind looking at the hidden cost of AI-generated test code . Generation cost has collapsed. Maintenance cost hasn’t. In some cases, AI actually increases it because you now have more code than your team would have written manually. AI Pull Requests Need Different Review There’s another subtle problem. Humans tend to judge large AI-generated pull requests differently. When someone on your team writes 80 lines, you probably read them. When an AI assistant generates 1,800 lines? You skim. You look at the filenames. You check whether CI is green. Merge. That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months. There are good ideas in this guide to testing AI coding assistant pull requests , but the bigger principle is simple: AI-generated tests need validation just like AI-generated product code. “Generated successfully” does not mean “tests the right thing.” Agents Add Another Failure Mode Now we’re moving from AI that writes test code to AI that actually decides what actions to take. That introduces a new question: What if the model chooses the wrong tool? An agent m

Two pages on my site went live on July 22. On August 8 they had zero impressions in Google. Not low. Zero, across three weekly exports. URL Inspection didn't say "crawled, not indexed." It said Google could not recognise the URL. Referring sitemap: none detected. Referring pages: none detected. Last crawl: not applicable. Never discovered. Seventeen days. The pipeline was green the entire time My deploy is a small chain: rsync the file, import it into MySQL, restart the service, ping IndexNow. Every step returned success. The last step returned 200 on every URL, every deploy, for three weeks. Here's what I'd never examined: IndexNow doesn't feed Google. It's Bing, Yandex, Seznam, Naver. My green light was real — it was just about a different search engine than the one whose console I was reading. That's the whole bug, and it isn't an SEO bug. It's the generic one: system A returns 200 → I conclude something about system B → nothing in the response object ever objected If you've ever read a webhook 202 as "the downstream processed it," or a CDN purge 200 as "the edge is cold," it's the same shape. What actually broke Search Console's Sitemaps report: Submitted: 2026-07-22 Last read: 2026-07-22 ← seventeen days ago Discovered: 101 URLs ← the file has had 117 for weeks The two pages went live on July 22 — the same day as the only read. Google fetched the sitemap and moved on, within hours of the file changing. Then nothing brought it back, because a sitemap changing on your server notifies nobody. There is no push. It's a pull-only resource with no cache invalidation, and if the consumer doesn't happen to return, your new URLs live in a document no one is reading. Resubmitting took two minutes. Read immediately, 117 URLs. So I wrote the check. It doesn't catch the bug. This is the part worth more than the fix. I wrote a post-deploy verifier. It does two things: // 1. every published, non-redirected page appears in the live sitemap const missing = published.filter((p) =

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor

Last updated: August 2026 The first few orders of an online store rarely make anyone think seriously about infrastructure. When there are only a handful of orders, almost any problem can be handled manually: check inventory, correct a status, contact a supplier, or figure out why incorrect information is appearing on a product page. At 10 orders, this is still a perfectly workable model. At 100, manual tasks start consuming a noticeable amount of time. Once operations reach 1,000 and beyond, however, the nature of the problem changes: small inconveniences begin turning into systemic limitations. As we develop Droplox, we increasingly look at scaling from this perspective. Growth from 1 to 10, then 100, and eventually 1,000 orders may look like nothing more than an increase in a single number. For architecture, data, and operational processes, these are completely different operating conditions. 1–10 Orders: Almost Everything Can Be Fixed Manually At the earliest stage, manual work is not necessarily a problem. In fact, it can be useful. The team gets to observe real user scenarios and understand which processes are genuinely worth automating and which happen so rarely that building a dedicated system for them would be premature. An incorrect order status can be checked manually. Outdated product information can be corrected quickly. A supplier issue can be handled as an isolated case. The difficulty comes later. Temporary solutions have an unfortunate tendency to become permanent. A spreadsheet created “for a couple of weeks” is still being used months later. A manual check becomes a mandatory step. A field added for one specific scenario suddenly becomes involved in five more. As long as the number of operations remains small, the cost of these compromises is almost invisible. Around 100 Orders: Random Problems Start Repeating As order volume grows, it isn’t only the workload that increases. Situations that once seemed like isolated incidents begin appearing regula

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything? It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had. The benchmark was measuring the order I ran things in machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25% . That claim also shipped inside machin guide , which is what every coding agent reads to learn the language. When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output: for kernel in kernels : for lang in [ machin , rust , zig ]: for _ in range ( 5 ): # all 5 machin, THEN all 5 rust, THEN all 5 zig time ( binary ) It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last . Zig always went last. Zig always looked slowest. The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did: intsum 10^9 before (blocked) after (interleaved) machin 2832 ms 3079.7 ms rust 3764 ms 3223.8 ms zig 3556 ms 3189.7 ms "machin +20-25%" machin +3% = a TIE A 20-25% win became a tie. I deleted the claim from the README and from machin guide . The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start

The most consequential bug in this project had been there since the beginning, survived several full end-to-end runs, and was reported as a PASS every time. ✓ PASS slow-and-low detected within 30m (7.3m), never exceeding legit rate That line is true. The scorer flagged it correctly, well inside the bound. What the line doesn't say is that the attacker was served every single request it ever made . Zero non-allow decisions, across the entire scenario. Detected and never once stopped. This is one instance of a pattern that accounts for more real bugs in this project than every other cause combined: a component contributes nothing, no error is raised, and every surrounding number stays plausible. The bug The scorer computes windows at 1m, 5m and 1h, and publishes each result to a per-client key in Redis and OPA. Each result. To the same key. So the last writer won. And 1-minute windows close most often, so they always won. slow-and-low issues about two requests a minute. Its 1m windows fall below the minimum request count and score zero. Its 5m and 1h windows accumulate the miss ratio that earns a deny . Every one of those zeroes immediately overwrote the deny. The entire premise of a multi-scale pipeline — that different attacks are visible at different scales — was silently violated by the publication step. Any detection that only appeared at a coarser scale was discarded. The fix is a roll-up: publish the most severe verdict across window sizes within the freshness horizon the policy already uses. Afterwards, the same attacker is denied on 24–31 of its 44 requests. Why it survived so long Because the report could not express it. Detection latency was computed as the earlier of two very different facts: the scorer's first non-allow window, and the gateway's first non-allow decision. Printed under one heading — detected — a client that was noticed but never touched looked identical to one that was noticed and blocked. A report that averages over the distinction you ar

A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding models before you commit . That harness answers one question: which model should I use? It does not answer the harder follow-up: once a model generates a patch for my real codebase, when is it safe to merge? This week there was a great discussion on DEV about "understanding over origin" — the idea that it doesn't matter whether code came from a human or a model, only whether someone actually understands it. I agree with the principle, but principles don't survive contact with a busy afternoon. What survives is a checklist with teeth. So here is the pipeline I bolted onto my model harness: every AI-generated patch has to pass through a scripted review gate before I even read it, and the script produces a scorecard that tells me how carefully I need to read it. The problem with eyeballing diffs When a model produces a 40-line diff that looks idiomatic, my brain does a dangerous thing: it pattern-matches on style and skips semantics. The code reads like something I'd write, so I approve it like something I'd write. The failures I've actually shipped from AI-generated code were never syntax errors — the tests even passed. They were things like: A retry loop that retried on the wrong exception type, so real errors got swallowed. A query filter that was subtly wider than the one it replaced (tests passed because fixtures were too small to notice). A dependency added for a one-liner the standard library already covers. All three would have been caught by asking four boring questions before reading the code. So I scripted the questions. The review gate: a reproducible artifact The gate is a small shell script. It takes a patch file, applies it to a throwaway worktree, and runs four checks. It never touches my working branch, and it prints a one-line verdict at the end. #!/usr/bin/env bash # review-gate.sh <patch-file> <base-branch> set -euo pipefail PATCH = " $1 " BASE = " ${ 2 :

I had a podcast pipeline I was proud of. It took a transcript, turned it into a two-person conversation with text-to-speech, laid in the music, and produced an MP3 I could publish. I'd built it in one app, and it worked. I loved the output. So when I started a second app that needed the same flow, I didn't want to rebuild the pipeline. I already had one. I just wanted it over there. So I asked the AI to copy it. And it did. Here's the part that matters: I didn't just copy it and hope. I checked. I opened a fresh session (a clean one, no memory of the first) and told it to look at the new pipeline and make sure everything was right. It went and looked. It came back and told me everything was good. Everything looked good. Or so I was told. Then I loaded the first real transcript and ran it. It was wrong. Not a little wrong. The voices were wrong. The music didn't come in when it was supposed to. It didn't cut off when it was supposed to. It didn't fade. It just stopped. The words were all there, every one of them, in the right order. But everything that made the first pipeline good (the timing, the production, the feel) was gone. I walked away from my desk for a bit. It pissed me off, because I'd done what I was supposed to do. I'd asked. It had answered. The check was green. And the check was a lie. Here's what I think I actually got wrong, and it's not "I trusted the AI." It's subtler than that. When I asked a fresh session to "make sure everything's good," I got back a confident yes. But the session had no way of knowing what good sounded like. It never heard the first pipeline. It had no stake in whether the podcast was any good. It reported what it could see (the code looked reasonable) and what it could see was almost never the thing I actually cared about. That's the trap, and it isn't a beginner's trap. I have a whole process built to avoid exactly this: spec, adversarial review, a plan, a build, a code review. And I skipped it, on a task I decided was too sma

The last lesson was about validating what a model hands you. The story behind it: a set of prompts that had returned real, criteria-matched vendors for weeks came back in staging with placeholder junk, literally the words Vendor A, Vendor B, Vendor C. So I built the validation layer, and the last gate in it is a model checking a model. Then FromZeroToShip asked three questions in the comments, and all three were about the gate rather than the model. That's the harder thing to look at, and I hadn't written all of it down. Here's the long version. What was on the fail list that I hadn't already been burned by? More than the question assumes, and not because I got clever about imagining failures. The placeholder output changed what I do with a failure . I stopped fixing the instance and asked what class it belonged to, and that class is a lot wider than "the model emitted example data." It's a suggestion that looks fine and isn't usable. Two of those I had never hit went in on the back of it: A vendor that's wrong for the category. A vendor that's no longer in business. Neither has anything to do with placeholder text, and both would sail through a schema check looking like a perfectly real answer. They also changed the prompt that produces the suggestions, not just the gate. Fixing only the failure I actually met would have left both of them live. So the list isn't purely retrospective. It grows by generalizing from the one failure you hit to the class it sits in, and it keeps growing from what the running system actually throws at me rather than from what I remembered to imagine. Is it foolproof? No. What's left is the case worth worrying about: results that read as real, pass the schema, satisfy every criterion I gave, and are still wrong. You can't validate the truth of a guess from inside the system. You can only lower the cost of it being wrong. That means a human in the loop at the stage where being wrong is expensive, the confidence surfaced so the answer is ch

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed. That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something. A few places I keep finding the real cause hiding. Tests that quietly depend on each other Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47. The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually. The app is racing, not the test Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off. This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, acro

Lets start with a bit of back story. I am a full stack developer. Developer being the keyword here, not a QA developer. But in my current role, I was recently asked to come up with a testing suite for the web application and the Flutter app I was managing and maintaining. At that time, I didn’t have anything better to do and thought this would be a fun little project to work on for a couple of weeks. Boy o boy, I was wrong. People in QA are so opinionated. Everyone has their preferred framework, structure, naming convention, abstraction, folder structure and a very strong opinion about why your approach is wrong. Starting with the industry best practices I started by trying to follow the trends and best practices used in the industry. Page Object Models, reusable helpers, proper assertions and all the usual bits and bobs. For the web application, which was built with React, I chose Playwright. For the Flutter app, I went with integration_test . Sounded simple enough. The login test that took three hours The first test I tried to write was a simple login flow. Open the application Enter the username and password Press the login button Wait for the dashboard Easy, right? It took me ages. And by ages, I mean roughly three hours just to get the web test to pass reliably. The actual Playwright test ended up being around 300 lines once I included the boilerplate, setup, selectors, assertions, waits, Page Object Model structure and everything else needed around the actual journey. Then came the Flutter app. That one was worse. The app has its own custom way of starting different flavors, and both the web application and Flutter app are white-labelled products. That means there are a lot of variations to cover. Different branding, configurations, screens and sometimes slightly different user journeys. Before I could even test the login flow, I needed a pile of setup code just to launch the correct version of the app. The Flutter test eventually went beyond 500 lines, includ

I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. What Is an Agent Evaluation Harness? An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: Define success metrics — what does "good" mean? Create test suites — what scenarios will you test? Run evaluations — how does the agent perform across all scenarios? Measure regressions — did a change make the agent worse? Track improvements — is version 2 better than version 1? This is not optional. This is engineering 101 . Why Most Agent Evaluations Are Wrong I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: Mistake 1: Single-Task Testing What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. Mistake 2: No Edge Cases What they do: Test happy paths only. "Book a flight when everything works.

Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow. This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do? The evaluation trap Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like: "Add retry logic to this half-migrated HTTP client without breaking the old call sites." "Write tests for a function whose behavior depends on a config file three directories up." "Refactor this 200-line function, but the ORM calls must stay in the same transaction." These tasks share a trait: correctness is checkable, but only by you . Your test suite, your type checker, your lint rules. That's actually good news — it means evaluation can be automated against artifacts you already have. The artifact: a task-runner harness The core idea is dumb on purpose. Define a set of tasks as directories. Each task has a prompt, a snapshot of the relevant code, and a verification command. The harness applies a model's patch and runs the verifier. No scoring model, no LLM-as-judge — just your own build. eval/ ├── tasks/ │ ├── 001-retry-http-client/ │ │ ├── prompt.md │ │ ├── repo/ # snapshot of the relevant files │ │ └── verify.sh # exit 0 = pass │ ├── 002-test-config-loader/ │ └── 003-split-billing-fn/ └── run_eval.py Here's a minimal runner (Python 3.10+, stdlib only): #!/usr/bin/env python3 """ run_eval.py — apply a model-produced patch to each task and

If you've experimented with AI-driven testing, you've probably lived this cycle: you ask an AI agent to "test the checkout flow," and it does something — clicks around, declares success, and leaves you unsure what was actually verified. The next day you ask again and it does something different. The browser automation works; the testing discipline is missing. That gap is what Snagly is for. Rather than describe it, I pointed it at softwaretestingtrends.com — my own production site, nothing fixed beforehand — and recorded the whole thing. It found eleven issues, including a critical accessibility bug on my own signup page. One of its findings turned out to be wrong, and I'll come back to that, because it matters more than the ones it got right. 📺 Watch the full walkthrough — installed from an empty folder, run against production, ~20 minutes. What it is Snagly is a free, MIT-licensed set of 30 skills for AI coding agents — GitHub Copilot , Claude Code , Cursor, Codex and 70+ others — that turn "an AI that can drive a browser" into "an AI that tests like a QA professional." A skill, if you haven't met them yet, is a reusable instruction set that teaches the agent a specific working method — when to use it, what rigor it requires, what evidence to capture, and what it must never do. Each skill in Snagly has one job, and they hand off to each other the way a real testing practice does: start-testing is the front door — say "what can you test here?" and it routes you to the right skill, checking prerequisites before handing off. Discovery & strategy : scenario-mapper explores your site and produces a prioritized list of test scenarios; test-case-writer expands any of them into a reviewable spec; test-plan sets strategy, cadence, and release exit criteria; qa-onboarding writes the guide for your next hire. Execution : flow-runner drives real user journeys step by step, asserting outcomes (not just that clicks happened) and capturing evidence the moment anything fails. cru

I have been a huge Storybook and Chromatic fan for years. But at some point the bill got my attention, and when I looked into why, the fix turned out to be simple. This is the write-up of what I changed. It works on any per-snapshot tool, not just Chromatic. First, some backstory on how I got here, because it explains why the cost crept up in the first place. How I ended up paying for a lot of snapshots In the past I would build a gigantic end-to-end pipeline that was flaky as hell and made me spend time every week fixing it. It took 40 minutes to run, and when it went red someone would assume it was just flaky, merge the change anyway, and then find out it truly did break the system. So I stopped writing lots of E2Es and moved to Storybook for interaction and visual testing. Much better. But because I was rendering every state of every component as its own story to get the screenshots in place, I was generating a lot of screenshots. And every snapshot tool, Chromatic, Percy, Playwright screenshots, UI Verify, renders and bills per story. So the number of stories is the cost, and it is also the noise surface: more stories means more places for a diff to flake. I ended up paying a lot, which made me think about whether there were ways to optimise it. There were. Here they are. The core idea: combine states into one story The naive pattern is one story per variant times state times theme. A component with 5 sizes, 3 states, and 2 themes is 30 snapshots the naive way. The whole idea below is to collapse that matrix into a handful of stories while keeping full coverage. Move 1: one gallery story, not N stories For something like a Button, there is no need to have separate Primary, Secondary, and Tertiary stories. I prefer one AllVariants story that maps through the prop combinations and renders them in a grid. One snapshot then covers the entire matrix. As a bonus you get a nice grid that shows every permutation at a glance, with no extra clicks to see the variations. /

How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it. The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing. The 11 Grader Types exact_match — Compare output to expected output, optionally ignoring case. contains — Check if the output contains a substring. regex — Match against a regular expression. semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold. llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader. json_schema — Validate that the output is valid JSON matching a JSON Schema. json_path — Extract a value from JSON output using a JSONPath expression, then compare it. code — Run a function to evaluate the output. human — Defer to human review. model_graded — Compare the output to a reference answer using a model. custom — Escape hatch for graders not covered by built-in types. How Graders Connect to Test Cases A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately. Why This Design Works Self-describing: An eval suite carries everything a framework needs to execute it. Framework-agnostic: Any framework can implement any subset of grader types. Extensible: The custom type lets frameworks bring their own graders. Comparable: Results from different frameworks use the same grader IDs. Try It pip install evalport-sdk npm install evalport-sdk Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md Repo: https://gith

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. Did anyone ask for...

I started this project because an LLM annoyed me. I gave a very strong model 322 , a small Dota 2 drafting game. The choices looked like the kind of work a computer should enjoy: repeated packs of players and heroes, visible ratings, familiarity scores, chemistry, rerolls and a simulated tournament at the end. I was disappointed by how well the LLM did. I am not a Dota expert, and I had only started watching it occasionally again during the previous six months or year. I still seemed to be doing better. The interesting engineering question was not how to write a longer prompt. It was how to replace the card-by-card language-model judgement with a deterministic policy, then test that policy without confusing improvement with luck. A stochastic benchmark needs shared randomness The browser history gave us a useful irritation and almost no reliable comparison. My earlier manual record contained 50 runs with a 14% title rate. The LLM won once in nine attempts. Putting 14% beside 11% looks temptingly quantitative, but the random offers, rejected packs and opponent fields were not preserved. The samples were small, unpaired and produced under different choices. That is not a model benchmark. It is a reason to build one. The offline solver generated every random choice from indexed tapes. Policy A and policy B received the same player offers, hero samples, field candidates and tournament randomness for a given episode. We could then compare the paired result: did the new policy win this exact episode where the old policy lost it? This is the common-random-numbers idea in a practical form. Sharing the luck removes a large amount of noise that has nothing to do with the policy change. Keep the simulator separate from the policy Before evaluating a strategy, we reproduced the game. The public client and seven data files were frozen with SHA-256 hashes. Draft legality, automatic hero allocation, chemistry, scoring and the tournament were ported into a deterministic Python engi

Top scores increased by 5x.