标签：#Testing

If you’ve managed web servers or applications for any length of time, you’ve probably seen this happen: a new feature or campaign goes live, traffic suddenly spikes, and Website Load Testing becomes critical when your website starts returning 503 errors at exactly the moment you need it to perform. What happens next is usually a scramble, SSH into a server you haven’t checked in months, inspect running processes, restart services, and make infrastructure changes based on guesswork. Eventually, the traffic settles, the site recovers, and the immediate crisis is over. But that kind of incident is often preventable. Load testing helps you find your website’s limits before your users do. In this guide, we will cover what load testing is, why it matters at every scale, how to run your first test using loader.io (the most accessible free tool available), what your results actually mean, how to find and fix bottlenecks, and how to make load testing a normal part of how you ship software. TL;DR Load testing answers one critical question: how many concurrent users can your server handle before it falls over? Without it, you’re guessing about capacity, and guessing wrong right when it matters most loader.io is the simplest free tool to get started: no install, browser-based, generous free tier Your three essential numbers: concurrent user target, response time threshold, and peak traffic window Run load tests before every major deployment, not after your site goes down What Load Testing Actually Is Let me clear up some confusion first, because “load testing” gets thrown around interchangeably with a few related terms that mean different things. Load testing is specifically about simulating concurrent users hitting your site and measuring how your server behaves under a expected load. You’re asking: “When 500 people are on this site at the same time, what happens?” Stress testing pushes beyond that, you keep adding users until something breaks, then you figure out exactly wher

While exploring X360 AI Tech, I started thinking about something beyond just generating test cases-how much should we actually trust them? Creating a basic happy-path test with AI seems pretty easy, but things like business logic, edge cases, and whether the test is actually checking the right thing still need a human eye. I’m also wondering about what happens a few months down the line. The app changes, requirements change, and some tests that made sense earlier may not make sense anymore. So maybe the bigger challenge isn’t just generating tests, but keeping them useful over time. For me, AI feels more useful as a second pair of hands rather than something that makes all the testing decisions. Curious how others are using it in real projects-are you reviewing every AI-generated test, or trusting it for certain types of scenarios?

In the previous article , I argued that AI isn't simply changing the tools we use to develop software, but shifting our work to a new level of abstraction. In this one, I want to address the problem that immediately follows: if we're going to write less and less code directly and agents are going to produce an increasingly larger part of it, how the hell do we know whether what they code is actually right? Because the answer obviously can't be “trust the AI, it's very smart”. Even though I personally develop code with AI today with practically no review, I don't blindly trust AI. Just as I don't blindly trust an engineer on my team. I don't even blindly trust myself. Blind trust is a security hole. And not blindly trusting someone doesn't mean distrusting them, it means having mechanisms to prevent their mistakes, or mine, from causing problems. That's why we've spent decades building mechanisms and methodologies around software development to detect, and avoid as much as possible, our mistakes. XP. Scrum. Tests. Code reviews. Pair Programming. CI. Static analysis. Permissions. Observability. Environments. Containers. Auditing... The question, therefore, shouldn't be whether we can trust an AI. The question should be what system do we need to build so we can use it without needing to blindly trust it? It's not deterministic One of the first objections is usually that if you ask it the same thing twice, it generates two different pieces of code. True. But if you give the same task to two different programmers, or to the same programmer with enough time in between, we'll very probably get two different implementations too, depending on the complexity of what we're asking. And if we've never required two developers to produce exactly the same code, why do we expect AI to produce exactly the same code from the same request? Isn't it enough for the result to satisfy the requested requirements? That it does what it's supposed to do. That it passes all kinds of tests. That

Acceptance gaps Your monitoring tool has 33 tests. Every one of them passes. It has also never executed, not once, and nothing in your project will ever tell you. Correct, complete, and never started We built a tool that scans four public sources for conversations where our product belongs. It reads only. It never posts. The acceptance was thorough. 33 checks in total. No write access, no credentials, results filtered for relevance, duplicates removed. A hard cap on output, back off on HTTP 429, and a dead source must not swallow the other three. All 33 passed. The tool shipped. The task promised a list of conversations every morning. There was no schedule. So there was never a list. And a tool that does nothing also reports nothing, so nobody noticed. Why your test suite cannot see this Tests answer questions about behaviour. Given this input, does the code do the right thing? Whether anything ever supplies that input is a different kind of question. It lives in a scheduler, a workflow file, a systemd timer, a queue consumer. Your test suite has no opinion about it. This is why the gap survives review. Reviewers read the diff, and the diff is correct. The missing part is not in the diff at all. Note how ordinary the failure is. Nobody was careless. The work was good. It just was not connected to anything. The question to add to every acceptance What starts this, and how would I know if it stopped? Ask it about every tool you ship that is meant to run on its own. A report, a backup, a sync, a scanner, a cleanup job. It has two halves and both matter. Something must start it. And when it stops starting, that must be visible without anyone going to look. Make it answerable by a machine A question you have to remember to ask gets forgotten. So we wrote a guard that asks it for every tool at once. The rule: any script whose own header says it runs daily must appear in a workflow file that has a schedule. Twelve lines of code, and it covers every tool we will ever add. T

The Third Predicate: Argument-Space Verification, Tested Agent Determinism Illusions (Part 10) Part 8 ended with a three-stage pipeline — evidence gate → contract regex → per-requirement LLM — and a patched framing: the combination narrows the gap without closing it. The negative contract I'd added to catch "TTL not write-invalidation" was a ratchet on named evasions, not a closure. Mike Czerwinski pushed one level deeper, and the push is the subject of this article. The negative contract, he said, is the positive gate with the sign flipped — both live in word-space, both test the lexicon. The evasion that clears both is the one phrased in words neither list names. And the predicate that actually matches scope to claim isn't lexical at all: "Write-invalidation done honestly isn't 'says invalidate, doesn't say TTL-simpler,' it's 'exercises the write path and observes the invalidation on the key the claim names.' That's argument-resolution... Positive and negative both live in word-space. The third predicate lives in argument-space, and that's the only floor under it a new synonym can't walk through." This article tests that claim. Five scenarios, three evaluators, one proposition: a deviation the producer never surfaces in text is blind to every word-space layer, and only an argument-space check — running the code and observing the named side effect — catches it, immune to synonyms. 1. The proposition, made testable Strip the comment to a falsifiable claim: A non-surfaced deviation — one the producer never writes into any evidence file — is invisible to word-space layers (contract regex, per-requirement LLM reading evidence text). Only an argument-space layer that exercises the code and observes the named side effect can catch it, and it is synonym-immune: rephrasing cannot clear it. The contrapositive is where the experiment earns its keep: if I can construct a scenario where the producer fabricates compliant evidence text but the implementation does not comply, the

Last month a model I'd been watching dropped its token price by half, and three people sent me the announcement within an hour. The implied question was always the same: when are you switching? My answer, these days, is: after it survives quarantine. Because the last time I swapped a model based on announcement-day excitement, everything looked fine for nine days. Then a scheduled job started emitting subtly malformed JSON — valid enough to parse, wrong enough to corrupt downstream state — and I spent a weekend reconstructing which records had been poisoned. The money I saved on tokens wouldn't cover one hour of that cleanup. The economics of model swaps are lopsided. The upside is small and predictable (cheaper tokens). The downside is unbounded and sneaky (behavioral regressions in edge cases your happy-path tests never exercised). So I built a pipeline that treats every new cheap model like an untrusted dependency with an attractive changelog: it gets isolated, probed, and graduated in stages. Here's the whole thing. What the pipeline needs (and what it doesn't) Three ingredients: candidate model access, somewhere disposable to run the evaluation, and checks that don't require a second LLM to grade the first one. For model access and the throwaway compute, I'm currently using MonkeyCode's free model access together with its free server option — bursty evaluation workloads are exactly the kind of thing I'd rather not attach to a production billing account. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing in the pipeline below is tied to that provider, though. Every endpoint is an environment variable, and I'd encourage you to wire it to whatever you're actually evaluating. I want to be explicit about two things I'm not assuming: that any particular model is on the free tier when you read this, and that any free offering stays available forever. Treat free infrastructure the way you treat a library's latest tag — convenient, n

يقوم Claude الآن بإرفاق بيانات تعريف العزو (provenance metadata) المشفّرة وفق C2PA بالملفات التي ينشئها. وينطبق الأمر نفسه على نماذج الصور من OpenAI و Gemini . هذا يعني أن إشارة العزو تصل سليمة إلى نقطة التحميل لأول مرة، لكن سلسلة معالجة الصور لديك قد تحذفها قبل أن يراها أي شخص. جرّب Apidog اليوم لا يحدث ذلك بنية سيئة؛ بل يحدث افتراضيًا. فمثلًا، تنشئ sharp().resize() ملفًا جديدًا بلا بيانات تعريف ما لم تطلب الاحتفاظ بها صراحةً. وينطبق ذلك أيضًا على ImageMagick وPillow ومعظم شبكات CDN الخاصة بالصور. يدخل الملف، ويخرج JPEG أصغر، ولا تخبرك السجلات أن بيانات العزو اختفت. هذه مشكلة قابلة للاختبار. ستتعلم هنا كيف تحدد المرحلة التي تحذف بيانات C2PA، وتثبت ذلك عبر رحلة رفع وتنزيل حقيقية، وتضيف فحصًا في CI يمنع عودة المشكلة. يتولى Apidog تنسيق سيناريو الـ API، بينما يتولى c2patool التحقق من صحة البيانات على مستوى البايت. ما الذي يتم تدميره بالفعل؟ بيان C2PA هو كتلة موقعة تشفيريًا ومضمّنة داخل حاوية الملف. يسجل من وقّع الأصل وما الذي ادعاه عنه. وبما أنه موقّع، فإن تغيير البايتات دون إعادة التوقيع يكسر التوقيع بطريقة يستطيع أي مدقق اكتشافها. النقطة المهمة هنا هي حاوية الملف : عندما تعيد كتابة الحاوية، قد يختفي البيان. العملية هل يبقى البيان افتراضيًا؟ نسخ أو نقل بايت ببايت نعم sharp().resize().toBuffer() لا ImageMagick عبر convert أو magick لا Pillow عبر Image.save() لا تحويل PNG إلى WebP أو JPEG إلى AVIF لا التحسين التلقائي في CDN للصور غالبًا لا لقطة شاشة لا إعادة الحفظ من محرر صور لا رفع إلى S3 دون تحويل نعم كل عنصر في عمود لا هو إجراء شائع في تطبيقات الويب: إنشاء صور مصغرة، توليد صور متجاوبة، التفاوض على التنسيق، أو إزالة EXIF لأسباب الخصوصية. كل خطوة منطقية بمفردها، لكنها قد تنهي سلسلة العزو بصمت. انتبه أيضًا إلى أن استخدام -strip لإزالة بيانات EXIF قد يكون مقصودًا، لأن EXIF قد يحمل إحداثيات GPS أو أرقامًا تسلسلية للكاميرا. لكن إزالة جميع البيانات الوصفية للتخلص من بيانات الموقع تزيل بيان C2PA كذلك. الحل هو إزالة البيانات الحساسة بشكل انتقائي، لا حذف الكتلة كاملة. أثبت المشكلة في دقيقتين قبل تغيير خط الأنابيب، تحقق من وجود المشكلة فعلًا. تحتاج إلى ملف واحد يحتوي على بيان

The worst test suite I ever inherited had 400 tests, and I trusted about six of them. The rest were either testing implementation details nobody cared about, duplicating each other, or so tightly coupled to internal function names that a harmless refactor broke thirty tests for no real reason. Reading that codebase taught me more about what not to do than any greenfield project ever has. So when you're starting from zero, the goal isn't "write a lot of tests fast." It's building a suite you'll still trust a year from now. If you're new to this, getting the software testing basics right early matters more than covering everything - learning how to build a first test suite from scratch teaches you what to prioritize in a way that inheriting someone else's bloated suite never will. Here's roughly how I'd approach it. Start with what would actually hurt if it broke Before writing a single test, list the handful of things that would be genuinely bad if they silently broke - checkout completing, auth working, the core thing your product does actually happening. Not every function, not every branch. Just the stuff where a silent failure costs you money, users, or trust. This list is usually shorter than people expect. Five to ten flows for most early-stage products. That's your actual test suite's job in the first few months, not "100% coverage." Unit tests for logic, not for plumbing Unit tests are for things with actual decision-making in them - pricing calculations, validation rules, state transitions, anything where "given this input, is the output correct" is a real question with a wrong answer possible. They're fast, they're cheap, and they should make up the bulk of your suite. Skip unit-testing pure plumbing: a function that just calls another function and returns its result doesn't need its own test. That's the kind of test that pads a coverage number without catching anything real, and it's exactly the kind of test that made that 400-test suite so hard to trust.

If you're testing an iOS build with real devices, you've got two main paths: Apple's TestFlight, or Expo's EAS Preview using Ad-Hoc provisioning. They solve the same problem — getting a build onto a real iPhone without the App Store — but the workflows are genuinely different, not just cosmetically. How each one works TestFlight uses Apple's official infrastructure. You upload your build to App Store Connect (often via npx testflight to speed this up), Apple processes/reviews it, and testers install the TestFlight app and accept an email or public link invite. No UDID collection needed — Apple handles device registration behind the scenes. Expo EAS Preview (Ad-Hoc) uses Ad-Hoc provisioning. You register each tester's device UDID against your Apple Developer account before building — either manually (eas device:create, eas device:list) or by having the tester scan a QR code that installs a temporary profile. Once devices are tied to your provisioning profile, you build with: bash eas build --platform ios --profile preview This generates a direct install link/QR code — no App Store account or TestFlight app required. Comparison table Feature Expo Preview / Ad-Hoc Apple TestFlight Device limit ~100 devices/device class/year (Apple Developer account tier) Up to 10,000 external testers Processing time Immediate after cloud build finishes Apple review/processing (mins to hours) UDID management Manual or profile-based registration required Not required, handled by Apple Best for Fast internal testing, client demos, strict ad-hoc distribution Larger-scale beta testing, staging before production Which one should you use? Fast internal iteration, client demos, small teams → Ad-Hoc. No waiting on Apple, instant install links. Wider beta testing before a production release → TestFlight. Built-in scale, no manual device management. Most teams I've worked with end up using both at different stages: Ad-Hoc during active development for quick feedback loops, TestFlight once the bui

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet. That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo. The eval set: eval/cases.json 21 cases, three buckets: Bucket Count Covers Easy 12 Shipping-status lookups, simple KB questions, a cancelled-order info request, one no-KB-match case that must escalate rather than fabricate Hard 6 Refund eligibility inside/outside the 30-day window, multi-item orders where only one item is refunded, boundary cases just past the window Edge 3 Legal-threat, fraud-flag, and duplicate-ticket patterns — must auto-escalate with zero tool/LLM calls A sample case, checking both the outcome and the trajectory that produced it: { "case_id" : "hard_03" , "bucket" : "hard" , "ticket" : { "ticket_id" : "hard_03" , "subject" : "Wrong size shoes, keep the socks" , "body" : "The running shoes from order ord_1004 are the wrong size. I want a refund for just the shoes, not the socks." , "customer_id" : "cust_002" , "order_id" : "ord_1004" }, "expected_trajectory" : [ "order_lookup" , "refund_eligibility" , "issue_refund" ], "expected_outcome" : "refund_proposed" , "expected_max_steps" : 4 , "policy_checks" : [ "refund amount reflects only the shoe item (~$74), not the full order total" , "issue_refund gated behind human approval" ] } Three things being checked per case, not just "did the answer loo

A free AI visibility score is auditable only when you can inspect the prompt, engine, raw answer, date, and denominator. Treat the score as a test result, not a property of your brand. This tutorial builds a six-check control you can run by hand, store as plain data, and compare with any tool's output. The workflow takes three buyer questions, runs them in two AI surfaces, and records the six answers without trying to force agreement. It will not estimate your entire market. It will tell you whether a dashboard's headline number has enough evidence to be investigated. What does an AI visibility score measure? An AI visibility score usually summarizes brand presence across a defined set of generated answers. That definition contains the trap: the question set is part of the metric. So are the engine panel, run date, session state, retrieval mode, and rule used to count a “hit.” Remove those inputs and the number is not reproducible. Imagine a tool asks three questions in two engines. That creates six cells. If your brand appears in two cells, the simple presence result is: presence = brand_present_cells / total_cells presence = 2 / 6 presence = 0.333... = 33.3% The arithmetic is trivial. The evidence is not. A different tool can ask five different questions in three engines and produce a different score without contradicting the first run. The two tools measured different grids. Keep the unit explicit: “present in two of six generated answers on this date” is defensible. “Our AI visibility is 33” is incomplete. Which evidence fields should you require? Require five fields for every result: prompt, engine, raw answer, timestamp, and counting rule. Use a sixth field for cited sources when the surface exposes them. A source-only appearance and a prose mention can signal different problems, so do not merge them silently. Here is one real saved result from Webappski's public 14 June 2026 tracker report: { "run_date" : "2026-06-14" , "prompt" : "beste Answer Engine Optimiz

You can load-test search latency with a script and a graph. Relevance has no such gauge by default, so most teams ship a new ranking rule, eyeball a handful of queries, and hope nothing important regressed. The fix is a small, boring relevance test suite: a fixed set of queries, human-judged expected results, and a metric you compute the same way every time — so "did this ranking change help?" becomes a number you can diff, not an argument you have in Slack. This post is a build guide. By the end you'll have a judgments file, a scorer that outputs precision@k, MRR, and nDCG, and a before/after comparison you can wire into CI. The examples use Postgres full-text search, but the harness is engine-agnostic — Elasticsearch, Meilisearch, or a vector store all slot into the same shape. Why can't I just load-test relevance the way I load-test latency? Latency is a property of the system. Relevance is a property of the match between a query and what a human expected to see — and that judgment lives outside the database. A commenter on an earlier post about running Postgres search in production put it well: latency can be load-tested, but quality needs query sets, expected result buckets, bad-query examples, and a way to compare changes before shipping a new ranking rule. That's the whole job, and none of it comes for free with your index. The trap is thinking a passing query proves relevance. SELECT ... WHERE tsv @@ query returning rows tells you the index matched. It says nothing about whether the right rows landed in the top 5, which is all a user ever sees. The takeaway: relevance is measured against human judgments, not row counts — so the first artifact you build is the judgments, not the query. Building the golden query set Start with 20–50 real queries. Pull them from your search logs if you have them (the head terms plus a long tail of specific ones), or write them from real user intents if you don't. For each query, mark which documents should come back and how rel

Here is every promise a to-do list makes. VARIABLE tasks Init == tasks = [i \in Ids |-> "absent"] Add(i) == tasks[i] = "absent" /\ tasks' = [tasks EXCEPT ![i] = "open"] Complete(i) == tasks[i] = "open" /\ tasks' = [tasks EXCEPT ![i] = "done"] Reopen(i) == tasks[i] = "done" /\ tasks' = [tasks EXCEPT ![i] = "open"] Delete(i) == tasks[i] # "absent" /\ tasks' = [tasks EXCEPT ![i] = "absent"] ClearCompleted == /\ \E i \in Ids : tasks[i] = "done" /\ tasks' = [i \in Ids |-> IF tasks[i] = "done" THEN "absent" ELSE tasks[i]] Not a summary. Not the important ones. All of them. A task cannot go from absent straight to done. Clearing completed items leaves the open ones alone. You cannot delete something that was never there. Nine lines, and when you've read them you have read the entire contract. Now go find that list for the system you work on. You can't. It doesn't exist. It's distributed across a test suite that asserts outcomes rather than rules, some validation scattered through handlers, and the memory of whoever's been there longest. The guarantees are real — your users depend on every one of them — and there is no file you can open to see them. That's the gap I want to talk about, because you can close it in an afternoon, and because something has changed recently that makes closing it pay for itself. The prime mark and two operators That's most of the syntax, so let's get it out of the way. tasks' means "tasks, in the next state." /\ is and . \E is "there exists." A definition like Complete(i) is a formula relating the current state to the next one — read it out loud: the task is open, and afterwards it is done. That's it. That's the language, near enough, for this purpose. The real file adds about eight lines of scaffolding around what you saw: a module header, a TypeOK saying a task is always in exactly one of the three states, and the two lines that tie the actions together — Next == \/ \E i \in Ids : Add(i) \/ Complete(i) \/ Reopen(i) \/ Delete(i) \/ ClearComplete

Here's a bug your test suite probably wouldn't catch. A backend developer refactors the user model. The id field — an integer since forever — starts coming back as a string: "42" instead of 42 . Every value is still "correct". Your assertion pm.expect(user.id).to.eql(42) fails, sure — but only on the one endpoint you asserted id on, not the other nine that return users. Meanwhile three client apps that did user.id + 1 are now computing "421" . That's structural drift , and it's what actually breaks API consumers: renamed fields, changed types, properties that quietly vanish. Field-by-field value assertions catch it patchily and by accident. Schema validation catches it systematically — and in Postman it costs about ten lines, because the ajv JSON-schema validator is built into the script sandbox. The ten lines In Scripts → Post-response on any request that returns a user: const userSchema = { type : " object " , required : [ " id " , " name " , " email " ], properties : { id : { type : " integer " }, name : { type : " string " }, email : { type : " string " , pattern : " @ " } } }; pm . test ( " Response matches the user schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userSchema ); }); That single test now fails if id becomes a string, if email disappears, if name becomes an object — every structural mutation, whether or not you thought to assert on that field's value. For an endpoint returning an array of users: const userListSchema = { type : " array " , minItems : 1 , items : userSchema // reuse the object schema }; pm . test ( " List matches schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userListSchema ); }); Share one schema across every endpoint The real power move: your API returns users from /users , /users/:id , /login , /teams/:id/members … and they should all be the same shape . Store the schema once as a collection variable (JSON, stringified), and every request validates against the sa

I have been building a thing that lets a language model propose an UPDATE , then executes it for real inside a transaction, measures the actual before and after values, and always rolls back. A human reads the measurement and decides. Only then does anything commit. The pitch is one sentence: what you approve is not the model's description of its SQL, it is what the database did when the SQL ran. Last week I found that the thing showing you that measurement was showing you a subset of it, and had been since the first release. The failure Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row: name = 'Tanaka' , postcode = '00100' . UPDATE customers SET name='Sato', postcode='00100' WHERE id=1 What this touches customers — Customer records. The postcode is used for billing address and delivery. 1 row would change, across 1 column: name Measured by running the statement and rolling it back id = 1 name: 'Tanaka' -> 'Sato' One row, one column. postcode is not mentioned, and that is correct — it is being assigned the value it already holds, so nothing about it changes. The card is describing the diff accurately. Approve it. Then, before it is applied, somebody else notices the postcode is wrong and fixes it: UPDATE customers SET postcode = '90210' WHERE id = 1 ; Now apply the approved plan: Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z. DB now: [{"name":"Sato","postcode":"00100"}] The fix is gone. Zero warnings. The word postcode never appeared on the approval card, never appeared in the audit record, and never appeared in the comparison the tool makes before it commits. One variable doing two jobs The diff was built like this: const changed : string [] = []; for ( const c of Object . keys ( before )) { if ( same ( before [ c ], after [ c ])) continue ; // drop what did not move if ( auto . has ( lower ( c ))) continue ; // drop what the DB maintains itself changed . push ( c ); } That is a correct answer to "what should the card sho

Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call? A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals: stop accepting new work; let safe work finish or reach a checkpoint; make ambiguous work visible instead of guessing; resume with an explicit recovery decision. 1. Model maintenance as a state transition Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine: RUNNING -> DRAINING -> QUIESCED -> STOPPED | +-> NEEDS_REVIEW DRAINING rejects new jobs but allows an active job to continue until its next checkpoint or deadline. QUIESCED means there are no unclassified side effects in flight. NEEDS_REVIEW is the safe outcome when the process died after sending a request but before recording the response. Persist the transition, not just an in-memory flag. A minimal record can look like this: { "runtime" : "agent-7" , "maintenance_id" : "mw-2026-08-10-001" , "state" : "DRAINING" , "started_at" : "2026-08-10T08:00:00Z" , "accepting_work" : false , "active_runs" : 2 } If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED . That is much more useful than inferring health from a missing PID. 2. Put checkpoints around side effects An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary: PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED On restart: PLANNED can be

I shipped a page that reported zero axe violations . It had button text at a contrast ratio of 1.04:1 — which is, for practical purposes, invisible text. The scan wasn't broken. It was answering a narrower question than I thought I was asking. The bug I had a theme system built the ordinary way. Tokens on :root , overridden in a prefers-color-scheme media query, and overridden again by an explicit [data-theme] attribute so a manual toggle wins in both directions. Buttons came in two flavours: a solid primary and a bordered secondary. .btn { background : var ( --accent ); color : var ( --panel ); } .btn.sec { background : transparent ; color : var ( --ink ); } In dark mode the accent goes light green, so white-on-accent stops working. I patched it the way you patch things at 1am: :root [ data-theme = dark ] .btn { color : #10241b } @media ( prefers-color-scheme : dark ) { :root:not ([ data-theme = light ]) .btn { color : #10241b } } Now count the specificity. Selector Specificity .btn.sec 0,2,0 :root[data-theme=dark] .btn 0,3,0 :root:not([data-theme=light]) .btn 0,3,0 :not() doesn't add specificity of its own, but its argument does. So :root (0,1,0) + [data-theme=light] (0,1,0) + .btn (0,1,0) lands at 0,3,0. My theme patch outranks the component modifier. In dark mode, every secondary button — transparent background, sitting on a #1a1c1f panel — got painted #10241b . Dark green on near-black. 1.04:1. The nasty part is that this class of bug is invisible in review. The rule looks correct. It is correct, for the buttons it was written for. It just also matched buttons it was never meant to touch, in one theme only. Why the scan didn't catch it axe-core evaluates the DOM as currently rendered . It reads computed styles, and computed styles resolve exactly one colour scheme: whichever one the browser is in right now. So npx axe https://example.com is not "does this page pass contrast." It's "does this page pass contrast in the scheme this headless browser happened to boo

Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it. The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production. Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury. This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?" The Problem With Ship-by-Feeling Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user. The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better

When building Model Context Protocol (MCP) servers or AI agents that consume them, traditional API testing tools fall short. An MCP server isn't just a basic REST endpoint—it's a dynamic interface exposed to non-deterministic LLMs through stdio, HTTP, or SSE transports. Testing tool schemas, transient network failures, and agent behaviors usually requires writing a mountain of boilerplate. I built bubblemcp-test-kit to eliminate that friction: no backend accounts, no complex test setup, and zero instrumentation required. What is bubblemcp-test-kit? bubblemcp-test-kit is a lightweight, standalone testing toolkit designed specifically for MCP server developers and AI agent engineers. Key features include: Transport Agnostic: Work with stdio, HTTP, or SSE behind a unified API. Fluent Assertions: Native matchers tailored for MCP response structures and JSON Schemas. Mocking & Replay: Fabricate tool outputs locally or record real server runs to replay in offline CI environments. Agent Trace & Fault Injection: Test if your AI agent calls tools in the right order and handles errors properly. Quickstart Example You can run a complete mock test suite without spinning up a live server: import { createMockMcpClient , expectMcp , validateAgainstSchema , withRecording , createReplayClient , } from ' bubblemcp-test-kit ' // 1. Define your tool contract const healthCheckTool = { name : ' health_check ' , description : ' Reports service health ' , inputSchema : { type : ' object ' , properties : { service : { type : ' string ' } }, required : [ ' service ' ], }, outputSchema : { type : ' object ' , properties : { service : { type : ' string ' }, status : { type : ' string ' , enum : [ ' ok ' , ' degraded ' , ' down ' ] }, latencyMs : { type : ' number ' }, }, required : [ ' service ' , ' status ' , ' latencyMs ' ], }, } // 2. Set up a mock MCP client const mock = createMockMcpClient ({ tools : [ healthCheckTool ] }) mock . mockTool ( ' health_check ' ). resolves ({ service : ' weat

Originally published on hexisteme notes . A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap. That note answered three cases I already knew about, because I'd already tripped over them. It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet? This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken. The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: SELECT COUNT ( * ) AS total , SUM ( some_column IS NOT NULL ) AS filled FROM some_table ; If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it. Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing. Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword. In one afternoon it produced both a false positive and a false negative. The sharper miss: a literal grep for a write path failed to find an INSERT OR REPLACE statement that was, in fact, live and doing exactly the writing I was looking for. Grep matched the shape of the bug I expected walking in, not the shape the code actually had. Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled. The question that works is: of all the rows that exist, how many have this column fi