标签：#test

يقوم Claude الآن بإرفاق بيانات تعريف العزو (provenance metadata) المشفّرة وفق C2PA بالملفات التي ينشئها. وينطبق الأمر نفسه على نماذج الصور من OpenAI و Gemini . هذا يعني أن إشارة العزو تصل سليمة إلى نقطة التحميل لأول مرة، لكن سلسلة معالجة الصور لديك قد تحذفها قبل أن يراها أي شخص. جرّب Apidog اليوم لا يحدث ذلك بنية سيئة؛ بل يحدث افتراضيًا. فمثلًا، تنشئ sharp().resize() ملفًا جديدًا بلا بيانات تعريف ما لم تطلب الاحتفاظ بها صراحةً. وينطبق ذلك أيضًا على ImageMagick وPillow ومعظم شبكات CDN الخاصة بالصور. يدخل الملف، ويخرج JPEG أصغر، ولا تخبرك السجلات أن بيانات العزو اختفت. هذه مشكلة قابلة للاختبار. ستتعلم هنا كيف تحدد المرحلة التي تحذف بيانات C2PA، وتثبت ذلك عبر رحلة رفع وتنزيل حقيقية، وتضيف فحصًا في CI يمنع عودة المشكلة. يتولى Apidog تنسيق سيناريو الـ API، بينما يتولى c2patool التحقق من صحة البيانات على مستوى البايت. ما الذي يتم تدميره بالفعل؟ بيان C2PA هو كتلة موقعة تشفيريًا ومضمّنة داخل حاوية الملف. يسجل من وقّع الأصل وما الذي ادعاه عنه. وبما أنه موقّع، فإن تغيير البايتات دون إعادة التوقيع يكسر التوقيع بطريقة يستطيع أي مدقق اكتشافها. النقطة المهمة هنا هي حاوية الملف : عندما تعيد كتابة الحاوية، قد يختفي البيان. العملية هل يبقى البيان افتراضيًا؟ نسخ أو نقل بايت ببايت نعم sharp().resize().toBuffer() لا ImageMagick عبر convert أو magick لا Pillow عبر Image.save() لا تحويل PNG إلى WebP أو JPEG إلى AVIF لا التحسين التلقائي في CDN للصور غالبًا لا لقطة شاشة لا إعادة الحفظ من محرر صور لا رفع إلى S3 دون تحويل نعم كل عنصر في عمود لا هو إجراء شائع في تطبيقات الويب: إنشاء صور مصغرة، توليد صور متجاوبة، التفاوض على التنسيق، أو إزالة EXIF لأسباب الخصوصية. كل خطوة منطقية بمفردها، لكنها قد تنهي سلسلة العزو بصمت. انتبه أيضًا إلى أن استخدام -strip لإزالة بيانات EXIF قد يكون مقصودًا، لأن EXIF قد يحمل إحداثيات GPS أو أرقامًا تسلسلية للكاميرا. لكن إزالة جميع البيانات الوصفية للتخلص من بيانات الموقع تزيل بيان C2PA كذلك. الحل هو إزالة البيانات الحساسة بشكل انتقائي، لا حذف الكتلة كاملة. أثبت المشكلة في دقيقتين قبل تغيير خط الأنابيب، تحقق من وجود المشكلة فعلًا. تحتاج إلى ملف واحد يحتوي على بيان

The worst test suite I ever inherited had 400 tests, and I trusted about six of them. The rest were either testing implementation details nobody cared about, duplicating each other, or so tightly coupled to internal function names that a harmless refactor broke thirty tests for no real reason. Reading that codebase taught me more about what not to do than any greenfield project ever has. So when you're starting from zero, the goal isn't "write a lot of tests fast." It's building a suite you'll still trust a year from now. If you're new to this, getting the software testing basics right early matters more than covering everything - learning how to build a first test suite from scratch teaches you what to prioritize in a way that inheriting someone else's bloated suite never will. Here's roughly how I'd approach it. Start with what would actually hurt if it broke Before writing a single test, list the handful of things that would be genuinely bad if they silently broke - checkout completing, auth working, the core thing your product does actually happening. Not every function, not every branch. Just the stuff where a silent failure costs you money, users, or trust. This list is usually shorter than people expect. Five to ten flows for most early-stage products. That's your actual test suite's job in the first few months, not "100% coverage." Unit tests for logic, not for plumbing Unit tests are for things with actual decision-making in them - pricing calculations, validation rules, state transitions, anything where "given this input, is the output correct" is a real question with a wrong answer possible. They're fast, they're cheap, and they should make up the bulk of your suite. Skip unit-testing pure plumbing: a function that just calls another function and returns its result doesn't need its own test. That's the kind of test that pads a coverage number without catching anything real, and it's exactly the kind of test that made that 400-test suite so hard to trust.

If you're testing an iOS build with real devices, you've got two main paths: Apple's TestFlight, or Expo's EAS Preview using Ad-Hoc provisioning. They solve the same problem — getting a build onto a real iPhone without the App Store — but the workflows are genuinely different, not just cosmetically. How each one works TestFlight uses Apple's official infrastructure. You upload your build to App Store Connect (often via npx testflight to speed this up), Apple processes/reviews it, and testers install the TestFlight app and accept an email or public link invite. No UDID collection needed — Apple handles device registration behind the scenes. Expo EAS Preview (Ad-Hoc) uses Ad-Hoc provisioning. You register each tester's device UDID against your Apple Developer account before building — either manually (eas device:create, eas device:list) or by having the tester scan a QR code that installs a temporary profile. Once devices are tied to your provisioning profile, you build with: bash eas build --platform ios --profile preview This generates a direct install link/QR code — no App Store account or TestFlight app required. Comparison table Feature Expo Preview / Ad-Hoc Apple TestFlight Device limit ~100 devices/device class/year (Apple Developer account tier) Up to 10,000 external testers Processing time Immediate after cloud build finishes Apple review/processing (mins to hours) UDID management Manual or profile-based registration required Not required, handled by Apple Best for Fast internal testing, client demos, strict ad-hoc distribution Larger-scale beta testing, staging before production Which one should you use? Fast internal iteration, client demos, small teams → Ad-Hoc. No waiting on Apple, instant install links. Wider beta testing before a production release → TestFlight. Built-in scale, no manual device management. Most teams I've worked with end up using both at different stages: Ad-Hoc during active development for quick feedback loops, TestFlight once the bui

Microsoft today released updates to remedy at least 398 security vulnerabilities in its Windows operating systems and supported software, including one weakness that is already being actively exploited and two others that were publicly detailed prior to today.

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet. That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo. The eval set: eval/cases.json 21 cases, three buckets: Bucket Count Covers Easy 12 Shipping-status lookups, simple KB questions, a cancelled-order info request, one no-KB-match case that must escalate rather than fabricate Hard 6 Refund eligibility inside/outside the 30-day window, multi-item orders where only one item is refunded, boundary cases just past the window Edge 3 Legal-threat, fraud-flag, and duplicate-ticket patterns — must auto-escalate with zero tool/LLM calls A sample case, checking both the outcome and the trajectory that produced it: { "case_id" : "hard_03" , "bucket" : "hard" , "ticket" : { "ticket_id" : "hard_03" , "subject" : "Wrong size shoes, keep the socks" , "body" : "The running shoes from order ord_1004 are the wrong size. I want a refund for just the shoes, not the socks." , "customer_id" : "cust_002" , "order_id" : "ord_1004" }, "expected_trajectory" : [ "order_lookup" , "refund_eligibility" , "issue_refund" ], "expected_outcome" : "refund_proposed" , "expected_max_steps" : 4 , "policy_checks" : [ "refund amount reflects only the shoe item (~$74), not the full order total" , "issue_refund gated behind human approval" ] } Three things being checked per case, not just "did the answer loo

A free AI visibility score is auditable only when you can inspect the prompt, engine, raw answer, date, and denominator. Treat the score as a test result, not a property of your brand. This tutorial builds a six-check control you can run by hand, store as plain data, and compare with any tool's output. The workflow takes three buyer questions, runs them in two AI surfaces, and records the six answers without trying to force agreement. It will not estimate your entire market. It will tell you whether a dashboard's headline number has enough evidence to be investigated. What does an AI visibility score measure? An AI visibility score usually summarizes brand presence across a defined set of generated answers. That definition contains the trap: the question set is part of the metric. So are the engine panel, run date, session state, retrieval mode, and rule used to count a “hit.” Remove those inputs and the number is not reproducible. Imagine a tool asks three questions in two engines. That creates six cells. If your brand appears in two cells, the simple presence result is: presence = brand_present_cells / total_cells presence = 2 / 6 presence = 0.333... = 33.3% The arithmetic is trivial. The evidence is not. A different tool can ask five different questions in three engines and produce a different score without contradicting the first run. The two tools measured different grids. Keep the unit explicit: “present in two of six generated answers on this date” is defensible. “Our AI visibility is 33” is incomplete. Which evidence fields should you require? Require five fields for every result: prompt, engine, raw answer, timestamp, and counting rule. Use a sixth field for cited sources when the surface exposes them. A source-only appearance and a prose mention can signal different problems, so do not merge them silently. Here is one real saved result from Webappski's public 14 June 2026 tracker report: { "run_date" : "2026-06-14" , "prompt" : "beste Answer Engine Optimiz

You can load-test search latency with a script and a graph. Relevance has no such gauge by default, so most teams ship a new ranking rule, eyeball a handful of queries, and hope nothing important regressed. The fix is a small, boring relevance test suite: a fixed set of queries, human-judged expected results, and a metric you compute the same way every time — so "did this ranking change help?" becomes a number you can diff, not an argument you have in Slack. This post is a build guide. By the end you'll have a judgments file, a scorer that outputs precision@k, MRR, and nDCG, and a before/after comparison you can wire into CI. The examples use Postgres full-text search, but the harness is engine-agnostic — Elasticsearch, Meilisearch, or a vector store all slot into the same shape. Why can't I just load-test relevance the way I load-test latency? Latency is a property of the system. Relevance is a property of the match between a query and what a human expected to see — and that judgment lives outside the database. A commenter on an earlier post about running Postgres search in production put it well: latency can be load-tested, but quality needs query sets, expected result buckets, bad-query examples, and a way to compare changes before shipping a new ranking rule. That's the whole job, and none of it comes for free with your index. The trap is thinking a passing query proves relevance. SELECT ... WHERE tsv @@ query returning rows tells you the index matched. It says nothing about whether the right rows landed in the top 5, which is all a user ever sees. The takeaway: relevance is measured against human judgments, not row counts — so the first artifact you build is the judgments, not the query. Building the golden query set Start with 20–50 real queries. Pull them from your search logs if you have them (the head terms plus a long tail of specific ones), or write them from real user intents if you don't. For each query, mark which documents should come back and how rel

Here is every promise a to-do list makes. VARIABLE tasks Init == tasks = [i \in Ids |-> "absent"] Add(i) == tasks[i] = "absent" /\ tasks' = [tasks EXCEPT ![i] = "open"] Complete(i) == tasks[i] = "open" /\ tasks' = [tasks EXCEPT ![i] = "done"] Reopen(i) == tasks[i] = "done" /\ tasks' = [tasks EXCEPT ![i] = "open"] Delete(i) == tasks[i] # "absent" /\ tasks' = [tasks EXCEPT ![i] = "absent"] ClearCompleted == /\ \E i \in Ids : tasks[i] = "done" /\ tasks' = [i \in Ids |-> IF tasks[i] = "done" THEN "absent" ELSE tasks[i]] Not a summary. Not the important ones. All of them. A task cannot go from absent straight to done. Clearing completed items leaves the open ones alone. You cannot delete something that was never there. Nine lines, and when you've read them you have read the entire contract. Now go find that list for the system you work on. You can't. It doesn't exist. It's distributed across a test suite that asserts outcomes rather than rules, some validation scattered through handlers, and the memory of whoever's been there longest. The guarantees are real — your users depend on every one of them — and there is no file you can open to see them. That's the gap I want to talk about, because you can close it in an afternoon, and because something has changed recently that makes closing it pay for itself. The prime mark and two operators That's most of the syntax, so let's get it out of the way. tasks' means "tasks, in the next state." /\ is and . \E is "there exists." A definition like Complete(i) is a formula relating the current state to the next one — read it out loud: the task is open, and afterwards it is done. That's it. That's the language, near enough, for this purpose. The real file adds about eight lines of scaffolding around what you saw: a module header, a TypeOK saying a task is always in exactly one of the three states, and the two lines that tie the actions together — Next == \/ \E i \in Ids : Add(i) \/ Complete(i) \/ Reopen(i) \/ Delete(i) \/ ClearComplete

Here's a bug your test suite probably wouldn't catch. A backend developer refactors the user model. The id field — an integer since forever — starts coming back as a string: "42" instead of 42 . Every value is still "correct". Your assertion pm.expect(user.id).to.eql(42) fails, sure — but only on the one endpoint you asserted id on, not the other nine that return users. Meanwhile three client apps that did user.id + 1 are now computing "421" . That's structural drift , and it's what actually breaks API consumers: renamed fields, changed types, properties that quietly vanish. Field-by-field value assertions catch it patchily and by accident. Schema validation catches it systematically — and in Postman it costs about ten lines, because the ajv JSON-schema validator is built into the script sandbox. The ten lines In Scripts → Post-response on any request that returns a user: const userSchema = { type : " object " , required : [ " id " , " name " , " email " ], properties : { id : { type : " integer " }, name : { type : " string " }, email : { type : " string " , pattern : " @ " } } }; pm . test ( " Response matches the user schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userSchema ); }); That single test now fails if id becomes a string, if email disappears, if name becomes an object — every structural mutation, whether or not you thought to assert on that field's value. For an endpoint returning an array of users: const userListSchema = { type : " array " , minItems : 1 , items : userSchema // reuse the object schema }; pm . test ( " List matches schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userListSchema ); }); Share one schema across every endpoint The real power move: your API returns users from /users , /users/:id , /login , /teams/:id/members … and they should all be the same shape . Store the schema once as a collection variable (JSON, stringified), and every request validates against the sa

I have been building a thing that lets a language model propose an UPDATE , then executes it for real inside a transaction, measures the actual before and after values, and always rolls back. A human reads the measurement and decides. Only then does anything commit. The pitch is one sentence: what you approve is not the model's description of its SQL, it is what the database did when the SQL ran. Last week I found that the thing showing you that measurement was showing you a subset of it, and had been since the first release. The failure Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row: name = 'Tanaka' , postcode = '00100' . UPDATE customers SET name='Sato', postcode='00100' WHERE id=1 What this touches customers — Customer records. The postcode is used for billing address and delivery. 1 row would change, across 1 column: name Measured by running the statement and rolling it back id = 1 name: 'Tanaka' -> 'Sato' One row, one column. postcode is not mentioned, and that is correct — it is being assigned the value it already holds, so nothing about it changes. The card is describing the diff accurately. Approve it. Then, before it is applied, somebody else notices the postcode is wrong and fixes it: UPDATE customers SET postcode = '90210' WHERE id = 1 ; Now apply the approved plan: Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z. DB now: [{"name":"Sato","postcode":"00100"}] The fix is gone. Zero warnings. The word postcode never appeared on the approval card, never appeared in the audit record, and never appeared in the comparison the tool makes before it commits. One variable doing two jobs The diff was built like this: const changed : string [] = []; for ( const c of Object . keys ( before )) { if ( same ( before [ c ], after [ c ])) continue ; // drop what did not move if ( auto . has ( lower ( c ))) continue ; // drop what the DB maintains itself changed . push ( c ); } That is a correct answer to "what should the card sho

Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call? A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals: stop accepting new work; let safe work finish or reach a checkpoint; make ambiguous work visible instead of guessing; resume with an explicit recovery decision. 1. Model maintenance as a state transition Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine: RUNNING -> DRAINING -> QUIESCED -> STOPPED | +-> NEEDS_REVIEW DRAINING rejects new jobs but allows an active job to continue until its next checkpoint or deadline. QUIESCED means there are no unclassified side effects in flight. NEEDS_REVIEW is the safe outcome when the process died after sending a request but before recording the response. Persist the transition, not just an in-memory flag. A minimal record can look like this: { "runtime" : "agent-7" , "maintenance_id" : "mw-2026-08-10-001" , "state" : "DRAINING" , "started_at" : "2026-08-10T08:00:00Z" , "accepting_work" : false , "active_runs" : 2 } If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED . That is much more useful than inferring health from a missing PID. 2. Put checkpoints around side effects An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary: PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED On restart: PLANNED can be

I shipped a page that reported zero axe violations . It had button text at a contrast ratio of 1.04:1 — which is, for practical purposes, invisible text. The scan wasn't broken. It was answering a narrower question than I thought I was asking. The bug I had a theme system built the ordinary way. Tokens on :root , overridden in a prefers-color-scheme media query, and overridden again by an explicit [data-theme] attribute so a manual toggle wins in both directions. Buttons came in two flavours: a solid primary and a bordered secondary. .btn { background : var ( --accent ); color : var ( --panel ); } .btn.sec { background : transparent ; color : var ( --ink ); } In dark mode the accent goes light green, so white-on-accent stops working. I patched it the way you patch things at 1am: :root [ data-theme = dark ] .btn { color : #10241b } @media ( prefers-color-scheme : dark ) { :root:not ([ data-theme = light ]) .btn { color : #10241b } } Now count the specificity. Selector Specificity .btn.sec 0,2,0 :root[data-theme=dark] .btn 0,3,0 :root:not([data-theme=light]) .btn 0,3,0 :not() doesn't add specificity of its own, but its argument does. So :root (0,1,0) + [data-theme=light] (0,1,0) + .btn (0,1,0) lands at 0,3,0. My theme patch outranks the component modifier. In dark mode, every secondary button — transparent background, sitting on a #1a1c1f panel — got painted #10241b . Dark green on near-black. 1.04:1. The nasty part is that this class of bug is invisible in review. The rule looks correct. It is correct, for the buttons it was written for. It just also matched buttons it was never meant to touch, in one theme only. Why the scan didn't catch it axe-core evaluates the DOM as currently rendered . It reads computed styles, and computed styles resolve exactly one colour scheme: whichever one the browser is in right now. So npx axe https://example.com is not "does this page pass contrast." It's "does this page pass contrast in the scheme this headless browser happened to boo

Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it. The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production. Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury. This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?" The Problem With Ship-by-Feeling Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user. The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better

When building Model Context Protocol (MCP) servers or AI agents that consume them, traditional API testing tools fall short. An MCP server isn't just a basic REST endpoint—it's a dynamic interface exposed to non-deterministic LLMs through stdio, HTTP, or SSE transports. Testing tool schemas, transient network failures, and agent behaviors usually requires writing a mountain of boilerplate. I built bubblemcp-test-kit to eliminate that friction: no backend accounts, no complex test setup, and zero instrumentation required. What is bubblemcp-test-kit? bubblemcp-test-kit is a lightweight, standalone testing toolkit designed specifically for MCP server developers and AI agent engineers. Key features include: Transport Agnostic: Work with stdio, HTTP, or SSE behind a unified API. Fluent Assertions: Native matchers tailored for MCP response structures and JSON Schemas. Mocking & Replay: Fabricate tool outputs locally or record real server runs to replay in offline CI environments. Agent Trace & Fault Injection: Test if your AI agent calls tools in the right order and handles errors properly. Quickstart Example You can run a complete mock test suite without spinning up a live server: import { createMockMcpClient , expectMcp , validateAgainstSchema , withRecording , createReplayClient , } from ' bubblemcp-test-kit ' // 1. Define your tool contract const healthCheckTool = { name : ' health_check ' , description : ' Reports service health ' , inputSchema : { type : ' object ' , properties : { service : { type : ' string ' } }, required : [ ' service ' ], }, outputSchema : { type : ' object ' , properties : { service : { type : ' string ' }, status : { type : ' string ' , enum : [ ' ok ' , ' degraded ' , ' down ' ] }, latencyMs : { type : ' number ' }, }, required : [ ' service ' , ' status ' , ' latencyMs ' ], }, } // 2. Set up a mock MCP client const mock = createMockMcpClient ({ tools : [ healthCheckTool ] }) mock . mockTool ( ' health_check ' ). resolves ({ service : ' weat

Originally published on hexisteme notes . A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap. That note answered three cases I already knew about, because I'd already tripped over them. It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet? This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken. The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: SELECT COUNT ( * ) AS total , SUM ( some_column IS NOT NULL ) AS filled FROM some_table ; If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it. Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing. Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword. In one afternoon it produced both a false positive and a false negative. The sharper miss: a literal grep for a write path failed to find an INSERT OR REPLACE statement that was, in fact, live and doing exactly the writing I was looking for. Grep matched the shape of the bug I expected walking in, not the shape the code actually had. Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled. The question that works is: of all the rows that exist, how many have this column fi

A Practical Guide to Faster, Deterministic Flutter & Dart Unit Testing If you’ve ever written unit tests for classic package:bloc applications using bloc_test , you know the drill: build your BLoC, dispatch an event in act , and assert state emissions in expect . Under the hood, classic BLoC processes state updates asynchronously via Dart microtask-queue Streams . While robust, testing asynchronous streams can introduce microtask timing headaches, race conditions, or the need to drain queues or use fakeAsync when testing complex side-effects. In BlocSignal , state updates propagate synchronously . Calling emit(newState) updates the underlying signal graph in the exact same call stack frame. This handbook is a practical, recipe-based guide to testing BlocSignal and CubitSignal applications using package:bloc_signals_test . Whether you’re coming from classic BLoC or brand new to Signals, this guide shows you how to test every scenario cleanly—and why it’s significantly easier than classic stream-based testing. 🤖 AI Assistant Tip : Working with an AI coding assistant (like Antigravity, Gemini CLI, or Cursor)? The official bloc-signals plugin includes a pre-built testing skill ( plugins/bloc-signals/skills/bloc-signals/ ) that automatically teaches your AI assistant these exact testing conventions, observer scoping rules, and declarative blocSignalTest patterns! 🛠️ Quick Reference: BLoC Streams vs. BlocSignal Testing Testing Task Classic BLoC ( package:bloc_test ) BlocSignal ( package:bloc_signals_test ) Why it’s easier in BlocSignal Execution Environment Often requires flutter test engine Pure dart test execution Blazing Speed : Business logic tests run in pure Dart CLI without booting Flutter UI engine. Simple State Assertions Requires async stream listener or blocTest Direct expect(cubit.state, 1) or blocSignalTest Synchronous : State updates on the next line of code without microtask delay. Failure Diagnostics Legacy Instance of 'CounterCubit' Built-in toString() :

You're Not Comparing Models. You're Comparing Contracts. Two teams publish scores on the same agent benchmark. One lands in the low sixties. The other clears seventy. A procurement team reads the spread and makes a call. What they do not see: both teams may be running the same model. They did not need to change the weights for the gap to appear. The spread can come from scaffold alone. One team wrapped the model in a harness with better retries. Different tool defaults. A planner step the other team had skipped. None of that appears on the leaderboard. The comparison that drove the decision was not between two agents. It was between two contracts. There Is No Benchmark The mistake hiding behind this story is a category error. People talk about agent benchmarks as if they measure a thing called “the model.” They do not. They measure a coupled system. The model is one component. The rest is a stack of protocol decisions that are almost never disclosed and almost always matter. The score is the output of that stack. Change any layer and you change what the number means. Recent research on agent evaluation has named those layers explicitly. There are at least seven. Deployment regime. Observation channel. Harness and scaffold. Metric and action. Configured evaluator. Grader protocol. Audit bundle. Each is a contract. Each is negotiable. And each can silently change the verdict while the headline looks the same. That is what a benchmark actually is. Not a measurement of a model. A measurement of an entire testing contract, of which the model is one slot. There is structural reason the seven layers are the seven layers. They cluster into three corners that show up in almost every published agent-evaluation failure. What the model is rewarded for. How that reward is optimised. And how the test contract differs from production. Once you hold those three corners in view, the seven-layer stack stops feeling like a checklist and starts behaving like the actual shape of what is

This article is about a poker-analysis framework, but the engineering problem is common to LLM tool use. The framework uses an LLM as an input and control layer. It reads a natural-language poker question, chooses a local calculator, and proposes typed fields. A Python program, not the LLM, performs the numerical calculation and returns a structured result with verification data. In this evaluation, a coordinator manually passed each calculator-eligible saved proposal to the command-line calculator; no automatic runtime bridge connected them. The design intent was to reduce manual arithmetic checking by sending numerical claims to deterministic, internally verified local software. The evaluation below tests whether those claims were checked and whether the handoff remained auditable. It did not measure time saved or the overall quality of the resulting poker analysis. The calculator catalog is poker-specific. This is not a poker strategy guide, and you do not need to know poker strategy to follow the failure. The question is whether a correct calculator can produce a verified result after the LLM chooses an interpretation without asking the user to confirm it. The workflow was tested with 25 hand-authored cases that were fixed before execution. They are labeled C01 through C25: C01–C23 tested the LLM's routing, proposed input, and boundary decisions; C24 and C25 repeated two accepted inputs to check non-volatile result semantics. The labels are test numbers, not poker terminology. The main example uses one small pot-odds model. In this model, the pot is the shared pool of chips the players are competing for. The calculator's inputs are: pot_before_bet : the amount already in the pot before the opponent's new bet; opponent_bet : the amount the opponent adds; call_cost : the amount the player must add to continue; expected_rake : an optional amount removed from the final pot. The calculation is: net final pot = pot_before_bet + opponent_bet + call_cost - expected_rake

You just installed Cursor, opened a TypeScript file, and asked the agent to fix a bug. Ten seconds later it handed you a type SomeType = any and a @ts-ignore above the line that wouldn't compile. This is the moment most developers discover that AI coding agents are powerful but undisciplined. The fix isn't a better model. It's rules. Most AI coding agent best practices boil down to a single idea: tell the agent what good looks like before it starts typing. Cursor lets you define rules files in .cursor/rules/ that load alongside your project context and tell the agent how to behave. Claude Code has its own rules system, Windsurf has a rules directory, Copilot reads .github/copilot-instructions.md . Learn to configure cursor rules properly and your agent starts behaving like a careful senior engineer instead of an eager intern. What cursor rules files are Cursor rules are markdown files with a .mdc extension stored in .cursor/rules/ at your project root. Each file is a set of instructions the agent reads before it starts working. When a rule's conditions match the file being edited, the instruction is injected into the model's context window. A cursor rules file has two parts: a YAML frontmatter block between --- markers, and a markdown body with the actual instructions. How to configure cursor rules: the frontmatter fields Three fields matter. description (required). A short summary of what the rule enforces. Cursor surfaces this when you toggle rules, so make it specific. globs (optional). File patterns the rule applies to. Without globs, the rule applies to everything, which wastes context and creates conflicts. alwaysApply (optional). Set to true for rules that should load in every session, regardless of the files involved. Leave it false for rules that only trigger when matching files are touched. Real example: --- description : Enforce strict TypeScript, no any, no ts-ignore globs : ** /*.{ts,tsx} alwaysApply : false --- # Strict TypeScript ## Context This codeb

Every AI coding agent I've used shares one habit: it writes the plausible thing. The code compiles. The tests pass. And a senior engineer reviewing it would reach for a red pen. any where a union belongs. Tests that assert on implementation so they survive any refactor. catch (e) {} blocks that quietly swallow production errors. The fix isn't a better model or a cleverer prompt. It's a set of rules at the repo level, written in a format the agent is guaranteed to read. What rules files are Cursor reads .cursor/rules/*.mdc . Claude Code reads CLAUDE.md and AGENTS.md . The .mdc format is plain markdown with YAML frontmatter. Here's the opening of the TypeScript rule file from the AgentForge sample pack: --- description: "Strict TypeScript discipline for production code" globs: "**/*.{ts,tsx}" alwaysApply: true --- Three fields carry the weight. description tells the agent in one sentence what the file is for. globs scopes it, so a TypeScript rule never fires on a Python file. alwaysApply: true loads it into every session. Agents skip a 2,000-line rules file. They read a 40-line one. Write a discipline contract, not a wish list Claude Code follows instructions frighteningly well, and that cuts both ways. Tell it "write good code" and it will be confidently, grammatically wrong. An AGENTS.md contract fixes that. Start with a baseline of rules: think before you act, take small verifiable steps, never claim what you haven't verified, no drive-by refactoring. Then add a verification ladder with six rungs. Does it compile? Does the changed behavior work? Does it break anything adjacent? Does it follow the codebase's conventions? Does it hold at the boundaries? Is it observable in production? The first three are mandatory for every change. Then the failure protocol, which is the line that pays for itself: First failure: fix and re-verify. Second failure: re-derive, your mental model is wrong, form at least two new hypotheses. Third failure: stop, revert to last known-good, d