One Passing Agent Run Is Not a Release Signal

I can make almost any agent change look good with one carefully chosen prompt. Run it once. Watch the right tool fire. Read a polished answer. Record the demo. Ship it? Not yet. A single trace is excellent evidence for one execution: what ran, what failed, how long it took, and whether declared invariants held. A release decision asks a different question: Does this change behave acceptably across the representative cases we care about? That shift—from run to set —is where agent testing starts to resemble engineering instead of demonstration. Give each evidence layer one job In agent-inspect@6.17.6 , I use three local, deterministic layers: Layer Question Suite Did each named case satisfy its expected checks? Cohort What moved between baseline and candidate groups? Gate Should CI accept the recorded evidence? representative traces | v suite --------> named case results | v baseline vs candidate -> cohort deltas | v gate --------> CI exit code + evidence These commands read persisted traces. They do not rerun the agent, invoke a model, or decide whether prose is semantically good. Start with named cases, not random traffic A compact suite makes the review set explicit: { "name" : "refund-release" , "traces" : "./.agent-inspect" , "cases" : [ { "id" : "eligible-order" , "runId" : "refund-eligible" , "requireTools" : [ "lookup_order" , "refund_order" ] }, { "id" : "unknown-order" , "runId" : "refund-unknown" , "forbidTools" : [ "refund_order" ] }, { "id" : "approval-required" , "runId" : "refund-approval" , "requireTools" : [ "request_approval" ] } ] } Validate the configuration and run it: npx agent-inspect suite validate \ --config agent-inspect.suite.json npx agent-inspect suite run \ --config agent-inspect.suite.json \ --markdown The suite distinguishes a violated expectation from missing evidence. If refund-unknown does not exist, the case is skipped with a diagnostic; it is not silently treated as a pass. An all-skipped suite cannot become a green release signal.