标签：#Testing

I once collected about 22,000 comments from roughly 140 Korean YouTube videos about AI coding tools and classified them. (Quotes below are translated from Korean.) I wanted to see what people were asking. What came out was something else. What the videos teach Put the titles and tags of those 140 videos in one pile and they say: How to install. How to get started. How to build an app. Which tool is best. All of it is "starting." Follow along, a result appears on the screen, the video ends. What the comments say The comments sweeping up the likes were telling a different story. "Verifying AI mistakes takes so much time. Checking every answer for nonsense got so tiring I just do the work myself now." (👍598) "Coding with AI makes me anxious. If one bug ships, I'm the one responsible. Checking and debugging everything one by one ends up being more work." (👍265) "I pay every month and it lies about work matters like it's nothing." (👍72) "Tokens burn too fast… added $50 and it was gone in half a day." (👍30) It compresses into three complaints: expensive, can't trust it, can't fix it. The videos teach the start. The people are dying right after the start. The scariest comment "Asked it for shampoo recommendations and it recommended one that doesn't exist. Slipped it in between real products — with the weight, the benefits, even a price." (👍49) That comment is the essence of the problem. When AI is wrong, it doesn't look wrong. The fake sits among the real ones, wearing plausible numbers. This is why "just write better prompts" is half an answer. Better prompts lower the odds of being wrong. They don't create a way to know when it's wrong. Drop the error rate from 10% to 3% and you still don't know where the 3% is hiding. If that 3% detonates inside payment logic, money leaves the building. One more finding — where the real questions live While collecting, I noticed the nature of comments changes with channel size. multi-million-sub videos real questions/needs = 12% of comm

A new arXiv paper watched coding agents get denied the facts they needed. They did not stop. They invented. On August 17th, a group of researchers posted a paper to arXiv with an unglamorous title and a genuinely unsettling core finding. The paper is "The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks" (arXiv:2608.16630), by Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, and Laurent Bindschaedler. Before going further, one honesty note that will hold for this whole piece: I have read the paper's abstract, not its full text, and every quotation below comes from that abstract. It is enough for what this essay is about, because what this essay is about is one sentence. The setup first. The authors model repository-scale coding as reconstructing a web of coupled facts. Every edit an agent makes needs certain facts, and each fact arrives through one of two channels: it is either in the recent context, or it is in the model's memorized knowledge. Facts covered by neither channel are what the authors call coherence debt. Their experiment supplies and withholds each channel deliberately, injecting faults across "seven models and five harnesses" (abstract), and then watches what the agents do when a needed fact simply is not there. The comfortable prediction is that a competent agent, denied a fact, stops and says so. Here is what the authors report instead: "A missing fact produces wrong work rather than absent work" (abstract). The agent asked to act, acts. In the paper's words, "an agent asked to act acts, fabricating the file or guessing the value" (abstract). That much is alarming in a familiar way. Everyone who works with these systems has a story about an invented function or a guessed constant. The abstract has sharper findings than the fabrication itself, though. When the researchers renamed a real library to defeat memorized knowledge, the failure was collective: "all seven fail in the same place, passing and missing the same tests

I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) ....

Originally published on hexisteme notes . A user complaint sent me into a caption pipeline: "the subtitles cut to two words in places where the sentence doesn't make sense." The fix I shipped for that complaint introduced a second bug, one word narrower and easy to miss, because the code that measured whether a line of text would fit reproduced an assumption about the text that the code drawing the line didn't share. Every test passed the whole time. I only found it by watching the rendered video. The bug the complaint pointed at The captioning system splits a transcript into short chunks that pop onto screen a few words at a time. The chunking function was doing fixed-size slicing — take the next N words, regardless of what came before or after. That's blind to sentence boundaries, so two unrelated sentences could land in the same chunk: loss. Today reads as one visual unit even though it's the tail of one sentence and the head of the next. The fix was a rule set, not a single tweak: hard break after terminal punctuation ( . ! ? … ) soft break at commas, semicolons, and em-dashes extend or push a chunk rather than let it end on a function word ( of , the , than , is , and about thirty others) target three words per chunk, four as a ceiling a pixel-width cap on the rendered chunk, measured against the actual caption font (Montserrat ExtraBold), with a budget of 1080 × 0.92 = 993.6px The first four rules are about where a line is allowed to break. The fifth is a physical constraint: however good the break points are, a chunk still has to fit on screen at the font size actually in use. That's the one that went wrong. What the width guard actually measured To get the pixel width of a candidate chunk, the guard rendered the chunk's text through the font and measured the result — which is the correct approach in principle, not a shortcut. Text width isn't a fixed number of pixels per character; it depends on the specific glyphs, so measuring the real string through the r

Background I run a fleet of static sites that publish new content every day, mostly unattended. One of the house style rules is simple: no emoji anywhere in our own copy. That rule is impossible to hold by hand. A single site builds a few hundred HTML files, and emoji can slip into nav icons, button labels, <title> , the RSS feed, or JSON-LD (the JSON-formatted metadata embedded in a page to describe its structure to search engines). Nobody is going to review all of that before every deploy. So I wrote emoji-lint , a check that exits 1 the moment it finds a single emoji . It sits in the pre-deploy gate, which means a failure stops that day's publish. This post is not about the regex. It's about what happens when you put a failing check into real operation: you immediately discover the places where the rule must not apply. How it works The core is unremarkable. A regex holds the emoji code point ranges, the scanner walks each file line by line, and matching lines are reported as JSON. const EMOJI_RE = / [\u {1F000}- \u {1FAFF} \u {2600}- \u {27BF} \u {2B00}- \u {2BFF} \u {1F1E6}- \u {1F1FF} \u {FE0F} \u {200D} \u {2049} \u {203C} \u {2122} \u {2139} ] /u ; \u{FE0F} (variation selector) and \u{200D} (ZWJ) are in there because emoji are not always a single code point. Arrows and similar symbols used in ordinary technical writing are deliberately left out. Catch everything and the check drowns in false positives, at which point people stop reading it. The interesting part came later. Three categories of content look exactly like a violation but must not be treated as one: Verbatim quotes from other people Real proper nouns whose official spelling contains a symbol Passages where the emoji itself is the subject being explained Delete the emoji in any of those and you break something more important than the style rule. One term up front: "masking" here means replacing a range with spaces so the scanner cannot see it. Nothing is deleted from the file. Implementation Scope

This is a submission for DEV's Summer Bug Smash: Clear the Lineup , powered by Sentry . Project Overview I was working on a small Python component that performs a preflight check and then, if the check succeeds, invokes one synchronous operation callback. A counter records whether that callback invocation returned normally. The counter is used for diagnostics, so it must follow the control flow rather than the expected happy path. Bug Fix or Performance Improvement When a handled failure occurred, the old implementation still returned one: return 1 That value was hard-coded because the successful path was expected to invoke exactly one operation. If the preflight check failed, however, the operation was never entered and the function still returned one. An offline reproduction produced: operation_entries=0 old_count=1 The failure was handled, but the counter contradicted the actual control flow. Code Reduced to the relevant lines, the old behavior was: # Simplified pre-fix behavior def buggy_completed_calls ( * , preflight , operation ): try : preflight () operation () except Exception : pass return 1 Here is the complete fixed function from the standalone reproducer: from collections.abc import Callable Callback = Callable [[], None ] def completed_calls ( * , preflight : Callback , operation : Callback ) -> int : """ Return one only when the cooperative operation returned normally. """ try : preflight () operation () except Exception : return 0 return 1 The essential regression assertion is shown below. Both callbacks are local, so the test performs no network request: # Abbreviated test excerpt def test_preflight_failure_does_not_count_an_unentered_operation (): operation_entries = 0 def refuse_preflight (): raise RuntimeError ( " controlled preflight refusal " ) def operation (): nonlocal operation_entries operation_entries += 1 result = completed_calls ( preflight = refuse_preflight , operation = operation , ) assert operation_entries == 0 assert result == 0 My

In June a reviewer on DEV who goes by @anp2network told me to stop storing a conclusion. I had a...

Opinion: Your Tests Can't See What a Migration Destroys — Dry-Run It on a Clone A green test suite is the wrong tool for judging an AI-generated migration, because tests run against the post-migration schema and never observe the intermediate states where data disappears. The up migration is the visible artifact that gets reviewed, while the down migration is treated as an afterthought even though it is the only safety net when the deployment goes wrong. Free model access makes the problem structural: generation cost drops to zero, so migration volume rises, and every additional migration multiplies the surface for unreviewed data loss. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Tests validate the destination, not the journey When a test suite runs against a migrated database, it confirms that the application can read the new schema, but it cannot confirm that the migration preserved the data it was supposed to preserve. The test runner connects after the migration has executed, so it never sees the moment when a column is dropped, a table is renamed, or a constraint is silently relaxed. A migration that passes every test can still destroy production data, because the tests were designed to validate application behavior, not migration safety. The standard mitigation is a staging database, but staging is a poor substitute for a dry run because it has different data, different volume, and different usage patterns. The dry run I recommend uses a clone of the production schema with a representative data sample, and it exercises both directions of the migration with data integrity checks at every step. The clone does not need to be large; a few thousand rows per table is enough to expose most destructive patterns. The dry-run workflow in five steps The workflow is deliberately mechanical, because the goal is to remove judgment from the verification process and reserve human attention for the migration's intent: Clone the schema and lo

This is a submission for DEV's Summer Bug Smash : Smash Stories. TL;DR. I bounded a database read to make my analyzer faster. I derived the bound carefully, wrote the reasoning into the KDoc, and shipped it behind five passing tests. The bound was wrong in a way none of those tests could see. The result: if a lifter deloaded once in the middle of a stall, which is the correct thing for a lifter to do, my app stopped telling them they had plateaued. No crash. No error. No log line. The feature just quietly stopped being true for the people using the app correctly. The setup WhyRep analyzes your training rather than just recording it. The core promise is that it tells you when you have stalled and what to change about it, and that every verdict traces back to a methodology document rather than to something a language model made up. The architecture decision underneath that promise is that nothing is precomputed . Verdicts are derived from raw set logs on read, every time, so there is no cached judgement to go stale when the rules change. Which means every read walked the lifter's entire history for every exercise in the session. That is fine at ten sessions. It is not fine at three hundred. The obvious optimization is to bound the read. The obvious bound is "it only needs the last two weeks." That was my first wrong answer, and it is worth thirty seconds before I get to the interesting one. The plateau rules are not measured in calendar time. They are consecutive-miss counts, and the count varies by lifter tier and by whether the movement is a big or small joint action. The widest window in the signed methodology is an elite lifter on a small joint action: 14 consecutive sessions without progress. Train a lateral raise once a week and 14 sessions is over three months of data. A 14-day cutoff could never have fired a plateau for anyone above beginner tier. It would not have thrown. It would have quietly stopped detecting the exact thing the product exists to detect. Th

The .NET 10 JSON console logging change is small enough to miss during an upgrade: the formatted message still exists, but a typical record no longer duplicates it at State.Message . A collector, script, or snapshot test that reads only that nested property can start returning null while the application continues logging normally. I treat console JSON as a schema whenever another process parses it. That means a runtime upgrade deserves a contract test, not just a visual check in a terminal. The practical fix is to read the top-level Message , keep State for structured values, and retain a narrow fallback for older records. Why .NET 10 JSON console logging breaks nested-message parsers Before .NET 10, a normal AddJsonConsole record commonly repeated the rendered text: { "Message" : "Order 42 moved to ready." , "State" : { "Message" : "Order 42 moved to ready." , "OrderId" : 42 , "Status" : "ready" , "{OriginalFormat}" : "Order {OrderId} moved to {Status}." } } In .NET 10, the typical shape keeps one rendered message at the top level: { "Message" : "Order 42 moved to ready." , "State" : { "OrderId" : 42 , "Status" : "ready" , "{OriginalFormat}" : "Order {OrderId} moved to {Status}." } } Microsoft documents this as a behavioral breaking change and recommends that parsers use the top-level property. The official compatibility note also gives an essential caveat: State.Message may still appear when its content differs from the top-level value. I therefore do not reject a record merely because both properties exist. This is not a loss of structured logging data. OrderId , Status , and {OriginalFormat} remain useful fields inside State . The part that changed is where a consumer should get the rendered sentence. Prefer the top-level Message and keep State structured A legacy-only extractor is brittle because it assumes the duplicate is the contract: static string ? ReadLegacyOnly ( JsonElement root ) => root . TryGetProperty ( "State" , out var state ) && state . TryGetPro

Traditional test automation frameworks often carry heavy maintenance costs, slow release cycles, and high knowledge dependency. By transitioning from standard script creation to a governed AI Test Automation Factory , engineering teams can shift their focus from writing boilerplate code to high-value validation and architectural optimization. Here is an architectural breakdown of how multi-agent AI systems, governed telemetry, and Model Context Protocol (MCP) transform enterprise quality engineering. The Problem: The 45-Hour Manual Bottleneck Building a end-to-end BDD automation suite manually requires significant time per user story—often taking up to 45 hours across five distinct steps: Context Generation & Requirements Review (~8 hrs) Manual Test Case Design (~9 hrs) Cucumber Feature File Creation (~8 hrs) Page Object Model Generation (~8 hrs) Step Definition Implementation (~10 hrs) This traditional workflow creates coverage gaps, inconsistent code quality, and defect leakage. The Solution: Multi-Agent AI Automation Pipeline Instead of relying on single prompts, an AI Test Automation Factory routes requirement artifacts (BRDs / User Stories) through specialized agents: [BRD / User Story] │ ▼ [Context Agent] ──► [Test Case Agent] ──► [Feature File Agent] │ [Automation Suite] ◄── [Step Definition Agent] ◄── [Page Object Agent] Context Agent: Parses acceptance criteria and enterprise domain knowledge. Test Case Agent: Auto-generates exhaustive test scenario matrices. Feature File Agent: Drafts standardized BDD Cucumber feature files. Page Object & Step Def Agents: Constructs clean design patterns (POM) and matching step implementations. Measurable ROI: Before vs. After AI By replacing manual generation with agentic workflows, the effort to automate a scenario drops from 45 hours to 9.5 hours: Phase Manual Effort AI-Driven Effort Time Saved Context Generation 8 hrs 2 hrs 75% Test Design 9 hrs 2 hrs 78% Feature File Creation 8 hrs 0.5 hrs 94% Page Object Creation 8 h

1. What is a Test Case? A Test Case is a set of steps, test data, conditions and expected results used to check whether a particular functionality is working correctly or not. Example: For a login page, enter a valid username and password and click Login. The expected result is that the user should successfully log in. 2. What is a Test Scenario? A Test Scenario is a high-level functionality or condition that needs to be tested. Example: "Verify Login Functionality" is a Test Scenario. Under this scenario, we can create multiple test cases like valid login, invalid password, empty username, empty password, etc. 3. What are Negative Test Cases? Negative Test Cases are used to check how the application behaves when invalid or unexpected data is given. Example: Entering an incorrect password or leaving the username field empty. The application should not crash and should show the proper error message. 4. What are Positive Test Cases? Positive Test Cases check whether the application works correctly with valid and expected input. Example: Entering a valid username and password should allow the user to log in successfully. 5. Relationship Between Test Case and Test Scenario A Test Scenario is a high-level requirement or functionality, while a Test Case contains detailed steps to test that scenario. Example: Test Scenario: Verify Login Functionality. Test Cases: Login with valid username and password. Login with invalid password. Login with empty username. Login with empty password. So, one Test Scenario can have multiple Test Cases. 6. What is Unit Testing? Unit Testing is testing individual units or components of software separately. Usually, developers perform Unit Testing. Example: If there is a function that calculates the total price, we can test that function separately to check whether it returns the correct result. 7. What is Integration Testing? Integration Testing is used to check whether two or more modules work correctly after they are combined. It mainly foc

I founded Krasyn, an outpatient EMR with an AI scribe inside it. Krasyn has run a working outpatient clinic's real patient records since March 2026, so what our scribe drafts ends up in charts that real clinicians sign. This post covers two things we shipped in August: a published benchmark of how faithful those drafts are to the transcript, and Note Check, a tool that reads any scribe's note against its transcript and lists what the transcript does not support. Why a fluent note is the problem A faithful note and a note with one invented blood pressure look the same on the screen, and the clinician who signs it owns every sentence. Published evaluations put ambient-scribe hallucination at about 1 to 3 percent of notes. A March 2026 analysis of 71,173 AI-drafted and finalized note sections found a confirmed edit in 5.8 percent of them. The drafting got automated. The checking did not. I wanted a number for our own scribe that I could defend, with the definitions printed next to it. A benchmark without definitions is marketing. The unit: a clinical assertion We measure at the level of a clinical assertion, one atomic statement about the patient that could be true or false on its own. "Denies fever, chills, and nausea" is three assertions. A measurement and its value are one. Hedging is kept verbatim. Every assertion gets exactly one label against the transcript: Supported: the transcript says it, or it is a faithful paraphrase or clinical translation. Inferred: not stated, but a reasonable clinical inference with a basis in the transcript. Tracked separately because it is the contested category. Unsupported: no basis in the transcript at all. Contradicted: the transcript says the opposite, including a symptom the patient denied, a treatment the clinician declined, or another person's symptom attributed to the patient. Hallucination rate is unsupported plus contradicted over all assertions. Coverage is measured separately against key facts per case, because a note tha

If you are building AI-powered test automation, you may eventually run into this question: Should we use RAG or MCP? The question sounds reasonable, but it is slightly misleading. RAG and MCP solve very different problems. In testing, you will probably need both. The Problem With AI-Generated Tests LLMs can already generate Selenium, Cypress, and Playwright tests from natural-language prompts. Ask: Test the login flow with valid credentials. and an AI can produce a reasonable script. But there is a problem. The AI does not automatically know: Your actual business rules Existing test cases Previous defects Test data API behaviour High-risk workflows Team-specific automation standards It knows how testing works , but not necessarily how your product works . That is where RAG becomes useful. What RAG Actually Solves RAG gives the AI access to project-specific information. Instead of working from a generic prompt, the model can retrieve relevant: Requirements Test Cases API Docs Bug History Business Rules Existing Automation Test Data Now consider the same request: Test the checkout flow. Without RAG, the AI may create a fairly standard checkout process. With RAG, it could first learn: Which payment methods are supported Whether guest checkout is allowed Which validations are required Which checkout bugs appeared previously Which scenarios already exist The generated test becomes much more relevant. But there is still a limitation. Knowing what should happen does not mean the AI can actually test it. That Is Where MCP Comes In MCP gives an AI system access to external tools. For browser testing, that could mean allowing an AI agent to use Playwright capabilities to: Open Page ↓ Inspect UI ↓ Enter Data ↓ Click ↓ Observe Result ↓ Validate So the difference is simple: RAG gives the AI context. MCP gives the AI capabilities. Or even shorter: RAG = What does the AI know? MCP = What can the AI do? Why This Matters for Test Automation Imagine an AI receives this instruction: C

A manual psql check answers exactly one question: does this policy work right now, against today's schema, with today's roles. It says nothing about tomorrow. Three ordinary changes are enough to quietly break tenant isolation without anyone noticing at review time. A migration that drops and recreates a table loses RLS entirely, since it's a per-table flag, not something that travels with column definitions. A new service role for a background job can skip the policy if nobody remembers to apply it. And the most common one: someone grants BYPASSRLS during an incident and never revokes it. Most guides point you at pgTAP here and stop. pgTAP is fine, but it's a separate SQL-based framework with its own runner. If your backend is already on Jest, you don't need a second test framework, you need a Jest test that actually proves a leak can't happen. The core pattern: seed a row as tenant A, query as tenant B, assert the result is empty. Run it through a dedicated low-privilege role, since table owners and superusers bypass RLS by default even with FORCE enabled for the owner. I break down the full pattern, the queryAsTenant helper, testing WITH CHECK on INSERT/UPDATE, catching accidental BYPASSRLS grants, and wiring it into GitHub Actions here: https://devencyclopedia.com/blog/postgres-rls-testing-jest If you're doing this across more than one or two tables, I also built RLSBuilder, a browser tool that generates the CREATE POLICY SQL and a matching Jest test from the same three inputs so they can't drift apart: https://devencyclopedia.com/tools/rls-builder

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The series checkpoint The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. What coverage percentage actually measures Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of b

Originally published on hexisteme notes . I run pre-registered checks against a live system, read the verdict, and move on — that's the whole point of pre-registering them, so I don't get to argue with the result after the fact. Most of the time the discipline pays for itself. This time it passed, and the pass was wrong, and the reason it was wrong is more interesting than the failure itself: the check could not have returned anything else, whatever had actually happened to the file under test. The question I was probing something narrow: does a hand-made audio crossfade survive a round trip through DaVinci Resolve? Build a timeline with a crossfade sitting on a cut, export it to FCPXML 1.10, re-import it, and see whether the crossfade is still there. Third-party documentation says transitions are invisible to and unmodifiable by the scripting API. Believing that, I pre-registered a judgment method that never looks at timeline structure at all: render audio around the splice and classify it by waveform shape. The judge, exactly as pre-registered: render two seconds either side of the cut, downsample to 8 kHz mono, compute a 20 ms sliding-window RMS envelope — 202 windows across the render — and take the largest normalized step between adjacent windows. Above 0.5, call it a hard cut: the fade is gone. Below 0.5, call it a gradual ramp: the fade survived. The probe came back pass — gradual ramp, max step 0.4761, under the 0.5 threshold. Exit 0, all green. The crossfade had actually been lost at the export step. The pass was a false confirm, and I only found that out by going back in with a second, read-only inspection after the fact. Why the check could not fail The prep instructions for this probe — which I also wrote — said the easiest way to get two adjacent audio items with enough handle to build a crossfade is to take one continuous clip and blade-split it in the middle. That's a completely reasonable instruction on its own. A crossfade needs overlap media on bot

I built a pytest harness that runs the same set of questions through five language models at once - a free local Llama, plus GPT, DeepSeek, and two Claude models - and compares them on the three things a team pays for: cost per query, speed, and answer quality. The plan was simple. Run the grid, read the scoreboard, say which model to use. The scoreboard came back clean and easy to read. This is the story of why I didn't trust it, and what I found when I checked. The thing I stopped trusting wasn't any of the models. It was the tool I was using to score them. It's also the first project in this series that spends real money. Every one before it ran locally, for free. Here each call costs something, and the whole comparison came to about 21 cents. That price is small, but it changed how I tested, and not in the way I expected. The scoreboard, and why I didn't stop there Five models, the same ten questions, twice each, every call measured. Here is the run, ordered by quality score (a second model grades each answer on correctness and relevance, combined into a 0-1 score, pass line 0.7): model quality mean $/query mean latency out-tokens deepseek-v4-pro 0.970 $0.000138 2713 ms 113 claude-haiku-4-5 0.967 $0.000537 1597 ms 104 gpt-5.6-luna 0.962 $0.000082 1323 ms 65 claude-sonnet-5 0.937 $0.002426 4093 ms 239 llama3.2 (local) 0.922 $0.000000 7859 ms 130 Read it straight and it looks finished. The whole quality column sits in a tiny band, 0.92 to 0.97. The cheapest, fastest paid model scores right in there with the rest. The most expensive one, Sonnet, at about thirty times the price per query, sits no higher than the others - its answers are just longer (239 tokens to GPT's 65), which costs more and takes longer without scoring better. So the easy takeaway is: use the small cheap model, skip the expensive one. I want to be careful with that, because it's the kind of tidy result I've learned to distrust. The gaps between the top models are tiny, and a ranking built on tin

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. The...

The Forked History: Byzantine Witness and the 3-of-4 Quorum — Tested Agent Determinism Illusions (Part 19) 2026-08-20 Where this fits: Part 18 closed the runtime face of C3's boundary at capability isolation — the oracle reads from a surface the producer cannot write. Part 18's §6 named the residual this part answers: Byzantine authority . Sealing one honest oracle's history says nothing about whose view is the truth when a compromised authority can present forked views to different observers. This part maps the witness layer — the answer is not a stronger single authority, but a witness set with three separable properties, an explicit fault bound, and a governed membership surface. Part 18 ended with the oracle isolated from the producer's writable surface. Isolation answers "can the producer fake the read?" It does not answer "whose read is the truth when the authority itself equivocates?" A trusted parent that closes the verdict channel still presents the approval history. If that authority is compromised — or the CI choosing the harness is — it can show job A a signed checkpoint and job B a different fork; each job sees a locally valid tree head with an inclusion proof, and no single history exists. This part tests that shape, then the witness machinery that answers it. 1. Sealed floor ≠ global history Peter's pin-rollback reply sealed a monotonic minimum-version floor: CI could no longer resurrect an older harness with a known false-green channel. The sealed floor is honest for what it claims. It is not what it looks like at first. The split into two predicates. The sealed floor proves this job did not go backwards on the view it was shown. It does not prove the approval history itself is one global append-only log. A compromised authority can hand job A a signed checkpoint whose minimum is 2 and job B a fork whose minimum is still 1; each view carries a locally valid signature and an inclusion proof while no single history exists. cell setup result A local sea