标签：#testing

Agent Plugins 1.0.0 ships a JSON Schema for plugin.json . It sets additionalProperties: false . So the obvious loader is four lines: const manifest = JSON . parse ( await readFile ( join ( dir , ' plugin.json ' ))); if ( ! validate ( manifest )) return reject ( ' invalid manifest ' ); That loader is wrong, and the specification says so in a sentence most people never reach. §5.2: Clients MUST report and ignore each unknown field and MUST continue loading the plugin if the manifest otherwise satisfies this section. An unknown top-level field is a schema violation you have to tolerate . §8.1 says the same for an extensions field that isn't an object. Every other schema violation is fatal. So a validator gives you one boolean where the spec wants three different outcomes, and the natural implementation is non-conformant in exactly two cases and correct everywhere else. That is the kind of bug that doesn't show up in your tests. It shows up as a plugin that works in one client and not another, six months later, in someone else's bug tracker. This has already happened, repeatedly I went looking before building anything. In the last few months: Codex loaded any directory with a root plugin.json through its Agent Plugins loader, which had no hook support. Every hook in .codex-plugin/plugin.json silently stopped running. Two plugins were dead for a week before anyone noticed. oh-my-pi routed packages declaring an agent-plugins.org $schema to a strict provider that dropped any SKILL.md with an extra frontmatter key. Downstream, a plugin went from 33 skills to 3. The fix was to delete $schema from the manifest, so conforming to the standard cost them the standard. dotnet/skills shipped manifests with no $schema and with skills , agents and mcpServers as top-level fields. Kiro refused them. Adding $schema got past the rejection and then loaded the package with every functional component excluded. VS Code , the largest shipping client, has no validation surface at all. Its trou

aimock crossed 2.5M weekly installs! Here's what it does and what's new If you've been...

The Problem: Testing Across Regions Is Hard If you're a developer, you've probably faced...

Every company knows when it revoked access. None knows when access stopped. I built this for the All Things Agentic Hackathon , and I wrote this post for the purposes of entering that hackathon. Code: github.com/NexuChat/parallax The chore I was actually trying to kill I maintain a web application with two roles, two languages, one of them right-to-left, a dark theme, and three viewport sizes. Every release, I would open it as the owner, click through, sign out, sign in as a member, click through again, switch to Arabic, reload, shrink the window, reload — and try to remember what a page had looked like ten minutes earlier. The worst defects never survived that process, because they are not visible in any single session. A member opening a page they should have been denied sees nothing wrong. Nothing on the page says "you should not be here." The information is not in their session at all. It is in the difference between their session and the owner's. So I stopped testing sessions and started comparing them. Seven witnesses, one axis apart Parallax opens seven isolated browser contexts at the same instant against the same application. One is a baseline — owner, English, light, desktop. The other six each change exactly one axis from it: privilege, locale, theme, viewport. The full product of those axes is thirty-six combinations. Seven one-axis derivations is not just cheaper; it is the only version that can attribute a cause. When the Arabic witness disagrees with the baseline and locale is the only thing that changed, locale is the reason. With thirty-six combinations you get a bigger table and less knowledge. Each axis carries a contract about what must change and what must not: Axis Contract A finding is Privilege access must differ sameness — an escalation Locale access constant, layout mirrors access drift, or geometry that did not mirror Theme access constant, layout does not move any positional shift Viewport access constant, reflow allowed access drift That

This is part four of the Defender Access series. Each part is standalone, but here is the thread if...

OpenAI Usage API api_key_id grouping solves a practical reporting gap: I can see which API key produced completion-token activity and which key accumulated cost. The tricky part is not making the two requests. It is joining their daily buckets without dropping unattributed or unmatched data. I want a reconciliation report to expose gaps, not smooth them over. A missing cost row, a cost-only row, or a null key ID can each be useful evidence. This pattern keeps those cases visible with a deterministic .NET sample that needs no credentials or paid calls. Why OpenAI Usage API api_key_id needs a full-outer join OpenAI's August 4, 2026 API changelog added API-key filtering and grouping to the usage and cost APIs. That gives both responses a shared operational dimension, but it does not make them identical datasets. The completions usage endpoint reports measures such as input tokens, output tokens, and model requests. Its api_key_id can be null. The costs endpoint returns monetary amounts and currency, also with a nullable API-key dimension. An inner join would retain only rows present in both responses. That is attractive for a tidy chart, but unsafe for reconciliation. It can hide a key that has token usage but no matching cost row, a key with cost but no completion row, or an unattributed bucket. I use a full-outer join keyed by (start_time, end_time, api_key_id) instead. Null or blank IDs become an explicit display value such as <unattributed> ; they do not disappear. Query both APIs at the same daily grain The Costs API supports daily buckets, so I request bucket_width=1d from both endpoints. I also group by the same single dimension: GET /v1/organization/usage/completions ?start_time=... &end_time=... &bucket_width=1d &group_by=api_key_id GET /v1/organization/costs ?start_time=... &end_time=... &bucket_width=1d &group_by=api_key_id Both resources paginate with has_more and next_page . I keep requesting pages until has_more is false. If a response says more data exis

Small automations often look easy: take information from one place and turn it into a task somewhere else. The hard part is what happens when the information is incomplete, someone submits the same request twice, or the workflow sees something it was never meant to use. A useful automation should handle those situations without creating extra cleanup for the owner. I built a small runnable example around four simple checks. 1. Make sure the important information is there If a request is missing something the team needs, the workflow does not create a half-finished task. It places the request on a short review list and explains what is missing. 2. Do not create the same work twice Repeated submissions happen. The example recognizes a repeated request and creates only one task instead of making the team sort out duplicates later. 3. Keep out information the workflow does not need The example copies only the agreed fields into its output. An unexpected column in the input is ignored instead of being passed along automatically. 4. Let a person review the result The example creates an owner-review list. It does not contact customers, connect to outside services, or turn on a live process. A person stays in control of what happens next. The repository includes four made-up requests, the expected result files, and ten automated checks. Those checks cover missing information, repeated requests, unexpected fields, broken input files, and repeatable results. This is an Allure Labs demonstration, not client work and not a claim about business results. You can see the code and sample output here: https://github.com/Allura-Gensin/small-workflow-automation-demo If one small file-based process is creating repeated or incomplete work, start with a $125 written workflow plan or a $500 tested small build. Describe one starting event, one result, and what the workflow must never do. The fixed-scope options and limits are here: https://offers.allurelabs.ai/workflow-automation/ Or use t

A healthcare platform I worked with needed two and a half weeks to complete a regression pass. Smoke testing alone took seven days. The obvious recommendation — the one everyone reaches for — was "automate everything." It would also have been the wrong place to start. Here's the trap. If you point automation at a bloated, outdated, UI-heavy suite, you don't get fast regression. You get an oversized, expensive, automated version of the same slow process, plus a maintenance bill that grows every sprint. A faster test runner does not create a faster regression process — it just runs the wrong tests more quickly. The teams that actually go from weeks to hours don't start with the runner. They start by asking where the time is really going — and most of it is not in test execution. On that healthcare platform, we eventually got regression down from 2.5 weeks to a single day while raising coverage from 50% to 90%. Automation was part of it. It was nowhere near the whole story. This is a guide to the whole story: what to cut, where to test, when to run, and how to investigate failures — so that automation removes specific bottlenecks instead of freezing an inefficient manual suite into code. Why regression testing takes weeks The first thing to fix is a measurement mistake. Most teams track test execution time and quietly assume it's the same thing as regression lead time. It isn't, and the gap between them is where your weeks disappear. Regression lead time = preparation + environment setup + queue time + execution + failure investigation + reruns + reporting Execution is one term in that sum, and often not the biggest. You can halve your runtime and still ship on the same day if the other six terms are untouched. Before you optimize anything, break your lead time into these buckets and see which one actually hurts. In my experience it's rarely the one people complain about. Here's where the time usually leaks. The suite grows but never shrinks Every team is good at addin

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions. This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all. Why Memory Changed the Review Game Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw. Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise. The Fixture Repository fixture/ ├── docs/decisions/0001-metrics-pipeline.md # accepted 2026-07-02 ├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14 ├── src/metrics_service.py # legacy module, 120 lines ├── src/telemetry_service.py # replacement module, 140 lines └── pyproject.toml # lint: E501 disabled for telemetry only The fixture encodes a deliberate conflict: the team renamed the metrics pipeline

A Vue component's job is to produce DOM in a browser. Most component tests ask it to do that somewhere else: in Node, against a DOM that jsdom simulates. That has been the default since npm create vue@latest started offering Vitest with jsdom, and for plenty of tests it is the right trade. It does set a ceiling on what a green test proves, though. Nothing is ever drawn. Your CSS never runs and nothing has a size or a position, so a component can pass every assertion in the file and still be broken on screen. I co-maintain twd-js , which runs tests inside your actual dev server, in a sidebar, next to the app. It was built for flow testing: visit a route, click through the app, assert on what the user sees. Component testing was the thing it did not do. Then I tried calling render() from @testing-library/vue inside a TWD test. import { afterEach , describe , it } from " twd-js/runner " ; import { twd , userEvent } from " twd-js " ; import { render , screen , cleanup } from " @testing-library/vue " ; import HomeView from " ../../views/HomeView.vue " ; import { componentHost , restorePage } from " ../support/componentHost " ; describe ( " HomeView component " , () => { afterEach (() => { cleanup (); restorePage (); }); it ( " increments the counter on click " , async () => { // componentHost() is a blank div on an empty page. More on it below. render ( HomeView , { container : componentHost () }); const button = await screen . findByTestId ( " counter-button " ); twd . should ( button , " contain.text " , " Count is 0 " ); await userEvent . click ( button ); twd . should ( button , " contain.text " , " Count is 1 " ); }); }); Nothing broke. The component mounts into the page, the sidebar shows it running, and reactivity does what reactivity does, in a browser, against a DOM nobody had to simulate. Why this works at all Vue Testing Library is a thin layer. render() mounts your component with @vue/test-utils and binds @testing-library/dom queries to the result. Neither of

Probe vs Prose: what the verifier-sharing-your-text-channel really costs Agent Determinism Illusions (Part 13) 2026-08-31 Where this fits: This part doesn't extend the C3 / key-space mechanism line of Parts 10–12. It returns to an earlier thread — Part 4's runner-independence (Mike Czerwinski's point that "verifiable" is a property of the check's independence from the generator, not of the output) and Theorem 2 (the Data Processing Inequality bound on text-channel verification). A comment from nexus-lab-zen gives that thread a name on the assumption side, and an experiment forces a refinement of what "prose rots" actually means. 1. nexus-lab-zen and the third face of the hatch In the comments on Part 2, a many-round thread with nexus-lab-zen arrived at a useful piece of vocabulary. The thread started on segregation-of-duties and common-mode failure ( Part 2 comments ); several rounds in, nexus-lab-zen had moved from theory to something their team shipped that week: We don't have [per-assertion TTL] either… What we shipped this week is a third face of the hatch[…]: a binding map. Every rule in our registry — 39 right now — must either name the detector that physically enforces it or carry an explicit reason why it's unbound; a fail-closed lint breaks on rules that have neither. Result: 9 bound, 30 unbound-with-reason… On making [TTL] real, one lesson from our timestamp incidents generalizes: fields humans transcribe rot; fields machines embed don't. An invalidation condition written as prose ("assumes transport X is live") goes stale like any prose. Written as a probe — the one command whose changed output falsifies the assertion — the TTL re-check becomes a runner, not a reader. Two things in that comment are worth pulling apart, because one of them survives an experiment and the other gets refined by it. The first is the binding map : 39 rules, of which 9 name a physical detector and 30 carry an explicit "unbound-with-reason." That's not TTL — it can't tell you a p

I had twenty-two passing tests and two separate reviewers on a piece of code. None of it objected. Then I pointed it at a real API owned by somebody else and it broke on the first live read. The mismatch fit in one sentence: my parser required ISO 8601, the documented API returned Unix seconds. The repair was not one line. It touched five files, 74 lines of parser and 52 lines of tests. The assumption was small; making it safe was not. Here is why nothing caught it, and it is the part worth keeping: My tests used ISO because my code used ISO, so they agreed with each other and never checked reality. The fixtures were written by the person who wrote the parser. They encoded the same assumption. The suite confirmed internal behaviour without ever challenging the ISO assumption, because both halves of it came from one head. Internally consistent is not the same claim as right, and nothing in that suite could tell the difference. Two separate reviewers missed it too. I cannot prove why, and I am not going to invent a reason. What I can show is that the parser and every fixture encoded the same ISO assumption, so none of the artifacts in front of anyone supplied the live contract that contradicted it. The second one was worse Working against a real system made redirect containment matter, so an independent breaker went at it. In Python 3.13 the default redirect handler rebuilds the redirected request from req.headers , dropping only content length and type. My X-API-Key sat in that header set, so the redirected request inherited it. Python has Request.add_unredirected_header() for exactly this, which marks a header as one that will not be added to a redirected request. I was not using it. The breaker reproduced it offline with a sentinel value and a cross-origin Location , and the sentinel crossed. No live FIPSign credential was ever shown to have crossed an origin. The defect was real and unshipped. I did not find it by auditing my own code, and I did not find it myself

Latest release: v0.2.2 — Aug 29, 2026 I did something I usually try hard not to do in a field test. I changed the design after it had already started. Halfway through validating AdversarialDebate, I realized the model set was too narrow to answer the most important question in the project. So I added a fourth model in the middle of the run. That was messy. It wasted work. It made the corpus inconsistent for a while. It also turned out to be one of the best decisions in the whole release. This post is about a lesson I trust far more now than I did before building this project: a field test is not just there to produce numbers. It is there to reveal whether your experiment can actually answer the question you think it is answering. The Setup I Started With I began with three models: GPT-4o-mini, Gemini 2.5 Flash, and DeepSeek-V3. That gave me three useful pairings — GPT + Gemini, Gemini + DeepSeek, and GPT + GPT as a homogeneous control. Three labs, two regions, one same-model control. Reasonable spread. I ran the small corpus first, just 3 PRs, to validate the pipeline. Pair Small-corpus score Verdict rate Gemini + DeepSeek 0.835 33% GPT + GPT 0.667 33% GPT + Gemini 0.148 0% The diverse pair was ahead. The weak pair was struggling. The homogeneous control was doing something interesting. If I had stopped there, I would have told a clean story — and it would have been the wrong one. The Problem Was Not The Data. It Was The Coverage. The issue was not that the first three models were bad. The issue was that the experiment could only see part of the diversity spectrum. With those three models, the farthest useful pairing I had was US + China. I did not have a genuinely cross-continent pair that could show what happened at the far end of diversity. The test could suggest whether diversity helped. It could not show whether maximum diversity behaved differently from moderate diversity. That is a major blind spot when the whole thesis is about pairing behavior. I needed a f

Originally published at nlocoding.com 92%of regression bugs in SaaS platforms go undetected until production without AI-based testing (Source: Capgemini World Quality Report 2026) Most companies spend more on fixing bugs post-release than on their entire automated testing stack. According to the Testing Intelligence Survey 2026, the average cost to fix a bug in production is $3,800—triple what it costs to catch it during automated testing. This is why 2026 trends in AI-driven software testing matter: the cost of ignoring them is rising fast. AI-driven test coverage is replacing manual scripts in 2026 AI-driven test coverage now exceeds traditional manual scripting by 64% in efficiency (SmartBear State of Quality 2026). Companies like Atlassian cut manual test creation time by 71% after switching to AI-powered tools such as Testim and Mabl, which both cost around $100/user/month. Manual testers are not obsolete, but they are now orchestrators, not script jockeys. 💡 Pro Tip: Start by identifying repetitive UI tests. AI tools excel at these and deliver instant ROI. Self-healing tests are solving flaky pipelines Self-healing tests reduce flaky test failures by 83%, according to Sauce Labs' 2026 industry report. This matters: Netflix slashed CI/CD pipeline downtime from 14 hours/month to under 2 by using Functionize, which auto-fixes selectors and waits for dynamic elements. The technology isn't magic, but it is relentless. 83%fewer flaky failures with self-healing AI (Sauce Labs 2026) You’ll notice fewer midnight Slack panics. Give your team back their weekends. Adopt a self-healing platform with robust change detection. GenAI is writing—and maintaining—test cases in 2026 Generative AI wrote 54% of all new test cases at Fortune 500 companies in Q1 2026 (TestOps Pulse). Copilot for Test Automation, released by GitHub in February 2026, costs $19/month and supports Cypress, Playwright, and Selenium. The result? Test coverage expands, but more importantly: maintenance shrin

An extension can work perfectly in development and still fail after packaging. The risky change is often not in the feature code itself. It can be a permission that moved, a host pattern that expanded, a content script that now runs somewhere new, or a browser surface that was never included in the release checklist. Here is the small preflight review I now use before testing an MV3 release. 1. Compare the packaged manifests Compare the last version you actually shipped with the new packaged version, not only the source manifest. Check separately: required permissions; optional permissions; required host access; optional host access. A permission moving from optional to required deserves attention even if the set of permission names looks familiar. 2. List every browser surface Turn the manifest into a list of things a person can interact with or that Chrome can start: action popup; options page; side panel; background service worker; content scripts; commands; externally connectable pages; declarative network rules; web-accessible resources. If a surface changed, add at least one release check for it. This sounds obvious, but it is easy to review the main popup while forgetting an options page or a host-specific content script. 3. Check where code can now run For every content script, compare: match patterns; excluded matches; frames; execution world; run timing. The JavaScript file can be unchanged while one of these settings changes the extension's behavior on real sites. 4. Test the packaged build Run the checklist against the same build directory that will be uploaded. A development build can hide packaging, path, minification, or generated-manifest differences. At minimum, reload the packaged extension and exercise one path through each changed surface. 5. Record why each check exists Instead of keeping a generic list such as “test the popup,” connect each check to a release change: host access expanded → test the new host and confirm the old hosts still

Originally published on hexisteme notes . I set a threshold from measurement instead of guessing. The measurement was clean: zero overlap between the two clusters, a 33x gap between them. I wrote the numbers into a comment with their sample sizes, feeling good about not having guessed. It was wrong, because the sample I had labelled "known good" was one of the bad ones. I've written before about checks that cannot fire — guards whose thresholds were miscalibrated for the scale of their input, so nothing you fed them ever tripped the line. This is a different animal. My threshold was calibrated from data . That's exactly what made it convincing, and it's why the calibration itself is where the bug lived. The check A video pipeline burns captions onto a rendered preview. A gate then diffs the burned output against the preview and treats every changed pixel as "text we drew," so it can ask whether our captions intrude into the platform's UI safe area. That reading only holds if the two files are a pair — if this output was burned from this preview. Nothing verified that. The only guard compared the number of sampled frames . Sampling is time-uniform, so two generations whose durations differ by 0.1s both yield exactly 60 samples. The guard was structurally incapable of noticing the thing it was nominally there to notice. Setting the threshold I wanted a statistical backstop: if the whole-frame difference between the two files is too large, they probably aren't a pair, so refuse to render a content verdict at all. Exactly one episode in the repo had both files sitting on disk. I used it as my positive control. sample median whole-frame abs diff "correctly paired" episode 19.51 known-mismatched pair 98.65 Threshold: 55.0. Zero overlap, a 33x gap. Two clusters, cleanly separated. Done. The control was a negative That episode's preview file had an mtime nine hours later than its output — and later than the gate run that had already approved it. The preview on disk had been

On 2026-08-25 at 21:27 EDT I pushed this to self-correcting-integration-maintainer: fix: repair...

I created this article for the purpose of entering the All Things Agentic Hackathon. TL;DR — An ADK output_key writes into the session of the agent that declares it. In-process that session is shared, so it looks like state flows. Across a RemoteA2aAgent hop it is the worker's session, and it never comes back. Nothing raises. Nothing warns. Every local run and every CI job exercises the working topology, so the failure is invisible to an offline test suite by construction — including at 100% coverage. The system that passed Bastion is a three-agent access-governance fleet built with Google ADK and A2A. An Orchestrator owns investigation state, an Access Auditor reads production IAM through a read-only identity, and a model-free Escalation Agent delivers validated count-only reviews. The local graph passed its configured core statement and branch coverage gate. Every branch, every seam. Then the same graph was split across deployed A2A workers, and an assumption that looked natural in-process became false. The boundary we had not modeled In-process, the previous step's result is simply there : # The Auditor declares output_key; the Orchestrator reads it back. report = ctx . session . state . get ( AUDIT_FINDINGS_KEY ) Deploy the same sequence and only the construction changes. The graph is identical: RemoteA2aAgent ( name = " access_auditor " , agent_card = card_url ( auditor , " access_auditor " ), description = " Reads the live IAM policy and flags anomalies. Read-only. " , httpx_client = private_a2a_client ( auditor ), a2a_request_meta_provider = _forward_investigation , ) output_key still writes. It writes into the worker's session, which never crosses back. The deployed Orchestrator saw an empty state key while every local run and every test saw a populated one. Observed 2026-08-22: the Auditor completed a full sub-trail, and the next step then refused with "returned no structured report." No exception at the boundary. No warning at construction. The run still r

The AI Wrote the Diff. The Tests Wrote the Verdict. AI refactor suggestions are hypotheses. Not facts. A free coding model rewrites your messy legacy function. The diff looks clean. CI stays green. Then a customer hits an edge case you forgot. This article shows a small workflow. Characterize legacy behavior first. Let the model propose a refactor. Run the same tests against both versions. The verdict: safe or not safe. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Why Characterization Comes First Legacy code has no spec. The only reliable spec is current behavior. Even bugs are behavior. If your refactor changes a bug, you need to know. A characterization test records inputs and outputs. It does not judge right or wrong. It freezes the current contract. After freezing, every difference becomes visible. Step 1: Capture Real Inputs and Outputs Pick one messy function. I used a shipping calculator. Nested conditionals, magic numbers, zero tests. Write a probe script. Call the function with realistic cases. Save outputs as JSON. import json from legacy import calculate_shipping cases = [ { ' items ' : [{ ' weight ' : 2.0 , ' qty ' : 3 }], ' region ' : ' US ' }, { ' items ' : [{ ' weight ' : 0.5 , ' qty ' : 10 }], ' region ' : ' EU ' }, { ' items ' : [{ ' weight ' : 0.2 , ' qty ' : 1 }], ' region ' : ' US ' }, { ' items ' : [{ ' weight ' : 5.0 , ' qty ' : 2 }], ' region ' : ' JP ' }, ] for c in cases : result = calculate_shipping ( c [ ' items ' ], c [ ' region ' ]) print ( json . dumps ({ ' input ' : c , ' output ' : result })) Save output to captured.json . That becomes ground truth. Step 2: Ask the Model for a Refactor MonkeyCode's free model access lets me prompt from the CLI. I gave the model one strict instruction: keep behavior identical. Refactor calculate_shipping into smaller functions. Do NOT change edge cases. Do NOT change rounding. Extract private helpers only. The model returned a diff. It split the function into three he

I keep finding git behaviours the same way: set up the case, run the command, then check the state I...