标签：#Testing

My upload CLI looked restart-safe until I tested two failures together: the process crashed at 63%, and the signed resume URL expired before restart. The checkpoint needs identity, not credentials: { "file" : "release.tar.gz" , "fingerprint" : "sha256:..." , "uploadId" : "up_123" , "localOffset" : 66060288 , "updatedAt" : "2026-07-19T12:00:00Z" } Do not persist the signed URL. On restart, exchange the durable upload ID for fresh authorization, query the server offset, then reconcile: const remote = await headUpload ( freshUrl ); if ( remote > file . size ) throw new Error ( " invalid remote offset " ); if ( remote !== checkpoint . localOffset ) { await saveCheckpoint ({ ... checkpoint , localOffset : remote }); } await sendFrom ( file , remote , freshUrl ); The server offset wins because a crash can occur after bytes are accepted but before the local checkpoint is renamed. Save checkpoints through write-to-temp plus atomic rename so a partial JSON file cannot destroy recovery. My fixture matrix includes: Failure Expected behavior crash after remote commit rewind/advance to remote offset expired URL refresh without creating a second upload changed local file stop on fingerprint mismatch missing remote upload ask before starting over checkpoint write interrupted retain previous valid checkpoint The tus resumable upload protocol specifies offset discovery and conflict handling that are useful even if the service uses a smaller custom protocol. The key idea is explicit reconciliation, not assuming client memory is authoritative. I also shipped upload status , upload forget , and an exportable checkpoint directory. Recovery is a user-facing feature; if users cannot inspect or remove state, “resumable” becomes hidden lock-in.

Multipart finalization is not the only retry boundary. Abort can succeed in object storage while the HTTP response is lost, leaving your database convinced the upload is active. Model abort as a state transition, not a button wired directly to an SDK call: active → aborting → aborted └──→ cleanup_pending The endpoint should own an idempotency key and durable intent: await db . transaction ( async tx => { const upload = await tx . lockUpload ( uploadId ); if ( upload . state === " aborted " ) return ; await tx . markAborting ( uploadId , idempotencyKey ); await outbox . enqueue ( " abort-multipart " , { uploadId , storageKey }); }); A worker calls storage. If a retry receives NoSuchUpload , do not blindly fail: reconcile whether the upload was already completed, already aborted, or expired. Only the “already absent because abort succeeded” path may converge to aborted ; completion requires a separate terminal state. Test this sequence: Step Fault Required invariant persist abort intent process dies outbox resumes work storage abort succeeds response is lost retry does not reactivate upload duplicate message delivered twice one terminal state client retries same key same operation result cleanup scan stale active row reconcile before deleting Amazon S3’s AbortMultipartUpload API notes that in-progress part uploads may still succeed around an abort and recommends verifying that parts are gone. That makes a post-abort verification pass part of correctness, not optional housekeeping. Expose abort_requested_at , attempt count, last storage result, and remaining-part count. Alert on age in aborting , then run a lifecycle cleanup policy as a backstop—not as a substitute for application reconciliation.

Removing EXIF GPS protects one privacy boundary. It does not prove that an uploaded JPEG contains only a JPEG. A useful hostile fixture is a valid image followed by bytes for another format. Many decoders stop at JPEG’s end marker and display the picture successfully, while downstream scanners, content sniffers, or accidental downloads may interpret the trailing payload differently. Build a fixture set rather than trusting extensions: clean.jpg valid JPEG exif-gps.jpg valid JPEG with location jpeg-plus-zip.jpg JPEG followed by ZIP bytes jpeg-plus-html.jpg JPEG followed by HTML truncated.jpg missing end marker oversized-dimensions.jpg small file, dangerous decode cost My upload gate has independent layers: Store the original in a non-public quarantine location. Enforce byte-size and decoded-dimension limits. Decode with a maintained image library. Re-encode pixels into a new file, discarding metadata and trailing bytes. Verify the output’s signature, dimensions, and complete parse. Publish only the derived object under a server-generated name. A boundary check can detect bytes after the end marker, but re-encoding is the stronger transformation because it creates a new representation from decoded pixels. Keep the original inaccessible and delete it according to a tested lifecycle. assert ( outputSize <= limit ); assert ( decoded . width * decoded . height <= pixelBudget ); assert ( parseConsumesEntireFile ( output )); assert ( noLocationMetadata ( output )); The OWASP File Upload Cheat Sheet recommends defense in depth: allowlisted types, generated filenames, size limits, storage separation, and content validation. No single MIME header or metadata scrubber replaces that chain. The important privacy lesson is broader than EXIF: inspect every representation that survives the upload pipeline, and publish only a constrained derivative.

An accessible combobox can follow the correct ARIA pattern and still become unusable when two search responses arrive in the wrong order. Reproduce this sequence: Type ca , then quickly type cat . The cat response arrives first and highlights cat facts . The slower ca response arrives and replaces the list. aria-activedescendant now points to an option that no longer exists. IME input adds another boundary: searching during composition can send partial text the user has not committed. I would model the request generation explicitly: let generation = 0 ; let composing = false ; async function search ( query : string ) { const mine = ++ generation ; const options = await fetchOptions ( query ); if ( mine !== generation || composing ) return ; render ( options ); restoreActiveOptionByKey (); } The stable key matters. An array index cannot preserve the active option when ranking changes. Regression matrix Input Injected failure Expected evidence ca → cat first request delayed only cat results render Arrow Down result refresh active key survives or resets visibly Escape response arrives afterward popup stays closed IME composition network is fast no request until compositionend A Playwright test should assert focus remains on the input, every aria-activedescendant resolves to a live element, and Escape invalidates outstanding generations. A manual screen-reader pass should confirm result-count announcements are not emitted for discarded responses. The WAI-ARIA Authoring Practices combobox pattern defines the keyboard contract. The missing production step is testing that contract under asynchronous replacement, not only against static example data. Which stale-response failure has been hardest to reproduce in your search UI?

GenPage is a generative AI system developed by Netflix to replace its traditional multi-stage recommendation pipeline by directly generating personalized user homepages. GenPage leverages user history and request context as a prompt to produce the entire page, resulting in improved user engagement and reduced serving latency. By Sergio De Simone

This is a submission for DEV's Summer Bug Smash: Clear the Lineup , powered by Sentry . Project Overview I built LeakLens Control Room to keep AI away from the financial math. Python calculates every dollar. The optional model explains the evidence. A human decides whether to act. LeakLens is a live, functional restaurant margin-review application built with Python, SQLite, HTML, CSS, and JavaScript. It uses synthetic data in the public demo. The optional application narrative provider defaults to GPT-5.6, but every rate, threshold, ranking, and dollar scenario comes from deterministic code. Then a date string proved that deterministic code can still lie. I had let a raw CSV field decide which operating day counted as "current." Bug Fix or Performance Improvement The symptom LeakLens groups channel metrics by date, sorts those dates, and analyzes the last one against prior days. That works when every input is canonical ISO data such as 2026-07-09 . The loader only checked that the date field was not blank. This input slipped through: metrics = [ DailyMetric ( " 2026-7-9 " , " direct " , 500 , 250 , 0 , 0 , 125 ), DailyMetric ( " 2026-07-10 " , " direct " , 1000 , 50 , 0 , 0 , 250 ), ] July 10 is later than July 9. Python's string ordering sees something else: sorted ([ " 2026-07-10 " , " 2026-7-9 " ]) # ['2026-07-10', '2026-7-9'] LeakLens selected the final string, 2026-7-9 , as the current day. That mistake spread through the report: expected current date: 2026-07-10 actual current date: 2026-7-9 actual gross sales: 500.00 sales-loss scenario: 500.00 discount scenario: 225.00 The input had not shown a current sales collapse or discount leak. The analyzer compared the days backward and produced $725 of false scenario impact from synthetic data. For a restaurant operator, this is worse than a visible crash. A crash asks for attention. A plausible financial report can send someone to audit the wrong promotion, question the wrong manager, or change a schedule that was

AI coding agents have dramatically shortened the distance between an idea and working software. They can inspect a project, create files, run commands, write tests, and help diagnose failures. What they have not eliminated is judgment. A polished screen is not proof that data survives a reload. A passing unit test is not proof that keyboard users can complete the core task. A successful deployment is not proof that the intended commit reached production. This is why I use proof gates : observable conditions that must be satisfied before a product claim becomes stronger. A proof gate is not a meeting, a long document, or an excuse to slow down. It is a compact question: What evidence would let another person verify that this claim is true? Here are five gates that separate a persuasive AI demo from a small MVP you can responsibly ship. Gate 1: Prove One Valuable User Loop AI makes feature generation cheap, which makes uncontrolled scope especially dangerous. Before requesting code, define one primary user in one specific situation. Then describe: Their observable before-state The smallest useful action they can take The immediate result The reason they might return This becomes the product’s core loop. For example, “build a productivity platform” is too broad. A more testable loop might be: A freelancer remembers a useful client outcome. They record the outcome and supporting evidence. The record appears in a searchable library. They can retrieve it later for a proposal or review. The gate is not passed because a form exists. It is passed when a new user can complete the entire loop and explain what changed without coaching. Write a Not Today list alongside the required capabilities. Authentication, dashboards, collaboration, billing, and AI-generated summaries may all be reasonable later. They should not compete with proof of the first useful loop. The goal is not the fewest possible features. It is the smallest complete behavior that tests whether the product creat

Originally published on hexisteme notes . I have a small pipeline that crossposts my notes to dev.to. It parses a Markdown file's front matter, builds a payload, and calls the dev.to API to publish. It has 19 gate tests, and every one of them was green the whole time it was shipping. It published nine articles. All nine went live with their titles broken — the front-matter quotes were sitting right there in the title, visible to anyone who looked, for nine days, and nothing in the pipeline noticed. I didn't notice either. A human had to open the dev.to profile page by accident before anyone found out. This is the postmortem, and the reason I'm writing it up as a general essay rather than just a fixed-bug log is that the root cause isn't specific to dev.to, or to Markdown front matter, or to Python. It's a category of mistake that any pipeline with an external endpoint on the other end can make: testing the payload you build, and never testing what the other system does with it. The pipeline that had "passed everything" The shape of it is ordinary. A draft file has YAML-style front matter — title: "Some Title" — because that's the convention. A parser reads the front matter and pulls out the title. A payload builder takes that title and a few other fields and assembles the JSON body for the dev.to API. The API gets called, dev.to accepts it, the article is live. Nineteen gate tests cover this path — the front-matter parsing and the payload/API contract of the pipeline's own code. All green, every publish. The gap is in what "parses the front matter" actually means. The parser isn't a real YAML parser. It's closer to line.partition(":") — split each line on the first colon, take the right-hand side as the value. That works fine for tags: testing, devops where there's nothing to unwrap. It does not work for title: "Some Title" , because the quote characters are part of the string on the right-hand side of the colon, and a partition-based parser has no concept of "this

Most A/B testing happens in the browser: a script swaps a headline or button color after the page loads. That works for surface-level UI changes, but it cannot test the logic that runs before a page is ever rendered — a pricing algorithm, a search ranking model, a checkout flow, or a backend API response. Server-side A/B testing moves the experiment decision into your application code, where you control the full request lifecycle. This guide explains when to test server-side, how it differs from client-side testing, and how to implement it with Optimizely Feature Experimentation, including working SDK code for Node.js and Python. What Server-Side A/B Testing Is In a server-side A/B test, your application server decides which variation a user sees and renders the response accordingly. Instead of shipping the control experience and patching it in the browser, the server already knows the assignment by the time it builds the HTML, the JSON payload, or the rendered component. The decision is deterministic: a given user ID is consistently bucketed into the same variation, so the experience stays stable across requests and devices. Your code branches on that assignment, serves the corresponding experience, and reports a conversion event when the user does something that matters — a purchase, a signup, a search that returns a click. This is the model Optimizely calls Feature Experimentation . If you have used Optimizely before, you may know this product by its former name, Full Stack — the SDKs, datafile, and decision model are the same lineage, now under the Feature Experimentation name. Searchers still look for "Optimizely full stack," but the current product and documentation use Feature Experimentation. When to Test Server-Side Server-side testing is the right tool when the thing you are changing is not a cosmetic, post-render tweak. Reach for it in these situations: Backend logic and algorithms. Recommendation engines, search ranking, fraud scoring, feed ordering, and

I wrote a cleanup routine that matched processes by command line with a wildcard pattern. It reported success on every run. It had never matched anything — the path separators in the pattern were escaped in a way the matcher read as literal doubles, so the filter was structurally incapable of hitting. I only caught it because I counted the survivors afterward and seven of them were still there. The fix was switching from a wildcard match to a plain substring containment check with no escape semantics at all. A filter that cannot fail loudly will lie to you politely forever. Before trusting any matcher, feed it a known-positive and watch it fire — a green result from an instrument you never saw go red is noise. What's the equivalent lesson your worst bug taught you?

The moment we realized our staging environment was broken It was 3 PM on a Thursday, and our team was scrambling. A critical API change had just been merged to main, but the staging environment—our supposed safety net—was showing false positives. The integration tests passed, but the mobile app was completely broken in production. That's when we knew: our shared staging environment was failing us. The Problem: Shared Staging Is Broken by Design Like many engineering teams, we operated with a single, shared staging environment. Every developer deployed their changes to the same place, leading to: Deployment conflicts: "Who deployed that breaking change?" Cascading failures: One broken PR would block the entire team Test contamination: Data from one test would leak into another Delayed feedback: You'd only discover issues after merging your PR and deploying to staging The "works on my machine" syndrome, now at scale The worst part? Our API contracts were changing constantly, but we only discovered breaking changes during integration testing—often too late. The Solution: Ephemeral Environments per PR We made a radical change: every PR gets its own isolated, short-lived environment. Here's our architecture: Our Implementation Stack Infrastructure: Kubernetes (EKS) with namespace-per-PR Orchestration: Custom GitHub Action workflow Database: Isolated RDS instance per environment Contract Testing: Pact flow + OpenAPI validation Cleanup: AWS Lambda that runs every hour, destroying environments older than 2 hours The Game Changer: Automated Contract Testing The magic wasn't just in isolated environments—it was in what we did with them. Every time a PR deployed to its ephemeral environment, we ran: Consumer-Driven Contract Testing (Pact) Our mobile and web clients would verify their expectations against the actual deployed API. If a change broke what the client expected, the PR would fail. Provider Contract Validation We'd automatically verify that the deployed API matched ou

A browser test can be green and still be wrong. It can pass because a mock returned an outdated response. It can fail because staging enabled a feature flag that no one documented. It can become flaky after a React upgrade even though the user-facing behavior looks unchanged. And when the same failure appears only in a minified build, the stack trace may be so unhelpful that the team blames the test before investigating the application. These problems look unrelated, but they usually share one root cause: the test is running against a different system than the one the team thinks it is testing . The difference may be configuration, data, rendering behavior, build output, infrastructure, or timing. Reliable browser testing therefore requires more than stable selectors. It requires evidence that the environment, application state, and execution path are what you expect. Feature flags create multiple versions of the same application Feature flags are useful because they let teams release functionality gradually. They are also one of the easiest ways to create staging-only failures. A test written against the default interface may encounter a completely different component tree when a flag is enabled. A button can move into a menu, a form can become a wizard, or an API request can be delayed until the user completes an additional step. The difficult part is that the URL may remain identical. From the test runner's perspective, it is visiting the same page. From the application's perspective, it is executing a different product variant. A useful starting point is this breakdown of why browser tests fail only in staging when feature flags change runtime UI state . For important workflows, record the active flag state with every run. Do not limit the log to a generic environment name such as staging . Capture the actual configuration that influenced the UI. A failed run should answer questions such as: Which flags were active? Which account or cohort received them? Did the

The most important SaaS workflows rarely stay inside one clean browser tab. A user starts on the application, opens an OAuth popup, completes MFA, returns to the original tab, receives an email, follows a verification link, and lands on a different domain. Their account belongs to one tenant, their data is stored in a particular region, and their locale changes the date format that the test expected. Each step may work in isolation. The complete journey still fails. That is why testing SaaS applications requires more than a collection of page-level tests. The real risk lives in the handoffs between systems, identities, windows, tenants, regions, and communication channels. Authentication is a state machine, not a login form A simple login test usually covers one path: Enter email and password. Submit. Reach the dashboard. Real authentication has many branches: OAuth consent already granted. OAuth consent required. Popup blocked. Identity provider opens in a new tab. MFA requested. MFA remembered on the device. Session expired during the handoff. User belongs to multiple organizations. Original tab resumes before the token is available. Callback lands on the wrong environment. This review of testing OAuth popups, MFA prompts, and cross-tab login handoffs with Endtest highlights the operational difficulty of these flows. Model authentication as a state machine. Record the expected transitions and test the failure paths between them. For example: Unauthenticated → OAuth opened → Provider authenticated → Callback received → Session created → Tenant selected → Application ready A test that only checks the last page cannot tell you where the handoff failed. Capture: Current window and newly opened windows. Redirect URLs and callback parameters. Cookie and storage changes. Network failures during token exchange. Visible provider errors. The tenant selected after authentication. Whether the original tab updates automatically or requires refresh. Multi-tenant testing must pr

AI-powered web applications look familiar on the surface. They have text boxes, buttons, menus, loading indicators, and API calls. That makes it tempting to test them like any other web application: submit an input, wait for a response, and compare the output with an expected string. That approach breaks quickly. Model output is variable. Safety behavior depends on context. A response can be semantically correct but displayed in the wrong conversation. An agent can produce a convincing final message after calling the wrong tool. A prompt-injection defense can block obvious attacks while failing when malicious instructions arrive through a webpage, document, image, or previous message. Testing these applications requires two kinds of evidence at the same time: Deterministic product evidence: the UI, state, permissions, tool calls, and workflow behaved correctly. Probabilistic model evidence: the output stayed within an acceptable range across repeated and adversarial inputs. Prompt injection is a workflow problem Prompt injection testing is often reduced to pasting “ignore previous instructions” into a chat box. That is a useful smoke test, but it does not represent how browser-based agents encounter untrusted content. An agent may read instructions from: A webpage. A support ticket. A PDF. A hidden DOM node. An email. A retrieved knowledge-base entry. A tool response. A previous conversation turn. The guide on testing prompt injection defenses in AI-powered browser workflows provides a good foundation. The test should verify more than the final sentence. It should inspect whether the agent: Treated external content as data rather than authority. Attempted a prohibited tool call. Exposed secrets in an intermediate step. Navigated to an unapproved domain. Changed its goal after reading untrusted content. Requested confirmation before a sensitive action. Preserved the original user instruction. A safe final answer does not prove that the workflow was safe. The agent ma

Frontend testing used to sound simple: open a page, find an element, click it, and verify the result. That description still works for basic workflows, but modern interfaces are no longer a single static DOM that changes in obvious ways. Components can render inside Shadow DOM. Modals can be portaled to a different part of the document. Server-rendered HTML can be replaced during hydration. Content can move because of CSS container queries. A page can look finished while several progressive-loading states are still changing underneath it. The hardest frontend bugs now tend to sit at the intersection of three things: State: what the application believes is happening. Timing: when the browser and framework apply changes. Geometry: where elements appear and whether users can actually interact with them. A stable test strategy has to observe all three. Shadow DOM and portals break naive assumptions about element location Component encapsulation is useful, but it changes how automation finds and interacts with elements. A control inside an open shadow root is not always reachable through the same selector strategy used for the main document. A portaled modal may appear visually next to a component while being rendered near the end of document.body . Focus can move into the modal even though the DOM hierarchy suggests that it belongs elsewhere. This guide to testing Shadow DOM and portaled modals without breaking browser automation suites covers the key challenges. The test should verify behavior, not merely the presence of a node: Can the user reach the control? Is the expected element visible above overlays? Does keyboard focus enter the modal? Is focus trapped correctly? Does Escape close it? Does focus return to the triggering element? Can a screen reader identify its label and role? Selectors still matter, but interaction boundaries matter more. A test that locates a button hidden behind another layer is not testing what the user experiences. Hydration creates a peri

I've been tuning Codenames AI , a small web game where an LLM plays Codenames with you. Clue generation is tightly constrained: one word, a count, optional intended targets, JSON on the wire, then deterministic validation before anything reaches the board. As the project started attracting regular players, I wanted to improve the gameplay experience without blowing out costs. Moving one model generation from gpt-4o-mini to gpt-5-mini was my first instinct. The default reasoning setting made responses an order of magnitude slower for this workload. Minimal reasoning looked like the obvious compromise: newer model, responsive gameplay. I expected to compare clue quality, latency, and cost while the surrounding prompt, validator, and consumer contracts stayed put. That last part was wrong. The experiment stopped behaving like an A/B test What showed up was structural, and it showed up in places that had been stable for months. Validation failures started rising. Retries started rising. Entire candidate batches started failing before the game ever saw a clue. The sharpest signal came from a clue-selection path that had run untouched for months, and it hard-failed for the first time. They weren't latency regressions so much as architectural ones. It is easy to read that as "minimal reasoning made the model worse." More often, the failures were exposing gaps in contracts that had looked fine under the previous model. What each failure actually invalidated Eventually every failure traced back to one of three layers: Prompt contracts ask for exactly count targets and, in batch mode, several distinct candidates. Deterministic validators reject target/count mismatches and filter invalid candidates before anything downstream runs. Downstream consumers only see survivors. Empty batches retry with rejection feedback, then fall back if needed. Those layers share one job: enforce the same invariants. The failures below cut across all three rather than mapping one to one. Side comm

Overview lead-quorum was a strong pilot repo because it was small enough to reason about and real enough to fail honestly. It has: repo-local Python environment ownership pinned dependency installation env bootstrap from example truth a deterministic local test surface live external verification a local web runtime a distributed demo path a Docker build lane That is exactly the kind of repo where a contract can look clean while still hiding real setup and execution drift. Why this repo mattered The useful pressure here was not “can Ota run one Python command.” The useful pressure was whether Ota could stay truthful when the repo itself owns: the .venv the dependency install lane the local executable path the runtime listener truth If Ota probes or fulfills those in the wrong order, the contract is not trustworthy even if the repo itself is valid. That is what made lead-quorum valuable. What the contract now models The final contract is explicit about the repo’s real setup split. Setup is not one opaque shell step. It is three different ownership surfaces: copy .env from .env.example only if missing create the repo-local virtual environment hydrate dependencies through typed uv requirements-file installation That looks like this in the contract: setup : aggregate : tasks : - setup:env - setup:venv - setup:deps setup:env : action : kind : copy_if_missing from : .env.example to : .env setup:venv : action : kind : ensure_virtualenv path : .venv python : " 3.12" setup:deps : prepare : kind : dependency_hydration medium : package_dependencies source : kind : uv cwd : . mode : pip_requirements requirements_file : requirements.txt The contract also keeps verification and external-runtime claims separate: verify for deterministic local validation live for Gemini-backed end-to-end testing app for the local web service distributed for the A2A demo path That matters because a working local scoring test and a live distributed runtime are not the same readiness claim. What lead-q

When building Cypress automation that interacts with AWS services, the first step is verifying that your test framework can successfully authenticate and communicate with your AWS account. In this article, you'll learn how to connect Cypress to AWS and perform a simple authentication test using AWS Security Token Service (STS) and the GetCallerIdentity API. This approach helps confirm that: AWS credentials are correctly configured. Cypress can invoke AWS SDK operations through Node.js tasks. The automation environment is connected to the expected AWS account. Establishing this connection first provides a solid foundation before automating interactions with services such as AWS Lambda, Amazon S3, Amazon DynamoDB, Amazon SNS, or Amazon SQS. Prerequisites Before getting started, ensure you have: Node.js installed A Cypress project Valid AWS credentials: AWS Access Key ID AWS Secret Access Key AWS Session Token (if using temporary credentials) AWS Region Note:If you're running Cypress in an AWS environment (such as AWS CodeBuild, an EC2 instance with an IAM role, or GitHub Actions using OpenID Connect), you may not need to provide credentials manually. The AWS SDK can automatically retrieve credentials from the execution environment. Step 1: Install the AWS SDK Install the AWS STS client package: npm install @aws-sdk/client-sts For this connectivity test, we only need the AWS Security Token Service (STS) client. The package provides: STSClient – Creates a client for communicating with AWS STS. GetCallerIdentityCommand – Returns details about the authenticated AWS identity associated with the configured credentials. Step 2: Configure AWS Credentials For local development, create or update your cypress.env.json file. { "AWS_ACCESS_KEY_ID" : "" , "AWS_SECRET_ACCESS_KEY" : "" , "AWS_SESSION_TOKEN" : "" , "AWS_REGION" : "" } Populate the file with your AWS credentials. Example: { "AWS_ACCESS_KEY_ID" : "your-access-key" , "AWS_SECRET_ACCESS_KEY" : "your-secret-key" , "AWS_SES

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. Model task state separately from model result task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure_error | missing Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task_id" : "repo-issue-17" , "dataset_revision" : "sha256:..." , "harness_revision" : "git:..." , "model_config" : "immutable-config-id" , "validity" : "disputed" , "result" : "pass" , "review_revision" : 3 , "evidence" : [ "fixture.log" , "review.json" ] } Publish three denominators Let: P_v , N_v : passes and total among reviewed-valid tasks; P_a , N_a : passes and total across all attempted tasks; D : disputed tasks. Report: valid-only score = P_v / N_v all-attempted score = P_a / N_a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. A tiny sensitivity calculator #!/usr/bin/env python3 import json , sys rows = [ json . loads ( line ) for line in open ( sys . argv [ 1 ]) if line . strip ()] valid = [ r for r in rows if r [ " validity " ] == " valid " ] disputed = [ r for r in rows if r [ " validity " ] in ( " unreviewed " , " disputed " )] attempted = [ r for r in rows if r [ " result " ] in ( " pass " , " fail " )] rate = lambda passed , total : pa

OpenAI introduced GPT-Live on July 8, 2026 and describes it as a full-duplex voice architecture: it can listen and speak at the same time. GPT-Live-1 and GPT-Live-1 mini are rolling out in ChatGPT Voice, while API availability was described as coming later. Primary source: OpenAI, “Introducing GPT-Live” . Full duplex changes the mobile failure surface. The app may hold microphone input, play output, detect interruptions, and continue background work concurrently. A polished desk demo does not answer what happens when a phone call arrives, Bluetooth disconnects, the app backgrounds, or permission changes. This is the test plan I would run before shipping a full-duplex voice workflow. It is a plan, not a report of measured GPT-Live API behavior. Record the environment device : " physical device model" os : " exact OS version" app_build : " immutable build" voice_model : " product/model label visible during test" audio_route : " speaker | wired | bluetooth" network : " wifi | 5g | constrained" battery_start : " percent" low_power_mode : false microphone_permission : granted background_permission : " record actual setting" A result without device, route, and network context is difficult to reproduce. Define observable states idle -> connecting -> listening <-> speaking -> reconnecting -> paused_by_system -> ended -> error Track three channels separately: audio input — is the microphone active? audio output — what route is speaking? task state — is a delegated/background operation still running? The UI should never imply “listening” after the OS revoked microphone access. Interruption sequence Use a harmless scripted conversation and timestamp every event: T+00 connect on phone speaker T+10 user speaks while assistant is speaking T+20 switch to Bluetooth headset T+35 background the app for 30 seconds T+65 return to foreground T+80 trigger an incoming call or system audio interruption T+95 decline/end interruption T+110 disable microphone permission in Settings T+130 retu