标签：#test

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

Most teams already keep Playwright or Robot Framework specs next to the product. Manual cases often live somewhere else — a spreadsheet, a cloud TMS, or a wiki page nobody updates after the first release. That is why automation coverage is usually a quarterly guess: the catalog and the specs never share an id. Put both in the same Git repo , give every manual case a stable id, and linking becomes a text match. Any IDE agent can do that if the cases are files it can already see — no custom AI product inside a test tool. One repo, two layers of the same suite Manual cases are YAML files under .gitoza-lite/test/cases/ (VS Code / Cursor extension) or .gitoza/test/cases/ (Desktop app). The filename without .yaml is the case id . Automated tests live wherever your team already puts them — tests/ , e2e/ , robot/ — in the same repository. Traceability is a shared id: put that case id on the automation side as a tag (or name), and on the YAML side as automated: true plus a params pointer. A PR can change the case, the Playwright spec, and the link in one review. Step 1 — Draft manual cases in the IDE Open the repo in Cursor or VS Code. Point the agent at a few existing case files so it learns your title style, tags, and step length. Then feed it a user story, a ticket, or a screenshot. Ask for several cases at once, not one shallow step. A useful prompt looks like: Read .gitoza-lite/test/cases/shopflow/auth/ for format. From ticket SHOP-184, draft three YAML cases (happy path, invalid password, locked account). Filename = case id. Use tags auth and smoke where it fits. Save the files under the suite folder. Then open the Gitoza Lite Test Repository tab to browse, edit, and — when you are ready — run them as a manual suite with Pass / Fail / Skip. The extension does not ship its own model. It keeps cases as plain YAML so whatever assistant you already use can read and write them. A minimal case: --- title : Login with valid credentials priority : high tags : [ smoke , auth ]

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time. The Problem Everyone Has Your test suite runs for 45 minutes. You change one file. Should you: Run all 2,000 tests? (Safe but slow) Run tests with matching filenames? (Fast but broken) Manually tag tests? (Gets stale immediately) We tried all three. They all failed. The Insight: Stop Guessing, Start Measuring What if we just measured what each test actually executes? Coverage tools have done this for years: { "src/MyComponent.js" : { "statements" : { "0" : 5 , "1" : 12 , "2" : 0 } } } Key insight: Coverage tools report per test run, not per test file. All we needed: save coverage after each individual test. The Architecture (5 Steps) Step 1: Instrument the Code // Build config (webpack/vite/rollup/etc) export default { plugins : [ process . env . COLLECT_COVERAGE && coveragePlugin ({ include : ' src/**/* ' , exclude : [ ' **/*.test.* ' ] }) ] } Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java) Step 2: Collect Coverage Per Test // Test framework afterEach hook afterEach ( async () => { const coverage = getCoverageData (); saveCoverage ( `coverage- ${ testName } .json` , coverage ); }); Step 3: Build the Reverse Map const map = {} ; for (const coverageFile of allCoverageFiles()) { const testName = extractTestName(coverageFile); const coverage = JSON.parse(read(coverageFile)); map [ testName ] = [] ; for (const [ sourceFile , data ] of Object.entries(coverage)) { if (wasExecuted(data)) { map [ testName ] .push(sourceFile); } } } Example output: { "tests/user-profile.test.js" : [ "src/components/Profile.jsx" , "src/components/Card.jsx" , "src/utils/formatters.js" ] } The magic: The test told us what it executed. No parsing, no guessing. Step 4: Detect Tests CHANGED = $( git diff origin/main...HEAD --name-only ) TESTS = $( lookupTestsInMap " $CHANGED " ) npm test $TESTS Step 5: Auto-Up

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why. That trust is a liability. An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness. Here is how to build one. The Problem: Self-Reported Truth Is Not Truth Most teams validate agent behavior by reading the agent's own output. They check the tool_calls field in the response, match it against an expected schema, and call it done. This works until it doesn't. Consider a common failure mode: the agent decides to call search_knowledge_base but the LLM formats the tool name as searchKnowledgeBase . The routing layer silently normalizes it, the call succeeds, and the agent logs search_knowledge_base . Your test passes. The actual execution path was different from what you verified. Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift. The root cause is the same. You are testing the agent's intent , not its execution . Intent is cheap to fake. Execution leaves fingerprints. The Solution: An External Observer You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched. The observer does not trust the agent's logs. It trusts what it sees on the wire. Here is the architecture at a high level: Intercept every outbound call from the agent to a tool. Record the raw request before any normalization or routing. Compare the recorded call against

Agent Determinism Illusions (Part 6) Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed. Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification. But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline. §§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place. 1. Alexey Spinov & Manuel Bruña: Layer Before You Judge Alexey's comment identified the most fundamental design flaw in my experiments: "G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM." Manuel added the constructive direction: "Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual." I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4): ID Output Type Could code catch it? G1 "I am a little duck, quack quack" nonsense ✅ very short + no keywords G2 "。" (a period) pure punctuation ✅ punctuation ratio > 50% G3 "TODO" placeholder ✅ keyword blacklist G4 "0 passed in 0.00s (no tests collected)" zero-test pass ✅ regex 0 passed + no tests All fo

Overview Open WebUI exposed a real Ota lifecycle boundary. This was not mainly a parsing or contract-shape repo. The contract was already strong enough to model: source-checkout verification packaged native runtime through uv run open-webui serve frontend development runtime default Docker Compose runtime What the repo exposed was operational truth after proof: a successful native proof still left a host workload alive the first cleanup fix then widened too far and treated a Compose-owned runtime as the same class of host workload That made Open WebUI a valuable pressure repo. It forced Ota to get more precise about cleanup ownership instead of treating all successful runtime proof as one generic teardown problem. The current pressure contract pins released Ota v1.6.24 . Its latest green matrix run proves the release surface at the exact contract and workflow revision linked below. What Open WebUI exposed in Ota This repo exposed four meaningful weaknesses. 1. proof success was weaker than it looked The first issue was not that runtime proof failed. It was that runtime proof succeeded and still left the native workload alive afterward. In this repo, the packaged native workflow launches: serve:native : launch : kind : command exe : uv args : - run - open-webui - serve - --host - 0.0.0.0 - --port - " 8080" Ota proved that workflow, but the launched process tree was still alive after proof completed. In GitHub Actions, that surfaced through setup-uv post-job cleanup, which blocked while the uv cache was still in use. That was an Ota gap. If proof succeeds but leaves behind repo-owned runtime state that later breaks CI cleanup, the proof surface is still incomplete. 2. native service cleanup widened past its real ownership boundary The first core fix made Ota clean selected native service workloads after successful proof. That was directionally correct, but Open WebUI immediately exposed the next boundary. The Docker workflow uses a native task shape to launch Compose:

You have been here. The test suite is green. The deployment pipeline reports all checks passed. Then a customer opens a ticket with a screenshot that shows something your test never caught. The test passed in staging. It fails in production. And you cannot reproduce it locally. This is not a flaky test problem. It is a fidelity problem. Your test environment and your production environment are not the same thing. The gap between them is where real bugs live. Let me walk through one concrete example, the fix, and what it teaches about writing tests that survive the handoff to a real user. The Problem: Environment Drift A fintech team I worked with had a checkout flow. The test clicked "Pay Now", waited for a success message, and asserted the text "Payment successful" appeared on screen. It passed every time in staging. Customers reported that after paying, they saw a blank white page for several seconds before the success message appeared. Some of them closed the tab during that blank period, thinking the payment failed. The transaction went through. The customer never saw the confirmation. Support tickets piled up. The test never caught this because the staging environment served the success page in under 200 milliseconds. The blank period did not exist there. Production had a slower downstream service that introduced a three-second delay between the payment confirmation and the page render. The test was correct in what it checked. It was wrong in what it assumed about timing and state. The Fix: Test the Experience, Not Just the Outcome The fix was not to add a longer wait. The fix was to test what the user actually experiences during that gap. Here is a minimal Playwright test in TypeScript that catches this class of problem: import { test , expect } from ' @playwright/test ' ; test ( ' checkout shows loading state before success ' , async ({ page }) => { await page . goto ( ' /checkout ' ); await page . fill ( ' #card-number ' , ' 4111111111111111 ' ); await page

Gherkin is not a test automation tool. It never was. Yet here we are, five years into your SDET career, and you're staring at a feature file that reads like a step-by-step manual for a human tester. Given I log in with username "admin" and password "password123" . When I click the "Submit" button . Then I see the text "Welcome" on the screen . You've written two years of these. Your team calls it BDD. Your manager calls it "living documentation." And somewhere in the back of your mind, a quiet voice whispers: This is just a manual test case with extra steps. That voice is right. Let me say it plainly: if your Gherkin scenarios describe how the system works instead of what it should do, you are not doing BDD. You are writing manual test cases in a structured English format and calling it automation. The only thing you've automated is the illusion of progress. The problem isn't Gherkin. The problem is how we use it. Most teams adopt BDD because someone read a blog post about "collaboration" and "shared understanding." They install Cucumber or SpecFlow. They write feature files. They map steps to Selenium or Playwright code. And they call it a day. But look closely at what happens next. The product owner never reads the feature files. The developer skims them once and goes back to writing code. The QA engineer — that's you — becomes the sole maintainer of a growing pile of Gherkin that nobody else touches. You're not facilitating collaboration. You're translating manual test cases into a format that requires a compiler. Here's the real test. Take any feature file from your project. Hand it to a developer who has never seen it. Ask them to implement the feature using only the Gherkin as a spec. If they can write production code from it, you have real BDD. If they ask you for clarification, you have documentation theater. I've seen teams with hundreds of feature files. Beautifully formatted. Perfect indentation. Tags for every regression cycle. And not a single one of th

Most engineers expect AI-assisted automation to be the easy part. You describe a test, the model writes it, you move on. The first week will prove you wrong. Not because the code is bad. Because the code is almost right. And almost-right code is harder to debug than wrong code. Wrong code fails loudly. Almost-right code passes on Monday, fails on Tuesday, passes again on Wednesday, and by Thursday you are questioning whether you understand your own application. I have watched teams adopt AI copilots into their Playwright suites and spend the first five days doing nothing but untangling false passes. If you are about to start this journey, here is what that week actually looks like. The Problem: The Model Does Not Know What "Stable" Means A language model has never waited for a network response. It has never watched a flaky selector survive three CI runs and then collapse on the fourth. It writes tests from a static understanding of your page, not from the dynamic reality of your application. You will ask it to write a test that clicks a button and waits for a confirmation toast. The model will produce something like this: await page . click ( ' button:has-text("Submit") ' ); await page . waitForSelector ( ' .toast-success ' ); Looks fine. Runs fine. Then your team deploys a new build where the toast takes 400ms longer to appear because of an analytics call. The test fails. Not because the feature broke. Because the model assumed a timing that was never guaranteed. This is the core problem. The model writes tests that match the page as it was when the model saw it . It does not write tests that match the page as it will be . The Solution: Treat AI-Generated Tests as Drafts, Not Deliverables The shift is mental before it is technical. You cannot review AI-generated tests the way you review human-written tests. Human tests come with intent. AI tests come with patterns. You need a different review lens. First, look for every hardcoded wait. Replace it with a state-based

Cursor's "Sand" project targets non-developers. Anthropic's Claude Cowork runs cross-device. OpenAI's ChatGPT Work delivers documents, spreadsheets, and presentations. When an agent produces a deliverable for someone who cannot read the code, the verification interface matters more than the output format. A download button is not verification. The problem When a developer reviews agent output, they can read the diff, run the tests, and check the types. When a non-developer receives an agent-produced spreadsheet or document, they have no equivalent verification path. They either trust it completely or reject it completely. Neither is useful. What a verification UI needs Element Purpose Implementation Source list Show what inputs the agent used Links to source documents with timestamps Confidence indicator Flag low-confidence outputs Per-section confidence derived from source coverage Change highlights Show what the agent generated vs. copied Diff view between source and output Audit trail Record who requested what and when Append-only log with request ID, timestamp, and result hash Rejection path Let the user say "this is wrong" without starting over Feedback record linked to specific output section A minimal verification component <!-- agent-output-verification.html --> <!DOCTYPE html> <html lang= "en" > <head> <meta charset= "UTF-8" > <title> Agent Output Verification </title> <style> .verification-card { border : 1px solid #ddd ; border-radius : 8px ; padding : 16px ; margin : 8px 0 ; font-family : system-ui , sans-serif ; } .source-list { list-style : none ; padding : 0 ; } .source-list li { padding : 4px 0 ; border-bottom : 1px solid #eee ; } .confidence-low { color : #c0392b ; font-weight : bold ; } .confidence-medium { color : #e67e22 ; } .confidence-high { color : #27ae60 ; } .reject-btn { background : #fff ; border : 1px solid #c0392b ; color : #c0392b ; padding : 4px 12px ; border-radius : 4px ; cursor : pointer ; } .reject-btn :hover { background : #c0392b

Tu prueba pasó el lunes: misma entrada, mismo código y temperature=0 . El martes falló sin cambios en tu aplicación. La aserción esperaba una cadena exacta, pero el modelo devolvió la misma respuesta con una redacción ligeramente distinta. El agente funciona; tu suite de pruebas no. Prueba Apidog hoy Este es el coste de probar sistemas que llaman a modelos de lenguaje. Incluso con temperatura cero, no obtendrás salidas idénticas byte a byte entre ejecuciones. En lugar de probar la redacción exacta, prueba el contrato: estructura, campos, rangos y efectos esperados. Este artículo profundiza en el tercer modo de fallo de nuestra guía sobre por qué los agentes de IA fallan en producción . Por qué temperature=0 no significa determinismo La temperatura controla cómo el modelo selecciona el siguiente token. Con temperature=0 , el modelo elige el token más probable, lo que parece reproducible. Sin embargo, esa configuración no garantiza resultados idénticos. La causa está debajo del modelo: Las operaciones de punto flotante en GPU no son asociativas: el orden de las sumas puede modificar mínimamente el resultado. Esa diferencia puede cambiar cuál token queda en primer lugar. El orden de ejecución puede depender de cómo el proveedor agrupa solicitudes, del hardware, de la región o de la versión del kernel. Los proveedores pueden cambiar GPUs, bibliotecas de inferencia o cuantización de pesos. Una discusión extensa en vLLM explica por qué una semilla fija y temperature=0 no bastan para lograr reproducibilidad bit a bit. La conclusión práctica es simple: el determinismo es una propiedad de toda la pila de servicio, no una opción de tu solicitud. Tu prueba debe aceptar variaciones de redacción cuando el significado y el contrato siguen siendo correctos. Por qué las aserciones de cadena exacta vuelven inestable tu suite Esta prueba es frágil: assert response == " Your order total is $42.00. " Puede pasar hoy y fallar mañana si el modelo responde: Your total comes to $42.00. La

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite

A helpful comment on my EXIF test suggested using a metadata scrubber. That is useful for ordinary still images, but a mobile upload contract must define every asset it sends. An iOS Live Photo can include both a photo resource and a paired video resource. The failure case is simple: the JPEG derivative has no GPS EXIF, while metadata or an original file associated with the paired video survives in the upload or retry path. Build the resource inventory first: let resources = PHAssetResource . assetResources ( for : asset ) for resource in resources { print ( resource . type , resource . originalFilename ) } Then make privacy assertions per output, not per UI selection: Artifact Required check still derivative no EXIF GPS/location fields paired video derivative no location metadata upload manifest only derived filenames retry queue no path to original resources temporary directory deleted after success or cancellation My lifecycle test would start an upload, force the app into the background after the still image is prepared, kill it while the paired video is processing, and relaunch. Recovery must either regenerate both safe derivatives or delete the incomplete pair. It must never mix a scrubbed still with an original video. Apple’s PHAssetResource API exposes the resources associated with a Photos asset. That enumeration should become evidence in the test: fail when an unexpected resource type is present rather than silently uploading it. Also verify what reaches the server. Device-side inspection alone misses multipart manifests, queued originals, and server-generated previews. Download the stored pair in a test environment and run the same metadata assertions again. A “remove location” button needs a precise scope. For Live Photos, the user reasonably expects it to cover the complete paired asset and every retry copy—not just the JPEG they can see.

A saga test that stops after “payment succeeded, inventory failed, refund called” assumes compensation is reliable. It is another distributed operation, so test it under the same faults. Use this sequence: 1. payment capture succeeds 2. inventory reservation times out 3. refund succeeds at provider 4. refund response is lost 5. compensation message is delivered again 6. late inventory-failed event arrives again The invariant is not “refund endpoint called once.” It is: captured amount - confirmed refunded amount = final charged amount and final charged amount is never negative Persist an inbox record for consumed message IDs and an outbox record for each intended side effect. Give the provider request a stable idempotency key derived from saga and compensation step: { "sagaId" : "order-42" , "step" : "refund-payment-v1" , "idempotencyKey" : "order-42:refund-payment:v1" , "amount" : 4900 } A deterministic simulator should permute duplicate delivery, delayed acknowledgement, worker crash, and out-of-order events. After every run, assert one terminal order state, at most one economic refund, and a complete evidence trail. “Already refunded” must reconcile to success only after amount and payment identity match. AWS describes coordination choices and rollback behavior in its Saga pattern guidance . The implementation detail that deserves its own test is durable compensation progress: a process restart cannot erase whether the external side effect occurred. Alert on sagas stuck in compensating , but do not let an operator click “retry” with a new key. The runbook should first query provider state, compare amount and currency, and resume the same operation identity. Exactly-once delivery is not required to preserve money. Stable operation identity plus reconciliation is.

My upload CLI looked restart-safe until I tested two failures together: the process crashed at 63%, and the signed resume URL expired before restart. The checkpoint needs identity, not credentials: { "file" : "release.tar.gz" , "fingerprint" : "sha256:..." , "uploadId" : "up_123" , "localOffset" : 66060288 , "updatedAt" : "2026-07-19T12:00:00Z" } Do not persist the signed URL. On restart, exchange the durable upload ID for fresh authorization, query the server offset, then reconcile: const remote = await headUpload ( freshUrl ); if ( remote > file . size ) throw new Error ( " invalid remote offset " ); if ( remote !== checkpoint . localOffset ) { await saveCheckpoint ({ ... checkpoint , localOffset : remote }); } await sendFrom ( file , remote , freshUrl ); The server offset wins because a crash can occur after bytes are accepted but before the local checkpoint is renamed. Save checkpoints through write-to-temp plus atomic rename so a partial JSON file cannot destroy recovery. My fixture matrix includes: Failure Expected behavior crash after remote commit rewind/advance to remote offset expired URL refresh without creating a second upload changed local file stop on fingerprint mismatch missing remote upload ask before starting over checkpoint write interrupted retain previous valid checkpoint The tus resumable upload protocol specifies offset discovery and conflict handling that are useful even if the service uses a smaller custom protocol. The key idea is explicit reconciliation, not assuming client memory is authoritative. I also shipped upload status , upload forget , and an exportable checkpoint directory. Recovery is a user-facing feature; if users cannot inspect or remove state, “resumable” becomes hidden lock-in.

Multipart finalization is not the only retry boundary. Abort can succeed in object storage while the HTTP response is lost, leaving your database convinced the upload is active. Model abort as a state transition, not a button wired directly to an SDK call: active → aborting → aborted └──→ cleanup_pending The endpoint should own an idempotency key and durable intent: await db . transaction ( async tx => { const upload = await tx . lockUpload ( uploadId ); if ( upload . state === " aborted " ) return ; await tx . markAborting ( uploadId , idempotencyKey ); await outbox . enqueue ( " abort-multipart " , { uploadId , storageKey }); }); A worker calls storage. If a retry receives NoSuchUpload , do not blindly fail: reconcile whether the upload was already completed, already aborted, or expired. Only the “already absent because abort succeeded” path may converge to aborted ; completion requires a separate terminal state. Test this sequence: Step Fault Required invariant persist abort intent process dies outbox resumes work storage abort succeeds response is lost retry does not reactivate upload duplicate message delivered twice one terminal state client retries same key same operation result cleanup scan stale active row reconcile before deleting Amazon S3’s AbortMultipartUpload API notes that in-progress part uploads may still succeed around an abort and recommends verifying that parts are gone. That makes a post-abort verification pass part of correctness, not optional housekeeping. Expose abort_requested_at , attempt count, last storage result, and remaining-part count. Alert on age in aborting , then run a lifecycle cleanup policy as a backstop—not as a substitute for application reconciliation.

Removing EXIF GPS protects one privacy boundary. It does not prove that an uploaded JPEG contains only a JPEG. A useful hostile fixture is a valid image followed by bytes for another format. Many decoders stop at JPEG’s end marker and display the picture successfully, while downstream scanners, content sniffers, or accidental downloads may interpret the trailing payload differently. Build a fixture set rather than trusting extensions: clean.jpg valid JPEG exif-gps.jpg valid JPEG with location jpeg-plus-zip.jpg JPEG followed by ZIP bytes jpeg-plus-html.jpg JPEG followed by HTML truncated.jpg missing end marker oversized-dimensions.jpg small file, dangerous decode cost My upload gate has independent layers: Store the original in a non-public quarantine location. Enforce byte-size and decoded-dimension limits. Decode with a maintained image library. Re-encode pixels into a new file, discarding metadata and trailing bytes. Verify the output’s signature, dimensions, and complete parse. Publish only the derived object under a server-generated name. A boundary check can detect bytes after the end marker, but re-encoding is the stronger transformation because it creates a new representation from decoded pixels. Keep the original inaccessible and delete it according to a tested lifecycle. assert ( outputSize <= limit ); assert ( decoded . width * decoded . height <= pixelBudget ); assert ( parseConsumesEntireFile ( output )); assert ( noLocationMetadata ( output )); The OWASP File Upload Cheat Sheet recommends defense in depth: allowlisted types, generated filenames, size limits, storage separation, and content validation. No single MIME header or metadata scrubber replaces that chain. The important privacy lesson is broader than EXIF: inspect every representation that survives the upload pipeline, and publish only a constrained derivative.

An accessible combobox can follow the correct ARIA pattern and still become unusable when two search responses arrive in the wrong order. Reproduce this sequence: Type ca , then quickly type cat . The cat response arrives first and highlights cat facts . The slower ca response arrives and replaces the list. aria-activedescendant now points to an option that no longer exists. IME input adds another boundary: searching during composition can send partial text the user has not committed. I would model the request generation explicitly: let generation = 0 ; let composing = false ; async function search ( query : string ) { const mine = ++ generation ; const options = await fetchOptions ( query ); if ( mine !== generation || composing ) return ; render ( options ); restoreActiveOptionByKey (); } The stable key matters. An array index cannot preserve the active option when ranking changes. Regression matrix Input Injected failure Expected evidence ca → cat first request delayed only cat results render Arrow Down result refresh active key survives or resets visibly Escape response arrives afterward popup stays closed IME composition network is fast no request until compositionend A Playwright test should assert focus remains on the input, every aria-activedescendant resolves to a live element, and Escape invalidates outstanding generations. A manual screen-reader pass should confirm result-count announcements are not emitted for discarded responses. The WAI-ARIA Authoring Practices combobox pattern defines the keyboard contract. The missing production step is testing that contract under asynchronous replacement, not only against static example data. Which stale-response failure has been hardest to reproduce in your search UI?

GenPage is a generative AI system developed by Netflix to replace its traditional multi-stage recommendation pipeline by directly generating personalized user homepages. GenPage leverages user history and request context as a prompt to produce the entire page, resulting in improved user engagement and reduced serving latency. By Sergio De Simone