标签：#Testing

Follow up to Part 1: How to Design AI Evaluations You Can Actually Trust At Google, we are...

Most data engineers write pipelines the way most people write shell scripts: run it, eyeball the output, ship it. That works right up until a schema changes upstream, a null slips through a join, or someone "fixes" a transformation and silently breaks three downstream tables. By then the bug isn't your problem anymore — it's a bad number in someone's dashboard. Software engineers solved this problem decades ago with automated testing. Data engineering has been slower to adopt the habit, partly because our code touches messy external reality (files, databases, clusters) in a way a typical web app doesn't. But that's exactly why testing matters more here, not less. This article is a practical, DE-flavored crash course in pytest — the dominant Python testing framework — plus the patterns you actually need for pandas, Polars, and PySpark pipelines. Why bother testing a data pipeline? A few concrete failure modes that tests catch before production does: A column gets renamed upstream and your join silently produces all-null matches instead of erroring. A "cleaning" function that's supposed to drop duplicates accidentally drops valid rows too. A date-parsing function works on your local machine's locale and breaks in the CI environment. A refactor changes an aggregation from sum to mean and nobody notices until finance asks why revenue looks 90% smaller. None of these require exotic testing techniques. They require the habit of writing small, deterministic checks against small, deterministic inputs — which is exactly what pytest is built for. Where pytest fits — and where it doesn't Before diving in, it's worth being precise about scope, because "testing a data pipeline" actually covers two different questions, and conflating them is a common source of confusion: Is my code correct? Given a known input, does the transformation logic produce the right output? This is a property of your code , and it doesn't change based on what day it is or what a source system decided to

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it. Some of the most misleading moments in building software happen when the page looks finished. The button is there. The layout is polished. The flow works in a test account. The code has been merged. It is very easy to look at that and think the product has moved forward. Then production reminds you that a product is larger than its frontend. I learned this repeatedly while building Eterna Clarity. A customer-facing change could depend on application code, a database function, authentication, storage rules, an email template, environment configuration and the way a demo account was isolated from real customer data. If one of those pieces stayed behind, the screenshot could be correct while the product was not. That changed the way I think about releases. A release is not “the code shipped.” A release is the smallest complete set of owned systems that have to advance together for the accepted behavior to become true in production. The browser can hide a lot of unfinished work Frontend work is unusually visible. That makes it easy to use as a proxy for progress. Back-end state is less visible. So are permissions, production configuration, storage policy, transactional email, tenant boundaries and data migrations. They tend to reveal themselves only when something goes wrong. That asymmetry can create a strange kind of false confidence. A team can spend hours polishing the thing a customer sees while the systems underneath it still describe an older product. In Eterna, the correction was to stop treating the repository as the whole release. Source code still matters. It is simply one owner among several. If a new customer flow requires a database change, the production database has to advance. If it requires a new authentication behavior, the production auth configuration has to advance. If it depends on storage permissions, those permissions have to exist in

A scheduled job of mine drives a real Chrome profile that stays signed in to DEV, because the API can read comments but cannot create them. One run came back with the dashboard replaced by the sign-in page: the log said it had opened https://dev.to/dashboard , and what it actually landed on was https://dev.to/magic_links/new , with zero links to my own profile anywhere in the DOM. The profile itself was fine. A probe against the debugging port at the same moment returned a live Chrome, and the dashboard fetched through that port rendered the account's own identity links normally. Two browsers, same machine, same minute, opposite answers. The three things worth checking first, and why they miss The session expired. That is the reflex, and it is also the one that makes you re-authenticate for no reason and burn the logged-in state you were trying to protect. Cookies got cleared by a Chrome update. Same family, same cost if you act on it. The debug port died and the tool fell back to something else. This one is close enough to be dangerous, because it names the right layer — which browser am I attached to — and then picks the wrong cause inside it. Where it actually goes wrong It is one argument. agent-browser attaches to an already-running Chrome when you pass --cdp <port> . Leave the flag off and it starts its own browser, with its own empty profile directory, and drives that one instead. Everything downstream still works — it navigates, waits, evaluates, returns a page. It just does all of that in a browser that has never logged in to anything. So the automation is not looking at an expired session. It is looking at a different browser's logged-out session, and reporting it in exactly the shape a real logout would take. The two failure modes do not look alike, and that is the trap Here is what I measured today, on Chrome 152.0.7977.65 with the current npx build. Pass the flag, but point it at a port nothing is listening on: npx -y agent-browser open "https://dev.to/

Originally published on hexisteme notes . I run a video pipeline where every shot carries a contract: a must_have list, a must_not_have list, and a serves_line field — the narration line the shot exists to support. One rule in that pipeline, which I call the enactment rule, checks each contract for self-contradiction: if must_not_have forbids the very thing serves_line requires, the shot can't honor its own contract and support its sentence at the same time, so the rule fails it hard at author time. It's a good rule to want. It had never once run. One layer deeper than the usual dead gate This adds another entry to the same line of investigation: Grep won't find your dead gates. A fill-rate query will. found rules nobody was calling — the wiring itself was absent; The Guard Passed on an Empty Table found a correctly-called rule whose target population was empty at runtime, so it passed on nothing left to check; and The check that cannot fire found a rule that ran against real data and silently disabled itself because its threshold was an absolute constant that didn't match the input's scale. This one sits a layer deeper than all three: the callers existed, the rule fired on every invocation, and what was actually missing was a producer for the one field the rule needed to read. The field nothing writes serves_line is a derived field — by definition, it's the narration of the beat the shot is attached to. I checked that derivation against reality before touching anything else: every stored value that existed matched its beat's narration exactly, 6 of 6, zero drift. Wherever the field existed, it was right. But almost nowhere did it exist. A repo-wide search for anything that writes serves_line returned zero producers. The archetype templates that generate contracts don't set it. So every contract created through the supported authoring path had serves_line = None , and the rule — which needs that field as its input — silently evaluated nothing. A published episode ha

We’re a small team building Sealshot, a free and open-source screenshot app for macOS. We started working on it because screenshots often become disposable files. We use them for bug reports, QA, documentation, support, and security work, but later they can be hard to find or reuse. They can also accidentally contain sensitive information such as emails, API keys, tokens, internal URLs, or customer data. Sealshot is built around a simple idea: Treat screenshots more like documents than temporary images. It supports: region, window, and scrolling capture screen recording editable annotations OCR and searchable screenshot archives sensitive information detection before sharing encrypted local storage local metadata generation Everything is processed locally on the Mac. It’s open source and free, and we’re still actively improving it. We’d really appreciate feedback, especially from developers, QA engineers, support teams, and people working in security. Website: https://seal-shot.com/ GitHub: https://github.com/ldeng83/Sealshot

Spent the past week wiring up CI and tightening a few decisions on a backend project. Nothing dramatic happened, but a few things stood out enough to write down. CI is worth setting up early, even on a solo project. First run caught a dependency that worked locally but was never actually declared in requirements.txt . Classic "works on my machine" gap. CI doesn't care what your machine has installed, only what the project actually declares, and that mismatch is exactly the kind of thing that's invisible until something forces the comparison. Pinned dependencies drift more easily than people expect. I had a specific package version pinned for a known compatibility issue, and a later, unrelated install silently bumped it past that pin. Caught it by chance during a review, not because anything alerted me. Worth adding an explicit check for that instead of relying on remembering. 404 over 403 for resources that belong to another user is a small choice with real weight. 403 confirms something exists and you're just not allowed to see it. 404 gives nothing away. Costs a bit of clarity for legitimate callers debugging their own mistakes, but that's a fair trade for not leaking what exists in the system. None of this is complicated. All of it is easy to miss quietly, and only shows up if something is actually checking. That's most of what good backend hygiene turns out to be.

Falsifier first: if you can find a fourth production call site that builds a Transformation and reconstructs its target field differently from the three I'm about to describe, this post is wrong about "all of them." I counted by grepping for the one function that computes a transformation's identity and checking every call site by hand. Three. If there's a fourth, the bug I'm describing isn't fully fixed. Here's the shape of it. Engine::plan_shape has a doc comment that says, more or less, "this isn't a second place where transformation identity gets defined, because it's the same code as the one true place." That claim was false, and it had been false since the field it's talking about was added. The actual second place was Engine::rehydrate_committed . Its job is to rebuild a Transformation from the journal when a fresh CLI process needs to undo something a previous process committed. Every gx undo call from a cold process goes through it. And for one field, target , it wasn't rebuilding anything. It wrote a hardcoded placeholder. Nobody noticed, because nothing disagreed with the placeholder. Every adapter shipping at the time also produced the placeholder for that field, by omission rather than by design, so the two sides matched by coincidence. A missing value that's always missing on both sides of a comparison is invisible. cargo check doesn't catch it because the type is Option<T> and None is a completely legal value of that type. Nothing was wrong, until something else became right. What made it right was landing the two adapters that finally do predict target , fs and git, so their production plan() calls started filling in the real value instead of leaving it empty. The moment that shipped, cold-process undo broke for every fs or git transformation: gx_code=INTERNAL detail="TransformationId(...) is Committed, and 43 §3 has no `rehydrate: the rebuilt transformation names another id, so the intent supplied is not the one this transformation was planned from`

I spent last weekend building a JSON toolkit in Rust under one rule: no third-party dependencies . Not "few". None. The [dependencies] table in Cargo.toml is present and empty, and Cargo.lock holds exactly one package — the project itself. No serde , No serde_json , No clap , No itoa , No ryu . That constraint is the premise of the Zero Dependency hackathon , and it is a good premise, because it forces you to find out what the standard library actually promises. Here is the thing I did not expect to find: About 10% of the JSON documents that RFC 8259 says a parser must reject are accepted by Rust's own number parser. Not a subtle 10%. NaN , Infinity , .5 , 5. , +1 and 012 are all invalid JSON, and f64::from_str and i64::from_str take every one of them. If you write a JSON parser the obvious way — scan to the end of the number token, hand the slice to from_str — you ship a parser that is silently non-conformant, with no warning anywhere. I have the number because I counted it against a real corpus before writing the parser. The rest of this post is what that measurement did to the design, and what generalizes to languages that are not Rust. What I built jaq-lite is a hand-rolled RFC 8259 parser, a serializer, and a jq-style query CLI with rustc -style caret diagnostics — 4,670 lines under src/ and 4,432 lines of tests, standard library only. $ echo '{"users":[{"name":"ada","age":36},{"name":"linus","age":54}]}' | jaq-lite '.users[] | .name' "ada" "linus" It supports identity, field access, quoted fields, indexes, iteration, pipes, commas, parentheses, the optional operator ? , and eleven builtins ( length , keys , keys_unsorted , type , to_entries , from_entries , flatten , first , last , reverse , not ). Exit codes follow jq: 2 for a bad flag, 3 for a filter that does not compile, 5 for input that is not JSON, 0 otherwise. The measurement JSONTestSuite is the standard conformance corpus: 318 files in test_parsing/ , named by what a parser is supposed to do with them

Agent Plugins 1.0.0 ships a JSON Schema for plugin.json . It sets additionalProperties: false . So the obvious loader is four lines: const manifest = JSON . parse ( await readFile ( join ( dir , ' plugin.json ' ))); if ( ! validate ( manifest )) return reject ( ' invalid manifest ' ); That loader is wrong, and the specification says so in a sentence most people never reach. §5.2: Clients MUST report and ignore each unknown field and MUST continue loading the plugin if the manifest otherwise satisfies this section. An unknown top-level field is a schema violation you have to tolerate . §8.1 says the same for an extensions field that isn't an object. Every other schema violation is fatal. So a validator gives you one boolean where the spec wants three different outcomes, and the natural implementation is non-conformant in exactly two cases and correct everywhere else. That is the kind of bug that doesn't show up in your tests. It shows up as a plugin that works in one client and not another, six months later, in someone else's bug tracker. This has already happened, repeatedly I went looking before building anything. In the last few months: Codex loaded any directory with a root plugin.json through its Agent Plugins loader, which had no hook support. Every hook in .codex-plugin/plugin.json silently stopped running. Two plugins were dead for a week before anyone noticed. oh-my-pi routed packages declaring an agent-plugins.org $schema to a strict provider that dropped any SKILL.md with an extra frontmatter key. Downstream, a plugin went from 33 skills to 3. The fix was to delete $schema from the manifest, so conforming to the standard cost them the standard. dotnet/skills shipped manifests with no $schema and with skills , agents and mcpServers as top-level fields. Kiro refused them. Adding $schema got past the rejection and then loaded the package with every functional component excluded. VS Code , the largest shipping client, has no validation surface at all. Its trou

aimock crossed 2.5M weekly installs! Here's what it does and what's new If you've been...

The Problem: Testing Across Regions Is Hard If you're a developer, you've probably faced...

Every company knows when it revoked access. None knows when access stopped. I built this for the All Things Agentic Hackathon , and I wrote this post for the purposes of entering that hackathon. Code: github.com/NexuChat/parallax The chore I was actually trying to kill I maintain a web application with two roles, two languages, one of them right-to-left, a dark theme, and three viewport sizes. Every release, I would open it as the owner, click through, sign out, sign in as a member, click through again, switch to Arabic, reload, shrink the window, reload — and try to remember what a page had looked like ten minutes earlier. The worst defects never survived that process, because they are not visible in any single session. A member opening a page they should have been denied sees nothing wrong. Nothing on the page says "you should not be here." The information is not in their session at all. It is in the difference between their session and the owner's. So I stopped testing sessions and started comparing them. Seven witnesses, one axis apart Parallax opens seven isolated browser contexts at the same instant against the same application. One is a baseline — owner, English, light, desktop. The other six each change exactly one axis from it: privilege, locale, theme, viewport. The full product of those axes is thirty-six combinations. Seven one-axis derivations is not just cheaper; it is the only version that can attribute a cause. When the Arabic witness disagrees with the baseline and locale is the only thing that changed, locale is the reason. With thirty-six combinations you get a bigger table and less knowledge. Each axis carries a contract about what must change and what must not: Axis Contract A finding is Privilege access must differ sameness — an escalation Locale access constant, layout mirrors access drift, or geometry that did not mirror Theme access constant, layout does not move any positional shift Viewport access constant, reflow allowed access drift That

This is part four of the Defender Access series. Each part is standalone, but here is the thread if...

OpenAI Usage API api_key_id grouping solves a practical reporting gap: I can see which API key produced completion-token activity and which key accumulated cost. The tricky part is not making the two requests. It is joining their daily buckets without dropping unattributed or unmatched data. I want a reconciliation report to expose gaps, not smooth them over. A missing cost row, a cost-only row, or a null key ID can each be useful evidence. This pattern keeps those cases visible with a deterministic .NET sample that needs no credentials or paid calls. Why OpenAI Usage API api_key_id needs a full-outer join OpenAI's August 4, 2026 API changelog added API-key filtering and grouping to the usage and cost APIs. That gives both responses a shared operational dimension, but it does not make them identical datasets. The completions usage endpoint reports measures such as input tokens, output tokens, and model requests. Its api_key_id can be null. The costs endpoint returns monetary amounts and currency, also with a nullable API-key dimension. An inner join would retain only rows present in both responses. That is attractive for a tidy chart, but unsafe for reconciliation. It can hide a key that has token usage but no matching cost row, a key with cost but no completion row, or an unattributed bucket. I use a full-outer join keyed by (start_time, end_time, api_key_id) instead. Null or blank IDs become an explicit display value such as <unattributed> ; they do not disappear. Query both APIs at the same daily grain The Costs API supports daily buckets, so I request bucket_width=1d from both endpoints. I also group by the same single dimension: GET /v1/organization/usage/completions ?start_time=... &end_time=... &bucket_width=1d &group_by=api_key_id GET /v1/organization/costs ?start_time=... &end_time=... &bucket_width=1d &group_by=api_key_id Both resources paginate with has_more and next_page . I keep requesting pages until has_more is false. If a response says more data exis

Small automations often look easy: take information from one place and turn it into a task somewhere else. The hard part is what happens when the information is incomplete, someone submits the same request twice, or the workflow sees something it was never meant to use. A useful automation should handle those situations without creating extra cleanup for the owner. I built a small runnable example around four simple checks. 1. Make sure the important information is there If a request is missing something the team needs, the workflow does not create a half-finished task. It places the request on a short review list and explains what is missing. 2. Do not create the same work twice Repeated submissions happen. The example recognizes a repeated request and creates only one task instead of making the team sort out duplicates later. 3. Keep out information the workflow does not need The example copies only the agreed fields into its output. An unexpected column in the input is ignored instead of being passed along automatically. 4. Let a person review the result The example creates an owner-review list. It does not contact customers, connect to outside services, or turn on a live process. A person stays in control of what happens next. The repository includes four made-up requests, the expected result files, and ten automated checks. Those checks cover missing information, repeated requests, unexpected fields, broken input files, and repeatable results. This is an Allure Labs demonstration, not client work and not a claim about business results. You can see the code and sample output here: https://github.com/Allura-Gensin/small-workflow-automation-demo If one small file-based process is creating repeated or incomplete work, start with a $125 written workflow plan or a $500 tested small build. Describe one starting event, one result, and what the workflow must never do. The fixed-scope options and limits are here: https://offers.allurelabs.ai/workflow-automation/ Or use t

A healthcare platform I worked with needed two and a half weeks to complete a regression pass. Smoke testing alone took seven days. The obvious recommendation — the one everyone reaches for — was "automate everything." It would also have been the wrong place to start. Here's the trap. If you point automation at a bloated, outdated, UI-heavy suite, you don't get fast regression. You get an oversized, expensive, automated version of the same slow process, plus a maintenance bill that grows every sprint. A faster test runner does not create a faster regression process — it just runs the wrong tests more quickly. The teams that actually go from weeks to hours don't start with the runner. They start by asking where the time is really going — and most of it is not in test execution. On that healthcare platform, we eventually got regression down from 2.5 weeks to a single day while raising coverage from 50% to 90%. Automation was part of it. It was nowhere near the whole story. This is a guide to the whole story: what to cut, where to test, when to run, and how to investigate failures — so that automation removes specific bottlenecks instead of freezing an inefficient manual suite into code. Why regression testing takes weeks The first thing to fix is a measurement mistake. Most teams track test execution time and quietly assume it's the same thing as regression lead time. It isn't, and the gap between them is where your weeks disappear. Regression lead time = preparation + environment setup + queue time + execution + failure investigation + reruns + reporting Execution is one term in that sum, and often not the biggest. You can halve your runtime and still ship on the same day if the other six terms are untouched. Before you optimize anything, break your lead time into these buckets and see which one actually hurts. In my experience it's rarely the one people complain about. Here's where the time usually leaks. The suite grows but never shrinks Every team is good at addin

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions. This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all. Why Memory Changed the Review Game Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw. Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise. The Fixture Repository fixture/ ├── docs/decisions/0001-metrics-pipeline.md # accepted 2026-07-02 ├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14 ├── src/metrics_service.py # legacy module, 120 lines ├── src/telemetry_service.py # replacement module, 140 lines └── pyproject.toml # lint: E501 disabled for telemetry only The fixture encodes a deliberate conflict: the team renamed the metrics pipeline

A Vue component's job is to produce DOM in a browser. Most component tests ask it to do that somewhere else: in Node, against a DOM that jsdom simulates. That has been the default since npm create vue@latest started offering Vitest with jsdom, and for plenty of tests it is the right trade. It does set a ceiling on what a green test proves, though. Nothing is ever drawn. Your CSS never runs and nothing has a size or a position, so a component can pass every assertion in the file and still be broken on screen. I co-maintain twd-js , which runs tests inside your actual dev server, in a sidebar, next to the app. It was built for flow testing: visit a route, click through the app, assert on what the user sees. Component testing was the thing it did not do. Then I tried calling render() from @testing-library/vue inside a TWD test. import { afterEach , describe , it } from " twd-js/runner " ; import { twd , userEvent } from " twd-js " ; import { render , screen , cleanup } from " @testing-library/vue " ; import HomeView from " ../../views/HomeView.vue " ; import { componentHost , restorePage } from " ../support/componentHost " ; describe ( " HomeView component " , () => { afterEach (() => { cleanup (); restorePage (); }); it ( " increments the counter on click " , async () => { // componentHost() is a blank div on an empty page. More on it below. render ( HomeView , { container : componentHost () }); const button = await screen . findByTestId ( " counter-button " ); twd . should ( button , " contain.text " , " Count is 0 " ); await userEvent . click ( button ); twd . should ( button , " contain.text " , " Count is 1 " ); }); }); Nothing broke. The component mounts into the page, the sidebar shows it running, and reactivity does what reactivity does, in a browser, against a DOM nobody had to simulate. Why this works at all Vue Testing Library is a thin layer. render() mounts your component with @vue/test-utils and binds @testing-library/dom queries to the result. Neither of

Probe vs Prose: what the verifier-sharing-your-text-channel really costs Agent Determinism Illusions (Part 13) 2026-08-31 Where this fits: This part doesn't extend the C3 / key-space mechanism line of Parts 10–12. It returns to an earlier thread — Part 4's runner-independence (Mike Czerwinski's point that "verifiable" is a property of the check's independence from the generator, not of the output) and Theorem 2 (the Data Processing Inequality bound on text-channel verification). A comment from nexus-lab-zen gives that thread a name on the assumption side, and an experiment forces a refinement of what "prose rots" actually means. 1. nexus-lab-zen and the third face of the hatch In the comments on Part 2, a many-round thread with nexus-lab-zen arrived at a useful piece of vocabulary. The thread started on segregation-of-duties and common-mode failure ( Part 2 comments ); several rounds in, nexus-lab-zen had moved from theory to something their team shipped that week: We don't have [per-assertion TTL] either… What we shipped this week is a third face of the hatch[…]: a binding map. Every rule in our registry — 39 right now — must either name the detector that physically enforces it or carry an explicit reason why it's unbound; a fail-closed lint breaks on rules that have neither. Result: 9 bound, 30 unbound-with-reason… On making [TTL] real, one lesson from our timestamp incidents generalizes: fields humans transcribe rot; fields machines embed don't. An invalidation condition written as prose ("assumes transport X is live") goes stale like any prose. Written as a probe — the one command whose changed output falsifies the assertion — the TTL re-check becomes a runner, not a reader. Two things in that comment are worth pulling apart, because one of them survives an experiment and the other gets refined by it. The first is the binding map : 39 rules, of which 9 name a physical detector and 30 carry an explicit "unbound-with-reason." That's not TTL — it can't tell you a p