标签：#Testing

You’ve spent weeks, maybe months, tracking down bugs, optimizing your user interface, and wrestling with backend security rules. You compile your native release build or run your final production compilations, thinking the hardest part of the journey is officially behind you. Then you open the Google Play Console, and you’re hit with the ultimate indie developer roadblock: the mandatory 12-tester and 14-day closed testing requirement . Many independent creators view this process as a simple download checklist. You might think, "I'll just find 12 people to download the app, leave it on their phones for two weeks, and wait it out." However, treating the testing phase as a static metric is the fastest way to get rejected during the final production access review. So, what is Google actually tracking in the background during these two weeks? Let’s take a deep dive into the core algorithmic requirement that determines your success: Continuous Engagement . 🔄 Decoding "Continuous Engagement" Google Play policies are not designed as a simple box-checking exercise. The underlying goal of the algorithm is to verify if your application is genuinely functional, stable, and being tested by an organic user base before it reaches millions of production users. To enforce this, Google's advanced systems actively monitor the devices connected to your closed test track over the 14-day timeline: Background Device Pings: Google Play Services regularly collects background automated signals (ping logs) from the devices where your test build is active. Real User Interaction: Leaving an app to rot in an application drawer without ever opening it is instantly flagged by the algorithm. Google measures whether the app is actively opened daily and tracks active interaction metrics within the build. Feedback Loops: The system monitors whether your test community is utilizing the internal testing channel on the Play Store to send private developer feedback and crash reports. 📉 The Illusion of "Ju

The biggest mistake teams make when comparing testing tools is treating the feature list like the decision. A tool can support API tests, visual checks, CI, reporting, and integrations, and still be the wrong choice if nobody adopts it, the runs are flaky, or the billing model turns into a budget surprise. Start with the workflow, not the brochure The first question is not “What does this tool support?” It is “Where will this tool sit in our actual delivery flow?” A tool that looks great in a demo can still fail if it does not fit how your team writes tests, reviews failures, shares results, and ships code. If your team lives in GitHub PRs, Slack, and CI pipelines, then the evaluation should center on how quickly a test result shows up where developers already work. If your team has QA specialists, product owners, and client stakeholders, then reporting and handoff matter as much as assertion syntax. This is why feature checklists can mislead. Two tools may both claim browser automation, API coverage, and dashboards, but one might require a heavy framework rewrite while the other can be adopted incrementally. The latter is usually the better tool, even if it looks less impressive on paper. Checklist item one, can people actually use it next week? Adoption beats capability. If a tool needs a long onboarding program, a specialist only one person on the team understands, or a custom setup that no one wants to own, the tool becomes shelfware fast. Look at who will author tests, who will maintain them, and who will interpret failures. A tool that lets QA write quickly but gives developers a painful review experience can still become a bottleneck. A good evaluation asks for the smallest realistic test case. Take one happy-path flow, one negative case, and one flaky UI interaction, then see how far each tool gets you without custom glue. That is usually more useful than a vendor demo with polished sample scripts. Checklist item two, what happens when the tests get messy? E

If your goal is faster releases with fewer flaky failures, the tool choice matters less than the testing strategy behind it. Teams usually start by asking, “Should we use Playwright, Selenium, Cypress, or a cloud platform?” A better question is, “What do we need to prove, in which browsers, at what cost to maintainability and reliability?” That shift changes the conversation. Browser automation is not only about writing scripts that click through a happy path. It is about building a test system that survives UI changes, covers the browsers your users actually have, and fails for the right reasons. This playbook walks through a practical sequence you can use to compare tools and make those tradeoffs explicit. Start with the outcomes, not the framework Before comparing tools, define the job your browser tests need to do. Most teams have a mix of goals, even if they do not write them down: Catch broken critical flows before merge Verify rendering in real browsers, not just headless simulations Keep test code readable enough that the team can maintain it Reduce flaky failures that waste review time and erode trust Avoid spending more time on infrastructure than on product quality Once you name those goals, tool comparison becomes simpler. A fast local developer feedback loop may point you toward one choice, while broad cross-browser coverage and managed execution may point you toward another. If a tool is fast but makes maintenance painful, that is not a win. If it supports many browsers but creates unstable runs, that is also not a win. Map your browser reality first The second step is to compare your user base with your test environment. Teams often say they support “all major browsers,” but the actual risk is usually narrower. Check which browser and device combinations matter for your product, then decide what needs automated coverage versus manual spot checks. This is where real browser execution becomes important. A headless run can be useful, but it does not repl

Hey, welcome back. Last time we talked about missing parameterization in test scenarios. Today's mistake is similar in spirit. The test runs. The numbers look great. But what you've built isn't a load test. It's a hammer. ⚠️ The script works. The test is inhuman. Real users don't fire requests like a machine gun. They log in. They pause. They read. They click. They pause again. A typical user journey that takes 60 seconds in real life? Without think time, your script does it in just a few seconds. What this breaks Your throughput numbers are fiction. If users complete journeys 30x faster than reality, your RPS is inflated by 30x. You're not measuring capacity — you're measuring endurance under abuse. You stress the wrong things. Realistic concurrency surfaces real bottlenecks. A firehose of instant requests just overloads your connection pool and calls it a day. Production behaves nothing like your test. Because real users think. Your script didn't. 🛠 The fix Add randomized pauses between steps. Every major tool supports it: JMeter: Gaussian Random Timer, Uniform Random Timer etc. k6: sleep(Math.random() * 5 + 3) Gatling: pause(3.seconds, 8.seconds) Locust: time.sleep(random.uniform(3, 8)) 3–8 seconds between actions is a reasonable starting point. Check your analytics for what real sessions actually look like. Before your next run: Pauses between every major action? Randomized, not fixed? Does the timing feel human? If not — you're not testing load. You're testing collapse. Think time is one piece of the puzzle. But realistic load modeling goes deeper — it's about understanding how real users behave, how to translate that into a load profile, and how to design a test that actually reflects production. That's not something you patch with a timer. It's something you build from the ground up. If you want to understand the full system — from load model design to test execution to results that mean something — that's exactly what Performance Testing Fundamentals course

สรุปสาระสำคัญ (TL;DR) ตัวแปรที่ตั้งค่าระหว่างการรันคำขอแบบแมนนวลใน Postman อาจ “หายไป” เมื่อรันผ่าน Collection Runner เพราะขอบเขตตัวแปรและพฤติกรรมการคงค่าระหว่างการรันไม่เหมือนกัน จุดที่ต้องตรวจสอบคือ pm.environment.set , การเลือก Environment, ค่า Initial/Current Value, ตัวเลือก “Keep variable values” และการเลือกใช้ Collection Variables ให้เหมาะกับสถานะภายในรันเดียวกัน ลองใช้ Apidog วันนี้ บทนำ คุณอาจเคยเจอสถานการณ์นี้: รันคำขอ Login ใน Postman แบบแมนนวล Post-response script ดึง access_token ตั้งค่า token ด้วย pm.environment.set คำขอถัดไปใช้ {{token}} ได้ตามปกติ แต่เมื่อกด Run Collection คำขอ Login ผ่าน แต่คำขอถัดไปได้ 401 Unauthorized ตัวอย่างสคริปต์ที่มักเป็นต้นเหตุ: pm . environment . set ( ' token ' , pm . response . json (). access_token ); สคริปต์นี้ไม่ได้ผิดเสมอไป แต่จะมีปัญหาเมื่อ: ไม่ได้เลือก Environment ใน Runner Runner รีเซ็ตค่าหลังรันเสร็จ ใช้ Environment Variables ทั้งที่ต้องการแค่ state ภายใน Collection Run ตั้งค่าเฉพาะ Current Value แต่ไม่ได้ตั้ง Initial Value บทความนี้สรุปวิธีดีบักและแก้ไขแบบลงมือทำได้ทันที ลำดับชั้นขอบเขตตัวแปรของ Postman Postman แก้ค่า {{variable}} ตามลำดับความสำคัญดังนี้: Local variables — ใช้เฉพาะในสคริปต์ที่กำลังรัน Data variables — มาจากไฟล์ CSV/JSON สำหรับ data-driven test Collection variables — ใช้ภายในคอลเล็กชัน Environment variables — ใช้ใน Environment ที่เลือก Global variables — ใช้ได้ข้ามคอลเล็กชันและ Environment ถ้ามีตัวแปรชื่อเดียวกันหลายขอบเขต เช่น token Postman จะใช้ค่าจากขอบเขตที่มี priority สูงกว่าก่อน ตัวอย่าง: pm . collectionVariables . set ( ' token ' , ' collection-token ' ); pm . environment . set ( ' token ' , ' environment-token ' ); console . log ( pm . variables . get ( ' token ' )); pm.variables.get('token') จะคืนค่าตามลำดับ priority ไม่ได้หมายความว่าจะอ่านจาก Environment เสมอไป ทำไมตัวแปรจึงหายไปใน Collection Runner 1. Current Value และ Initial Value ไม่เหมือนกัน ตัวแปรใน Postman มี 2 ค่า: Initial value : ค่าที่ซิงค์และแชร์กับทีม Current value : ค่า local ในเครื่องของคุณ เมื่อใช้: pm . environment . set (

Em resumo Variáveis definidas em scripts do Postman podem “sumir” quando você troca a execução manual pelo Collection Runner. Na prática, quase sempre é um problema de escopo: pm.environment.set escreve no ambiente ativo, variáveis de coleção têm outro ciclo de vida, e o runner pode descartar alterações ao final da execução. Experimente o Apidog hoje Neste guia, você vai ver como diagnosticar o problema, escolher o escopo correto e corrigir os casos mais comuns. Também verá como o Apidog lida com variáveis de forma mais explícita na interface. Introdução Você testa uma API manualmente no Postman: Executa a requisição de login. Um script salva o token. As próximas requisições usam {{token}} . Tudo funciona. Depois você clica em Run Collection . O login retorna sucesso, mas a próxima requisição falha com 401 Unauthorized . O token não foi encontrado. Esse comportamento é comum porque o modo manual e o Collection Runner não lidam com o estado das variáveis exatamente da mesma forma. A correção começa entendendo a hierarquia de escopos do Postman. Hierarquia de escopo de variáveis do Postman O Postman resolve variáveis seguindo uma ordem de prioridade. Da maior para a menor: Variáveis locais : existem apenas durante a execução do script atual. Variáveis de dados : vêm de arquivos CSV ou JSON usados em execuções orientadas por dados. Variáveis de coleção : pertencem à coleção e podem ser usadas por requisições dentro dela. Variáveis de ambiente : pertencem ao ambiente selecionado. Variáveis globais : ficam disponíveis para qualquer coleção e ambiente. Quando você usa: {{token}} o Postman procura token nessa ordem e usa o primeiro valor encontrado. Isso significa que o problema nem sempre é “a variável não existe”. Às vezes ela existe, mas em outro escopo, ou um escopo de maior prioridade está sobrescrevendo o valor esperado. Por que as variáveis desaparecem no Collection Runner 1. Valor inicial vs. valor atual Cada variável no Postman pode ter dois valores: Valor inicial

No trabalho, nenhum código mudou. O que mudou foi a forma como os clientes inserem os dados. E isso quebrou coisas que nenhum teste existente pegou. O bug que só aparece quando alguém usa A motivação pra montar E2E do zero veio de um problema específico. Você precisava acessar a aplicação pra quebrar. Não era um erro de lógica isolado que um teste unitário pegaria. Era uma combinação de dados reais num fluxo real produzindo um resultado errado que só aparecia na tela. Os clientes chegavam lá antes da gente. É uma categoria de problema que teste de código não resolve, porque o problema não está no código. Está na interação entre o código, os dados e o ambiente. A forma mais rápida de pegar antes é rodar o fluxo completo do jeito que o usuário roda. Ficou com smoke tests cobrindo os principais fluxos do produto, configuração pra rodar contra múltiplos ambientes, e notificação no Slack quando o nightly quebra. A parte mais útil não são os testes em si. É saber antes do cliente reportar. Autocrop: quando nenhuma ferramenta resolve tudo Num projeto paralelo que mantenho, passei o fim de semana montando autocrop automático pra imagens. A ideia inicial era usar o imgproxy Pro, que tem detecção de objeto embutida. Não ficou preciso o suficiente pra variedade de imagens que eu tinha. Fui pro Rekognition, que retorna bounding boxes. Mais controle, mas bounding box tem um limite: é um retângulo. Objetos não são retângulos. Aí descobri o rembg, que faz algo diferente. Em vez de delimitar uma área, ele cria uma máscara pixel por pixel usando uma rede chamada U2Net, treinada pra segmentação de primeiro plano. O resultado foi bem superior — ele recorta o objeto, não uma caixa em torno dele. Colocar isso em Lambda foi onde a semana ficou mais lenta. O modelo precisava estar acessível pro processo do Lambda, coloquei em /root , Lambda não lê de lá. Movi pro /opt , chmod 755. O NUMBA tentou escrever cache em diretório read-only, defini NUMBA_CACHE_DIR=/tmp . Depois OOM em imagens mai

In Chapter 2 we wrote our first tests and hit two bugs. Before we add more, we need the one skill everything else rests on: finding elements reliably . Get this right and your tests survive redesigns; get it wrong and they break every sprint. Code for this chapter is tagged ch-03 in the repo: https://github.com/aktibaba/playwright-qa-course — see src/tests/ui/locators.spec.ts . Locate the way a user perceives The brittle instinct is to grab elements by their structure — CSS classes, nth-child , XPath. All of that changes the moment a developer touches the markup. Playwright's recommended locators instead target what a user (and a screen reader) perceives: the role, the label, the visible text. Use them in this order of preference: getByRole — the role + accessible name (covers the vast majority of cases) getByLabel — form fields by their <label> getByPlaceholder — inputs without a label getByText — non-interactive content getByTestId — a deliberate data-testid , only when nothing semantic fits Here's the top of the priority list, live against Inkwell's home page: import { test , expect } from " @playwright/test " ; test ( " prefer role-based locators over CSS " , async ({ page }) => { await page . goto ( " / " ); await expect ( page . getByRole ( " button " , { name : " Global Feed " })). toBeVisible (); await expect ( page . getByRole ( " link " , { name : " Sign up " })). toBeVisible (); await expect ( page . getByRole ( " heading " , { name : " inkwell " })). toBeVisible (); }); getByRole("button", { name: "Global Feed" }) asserts two things at once — that an element with the button role exists and that its accessible name is "Global Feed". If a dev swaps the <div class="feed-btn"> for a real <button> , this locator keeps working; a CSS selector wouldn't. Strict mode is your friend Playwright locators are strict : if a locator matches more than one element, the action throws instead of silently picking the first. That catches ambiguous tests before they pick the

In Chapter 1 we argued that automation fails from a lack of structure , not a lack of tooling — and we met Inkwell , the dockerized app we test against. Now we install Playwright + TypeScript and write our first UI and API tests, deliberately simple. We'll also hit two real bugs along the way. I'm leaving them in on purpose — they're the exact problems the framework we build later is designed to prevent. Code for this chapter is tagged ch-02 in the repo: https://github.com/aktibaba/playwright-qa-course Before you start Make sure Inkwell is running (from Chapter 1): cd sut docker compose up -d --build --wait # web :3000, api :3001/api Install Playwright + TypeScript From the repo root: npm install -D @playwright/test typescript @types/node npx playwright install chromium That's it — Playwright bundles its own test runner, assertion library, and TypeScript support. No extra config to make .ts test files work. A minimal config — not a framework yet Two small files keep us honest from day one. First, never hard-code URLs in tests — put them in one place: // src/utils/env.ts export const env = { /** Inkwell SPA (nginx) — the UI base URL. */ webURL : process . env . WEB_URL ?? " http://localhost:3000 " , /** Inkwell API base, including the /api prefix. */ apiURL : process . env . API_URL ?? " http://localhost:3001/api " , } as const ; Then the Playwright config. We split tests into two projects — a fast api project and a Chromium ui project — because API tests need no browser and should run in milliseconds: // playwright.config.ts import { defineConfig , devices } from " @playwright/test " ; import { env } from " ./src/utils/env " ; export default defineConfig ({ testDir : " ./src/tests " , fullyParallel : true , reporter : " list " , use : { trace : " on-first-retry " , screenshot : " only-on-failure " }, projects : [ { name : " api " , testDir : " ./src/tests/api " , use : { baseURL : env . apiURL } }, { name : " ui " , testDir : " ./src/tests/ui " , use : { baseURL : e

Welcome to the first chapter of a hands-on course where we build a production-grade Playwright + TypeScript automation framework — covering both API and UI testing — against a real, dockerized web app you run on your own machine. This isn't a "here are 5 Playwright tips" post. By the end of the series you'll have a framework with the same shape a real QA team ships: layered, parallel-safe, authenticating once and reusing the session, seeding data through the API and verifying it in the UI, and running sharded in CI. We build it one chapter at a time, and every line of code is in a public repo you can clone and run. Who this is for You can read basic JavaScript (variables, functions, async/await ). That's it. No Playwright or TypeScript experience required — we introduce both from zero. You've maybe written a few UI tests before and felt them turn into a tangle. This course is about the structure that prevents that. How the course works Each chapter is one post in this series, in order. Read them top to bottom. There's a companion GitHub repo — the single source of truth for all code: 👉 https://github.com/aktibaba/playwright-qa-course The repo carries one git tag per chapter ( ch-01 , ch-02 , …) so you can check out the exact state of the code at any point and compare it to what you have. Every chapter ends with what changed, so you can either build along or just read the diff. Get the code and run the app We don't test toy pages. The course runs against Inkwell — a small but real React + Express + PostgreSQL blogging app (articles, comments, tags, follow/favorite, JWT auth). It lives in the same repo under sut/ ("system under test") and ships as a one-command Docker stack with deterministic reset / seed endpoints, so your tests never race startup or fight flaky data. You'll need Node.js 18+ and Docker . # 1. Clone the course repo git clone https://github.com/aktibaba/playwright-qa-course.git cd playwright-qa-course # 2. Start the app (db + API + web), wait until eve

Section 1: Getting Started with Cypress 1. Installing and Setting Up Cypress Prerequisites Before installing Cypress, ensure you have Node.js installed on your machine. Cypress requires Node.js 18.x or 20.x and above. You should also have an existing React project or create a new one. Check your Node.js version by running this command in your terminal: node --version # Should output v18.x.x or higher Creating a React Project (Optional) If you don't have an existing React project, create one using Vite which is the recommended approach for new React projects: npm create vite@latest my-react-app -- --template react cd my-react-app npm install Installing Cypress Navigate to your React project directory and install Cypress as a development dependency. Cypress is a fairly large package, so the installation might take a minute or two: npm install cypress --save-dev # Or using yarn yarn add cypress --dev Opening Cypress for the First Time After installation, open Cypress for the first time. This will create the initial folder structure and configuration files: npx cypress open When Cypress opens for the first time, you'll see a welcome screen where you can choose between E2E Testing and Component Testing. Select E2E Testing to get started with end-to-end tests. Cypress will then prompt you to choose a browser. You can select Chrome, Firefox, Edge, or Electron. Choose your preferred browser and click Start E2E Testing in [Browser] . Adding NPM Scripts Add convenient scripts to your package.json for running Cypress tests: { "scripts" : { "dev" : "vite" , "build" : "vite build" , "cy:open" : "cypress open" , "cy:run" : "cypress run" , "test:e2e" : "start-server-and-test dev http://localhost:5173 cy:run" } } Tip: Use npm run cy:open for interactive development with the Cypress Test Runner. Use npm run cy:run for headless execution in CI/CD pipelines. 2. Understanding Cypress Project Structure Project Directory Overview After initializing Cypress, you'll notice several new fold

If you're building or integrating Automated Test Equipment for aerospace or defence electronics, the technical requirements go well beyond "does the test pass." You need documentation that satisfies MIL-STD, AS9100, and DO-178C auditors — and an architecture that scales from prototype to production. Here's how modern Universal ATE systems are structured for defence-grade compliance. The Core Architecture A defence-grade ATE system has four functional layers: ┌─────────────────────────────────────────┐ │ Test Executive (LabVIEW) │ ← Orchestrates all test sequences ├────────────────┬────────────────────────┤ │ Instrument │ DUT Interface │ ← Hardware layer │ Control │ (ICT/JTAG/Func) │ ├────────────────┴────────────────────────┤ │ Data Management Layer │ ← Logging, traceability, reports ├─────────────────────────────────────────┤ │ Calibration & Verification │ ← Ensures measurement accuracy └─────────────────────────────────────────┘ Test Executive Design in LabVIEW The test executive controls the sequence, manages results, and handles failures. Key design principles: Test Sequence: 1. DUT identification (serial number scan or manual entry) 2. Pre-test self-check (verify instrument calibration status) 3. ICT phase — passive component verification 4. JTAG boundary scan — IEEE 1149.1 interconnect verification 5. Power-on functional test — operational verification 6. RF/signal analysis — if applicable to DUT type 7. Report generation — automatic, timestamped 8. Pass/fail disposition record JTAG Integration via IEEE 1149.1 For high-density boards where bed-of-nails is not viable, JTAG boundary scan is implemented via a JTAG controller (e.g., XJTAG, Corelis, or ASSET InterTech) integrated into the LabVIEW environment: LabVIEW → JTAG Controller API → Scan Chain → DUT ICs The boundary scan description files (BSDL) for each IC define the test vectors. Your test executive loads BSDL files, generates scan chain topology, and runs interconnect tests automatically. Data Traceabili

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.

The test hung for 30 seconds. The response had already fired. One moved line fixed it. The test hung for 30 seconds, then timed out. The browser had received the response. The page had loaded. The data was there. The test was still waiting. The wizard I was writing a helper to walk through a 4-step booking wizard. After clicking "Next" on step 1, the page does a full navigation — window.location.href to step 2. Step 2 immediately loads doctor data from the API. The helper looked like this: await Promise . all ([ page . waitForURL ( /step=2/ ), step1Next . click ()]); await page . waitForResponse ( r => r . url (). includes ( ' /doctors ' )); Standard pattern: wait for navigation, then wait for the data request. Timeout. Every time. What I checked first The URL pattern. Maybe /doctors wasn't matching. Opened the network tab. The request was there: GET /api/v1/doctors , 200, 47ms. Correct URL, correct response. The page looked fine. The data was rendered. The test said it was waiting for a response that had already happened. Added waitForLoadState . Still hung. Added an explicit waitForSelector for an element that was clearly on the page. That passed. Then waitForResponse hung again. The response existed. The test couldn't see it. What was actually happening page.waitForResponse() is not a query. It doesn't look at what happened. It registers a listener — from that exact moment forward — and waits for the next matching response. The sequence in my code: Promise.all resolves when the URL changes to step=2 By the time the URL changed, step 2 had already loaded Step 2 had already sent and received /api/v1/doctors Then waitForResponse registered its listener Now it's waiting for the next /doctors response Which never comes Playwright doesn't buffer missed events. If the response fired before the listener was registered — it's gone. The fix await Promise . all ([ page . waitForURL ( /step=2/ ), page . waitForResponse ( r => r . url (). includes ( ' /doctors ' )), step1Next

You just merged an AI-assisted feature branch, the code review looks clean, and the app works in your local smoke test. Now comes the real question: do you add another traditional browser test, let an AI tool generate the coverage, or spend the time improving the observability around the existing suite? That decision is where a lot of teams get stuck. AI-assisted development changes more than coding speed. It changes the shape of bugs, the pace of UI churn, the expectations for review, and the amount of test maintenance you can tolerate. If you treat AI testing as a magic replacement for your current process, you will probably add noise. If you ignore it entirely, you miss a chance to reduce repetitive work and catch gaps earlier. The real choice is not AI vs non-AI The useful decision is usually this, should AI help create and maintain tests, should it assist human review, or should it stay out of the critical path and only support investigation? That splits into three practical modes: 1. AI assists development, but humans own test strategy This is the safest default. AI can help draft test cases, suggest assertions, summarize failing traces, or propose missing edge cases, but the team still decides what belongs in the suite. If your product has regulated flows, complex permissions, or revenue-critical paths, that ownership matters more than any automation shortcut. 2. AI generates or heals tests inside a human-defined framework This is useful when the team already knows what it wants to cover, but not every selector, fixture, or assertion has to be hand-written. AI can reduce repetitive maintenance, especially for UI-heavy apps that change often. The hidden cost is that you still need a way to judge whether the generated test reflects product intent or just mirrors the current page state. 3. AI becomes part of the evaluation and triage loop Here the value is not test creation, it is speed of diagnosis. AI can summarize logs, cluster failures, or explain a flaky pa

Introduction: The Controversial Rise of AI in Software Development The software development industry is at a crossroads. On one side, the rapid advancement of AI tools promises to revolutionize coding, automate repetitive tasks, and accelerate project timelines. On the other, a growing chorus of experts, led by figures like George Hotz , warns that the integration of AI agents into software development could become "one of the most costly mistakes in the field’s history." This bold prediction isn’t just hyperbole—it’s a call to scrutinize the mechanisms by which AI adoption could deform the very foundation of software engineering. At the heart of this debate are three critical failure points: over-reliance on AI without human oversight , insufficient real-world testing , and misalignment between AI capabilities and software development demands . Each of these factors acts as a stressor on the system, threatening to heat up development costs, expand systemic vulnerabilities, and ultimately break the delicate balance between innovation and reliability. Consider the causal chain: over-reliance on AI leads to a degradation of human expertise , as developers become less engaged in problem-solving. This, in turn, creates a feedback loop where AI-generated code, lacking nuanced understanding, introduces errors that go unnoticed. Without proper oversight , these errors propagate through systems, causing observable effects like reduced software quality and increased maintenance costs. Similarly, insufficient testing of AI agents in real-world scenarios means their failure modes remain unknown until they’re deployed at scale, risking systemic collapse in critical applications. The stakes are high. If unchecked, AI integration could lead to a loss of institutional knowledge , escalating development costs , and vulnerabilities in critical systems . The question isn’t whether AI has a role in software development—it’s how to implement it without deforming the field’s core princi

Internal note to the team, we need to improve test coverage and keep shipping, which means we should treat AI as a helper in the workflow, not as a replacement for testing discipline. AI-assisted development changes the shape of our risk. It can produce more code faster, but it also increases the chance that small logic mistakes, brittle selectors, and shallow test cases slip through review. The answer is not to add more manual checking everywhere. The answer is to be more deliberate about what we review, what we automate, and where we let AI help. What changes when AI writes part of the code The first thing that changes is review. When a developer uses AI to draft a feature, a test, or a refactor, the reviewer is no longer only checking intent and style. The reviewer also needs to check whether the generated code matches the product rule, whether it introduced a hidden dependency, and whether it quietly weakened coverage. That does not mean every AI-assisted change deserves extra ceremony. It means our review checklist should shift from "does this look correct" to "what did the model assume, and did we verify those assumptions?" That is especially important for test code, because generated tests often look plausible even when they do not prove much. Coverage should move from volume to signal AI tends to produce more test cases, but more cases are not the same as better coverage. If a generated test suite repeats the same happy path under slightly different names, the team gets a false sense of safety. Coverage should answer a more practical question, where are we most likely to break the user experience, and where will a test actually catch it? For chat and other AI features, prompt-by-prompt manual checks are a trap. They do not scale, and they encourage a habit of eyeballing output instead of verifying behavior. A better pattern is to build assertions around expected properties, create eval sets for representative prompts, and add regression coverage for failure

Critics say Trump plan to test AI models is short-sighted, performative.

Here's a trace that reset how I think about evaluating tool-calling agents. An agent tries to book a flight. It calls search_flights with departure_date="next Friday" . The endpoint expected an ISO date, so it returns a 400 . The agent retries the same string four times, then apologizes to the user and gives up. Now the part that actually bothered me. Tool selection was correct. The model picked the right function out of a registry of 28. My tool-selection accuracy logged a clean 1.0 . The aggregate task-completion logged a 0 . And neither number told me which of three things broke: the argument was wrong, the model never read the 400 body, or the retry policy looped on the same input. My eval wasn't wrong. It was asking the wrong question. What "tool-call accuracy" actually grades If the only thing you measure is did the agent call the right tool , you're testing intent, not execution. Tool selection is necessary, not sufficient. It passes the moment the right function name shows up in the trace, completely blind to whether the arguments were garbage, whether the model read what came back, or whether it recovered from the 400 . That's the gap. The metric checks that the agent started the right way. Production needs to know whether it finished the right way. The reframe: it's four eval problems, not one The thing I had to internalize is that tool-calling eval is four problems stacked, each with its own root cause: Tool selection , right tool, or correctly no tool Argument extraction , schema-valid and semantically correct Result utilization , did it actually use what the tool returned Error recovery , did it retry, fall back, or escalate Score them separately and "the agent failed" collapses into "the argument extractor regressed on date strings on the flight-booking path." One bisect instead of three days. What I rebuilt Layer 1: Tool selection (with the bucket everyone drops) F1 on the tool name, so a 28-tool registry doesn't hide a regression on one rare endpoint

Software testing is changing quickly. Teams are dealing with faster release cycles, more AI-assisted development, more complex browser behavior, and higher expectations around product quality. I collected a few practical articles that cover different parts of modern QA, test automation, developer workflows, and testing strategy. Recommended reads How to Test AI Agents for Tool Use, Memory, and Recovery Paths A practical framework for testing AI agents for tool use, memory retention, retries, and recovery paths, with concrete strategies for QA and engineering teams. How to Evaluate a Test Automation Tool for Shadow DOM, iframes, and Other Hard-to-Test UI Surfaces A practical buyer guide for evaluating test automation tools for shadow DOM testing, iframe testing, resilient selectors, and dynamic UI edge cases. How to Reproduce a Flaky Browser Test with Video, Logs, and Network Traces A practical workflow to reproduce a flaky browser test using video, logs, and network traces, then turn intermittent failures into repeatable bug reports. Endtest Review for Small QA Teams: Where Editable Test Flows Save the Most Time A practical Endtest review for small QA teams focused on editable test flows, maintainable test steps, and where no-code QA automation actually saves time. Editable Test Steps vs Generated Test Code: Which Holds Up Better After UI Changes? A practical comparison of editable test steps vs generated test code for UI change resilience, maintenance overhead, debugging, and team handoff, with guidance for QA and engineering leaders. Managed QA Services vs Staff Augmentation: What Changes in Ownership, Speed, and Cost A practical comparison of managed QA services vs staff augmentation, focusing on ownership, ramp time, communication overhead, cost, and maintenance risk. Automation Payback Period: How Long Does QA Test Automation Take to Break Even? Learn how to estimate the test automation payback period, model QA ROI, account for maintenance cost, and identify wh