标签：#Testing

There is a quiet tax many engineering teams pay after their automated test suite starts to matter. The tests live in code. They run in CI. They already know the branch, commit, environment, failure message, stack trace, screenshot, trace file, retry status, and build URL. Then someone asks: Can we put these tests in Jira too? That request usually comes from a good place. Jira is where work is planned. Product, QA, engineering, and release stakeholders already use it. If quality signal is invisible there, people end up asking for screenshots from CI, links to failed builds, or spreadsheet summaries before every release. The mistake is assuming the answer is to recreate every automated test as a manual Jira test case. For many teams, that creates a second source of truth that starts decaying immediately. Automated tests are not manual test cases Manual test cases and automated tests have different jobs. A manual test case is a human-readable procedure. It often describes a workflow, expected result, and maybe some preconditions. It is useful when a person needs to execute or review a scenario. An automated test is executable behavior. It is versioned with the code, refactored with the product, reviewed in pull requests, and run repeatedly by machines. When teams try to manage automated tests by copying them into a test-case inventory, they usually create a translation problem: The test name changes in code, but the Jira case does not. The test is deleted or split, but the manual record still exists. The CI failure has rich evidence, but the test case only says "failed." The test belongs to a branch or commit, but the copied case does not. The release team sees a static inventory instead of the latest run signal. The test case becomes a label for a thing that actually lives somewhere else. The better question Instead of asking, "How do we turn all automated tests into Jira test cases?", ask: What does Jira need to know from each automated run? That changes the shape of

The Playwright Playbook — Part 3: Multi-User, Multi-Tab & Browser Context Testing "Most frameworks test one user at a time. Real apps don't work that way." In Part 1, we built the full foundation — POM, storageState , fixtures, and a clean project structure. In Part 2, we took control of the network layer — mocking APIs, simulating failures, and asserting on real network calls. Now we tackle the scenario that breaks most automation frameworks. Multiple users. Simultaneously. In one test. Think about your app for a moment. How many features actually involve just one user? Admin assigns a task → regular user sees it appear in real time User A edits a record → User B gets a notification Admin revokes access → User's session should invalidate Two users try to edit the same record → conflict resolution kicks in These are real features. They need to be tested. And page.goto('/login') in a beforeEach won't get you there. Playwright's browser context architecture was built exactly for this. Let's use it properly. 🎯 🏗️ Where We Left Off After Part 2, our project looks like this: playwright-playbook/ ├── tests/ │ ├── auth/ │ │ └── login.spec.ts ✅ Part 1 │ ├── tasks/ │ │ └── task-management.spec.ts ✅ Part 1 │ └── network/ ✅ Part 2 │ ├── api-mocking.spec.ts │ ├── error-simulation.spec.ts │ └── network-assertions.spec.ts ├── pages/ │ ├── LoginPage.ts ✅ Part 1 │ └── TaskPage.ts ✅ Part 1 ├── fixtures/ │ ├── auth.fixture.ts ✅ Part 1 │ ├── tasks.json ✅ Part 2 │ ├── empty-tasks.json ✅ Part 2 │ └── tasks-har.har ✅ Part 2 ├── scripts/ │ └── record-har.ts ✅ Part 2 ├── .auth/ │ ├── admin.json │ └── user.json ├── global-setup.ts ✅ Part 1 ├── playwright.config.ts ✅ Part 1 └── .env By the end of Part 3, we add: playwright-playbook/ ├── tests/ │ ├── multi-user/ ← NEW │ │ ├── role-permissions.spec.ts │ │ └── realtime-collaboration.spec.ts │ └── multi-tab/ ← NEW │ └── multi-tab-flows.spec.ts ├── pages/ │ └── DashboardPage.ts ← NEW ├── fixtures/ │ └── multi-user.fixture.ts ← NEW Every one of th

AI agents are making software development faster. That is great. But there is a problem I do not think we are talking about enough: testing is not speeding up in the same way. In many teams, testing is still held together by spreadsheets, meeting notes, screenshots, chat messages, and the memory of a few experienced QA engineers. That worked when delivery was slower. It becomes fragile when one developer can use multiple agents to change code across several modules in a single afternoon. The bottleneck is no longer "can we write more test cases?" The bottleneck is: Can the team prove what was tested, why it was tested, what failed, what was fixed, and whether the release is safe? That is the problem I built testboat for. The Most Dangerous Sentence Before A Release The sentence I worry about most is not: We did not test this. At least that is honest. The dangerous sentence is: I think we tested this. That sentence usually means the team has test artifacts, but they are disconnected: requirements live in a doc test cases live in a spreadsheet automation scripts live somewhere in the repo execution results live in CI logs or chat bugs live in an issue tracker release reports are written manually before sign-off Each piece may be useful on its own. But when a Tech Lead asks, "Which requirements are not covered?" or a founder asks, "Can we release today?", the team has to reconstruct the answer manually. That is not a testing process. That is institutional memory under pressure. AI Makes This Gap Worse AI agents are very good at increasing throughput. They can: implement a feature faster refactor code faster generate UI faster write automation faster fix bugs faster But faster change creates more testing uncertainty. If an agent changes the authentication module, what should be rerun? If a test fails, is it a product bug, a flaky automation script, or an environment issue? If a developer says "fixed", has the failed test actually been rerun? If a release report says "ma

AI for API Design & Testing in 2026: Speakeasy vs Swagger AI vs Postman AI — I Built 3 APIs with Each API design is one of the most tedious parts of backend development. Writing OpenAPI specs, generating SDKs, testing endpoints, maintaining documentation — it's all hours of work that could be automated. In 2026, three tools are fighting for your API workflow: Speakeasy , Swagger AI , and Postman AI . I spent 4 weeks building the same REST API with each one, tracking time savings, code quality, and actual developer experience. Here's what actually happened. The Test Setup: Building a Real API I built a simple but realistic ecommerce API (GET/POST products, orders, user management) three separate times, measuring: Time to spec — writing the OpenAPI definition Time to SDK generation — generating client libraries Time to testing — setting up test cases and running them Documentation quality — readability, examples, completeness Iteration speed — how fast you can modify the API and regenerate everything All three tools were given the same requirements. Same laptop, same environment, same skill level (senior backend engineer). Speakeasy: The SDK Generation King Setup time: 8 minutes (CLI install, auth, first config) Learning curve: Low — clear docs, straightforward CLI Speakeasy is laser-focused on one problem: turning your API spec into production-ready SDKs. It generates TypeScript, Python, Go, Java, and more from a single OpenAPI definition. The Good SDK quality is exceptional. The generated TypeScript SDK was production-grade immediately — proper error handling, retry logic, request/response typing, retries built in. No cleanup work needed. Multi-language generation is instant. After writing the OpenAPI spec once, I had Python, Go, and TypeScript SDKs in < 2 minutes total. Each one was framework-idiomatic (using popular libraries in each language). Versioning is smart. Speakeasy automatically versioned SDKs and generated changelogs. When I modified the spec, it detect

Agentic QA Pipelines: Why Your Test Scripts Are Already Obsolete You wrote the test. You maintained the test. The app changed. You rewrote the test. If that loop sounds familiar, you're not alone — and in 2026, you're also not competitive. Agentic QA pipelines are replacing script-based test automation not because AI is smarter than your QA engineers, but because describing goals is faster than maintaining instructions. Here's what's actually changing, why it matters, and how forward-thinking teams are shipping without the script debt. The Script Maintenance Tax Is Killing Velocity Traditional test automation follows a simple premise: write explicit instructions, run them, check results. It worked when applications changed slowly and test environments were stable. In 2026, neither is true. AI-generated code ships faster. Features change in days. UI components regenerate. And every change breaks a percentage of your carefully maintained test scripts — creating a maintenance tax that grows proportionally with your automation coverage. Quash's 2026 State of QA Automation Report found that teams spending more than 30% of QA bandwidth on script maintenance are shipping 2.4x slower than teams that have automated that maintenance layer away. The irony: the more test coverage you write, the more you're paying the tax. What Agentic QA Actually Means (Without the Buzzwords) An agentic QA system doesn't follow a script. It follows a goal. Instead of: Click the login button Enter " testuser@example.com " in the email field Enter "password123" in the password field Assert redirect to /dashboard An agentic QA agent receives: Goal: Verify that a registered user can successfully authenticate and access their dashboard. Context: Auth flow supports email/password and OAuth. Dashboard loads user-specific data. The agent then: Explores the auth flow autonomously Generates test scenarios, including edge cases it infers from the UI Executes tests, reads failures, and adapts to UI changes

Every developer who has written a Playwright test for OTP verification has written this line: const otp = email . body . match ( / \b\d{6}\b / )?.[ 0 ]; It works. Until it doesn't. The email body changes format. The OTP appears inside an HTML table. The sending service wraps it in a <span> . Your regex matches a phone number instead of the code. The test fails intermittently and you spend an hour debugging something that has nothing to do with the feature you're testing. The regex problem OTP extraction via regex is brittle by nature. You're pattern-matching against a string that your email sending service controls — not you. Any time the template changes, your tests break. Here's what a typical OTP test looks like today: import { test , expect } from ' @playwright/test ' ; import { ZeroDrop } from ' zerodrop-client ' ; const mail = new ZeroDrop (); test ( ' user can verify OTP ' , async ({ page }) => { const inbox = mail . generateInbox (); // 1. Trigger OTP send await page . goto ( ' /login ' ); await page . fill ( ' [data-testid="email"] ' , inbox ); await page . click ( ' [data-testid="submit"] ' ); // 2. Wait for email const email = await mail . waitForLatest ( inbox , { timeout : 15000 }); // 3. Extract OTP — the fragile part const otp = email . body . match ( / \b\d{6}\b / )?.[ 0 ]; if ( ! otp ) throw new Error ( ' OTP not found in email body ' ); // 4. Enter OTP await page . fill ( ' [data-testid="otp"] ' , otp ); await page . click ( ' [data-testid="verify"] ' ); await expect ( page ). toHaveURL ( ' /dashboard ' ); }); The test works — but line 14 is carrying all the risk. Change the email template and the test breaks. Add a phone number to the footer and the regex matches the wrong number. Send a 4-digit OTP instead of 6 and you need to update the pattern. OTP extraction at the edge ZeroDrop extracts OTPs before they reach your test. The Cloudflare Worker that catches incoming emails runs a pattern match on the plain-text body and stores the result alongsi

Vos tests API passent en local. Le vrai enjeu est de les exécuter automatiquement à chaque pull request, fusion et build nocturne, sans clic manuel. Pour cela, vous avez besoin d’un exécuteur CLI : il lance vos tests en mode headless, retourne un code de sortie exploitable par la CI et génère un rapport lisible par votre pipeline. Essayez Apidog aujourd'hui Deux outils reviennent souvent pour ce cas d’usage : le CLI Bruno et le CLI Apidog . Les deux exécutent des tests API depuis GitHub Actions, GitLab CI, Jenkins ou tout environnement Node.js. Les deux font échouer la build lorsqu’un test échoue. La différence principale se situe avant l’exécution : où vivent les tests, comment ils sont créés et comment la CI y accède. Cet article compare les deux outils au niveau commande, avec des exemples directement intégrables dans un pipeline. En bref CLI Bruno ( @usebruno/cli , binaire bru ) exécute des fichiers .bru présents dans votre dépôt Git. Il est open source, fonctionne hors ligne et ne nécessite ni compte ni jeton. CLI Apidog ( apidog-cli , binaire apidog ) exécute des scénarios de test créés visuellement dans Apidog, récupérés par ID avec un jeton d’accès. Les deux génèrent des rapports JUnit, JSON et HTML. Les deux retournent un code non nul en cas d’échec, ce qui permet à la CI de bloquer une fusion ou un déploiement. Choisissez Bruno si vous voulez des tests versionnés comme du code, dans le dépôt. Choisissez Apidog si vous voulez créer, chaîner et exécuter des scénarios visuels sans maintenir manuellement des fichiers de test. Le problème : des tests qui existent mais ne tournent pas Un test API lancé manuellement finit souvent par devenir obsolète. Il a été écrit, validé une fois, puis oublié pendant que l’API évoluait. La solution n’est pas seulement d’ajouter plus de tests. Il faut les exécuter automatiquement à chaque changement avec un signal clair : succès ou échec ; rapport exploitable ; code de sortie lisible par la CI. Un exécuteur CLI doit donc rempli

Seus testes de API passam no seu laptop. O que importa é se eles rodam em cada pull request, merge e build noturno sem intervenção humana. Para isso, você precisa de um executor de linha de comando: ele roda os testes em modo headless dentro do pipeline, retorna código de saída correto e gera relatórios que o CI consegue ler. Experimente o Apidog hoje Dois CLIs aparecem com frequência nessa configuração: Bruno CLI e Apidog CLI. Ambos executam testes de API em CI/CD, mas partem de modelos diferentes: Bruno é git-native , offline-first e open source. O CLI executa arquivos .bru versionados no repositório. Apidog é uma plataforma de API completa. O CLI executa cenários visuais criados no aplicativo, buscados por ID. Ambos funcionam em GitHub Actions, GitLab CI, Jenkins e qualquer runner com Node.js. Ambos falham a build quando um teste falha. A diferença principal está em como você cria os testes, onde eles ficam e como o CI os acessa. Em resumo Bruno CLI ( @usebruno/cli , binário bru ) executa arquivos .bru diretamente de uma pasta no seu repositório Git. Apidog CLI ( apidog-cli , binário apidog ) executa cenários de teste visuais do seu projeto Apidog usando um access token . Ambos geram relatórios JUnit, JSON e HTML. Ambos retornam código de saída diferente de zero quando há falha. Use Bruno quando quiser testes em texto simples, versionados no repositório, sem conta e sem dependência de rede. Use Apidog quando quiser criar cenários visualmente, encadear requisições, reutilizar ambientes e rodar testes data-driven sem manter código de teste manualmente. O problema: testes que existem, mas não rodam Um teste executado só manualmente tende a ficar desatualizado. A API muda, o teste continua parado e ninguém percebe até quebrar algo em produção. O objetivo do CLI é transformar esses testes em um gate automatizado: Rodar sem interface gráfica. Retornar erro quando uma asserção falhar. Gerar relatório para o CI. Permitir execução por ambiente, pasta, cenário ou tag. Func

Accessibility affects how millions of people interact with websites, applications, and digital services every day. Yet many digital experiences still create barriers for users with visual, auditory, cognitive, or motor impairments. To address this, organizations rely on the Web Content Accessibility Guidelines (WCAG) , the most widely recognized standard for building accessible digital products. WCAG provides a framework for designing, developing, and testing experiences that are usable by a broader range of people, regardless of ability. In this guide, we'll explore what WCAG compliance means, how the guidelines are structured, the different conformance levels, and the steps organizations can take to build more accessible digital experiences. What Is WCAG Compliance? WCAG compliance means a website, application, or digital product satisfies the accessibility requirements defined by the Web Content Accessibility Guidelines (WCAG) . These requirements are organized into testable success criteria that help organizations evaluate whether their digital experiences can be accessed and used by people with a wide range of abilities and assistive technologies. Compliance is typically measured against one of three conformance levels: A, AA, or AAA , with Level AA being the most commonly adopted standard. Who Created and Maintains WCAG? WCAG is developed and maintained by the World Wide Web Consortium (W3C) through its Web Accessibility Initiative (WAI) . The W3C is the international standards organization responsible for many of the technologies and best practices that power the web. Through the WAI, it publishes and updates accessibility standards that help organizations create more inclusive digital experiences. WCAG vs. WCAG Conformance: What's the Difference? These two terms are often used interchangeably, but they refer to different concepts. > WCAG refers to the accessibility guidelines themselves. > WCAG conformance refers to the degree to which a website, application

TL;DR: Pest PHP can test the structure of your code, not just its behavior. Write your team rules as architecture tests and CI enforces them on every commit. One such test caught a multi-tenant data leak that a human review had missed. We had a rule. Every model holding tenant-specific data must use our BelongsToTenant trait. That trait adds the global scope that keeps one clinic from seeing another clinic's data. The rule was in onboarding. It was in the code review checklist. Everyone knew it. A developer joined the team. Three weeks in they added a new model and forgot the trait. The reviewer was focused on the business logic, which was genuinely well written, and did not notice the missing trait. The model shipped. For two days one clinic could see fragments of another clinic's data in one specific report. A support ticket caught it. Our tests did not. That was the day architecture tests went into the project. What an Architecture Test Is Most tests check behavior. Given this input the function returns that output. An architecture test checks structure instead. It asserts things about how the code is organized rather than what it computes. Pest has an arch function for exactly this. // tests/Architecture/ArchTest.php arch ( 'tenant models must use the BelongsToTenant trait' ) -> expect ( 'App\Models' ) -> toUseTrait ( 'App\Traits\BelongsToTenant' ) -> ignoring ( 'App\Models\SystemSetting' ); arch ( 'controllers may not touch the DB facade directly' ) -> expect ( 'App\Http\Controllers' ) -> not -> toUse ( 'Illuminate\Support\Facades\DB' ); arch ( 'services may not depend on the HTTP request' ) -> expect ( 'App\Services' ) -> not -> toUse ( 'Illuminate\Http\Request' ); arch ( 'no env calls outside config files' ) -> expect ( 'App' ) -> not -> toUse ( 'env' ); These run in CI on every commit. Break a rule and the build fails with a message naming the rule and the file that broke it. The Tests That Earned Their Keep The tenant trait test caught four more models over

The Moment I Realised UI Testing Wasn't Enough Three years into my QA career, I thought I was doing well. I had a solid Selenium suite running. Regression coverage was green. Stakeholders were happy. Then a production incident happened. A payment API was returning incorrect amounts under a specific condition. The UI looked perfect — amounts displayed correctly after rounding. But the raw API response? Off by a significant margin. My entire test suite missed it. Every single test. Because I was only testing what users saw . Not what the system was actually doing . That incident changed how I approached QA forever. 👇 Why API Testing Is the Most Underrated Skill in QA Let me be direct about something. Most QA engineers treat API testing as a secondary skill. Something you do with Postman when a developer asks you to verify an endpoint. A quick sanity check before moving on. That's the wrong mental model entirely. Here's the truth after 7.5 years: The API layer is where your product actually lives. The UI is a presentation layer. It shows users a version of the truth. But the API? That's the truth itself. Data contracts, business logic, validation rules, error handling — all of it lives at the API layer. If you're only testing the UI, you're testing the packaging. Not the product. My API Testing Journey — Tool by Tool Let me walk you through exactly how my API testing practice evolved, and what each tool actually taught me. Stage 1 — Postman: Learning to Think in Requests Postman was my entry point. And it's still the tool I reach for first when exploring a new API. But most people use Postman wrong. They treat it like a manual testing tool — fire a request, check the response, move on. That's wasting 80% of what Postman can do. Here's how I actually use it: Collections + Environments = your real power combo // Environment variables — not hardcoded values {{ base_url }} /api/ v1 / users / {{ user_id }} // Switch between dev/staging/prod by changing one environment // No

Shifting from traditional application testing to serverless TypeScript engineering is all about shifting your perspective: you stop testing a running server, and you start testing how your function responds to events and SDK states. Here is a simple, practical example for each testing, using modern AWS SDK v3 syntax. 1. Unit Testing: Jest + Mock Payloads & SDK Client Mocking In unit tests, you don't call real AWS services. You pass a simulated API Gateway event to your handler, and you mock the AWS SDK so it returns predictable data instead of hitting live infrastructure. The Code Under Test ( src/handler.ts ) import { APIGatewayProxyEvent , APIGatewayProxyResult } from ' aws-lambda ' ; import { DynamoDBClient } from ' @aws-sdk/client-dynamodb ' ; import { DynamoDBDocumentClient , GetCommand } from ' @aws-sdk/lib-dynamodb ' ; const ddbClient = new DynamoDBClient ({}); const docClient = DynamoDBDocumentClient . from ( ddbClient ); export const handler = async ( event : APIGatewayProxyEvent ): Promise < APIGatewayProxyResult > => { const userId = event . pathParameters ?. id ; if ( ! userId ) { return { statusCode : 400 , body : JSON . stringify ({ message : ' Missing ID ' }) }; } // Fetch from DynamoDB const result = await docClient . send ( new GetCommand ({ TableName : process . env . USERS_TABLE , Key : { id : userId } })); if ( ! result . Item ) { return { statusCode : 404 , body : JSON . stringify ({ message : ' User not found ' }) }; } return { statusCode : 200 , body : JSON . stringify ( result . Item ), }; }; The Jest Unit Test ( tests/unit.test.ts ) Instead of the legacy aws-sdk-mock , the modern standard for AWS SDK v3 is aws-sdk-client-mock . import { handler } from ' ../src/handler ' ; import { APIGatewayProxyEvent } from ' aws-lambda ' ; import { mockClient } from ' aws-sdk-client-mock ' ; import { DynamoDBDocumentClient , GetCommand } from ' @aws-sdk/lib-dynamodb ' ; const ddbMock = mockClient ( DynamoDBDocumentClient ); describe ( ' Lambda Handler Unit

Originally published on the Prufa blog . In June 2026 we pointed Prufa's free audit at 50 products that had just launched on Show HN — every launch from the previous 30 days that earned at least 10 points. These are products at their moment of maximum attention: front page, real traffic, founders watching the comments. The headline numbers, from the 49 audits that completed (one site couldn't be reached by our runner): 100% of the 49 launches had at least one machine-verified finding. 78% — 38 of 49 — had at least one critical finding. 40 critical and 61 warning findings in total, every one verified by deterministic checks against captured browser evidence. No site is named in this post. The point isn't to embarrass anyone — it's that these failures are systematic, and if these teams have them on launch day, you probably do too. Methodology, briefly Each site got the same audit a free Prufa run does: a real browser loads the public pages, captures network traffic, console output, cookies, and response codes, and a fixed suite of deterministic checks grades the evidence. Same input, same verdict. Every number below is from a code-verified check — no LLM opinions are counted anywhere in this data. One honest caveat: our export keeps only the top findings per site, so the per-issue counts below are floors , not totals. The real numbers are equal or worse. What actually breaks at website launch: the numbers Sites affected (of 49) Finding Severity 38 No analytics events detected critical 24 No canonical link on entry page info 22 Cookies set without the Secure attribute warning 14 Broken links warning 12 No <h1> heading on entry page info 11 No robots.txt info 10 JavaScript console errors during page load warning 10 Missing meta description warning 8 Images missing alt text info 7 Missing Open Graph tags info 3 Tag container loads, but no analytics events fire warning 2 Canonical URL pointing to a different host critical The most common launch bug: analytics that record

Most of today went into a user-management overhaul in kickoff — my Laravel starter kit. Flyout CRUD panels, bulk actions, permission assignment, and the piece I want to talk about: account status . Active, suspended, unverified, deleted. The interesting part isn't the feature. It's the modelling decision underneath it. Where do those four states actually live ? The trap: a status column The obvious move is a status enum column on users . Set it to suspended when you suspend someone, active when you reinstate, unverified until they verify their email, deleted when they're soft-deleted. It works. Until it doesn't. Now you've got a status column and an email_verified_at column and a deleted_at from soft deletes — and all three encode overlapping truths. Soft-delete a user and forget to flip status ? Now your database says the account is both active and trashed. Verify an email but the status update fails mid-request? Drift. Every place that mutates a user becomes a place that has to remember to keep status in sync. That's not a feature, that's a maintenance tax. The signals already exist. email_verified_at tells you verified-or-not. deleted_at (soft deletes) tells you removed-or-not. The only genuinely new state is suspended — an admin deliberately blocking sign-in. So that's the only thing worth storing. What we actually store: one nullable timestamp Schema :: table ( 'users' , function ( Blueprint $table ) { $table -> timestamp ( 'suspended_at' ) -> nullable () -> after ( 'email_verified_at' ); }); That's the whole migration. Not a boolean is_suspended — a nullable timestamp. Null means not suspended; a value means suspended and tells you when. A boolean throws that second fact away for free; the timestamp keeps it at no extra cost. Same instinct as email_verified_at and deleted_at — Laravel models "did this happen, and when" as a nullable timestamp everywhere, so we follow the grain of the framework. The behaviour on the model stays tiny: public function isSuspended

Ein KI-Agent ist nur so zuverlässig wie die APIs, die er aufruft. Das Modell wählt ein Tool aus, füllt Argumente ein und sendet eine Anfrage. Wenn diese Anfrage fehlschlägt, die falsche Form zurückgibt oder hängen bleibt, trifft Ihr Agent eine selbstbewusste Entscheidung auf Basis schlechter Daten. Produktions-Agenten stehen und fallen deshalb mit einer getesteten Tool- und API-Schicht. Apidog noch heute ausprobieren Diese Anleitung zeigt, wie Sie einen Agenten erstellen, der reale Tools aufruft, und wie Sie Apidog als API-Schicht und Testumgebung verwenden. Sie definieren Tool-Endpunkte, mocken sie für die Offline-Entwicklung und schreiben Assertions, die fehlerhafte Tool-Aufrufe abfangen, bevor sie Benutzer erreichen. Was ein Agent auf der API-Ebene tatsächlich tut Reduziert auf die technische Schleife passiert Folgendes: Das Modell erhält ein Benutzerziel und eine Liste verfügbarer Tools. Es gibt einen Tool-Aufruf zurück: Tool-Name plus JSON-Argumente. Ihr Code führt den Aufruf aus, meist als HTTP-Request. Das API-Ergebnis geht zurück an das Modell. Das Modell ruft ein weiteres Tool auf oder antwortet dem Benutzer. Die kritischen Fehler entstehen fast immer in Schritt 3 und 4: Das Modell halluziniert ein Argument. Die API gibt 400 , 422 , 429 oder 500 zurück. Das Antwortschema hat sich geändert. Der Request läuft in ein Timeout. Eine Ratenbegrenzung greift mitten in der Agenten-Schleife. Wenn Sie KI-Agenten als neue API-Konsumenten betrachten, wird klar: Ihr Agent ist ein API-Client. Er braucht dieselbe Teststrenge wie jeder andere produktive Client. Die Arbeit besteht aus zwei Teilen: Tools als reale, testbare API-Operationen definieren. Prüfen, ob der Agent diese Tools unter guten und schlechten Bedingungen korrekt aufruft. Schritt 1: Tools als reale API-Operationen entwerfen Definieren Sie jedes Tool zuerst als API-Endpunkt in Apidog. Behandeln Sie Tool-Schema und API-Schema als denselben Vertrag. Beispiel: Tool: get_weather API-Operation: GET /weather Paramet

There's a class of bug that's maddening: it passes every test you have, then crashes in the user's face. I hit one in the admin UI of laravel-config-sso today, and the real fix wasn't changing an icon name — it was writing a test that could see the bug in the first place. The bug: wrong icon name, crashes only at runtime The admin UI uses Flux . Flux resolves icons through <flux:delegate-component> , and it throws for a name that doesn't exist: Flux component [icon.ellipsis] does not exist. It's an easy mistake. Flux ships Heroicons , not Lucide. So your Lucide reflexes lie to you: You type (Lucide) Flux wants (Heroicon) ellipsis ellipsis-horizontal trash-2 trash eye-off eye-slash Why feature tests don't catch it Here's the interesting part. I had a feature test that hits the admin route and asserts 200. Green. But the real UI crashes. How? Because in the headless test harness, Flux renders icons as no-ops. No real <flux:delegate-component> boots, so the icon name never gets resolved. The crash only surfaces under a full boot ( testbench serve ) — exactly where your automated tests don't go. Analogy: it's like a spell-checker that only runs when you print the document, not while you type. Your tests type away happily. The crash waits at the printer. The fix: a static test that reads the Blade and validates every icon Instead of relying on runtime, I wrote a Pest test that reads the Blade view, extracts every icon name (static and inside dynamic expressions), and asserts Flux actually ships a stub for each one: $fluxIconStubs = base_path ( 'vendor/livewire/flux/stubs/resources/views/flux/icon' ); it ( 'only references Flux icons that exist' , function () use ( $fluxIconStubs ) { expect ( is_dir ( $fluxIconStubs )) -> toBeTrue ( "Flux icon stubs not found" ); $view = file_get_contents ( __DIR__ . '/../../resources/views/livewire/sso-providers.blade.php' ); // Static `icon="name"` plus quoted tokens inside dynamic // `icon="{{ $cond ? 'eye-slash' : 'eye' }}"` expressio

Here's a fun one. You build a package that backs up an app's config — the .env plus the settings stored encrypted in the database — into a single password-protected ZIP. The whole selling point is portability : take a backup on server A, restore it on server B, even when the two servers have different APP_KEY s. Then you write a test that actually changes the key during a restore, and it fails. The DB settings come back garbled. Turns out the bug wasn't in the encryption at all. It was in a cache I forgot was there. Today I shipped 1.1.0 of laravel-config-backup and this portability fix was the headline. Let me walk through it, because the lesson generalizes way beyond this package. Why APP_KEY portability is even a thing Laravel encrypts things with APP_KEY . Encrypted Eloquent casts, signed cookies, sessions — all of it keys off that value. So if you naively mysqldump a table with encrypted columns and load it onto another server, every encrypted column is now ciphertext that the new key can't decrypt. Dead data. The trick this package uses is to store the archive contents decrypted . When I export the database, rows go out through their casts , so an encrypted column becomes a plain value inside the ZIP (the ZIP itself is AES-256 password-encrypted, so it's not sitting around in plaintext). On import, each row is written back through the model , which means the cast re-encrypts it with whatever APP_KEY is active on the destination. Server A (key A) Archive (decrypted) Server B (key B) ──────────────── ─────────────────── ──────────────── settings.payload ──decrypt──▶ "Portable" ──import──▶ settings.payload (ciphertext A) (cast) (cast) (ciphertext B) Think of it like shipping furniture: you don't ship the assembled wardrobe through a doorway it doesn't fit, you flat-pack it and reassemble at the destination with the screws you have there. The restore sequence A restore that also brings a new .env has to be careful about ordering . Here's the real flow: public func

SaaS teams using AI-driven experimentation platforms (also called A/B testing automation or CRO...

Two CI workers kick off at the same moment. Both sign up a test user, both poll the shared QA Gmail account for "the" verification email, and worker #7 grabs the message that belonged to worker #12. The test passes. The wrong test. You spend an afternoon staring at a green build that should've been red. Shared inboxes are the single biggest source of flakiness in email-dependent E2E tests, and every workaround — catch-all forwarding rules, label rules scoped per PR, OAuth tokens living on the runner — adds another moving part that breaks on its own schedule. The fix is structural: every test gets its own address, on infrastructure your suite provisions and destroys. One wildcard, infinite addresses The E2E email testing recipe sets this up with one CLI command: nylas inbound create e2e You get back an inbox ID and a wildcard pattern shaped like e2e-*@yourapp.nylas.email . From there, each test mints a unique address under the wildcard — e2e-<uuid>@yourapp.nylas.email — and there's nothing to provision per address. You don't pay or configure per address either; the wildcard is just a convention, so burn UUIDs freely. Mail flows through MX records hosted on the Nylas side, which means zero DNS work in your own zone (the tradeoff: addresses live under *.nylas.email ). The Playwright fixture is two pieces — an address minter and a poller: export const test = base . extend < Fixtures > ({ testEmail : async ({}, use ) => { await use ( `e2e- ${ randomUUID ()} @yourapp.nylas.email` ); }, pollInbox : async ({ testEmail }, use ) => { const poll = async ( timeoutMs = 30 _000 ) => { const deadline = Date . now () + timeoutMs ; while ( Date . now () < deadline ) { const out = execSync ( `nylas inbound messages ${ process . env . INBOX_ID } --json --limit 50` , ). toString (); const match = JSON . parse ( out ). find (( m ) => m . to . some (( t ) => t . email === testEmail ), ); if ( match ) return match ; await new Promise (( r ) => setTimeout ( r , 1500 )); } throw new Error (

Most testing tutorials ignore e-learning completely. Here's how to build a Playwright test suite that validates your SCORM packages actually work across LMS platforms. Why E-Learning Testing Is Different If you've ever published a SCORM package to an LMS and watched it silently fail — no completion recorded, quiz scores vanishing, navigation broken — you know the pain. E-learning content doesn't behave like a typical web app. It runs inside an LMS-provided iframe, communicates through a JavaScript API (the SCORM Runtime), and its behavior changes depending on which LMS hosts it. Manual QA across even 3-4 LMS platforms is slow and error-prone. In this tutorial, I'll walk you through setting up Playwright to automate SCORM package testing — from basic content loading to verifying API calls and completion status. Prerequisites Before we start, make sure you have: Node.js 18+ installed Playwright ( npm init playwright@latest ) A SCORM 1.2 or 2004 package (a .zip file containing your e-learning content) A local LMS for testing — we'll use SCORM Cloud (free tier) or a simple SCORM API shim Step 1: Set Up a Local SCORM Runtime Shim Testing SCORM content requires an API that mimics what an LMS provides. Rather than spinning up a full Moodle instance, we'll create a lightweight shim. Create a file called scorm-api-shim.js : // scorm-api-shim.js // Mimics the SCORM 1.2 Runtime API that an LMS would expose window . API = { _data : {}, _initialized : false , _calls : [], LMSInitialize : function ( param ) { this . _initialized = true ; this . _calls . push ({ method : ' LMSInitialize ' , param , timestamp : Date . now () }); console . log ( ' [SCORM] LMSInitialize called ' ); return " true " ; }, LMSGetValue : function ( key ) { this . _calls . push ({ method : ' LMSGetValue ' , key , timestamp : Date . now () }); return this . _data [ key ] || "" ; }, LMSSetValue : function ( key , value ) { this . _data [ key ] = value ; this . _calls . push ({ method : ' LMSSetValue ' , key