标签：#Testing

I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading. Grading gets its own post for a reason. Build the grading wrong, and the score lies to you. 5 wrong out of 29 — can I ship? No idea. Because "which 5" is missing. If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled. So don't grade by count. Grade by severity. Severity = "can a human undo this?" My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck. FATAL Wrong goods on the truck. Cannot be undone RISKY Confirmed something ambiguous without asking. Right this time — fatal next time MISSED Dropped an order. The customer calls. Fixable HARMLESS Over-asked "please confirm." Just slower One principle falls out of this: A wrong confirmation is worse than no confirmation. Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen. The same 28/29 splits two ways FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back Same score. Opposite fates. Two accidents my grader caused The grader is code I wrote. Like all code I write, it had bugs. Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace. The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo . Accident two — penalizing a good answer. For "250 b

Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's wasted points. Models rarely fail the normal cases. What fails is everything that isn't normal. My 29 questions broke down like this: Normal orders 4 Things that aren't orders 6 ← the biggest group Changes & cancellations 4 Ambiguous ones 5 Typos & extreme shorthand 3 After learning kicks in 7 Normal is the smallest group. On purpose. Why "not an order" gets the most questions The worst accident for this program is shipping something nobody ordered. So the exam should aim at that accident more than anything else. What are the dimensions of the 250 shipping box? Product name: present. Number: present. But it's not an order. It's a question. A program that treats "product name spotted" as "order detected" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request. Changes and cancellations are nastier. I ordered 5 boxes of the 250 — please send only 3 Two numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice. Plant traps in the catalog too It's not just about hard questions. Make the data itself messy. Two kinds of clear tape — 48mm and 60mm Five products starting with "250" Different pack sizes per box — 50, 40, 25, 10 sheets A few loose items with no box unit at all One reason: real data already looks like this. A real product catalog always has near-twins. Run the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's no

The MCP C# SDK hybrid sessions option solves an awkward upgrade boundary: some clients still use the 2025-11-25 initialize handshake and depend on sessions, while clients on 2026-07-28 expect every HTTP request to stand alone. I want both groups to reach one ASP.NET Core endpoint without making modern clients downgrade or stripping useful behavior from legacy clients. The stable C# SDK 2.2.0 release added exactly that path with HttpServerSessionMode.StatefulForInitializeClients . The release notes describe it as hybrid stateful/stateless serving, and the official session-mode guide spells out the per-request behavior. Why one global session switch fails The 2026-07-28 MCP revision removed the initialize handshake and Mcp-Session-Id from its wire format. Client identity, capabilities, and protocol version travel with each request instead. The final specification announcement explains why the core moved toward request/response statelessness. That creates a migration choice for an existing server. With HttpServerSessionMode.Stateful , initialize-era clients receive full sessions. A modern request is refused so a dual-path client can fall back to the older handshake. Compatibility is preserved, but the client does not use the new protocol natively. With HttpServerSessionMode.Stateless , every request is independent. That is the right default for servers that do not need session state, unsolicited notifications, resource subscriptions, or older server-to-client flows. It may be too abrupt when deployed clients still rely on those features. Hybrid mode makes the decision from the incoming request instead of applying one choice to the endpoint. Configure MCP C# SDK hybrid sessions The server configuration is deliberately small: builder . Services . AddMcpServer () . WithHttpTransport ( options => { options . SessionMode = HttpServerSessionMode . StatefulForInitializeClients ; }) . WithTools < DemoTools >(); app . MapMcp ( "/mcp" ); An initialize-era client sends an initial

You picked a free model because the answers looked good. Good answers are not an endpoint. An endpoint is the model plus the server plus the network. Demos pass. Pipelines stall. The model was rarely the problem. So why do we keep benchmarking only the model? Because it is easy. You paste a prompt. You read the output. You declare a winner. The server never gets a vote. This post is a reproducible benchmark. It measures the pair, not the model. Run it before you wire any free endpoint into CI. The Pair, Not the Model Most evaluations compare answers. You paste a prompt. You judge the output. You pick a winner. That measures the model. It ignores the server. Free model access usually means a shared endpoint. A free server option means shared tenancy. Other users share the CPU, memory, and network. Your latency is their latency. Your timeout is their timeout. Here is the scenario I keep seeing. A team evaluates a free model on Friday. The answers look great. They wire it into CI on Monday. By Wednesday, the pipeline is red. The model did not change. The server did. A neighbor started a batch job. Now every request queues behind it. I applied the same harness to MonkeyCode's free model access and their free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not trust the demo. I built a harness instead. The Harness A benchmark needs three things. A fixed prompt set. A concurrency ladder. A pass/fail table. Here is the harness I use. #!/usr/bin/env python3 """ Benchmark a model endpoint as a pair: model + server. """ import argparse import asyncio import json import statistics import time import httpx PROMPTS = [ " Say OK. " , " Classify this log line: ERROR disk full " , " Return one word: is 429 a retryable status? " , ] async def fire ( client , url , payload , sem , timeout = 30 ): async with sem : start = time . perf_counter () try : r = await client . post ( url , json = payload , timeout = timeout ) return r . sta

Conclusion first: a free model drafted a working C++17 directory hasher in one pass. The draft compiled, ran, and was still wrong. A differential test against standard system tools found three real bugs before the tool ever touched a production cache. Generation was the cheap part. Verification was the deliverable. Background I needed a deterministic hash of a directory tree. The use case was cache invalidation for a small build pipeline: if any file content, name, or symlink target changes, the cache key must change. If nothing changes, the key must stay identical across machines and across checkouts. Hand-writing the tool is maybe 200 lines of std::filesystem code. The happy path is easy. The risk lives in ordering, symlinks, and metadata leaking into the hash. I turned the task into an experiment. MonkeyCode's free model access and free server option meant the model ran on a remote server while I kept verification on my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The plan: let the model write the first version, then prove or disprove it against a reference oracle. The Contract The goal was not "a tool that compiles." The goal was a tool that matches a reference implementation on every input I could generate. I wrote the contract in three sentences: Same tree → same hash, on any machine. Different content, name, or symlink target → different hash. File metadata (mtime, inode) must not affect the hash. Implementation Step 1: the prompt. I gave the model the contract, the C++17 standard, and one constraint: a single file with no dependencies beyond the standard library. Step 2: the draft. The model returned one .cpp file in a single response. It compiled on the first try. That is the exact moment where most workflows stop. This one did not. Step 3: the reference oracle. Instead of reviewing the code line by line, I built a harness that compares the tool against a shell pipeline: find " $tree " -printf '%P\0' | sort -z | wh

MCP x-mcp-header validation is easy to miss because the annotation looks like ordinary JSON Schema metadata. On the 2026-07-28 Streamable HTTP transport, it is a wire contract: the client copies selected tool arguments into Mcp-Param-* headers, intermediaries can act on those headers, and the server checks them against the JSON-RPC body. I treat that contract as something to test before a tool reaches tools/list . A bad suffix, an unsupported type, or an unreachable annotation makes the whole tool definition invalid. Silently accepting it only moves the failure to a harder place to diagnose. Why the same value travels twice The final Streamable HTTP specification mirrors request metadata into HTTP headers so a load balancer, gateway, or WAF does not need to parse JSON-RPC. A server can add x-mcp-header to a tool property: { "type" : "object" , "properties" : { "region" : { "type" : "string" , "x-mcp-header" : "Region" } } } A call with "region": "us-west1" then carries: Mcp-Param-Region: us-west1 The official C# SDK can generate that schema from a parameter attribute: [ McpServerTool ] public static string ExecuteSql ( [ McpHeader ( "Region" )] string region , string query ) => $"Queued for { region } " ; Current C# SDK v2 tool documentation describes both schema generation and automatic header projection. The feature is on the stable v2 line; it is not necessary to pin an earlier preview or release candidate. MCP x-mcp-header validation rules The final tool definition rules are deliberately narrow. The annotation value must be a non-empty HTTP field-name token and must be unique without regard to case. Region and region therefore collide. Control characters, spaces, and separators such as a colon are not valid suffix characters. Only string , integer , and boolean properties can be mirrored. JSON Schema number is excluded, and integer values must stay between -(2^53 - 1) and 2^53 - 1 so every conforming implementation can represent the value exactly. Reachability i

Free model endpoints are seductive. Zero cost. Zero setup. Zero reason to trust them. I don't trust demos. I trust failure modes. So I built a small evaluation harness. It tests one thing: can a free model endpoint gate a pull request for secrets? This is not a benchmark. It's a repeatable experiment. You can run it in an afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model endpoint and the free server option for the test. No quotas. No hardware claims. Just a harness and a rubric. Why I stopped trusting free endpoints Free endpoints look great in a demo. You paste a diff. The model finds the secret. Everyone claps. Then you wire it into CI. The JSON breaks. The latency spikes. The model misses a private key. The demo didn't show that. An evaluation will. The experiment I designed a 40-minute test. It answers one question: where does the free endpoint perform well, and where does it break? The dataset is 30 synthetic diffs. Fifteen contain real-looking secrets. Fifteen are clean. Each diff is small. Each diff has one clear change. The prompt is strict. The model must return JSON. No prose. No apologies. Just a verdict. # eval_secret_gate.py # Simplified harness. Adapt to your client SDK. import json , time def classify ( client , diff : str ) -> dict : prompt = f """ You are a secret scanner for code review. Return ONLY JSON with this shape: {{ " contains_secret " : true, " line " : 12, " type " : " aws_access_key " }} Diff: { diff } """ start = time . time () response = client . complete ( prompt , model = " free " , server = " free " , # free server option ) latency = time . time () - start return { " latency " : latency , " raw " : response } def evaluate ( client , diffs , runs = 3 ): for i , diff in enumerate ( diffs ): for run in range ( runs ): yield i , run , classify ( client , diff ) The harness is deliberately small. It measures five things. Accuracy. JSON validity. Latency. Variance. Fa

I'd already done a lot right by the time I started writing tests for Beaver-Auth . Every module had gone through multiple rounds of deliberate review. Enumeration protection, hashed tokens, refresh rotation, TOTP replay defense — the design was solid, and I knew it was solid, because I'd thought hard about every piece of it. Then I wrote 238 tests against the actual code, and found 8 real bugs. Some of them were the kind that would have silently broken production on day one. This post isn't about the bugs specifically — it's about the gap between "I reviewed this carefully" and "this is shippable," and why that gap is bigger than most of us assume, even when the reviewing was genuinely careful. "Passing tests" and "shippable" are different claims Here's the trap I nearly walked into: I'd built a solid test suite covering the core auth flows — registration, login, verification — and every test passed. It felt done. But passing tests only tell you the code does what the tests expect. If the tests were written from the same mental model as the code, they'll happily confirm a bug is correct behavior, because both the code and the test agree on the same wrong assumption. The fix wasn't "write more tests." It was testing against the real, integrated system — not a hand-built mock of my own logic, and not testing modules in isolation from what actually calls them. A few of the bugs below only surfaced because a test exercised the real dependency chain instead of assuming it worked. Bug 1: TypeScript let an argument-shift bug compile clean This is the one that scared me most. Beaver-Auth dispatches background work (like sending a verification email) through a TaskDispatcher interface: interface TaskDispatcher { dispatch ( taskName : string , payload : unknown , handler : () => Promise < void > , onFailure ?: ( error : unknown ) => Promise < void > | void , ): Promise < void > } The default implementation had drifted to a different signature — missing the payload parameter e

Successfully onboarding an automated test suite to Harness CI requires configuring infrastructure placeholders, secrets, pipelines, and branch protection rules. Here is a 10-step checklist to help you onboard your end-to-end (E2E) automation pipelines seamlessly. Step 1: Replace Infrastructure Placeholders Ensure your pipeline YAML definitions (e.g., .harness/e2e-poc.yaml and .harness/e2e-regression-parallel.yaml) contain your specific environment values: ORG_ID: Harness Organization Identifier PROJECT_ID: Harness Project Identifier GIT_CONNECTOR: Harness Git Connector for GitHub Enterprise access APP_REPO_NAME: Target repository in owner/repo format K8S_CONNECTOR: Kubernetes connector for build infrastructure K8S_NAMESPACE: Kubernetes namespace where build pods run Step 2: Configure Environment Secrets In Harness, set up the following runtime secrets: CONNECT_URL CONNECT_USERNAME CONNECT_PASSWORD Step 3: Setup PR Validation Pipeline Import your short-run pipeline YAML into Harness. Save it as your PR Validation Pipeline. Run a manual validation test using runtime overrides: TargetEnv = qa cucumberTags = @smoke Step 4: Verify Artifact Generation Confirm that the initial execution correctly generates and uploads all required outputs: JUnit Report: reports/junit-report.xml Test Reports: reports/** Failure Artifacts: test-results/** (screenshots, traces) Step 5: Setup Nightly Parallel Pipeline Import your parallel pipeline YAML into Harness. Save it as your Nightly Regression Pipeline. Run a manual validation test with target concurrency parameters: TargetEnv = qa cucumberTags = @regression cucumberParallel = 4 Step 6: Configure Automated Triggers & Branch Protection PR Trigger: Configured on pull requests with cucumberTags= @smoke . Nightly Schedule Trigger: Configured on a nightly cron schedule with cucumberTags=@regression and cucumberParallel=4. GitHub Branch Protection: Enable branch protection on target branches requiring the Harness PR pipeline status check to p

Integrating end-to-end (E2E) automation suites into enterprise CI/CD pipelines requires robust reporting, dynamic execution controls, and seamless artifact management. Here is a guide on setting up a Node.js + Playwright + Cucumber.js test suite using Harness CI , configured with dual-repository dependencies, parallel execution capabilities, and dashboard-ready reporting. Key Architectural Setup Two-Repo Architecture: Repository A (Application Automation Repo): Contains application-specific feature files, page objects, and pipeline definitions. Repository B (Shared Framework Repo): Hosts core framework utilities, custom assertions, and base drivers consumed as a pinned dependency. Tech Stack: Node.js, Playwright, Cucumber.js, Allure/JUnit reporting. Step 1: Configure Harness Connectors & Secrets Set up these foundational resources within your Harness account: Connectors: GIT_CONNECTOR: Grants access to both application and framework GitHub repositories. K8S_CONNECTOR: Manages the Kubernetes build infrastructure. Secrets: CONNECT_URL, CONNECT_USERNAME, and CONNECT_PASSWORD (and proxy settings if required). Step 2: Configure Pipelines Import your execution configurations using YAML files inside .harness/: Standard Run (.harness/e2e-poc.yaml): Used for fast PR checks. Parallel Regression (.harness/e2e-regression-parallel.yaml): Used for scheduled, high-volume regression runs. Replace placeholders such as , , and to map to your cluster environment. Step 3: Define Pipeline Triggers Set up two primary execution workflows: Pull Request (PR) Trigger: Event: Pull Request to main/POC branch. Runtime Variables: cucumberTags= @smoke Scheduled Nightly Trigger: Event: Scheduled Cron. Runtime Variables: cucumberTags=@regression, cucumberParallel=4 Step 4: Test Report & Artifact Collection To ensure test metrics display properly on the Harness dashboard, configure both JUnit parsing and raw artifact archiving. Generated Outputs: reports/junit-report.xml (parsed by Harness for test

Optimizing test execution speed is essential for keeping build pipelines lean. Depending on how your framework is structured, you can achieve full parallel execution either natively using TestNG ** or dynamically using a **Custom Excel Allocator . Here is a step-by-step guide on configuring both approaches, along with a comparison to help you choose the right strategy. Strategy 1: Native TestNG Parallelization (Recommended for Code-Native Suites) TestNG natively supports parallel execution at the methods, classes, tests, or instances level using its XML configuration or Maven parameters. 1. Update testng_regression.xml Modify the tag to set the execution mode and thread pool size: <suite name= "Regression" parallel= "methods" thread-count= "10" > 2. Configure pom.xml for Dynamic Overrides Allow developers and CI pipelines to override execution settings without altering XML files by adding these lines inside the block of the maven-surefire-plugin: <parallel> ${parallel} </parallel> <threadCount> ${threadCount} </threadCount> 3. Execution Commands Default Run: mvn clean test -P runTestNGTests Override Thread Count Dynamically: mvn clean test -P runTestNGTests -DthreadCount = 15 Full Parallel Execution (Match CPU Core Count): mvn clean test -P runTestNGTests -Dparallel = methods -DthreadCount = 24 Strategy 2: Custom Allocator & Run Manager (For Excel-Driven Suites) If your framework relies on an Excel-driven Run Manager to parse keyword flows and data sheets dynamically, parallelism is managed via a custom ExecutorService fixed thread pool. Execution Command mvn clean test -P runAllocator How it works: The allocator reads active test rows (Execute=Yes), dynamically assigns thread pools based on target thread properties, and dispatches concurrent runs. Comparison: Allocator (Run Manager) vs. Native TestNG Feature Allocator (Run Manager) TestNG Native Entry Point allocator.Allocator.main() via Maven Exec Plugin maven-surefire-plugin executing testng.xml Test Selection Re

Accelerating test execution in a Hybrid Automation Framework (combining Data-Driven and Keyword-Driven architectures) requires an efficient parallel execution strategy. By dynamically mapping keyword actions and test data rows to concurrent threads, you can drastically reduce execution time without compromising framework design. Here is a guide on setting up parallel execution using a central Allocator and Run Manager. 1. Overview of the Setup The framework leverages a Run Manager sheet to map keywords to execution steps and pull test data dynamically. Parallelization works by assigning NumberOfThreads to match the exact number of active test cases marked for execution. Key parameters are configured globally inside the Global Settings.properties file. 2. Configuration Steps a. Set the Number of Threads Total the number of test scenarios marked with Execute=Yes across your target keyword and data sheets. Set NumberOfThreads equal to this count. Example: If your Run Manager sheet contains 42 test iterations set to Execute=Yes, update your configuration: NumberOfThreads = 42 b. Disable Profile-Based Execution (If Not Needed) For clean parallel browser execution, set EnableProfile=False. If user profiles are required to maintain session state across keywords, set UseMultiProfile=True and configure separate profile directories per thread to avoid file-lock conflicts. c. Prepare the Run Manager Flag every keyword test case intended for the current run with Execute=Yes. The allocator will read these rows, pair them with their corresponding data sets, and dispatch them to the thread pool. 3. Executing the Test Suite Trigger the allocator flow via Maven: mvn clean test -P runAllocator The allocator reads the mapped keyword sheets and test data, initializes the specified NumberOfThreads, and executes the tests in parallel. 4. Handling Multiple Keyword & Data Sheets Option 1: Use a Master Control Sheet (Recommended) Consolidate execution rows into a single master sheet (e.g.,

The concrete problem A security pull request can be green for the wrong reason. Unit tests may pass, the vulnerable endpoint may return a different status code, and a scanner may stop reporting the original finding. None of those results necessarily shows that the attacker lost the capability that mattered. The same identity might reach the sensitive action through another route, inherit a broader token, or trigger an equivalent workflow with slightly different input. This becomes especially uncomfortable when an automated tool proposes or reviews the fix. A plausible patch explanation is not behavioral evidence. The reviewer still needs to know which identity was used, which preconditions were established, which requests ran, where privilege was gained before the fix, and at which exact step the patched build denied it. Without that trace, “fixed” is partly an assertion about code rather than an observation of the attack path. The current signal On August 17, Wiz described a GitHub Actions script-injection flaw in a Snowflake repository. The vulnerable workflow change reached production on June 18 and Wiz reported exploiting it on June 23. The final squash commit credited Copilot Autofix as a co-author, while AI-assisted review did not flag the injection. Wiz later clarified that it could not determine whether the code change itself was AI-generated. That distinction matters: the lesson is about assurance around AI-assisted workflows, not proof that a model wrote the bug. The Hacker News discussion was active when RayTally captured it at 2026-08-18 00:33 UTC: 306 points, 123 comments, and rank 5. Those are historical attention numbers, not market validation. The useful engineering signal is narrower. Teams now have a concrete incident in which an apparently protective condition and an escaping routine still produced a reachable credential-exfiltration path. Bright STAR and StackHawk show that dynamic testing in CI is already real. Bright documents building and star

This is an English write-up of a post from my Japanese dev diary. Original: https://saas-diary.com/tech-log/backup-restore-drill-automation/ For over a year, my backup job has reported success every single night. Green check, every day, no exceptions. Then I asked myself one question and went cold: "How many times have I actually restored from it?" Zero. Not once. "It was backed up" and "it can be restored" are different states My setup has two paths. One mirrors all source to a private repo. The other packs the things I can never recreate — notes, config, and Android signing keys — into an encrypted bundle and ships it to a private channel every night. Both were green every day. But green only proved the upload finished . It never proved the contents were right, or that the archive could even be opened. Within one month, I had two failures that stayed green the whole time. Failure 1. The collector for signing keys used three hardcoded paths. I kept shipping new apps, so the number of keys kept growing — but the collector didn't. By the time I noticed, 7 of 10 keys were missing from the backup . Five of those apps were live on the store. If my machine had died, I could never have shipped an update for them again. The backup reported success every night through all of it. Failure 2. The mirror push failed 7 days in a row (a large binary hit the host's file-size limit). But the script printed "✅ done" and returned exit code 0 even when one half failed. A failure that isn't visible isn't a failure — it's a time bomb. So I automated a restore drill Once a month, a job now does this: Rebuild the encrypted bundle (without shipping it) Actually decrypt it with the stored passphrase Extract it and count what's inside Check the mirror is not stalled (latest commit timestamp via API) Delete the scratch folder and the generated bundle The encryption is openssl-compatible AES-256-CBC with PBKDF2 (SHA-256, 100k iterations). I deliberately avoided depending on the openssl binary,

What Is Software Testing? 🧪 Software testing is the process of checking software to make sure it works correctly and does what it is supposed to do. For example, when we use a login page, we can test: Correct username and password Wrong password Empty username Empty password Forgot password option The goal is to find bugs and problems before the software is used by customers. Why Is Testing Important? Testing helps developers and companies: Find bugs Improve software quality Provide a better user experience Prevent problems after release Even a small bug can sometimes cause a big problem, so testing is an important part of software development. Manual Testing In manual testing, a tester checks the application manually without using automation scripts. For example, a tester can open a website, enter different inputs, click buttons, and check whether the expected result appears. Automation Testing In automation testing, we use tools and programming to test software automatically. Some popular tools are: Selenium Playwright Cypress Automation is useful when the same tests need to be performed many times. Conclusion Software testing is an important part of creating reliable software. If you are a beginner, you can start with manual testing , then learn SQL, API testing, and automation testing .

xUnit 4.0.0 makes full test-case parallelization an explicit option. That is useful, but xUnit 4 ParallelMode.All changes a quiet assumption in many suites: tests in the same class, including separate rows of one theory, may now overlap. A static fake, shared fixture, temporary file, or database record that was safe under collection-level parallelism can become a race. I treat this as an isolation change, not a speed switch. Before enabling it across a suite, I want a deterministic failure that proves the risk and a deterministic check for each guardrail. What xUnit 4 ParallelMode.All changes The xUnit.net v3 4.0.0 release notes describe full test-case parallelization as a new feature. The default is still ParallelMode.Collections , so upgrading does not silently enable the broader mode. I have to opt in at the assembly level: using Xunit.Sdk ; using Xunit.v3 ; [ assembly : Parallelization ( Mode = ParallelMode . All , MaxThreads = 2 , Algorithm = ParallelAlgorithm . Conservative )] With Collections , tests within a collection are serialized. With All , every test case is eligible to run beside every other test case. That includes two cases from the same class and two pre-enumerated rows from the same theory. The official parallel test execution guide documents the modes, algorithms, and available opt-out scopes. I set MaxThreads = 2 in the sample so the scheduling condition is easy to inspect. It is a demonstration setting, not a recommendation for CI. The right value depends on available CPU, memory, and the external systems touched by the tests. Before changing the mode, I scan for mutable static fields, IClassFixture and ICollectionFixture implementations, fixed file names, environment-variable changes, test servers bound to fixed ports, and records addressed by shared IDs. I also check theory data sources for objects that rows can mutate. That inventory tells me whether the resource should become concurrency-safe, receive a unique per-test identity, or stay beh

AI evals and analysis Let's say you're testing out new AI tools. Perhaps you implement and...

Ultimate IDOR Testing Checklist Phase 1: Setup & Target Identification [ ] Create Test Accounts: Create two accounts (Attacker and Victim) for safe testing of destructive requests (POST/PUT/DELETE). [ ] API Identification: Find JSON endpoints over rendered HTML. [ ] Sensitivity Analysis: Target critical functions first (password reset, account recovery, financial data, DMs, user management). [ ] ID Audit: Check if endpoint is private or public and contains any kind of ID parameter. [ ] ID Leakage: Check for IDs leaked via other API endpoints or public pages (public profile pages, listings). [ ] Map Clients: Collect web/mobile clients, open APIs from decompiled mobile (jadx/apktool), and swagger/openapi if present. Phase 2: Direct ID Substitution & Enumeration Technique Scenario to Test (Attacker ID=10, Victim ID=9) Basic ID Flip GET /api/v5/users/10 -> GET /api/v5/users/9 Incremental Numeric Brute Force Loop over sequential numeric IDs (decrement/increment from own ID). Non-Numeric ID Substitution Replace param with email / username / UUID. Complex ID Brute Force Brute force short alphanumeric segments (last 1–4 chars). Predictable ID / Combined ID /user/2222/data/3333 — change one or both parts. Hashed/Derived IDs (MD5/SHA1 pattern) Detect hashed IDs, create accounts to infer mapping, try replacing derived hashes. Phase 3: Path and URL Manipulation Bypasses Technique Scenario to Test (Attacker ID=10, Victim ID=9) Trailing Slash GET /api/v5/users/9 -> GET /api/v5/users/9/ Double Slashes / Obfuscated Path GET /api/v5/users//9 or GET /api/v5/users/./9 Case Variation / Key Swapping /api/User?id=123 vs /api/user?id=123 or user_id ↔ userid Path Traversal / Mixed Paths POST /users/delete/my_id/../victim_id Wildcard Substitution GET /api/users/* or GET /api/users/user_id Fuzz Keywords in Path GET /api/v3/users/12345 -> /api/v3/users/all SQLi Quick Check GET /api/v3/users/12345' Phase 4: Logic & Endpoint Bypasses Technique Scenario to Test Version Downgrading GET /v3/user/1

When every unauthorized request becomes the same dashboard line, diagnosis turns into guessing. ASP.NET Core 10 authentication metrics give me a better split: did the handler have nothing to authenticate, reject supplied credentials, or accept them? That distinction matters because a client deployment that drops credentials needs a different response from a surge of malformed or expired credentials. ASP.NET Core 10 added built-in authentication and authorization instruments to System.Diagnostics.Metrics . I can collect them without rewriting each handler, and I can lock their behavior into an offline test before wiring up a production exporter. Why one 401 hides two different problems A protected endpoint normally challenges an unauthenticated caller. The final status is 401 whether the caller sent nothing or the handler rejected what it received. The authentication duration histogram exposes the missing context through aspnetcore.authentication.result : Result What the handler reported A common interpretation none No authentication result No applicable credentials were available failure Authentication failed Supplied credentials were rejected or processing failed success A principal was created Authentication completed successfully _OTHER Another framework result Preserve it as an explicit catch-all none is a handler result, not a universal synonym for “missing Authorization header.” A policy scheme or custom handler can make a different choice. I verify the behavior of the schemes I actually deploy instead of building an alert from the label alone. Likewise, success means the handler produced an authentication ticket. Authorization can still deny that principal, so it does not promise a 2xx response. The separate aspnetcore.authentication.challenges counter answers another question: how often was a scheme challenged? Both a none result and a failure result can be followed by a challenge, so challenge count cannot replace the result split. A challenge is an authent

Fragile element locators are one of the primary drivers of test flakiness in UI automation. Relying on auto-generated, deeply nested CSS selectors or long XPath expressions makes your test suite sensitive to minor layout changes, styling refactors, and DOM updates. Adopting a clear locator strategy simplifies maintenance and ensures tests remain reliable as applications evolve. Core Principles for Locator Selection Prioritize Intent-Revealing Attributes: Always prefer dedicated, stable testing attributes such as data-test, data-testid, or data-qa. Avoid Style-Driven Locators: Steer clear of brittle, structure-dependent CSS paths (e.g., div > div > span:nth-child(2)) and complex XPath queries unless absolutely necessary. Preferred Selector Patterns Buttons & Actions: button[data-test="login-submit"] Content & Inputs: [data-testid="product-name"] Practical Migration Tips Centralize Locators: Group and manage all selector definitions inside dedicated Page Object Model (POM) files rather than hard-coding strings within step definitions or tests. Collaborate for Testability: If a critical UI element lacks a distinct test attribute, submit a quick PR to your developer team to add a dedicated data-test attribute. Automate Audits: Implement a lightweight audit script in your workflow to scan and flag missing data-test attributes across key target pages before running full regressions.