Four false positives in one evening: telling a broken web app from a broken measurement

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor