200 OK Is Not Enough: Why Bot-Protected Sites Still Return Bad Data

Your crawl job finished successfully. That doesn't mean it got the data. Every scraping pipeline has a monitoring dashboard, and every monitoring dashboard has the same blind spot: it tracks whether requests succeeded, not whether the content that came back was real. A job that completes with a wall of green 200 status codes looks healthy. It can also be quietly wrong, page after page, for weeks, because a 200 response only tells you the server accepted the request. It says nothing about whether you're looking at the actual page or a version built specifically for visitors the site doesn't fully trust. That gap between "the request succeeded" and "the data is correct" is where most silent pipeline failures live, and it's getting wider as anti-bot systems get more sophisticated about what they serve instead of an outright block. What a "successful" response can actually contain A block used to be simple to detect: a 403, a 429, a connection reset. Modern anti-bot systems increasingly prefer a different approach, because an obvious block tells the requester exactly what happened and invites a fix. A soft block, served with a 200, doesn't. In practice, that 200 can be a challenge page, an interstitial that looks like real content in the raw response but is actually a JavaScript-driven verification step (a "just a moment" style page, a hidden CAPTCHA iframe, a redirect loop disguised as a normal page load). It can be a cached fragment, an old snapshot of the page served to anything that looks automated, so the price, availability, or listing you scraped is stale even though the request itself worked fine. It can be an empty state, a search results page or listing that legitimately returns "no results" to a request pattern the site doesn't recognize, even though a real visitor would see dozens of items. And increasingly, it can be a partial HTML shell: the server response contains the page skeleton, but the actual content only renders after JavaScript executes in a real