Your HTML is fine. The CDN still blocks the bot.

This started as a comment @wrencalloway left on my last post. It was sharp enough that it deserved more than a reply — so here's the full version. You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs. Except the crawler never fetches your URL. It asks the CDN, and the CDN says 403. Your content is perfect and unreachable. This is the layer underneath the one everyone talks about — and it's invisible in every tool you'd normally reach for. Two different kinds of "no" robots.txt is a note taped to the door. It says "please don't come in." A polite crawler reads it and turns around. A rude one ignores it and walks straight past. Either way, the note never touches your bytes — it's a request, enforced entirely by the visitor's own manners. A WAF or CDN rule is the door. It answers the request itself, at the edge, before anything reaches your origin: 403 Forbidden , 429 Too Many Requests , or a JavaScript challenge the bot can't solve. The HTML behind that door could be a masterpiece or a blank page. The bot sees neither. It sees the status code. People spend weeks perfecting the note and never check whether the door is locked. This isn't hypothetical — here's the data Cloudflare Radar publishes what actually happens to AI-crawler traffic across its network. From the AI Insights dashboard (7-day view, as of 18 July 2026): Of all HTTP responses served to AI bots and crawlers: 200 OK — 73.6% 403 Forbidden — 5.2% 429 Too Many Requests — 1.3% 503 — 1.1% AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026. Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused. And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a