Your DNS check is lying to you

Or: how a "this host is dead" verdict from a single net.LookupHost call quietly broke our crawler, and what we did about it. The setup We run a crawler that fetches tens of thousands of corporate websites a day from a datacenter. Before we spend any budget on a fetch — the actual HTTP request, the residential proxy hop, the S3 upload — we run a cheap reachability gate . The job of the gate is one thing: answer the question "is it even worth trying to fetch this host from here?" The first version of that gate was the obvious thing: resolve the host. If DNS returns an IP, the host exists. If it doesn't, mark the URL dead and move on. That gate was wrong often enough to matter. This is the story of the four ways it was wrong, and the gate we ended up with. Why "just resolve the host" isn't enough A naive reachability check has the shape: Call net.LookupHost . If it returns IPs, the host is reachable. If it errors, it isn't. Every clause in that sentence is a lie in production. Here are the four leaks we hit, in order of how painful they were. Leak 1 — CNAME chains the resolver doesn't finish in time A lot of corporate sites don't resolve directly. They sit behind a CDN, which sits behind a tenant-specific alias, which sits behind a regional load-balancer name. From DNS's point of view, that's a CNAME chain: ir.bigcorp.com → bigcorp.cdnvendor.net → edge-eu-west-3.cdnvendor.net → A 203.0.113.42 LookupHost is supposed to chase the chain transparently and hand you the final IP. It usually does. But "usually" hides two real failure modes: The resolver chases the chain in series under a single deadline. A slow hop two-thirds of the way down eats the whole budget; the call returns a timeout, not the IP it would have found with another 200ms. An intermediate hop misbehaves — wrong record type, NXDOMAIN at a tier the resolver doesn't expect, a stub that's been decommissioned. The lookup fails even though the host is registered and reachable through other paths . Both look ident