Two ceilings: taking a Go DNS server from 500 to 9,500 QPS
I run HydraDNS, an open-source DNS security gateway in Go. Last month I sat down to find out what one box could actually handle before I put it on anyone else's network. The plan had a rule I'd written for myself: every number we discover becomes either a sales claim or a fix ticket. No number, no claim. I expected to find one bottleneck. I found two, stacked on top of each other, and a third thing I wasn't looking for: a data structure in our own documentation that had never existed in the code. Everything below was measured on a 22-core dev machine with load generated inside the container, using dnspyre, so docker-proxy and host networking stay out of the numbers. It's not appliance hardware and I'm not making appliance claims. The shapes are what matter. The first ceiling: ~500 QPS, and it didn't care what I threw at it The first redline run capped at roughly 500 queries per second. Fine, servers have limits. What made it interesting was that the cap didn't move. Blocked queries: ~500. Cached queries that never touch upstream: ~500. Two code paths that do completely different work, hitting the same wall, with the CPU sitting under 30% of 22 cores. That signature is worth memorizing. When two very different paths hit the same ceiling and the CPU is bored, the bottleneck isn't in either path. It's in something they share, or something upstream of both. Ours was in the blocklist check. IsBlocked ran a SQL COUNT against a 92k-row blocklist_entries table on every query . Not just candidate blocks, every query, because the check sits in front of the cache, so even cache hits paid for it. And all of those reads were serialized through a single SQLite connection, MaxOpenConns=1 , which was also absorbing the async write traffic from query logging. The engine's self-measured latency under load: p50 of 50ms, p99 of 5000ms. Five full seconds at the tail, for DNS, which is supposed to be the fast part of the internet. The part where I found out our docs were lying Here's the