The Bloom filter that never existed, and the two ceilings it was hiding

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . The most expensive bug I fixed this year was not in the code. It was in the documentation, and it had been shaping what everyone believed the code did. The setup HydraDNS is an open-source DNS security gateway I build in Go. Router points at it, it filters every DNS query on the network against a 92k-domain blocklist, blocks the bad ones, forwards the rest. Before putting it on anyone else's network I wanted a real number for what one box could take, so I sat down with dnspyre and a rule I had written for myself: every number becomes a sales claim or a fix ticket. No number, no claim. Our feature sheet said the blocklist was backed by a Bloom filter, sub-millisecond lookups. Here is the uncomfortable part: at every load this system had ever run, that claim was indistinguishable from the truth. Normal-traffic latency sat at one or two milliseconds. There was nothing to doubt, because nothing observable disagreed. The first ceiling The redline test capped at about 500 queries per second. Odd, but fine, until I noticed the cap would not move. Blocked queries capped at ~500. Cached queries that never touch upstream also capped at ~500. Two paths doing completely different work, same wall, CPU sitting under 30% on a 22-core dev machine. That combination is worth memorizing: when two very different code paths hit the same ceiling and the CPU is bored, the bottleneck is not in either path. It is in something they share. Ours was the blocklist check. IsBlocked ran a SQL COUNT against the 92k-row table on every single query, because the check sits in front of the cache, so even cache hits paid for it. Every one of those reads was serialized through a single SQLite connection, MaxOpenConns=1 , which was also absorbing the async write traffic from query logging. Engine self-latency under load: p50 of 50ms, p99 of five full seconds. For DNS. And the Bloom filter? I went looking for it so I could