What 78K attack samples taught me about catching prompt injection

I spent the last while building a prompt-injection detector trained on 78,000+ attack samples. Here's what surprised me, and why I ended up going the unfashionable route. The trendy approach is to use an LLM. I didn't. The default move in 2026 is "use an LLM to judge whether input is an attack." It's appealing because models understand nuance. But once you try to run it inline on every request, the problems pile up fast: Latency. You've added a full model round-trip to every single call. Hundreds of milliseconds, minimum. Cost. Your security bill now scales with your traffic. Every request pays the token tax. Non-determinism. The same input can get a different verdict tomorrow. Try explaining that in an incident review. It's jailbreakable itself. Your security model is an LLM, which means it's vulnerable to the exact attacks it's supposed to catch. So I built the boring version instead: deterministic regex plus classical ML (TF-IDF character n-grams into logistic regression). No LLM in the detection path. It runs in about 7ms, costs nothing per call, and is fully deterministic. What the data actually showed Here's the part I want to be honest about, because most vendors quote one number and hide the rest. Measured on public benchmarks the model was not trained on (held out, non-circular): Real-world, in-the-wild jailbreaks: 0.895 recall at 1.00 precision Obfuscated / evasion attacks: 0.799 at 1.00 precision A frozen external split: 0.804 recall, 0.48% false-positive rate Subtle roleplay-framed jailbreaks: 0.324 That last number is bad, and it's the most important one on the list. The honest read is that deterministic detection is excellent on real-world and obfuscated attacks and weak on subtle roleplay framing. That's a real gap, and pretending otherwise just means someone finds it later and trusts you less. The false-positive rate is a moving target One thing I didn't appreciate going in: FPR is completely traffic-dependent. The same model reads roughly: ~0.4% fal