My Scanner Missed 93% of the Bugs — and That Was the Right First Result

The first time I ran my vulnerability scanner against the industry-standard benchmark, the bottom line of the scorer's report was this: $ python scripts/score_benchmark.py --findings out/java.findings.json \ --truth benchmark-java/expectedresults-1.2.csv OVERALL precision 0.60 recall 0.07 F1 0.13 # abridged Three numbers, and here is what each one means. Precision 0.60 — of all the alarms the scanner raised, 60% pointed at real bugs: when it spoke, it was right more often than not. Recall 0.07 — of all the real bugs in the benchmark, it found 7%. In the four vulnerability classes my scanner covers, the benchmark contains 777 real, labeled vulnerabilities; it missed 93% of the bugs it exists to find. F1 0.13 — precision and recall combined into one score (their harmonic mean), dragged down to almost nothing by that recall. My first instinct was to fix it before anyone saw it. Instead I saved the output, wrote the number into my benchmark log, and kept it — because that number was always going to be published, and this is the article that publishes it. The Context For the past months I've been deep in AI — reading, building, measuring. One of the projects that came out of it is an AI vulnerability scanner. The design in one sentence: deterministic static-analysis rules do all the searching, and an LLM judges each finding — is this a real bug or a false alarm? The full architecture gets its own article. This one is about the first measured number. The test set is the OWASP Benchmark — 2,740 labeled Java test cases, the standard exam for Java security scanners. In my scanner's four vulnerability classes (SQL injection, command injection, path traversal, XSS) there are 1,478 cases: 777 real vulnerabilities and 701 cases deliberately designed to bait scanners into raising false alarms. Every tool I compare against — Semgrep, CodeQL — takes the same exam, scored by the same scoring code. Same rules for everyone. New to this? Three words carry this article. A source is wher