Most open-source AI detectors can't hold a 0.5% false-positive rate [P]

We needed to know where the open-source AI-detection field actually stands, so we ran every notable open detector through the same protocol. Setup: - Public data only: Jabarian & Imas 2025 (NBER), Liang 2023 TOEFL essays, a 1,060-text frontier set (GPT-5.x, Claude Opus 5, Gemini 3.x), 5,000 pre-LLM (2018) FineWeb pages as human pool - Every model gets its threshold set on the same 6,930 human docs to a matched 0.5% FPR - Then measure recall per group: raw AI, humanizer-paraphrased AI, frontier models What surprised us: - 4 of 6 models effectively can't reach 0.5% FPR. MAGE scores >0.9999 on 26% of ordinary human web text. The old OpenAI RoBERTa detector lands at AUC 0.31 - worse than coin flip on modern generators. - Humanizer-paraphrased text is where everything collapses: best model catches 42%, second best 4%. - All models flag non-native essays at a higher rate than they flag native essays. This is a fundamental flaw in the entire class of models, not just one. model ROC-AUC raw AI humanized AI frontier models tropa-mini 0.968 93.2 % 41.6 % 33.6 % desklib/ai-text-detector-v1.01 0.875 83.9 % 4.0 % 1.8 % SuperAnnotate/ai-detector 0.824 0.5 % 1.4 % 0.6 % Hello-SimpleAI/chatgpt-detector-roberta 0.571 0.8 % 0.4 % 0.2 % yaful/MAGE 0.507 —* —* —* roberta-large-openai-detector 0.313 0.0 % 0.1 % 0.0 % More information and data on HF \ recall at matched 0.5% FPR; * MAGE can't reach 0.5% FPR at any threshold (flags 26% of human web text with score >0.9999)** Disclosure: one of the six is ours - we run a hosted detector and released this one as open weights (Apache-2.0). All datasets and the methodology are in the model card, so you can rerun everything: huggingface.co/wasitaigeneratedcom/ai-text-detector-small submitted by /u/grumpyp2 [link] [留言]