My Similarity Check Let the Same Story Through 3 Times. Here's How I Killed It.

I run a content pipeline that picks trending topics and publishes articles automatically. Last week I found out it had published the same story three times. Not the same title — the same exact topic, reworded each time. My dedup check was supposed to stop that. It didn't. Here's why, and how I killed the check. The Bug My pipeline had a similarity gate. Every candidate title got compared against the last 30 published titles, and anything scoring 0.58 or higher was rejected. Straightforward, right? from difflib import SequenceMatcher def jaccard_bigram ( a : str , b : str ) -> float : def bigrams ( s : str ) -> set [ str ]: return { s [ i : i + 2 ] for i in range ( len ( s ) - 1 )} x , y = bigrams ( a ), bigrams ( b ) return len ( x & y ) / len ( x | y ) if ( x | y ) else 1.0 def similarity ( a : str , b : str ) -> float : return max ( SequenceMatcher ( None , a , b ). ratio (), jaccard_bigram ( a , b )) THRESHOLD = 0.58 Here's the pair that slipped through. The candidate: 中国军队国际形象网宣片《当红》 And a title I had already published: 《当红》网宣片刷屏，普通人看到的中国军人是什么样 Same film. Same topic. Third time it was being covered. Watch what the algorithm did: candidate = " 中国军队国际形象网宣片《当红》 " published = " 《当红》网宣片刷屏，普通人看到的中国军人是什么样 " print ( similarity ( candidate , published )) # SequenceMatcher: 0.187 # jaccard bigram: 0.185 # max: 0.187 < 0.58 -> PASSED 0.187. The gate let it through with a five-fold margin to spare. Why It Failed The name 当红 is the same in both titles. That is the whole topic. But the algorithm does not care about that. SequenceMatcher matches in order. In the published title, 当红 sits at position zero. In the candidate, it is at the end. Reordered tokens break the match, so the ratio collapses to the shared fragments — 网宣片 plus the generic words around it. The bigram fallback does not save you either. Jaccard over character bigrams measures surface overlap, not meaning. Five shared bigrams out of twenty-seven total. 0.185. It "proves" the titles are unrelated because most of