RAG Retrieval Accuracy: 38%. After the Fix: 87%. The Model Was Never Touched.

That's a rebuild I shipped. The system: a RAG assistant for fraud analysts — ask it "how do we handle card testing followed by a successful auth?" and it should answer from the team's own SOPs and case history. The complaint: the answers were wrong, therefore the model must be dumb, therefore procurement should buy a bigger model. The model was fine. It was answering perfectly — from garbage context. Walk the forensic trail with me, because every step is checkable on your own system this week. Exhibit A: the chunking was destroying meaning before anything was embedded The ingestion split SOP documents every N characters, mid-sentence. Which means half the vectors in the index encoded fragments like this: chunk_147 = " ...ing to a freight forwarder. In these cases, do NOT " chunk_148 = " cancel the order immediately. First verify the customer via " The policy — don't cancel, verify first — exists in no single chunk. An embedding can't encode a meaning that isn't in its input. Retrieval was being asked to find semantics the pipeline had already shredded. Fix one: chunk on structure (sections, paragraphs), never on character counts, with enough overlap that no rule straddles a boundary. Exhibit B: dense-only retrieval, bimodal queries Fraud analyst queries split into two populations: pattern questions ("high-value order, new account, rushed shipping") and identifier questions ("what's the SOP for decline code 4863?", "rule VEL-013 rationale"). The system was dense-only — and embeddings treat a rare token like 4863 as noise, so identifier queries retrieved similar-feeling chunks instead of the literal match. Half the query population was structurally doomed regardless of model quality. Fix two: hybrid retrieval — BM25 for the identifiers, embeddings for the patterns, reciprocal rank fusion to merge. Exhibit C: nobody could see any of this, because quality was a rumor No golden dataset. No retrieval metric. The system's accuracy was whatever the loudest anecdote said it