How We Built Precise Translation and Language Identification for AI Book Translation

How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation. The Problem: Garbage In, Garbage Out When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation. Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits. We needed something far more robust—what we internally call 精准翻译与语言识别 (precise translation and language identification). Here’s how we built it. The Language Detection Pipeline: From 85% to 98% Accuracy Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection. We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation. Combining Multiple Detectors with Voting We created a LanguageDetector class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are: fastText with the official lid.176.bin model (loaded once, not per request) langdetect , which is lightweight and good for long texts cld3 (Compact Language Detector 3) fr