Building Real-Time AI Translation Assistance with FastAPI, Claude, and Server-Sent Events

How we added an on-demand translation help feature to our book translation platform, streaming LLM suggestions for tricky passages. At LectuLibre, our AI-powered book translation service allows users to upload EPUB or PDF files and get translations generated by large language models like Claude and DeepSeek. But we quickly noticed a pain point: automated translations, while fast, sometimes produced awkward or ambiguous results for culturally specific phrases, idioms, or technical jargon. Users wanted a way to get instant, contextual help for these tricky passages without leaving the platform. That’s when we set out to build the 翻译与转录求助 (Translation Assistance) feature — an interactive side panel where users can select any sentence or paragraph and receive alternative translations, explanations, and stylistic suggestions from an LLM in real time. In this article, I’ll walk you through the engineering challenge, the architecture we chose, and the specific code and trade-offs that made it work smoothly under production constraints. The Problem: Real-Time, Context-Aware Translation Help The core requirement was simple: a user highlights a piece of text in the translated book and clicks “Get Assistance”. Immediately, the system should stream back multiple translation options, a brief explanation of differences, and stylistic notes — all aware of the surrounding context, the author’s style, and the target language. Under the hood, this meant: Low latency : Users expect a response in under 2 seconds. Streaming : The LLM output can be long, so we needed to stream tokens as they are generated. Context awareness : We must include enough surrounding text from the book to ground the model’s response. No blocking : The main translation pipeline shouldn’t be affected; the assistance feature should exist as an independent async service. Cost efficiency : Avoid re-processing the entire book each time a user asks for help. Our Approach: Async FastAPI + SSE + Rate Limiting We run a P