Your LLM Trace Is Green. Why Is the RAG Answer Still Wrong?

TL;DR Many LLM observability setups capture prompts, outputs, tokens, and latency while leaving retrieval failures hidden. A single search call may conceal query rewriting, filtering, fetching, deduplication, reranking, and evidence selection. A useful trace connects the original question to the effective query, returned sources, selected passages, and final claims. Retrieval tracing helps distinguish missing, stale, or ignored evidence from a genuine generation failure. Production teams should measure freshness, duplicate evidence, citation coverage, and cost per grounded answer. A user asks your AI assistant whether a product still supports a particular feature. The assistant responds confidently and links to the company’s documentation. The model request succeeded. Latency was normal. Token usage stayed within budget. No tool call failed. Every indicator on the dashboard is green. The answer is also six months out of date. The model trace cannot tell you whether the system searched for the wrong phrase, preferred an old page, discarded a better result, or ignored the correct evidence. It only shows the context that eventually reached the model. That is the blind spot in model-centred observability. For RAG applications and web-connected agents, the useful unit of observation is not the model call. It is the complete evidence path. A Successful Model Call Can Still Be a Failed Request A typical LLM trace records the prompt, response, model name, token consumption, latency, errors, and perhaps a tool invocation. That is useful for diagnosing slow requests, malformed inputs, and unexpectedly expensive generations. It does not tell you whether the model received the right facts. In a retrieval application, the final prompt is assembled by an upstream system. That system may rewrite the query, choose a search provider, apply time or domain filters, fetch pages, extract text, remove duplicates, rerank candidates, and select passages for the context window. The model ca