Putting the trace before the loop

An observability-first approach for building an AI agent, and what it bought me. A couple of weeks ago I started implementing Kept, a self-hostable post-purchase support agent for e-commerce, with reliability as its core offering. Besides the product itself, my objective in building it is to delve into the depths of agentic system design, and see what it actually means to build an agent with "reliability at its core". Observability first I started from a theory my experience validated again and again throughout the years: observability is the bedrock of reliability. Proper logs and metrics beat an ideal architecture, industry-leading frameworks or best coding patterns. That was the case well before the agentic era, and I've always paid particular attention to this layer. But with the advent of non-deterministic LLMs sitting at the core of products, I decided to take that to the next level: designing and implementing the full tracing layer before writing a single line of code for the main agent loop/product. Turned out to be an insightful experience. Up next I'll walk through how I designed and implemented the tracing layer, then present 3 benefits and 2 drawbacks that I consider worth highlighting from the observability-first approach, and end with some thoughts on how to apply the approach if you don't have the luxury of starting with a fresh codebase. Tracing layer design and implementation I started with an AI-assisted research phase. Here's a list of resources I found particularly useful: Anthropic, "Building Effective Agents" OTel blog, "Inside the LLM Call: GenAI Observability with OpenTelemetry" John Hodge, "OpenTelemetry GenAI semantic conventions" OTel GenAI semantic conventions, the dedicated repo Langfuse data model 0. The guiding design decision The research produced the first important insight. My original idea was to have Langfuse as the source of truth that saves all spans and traces and serves them wherever they're needed. But then I realized I'd hav