Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch "Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step. The minimum trace payload Every tool call in this build logs a structured record — src/trace.ts : export interface TraceStep { trace_id : string ; step_id : number ; tool_name : string ; args_hash : string ; // hashed, never raw args duration_ms : number ; result_summary : string ; model : string ; token_usage : { input : number ; output : number }; } Two details here that look small and aren't: args_hash , not raw args . This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values: export function hashArgs ( args : Record < string , unknown > ): string { return " sha256: " + createHash ( " sha256 " ). update ( JSON . stringify ( args )). digest ( " hex " ). slice ( 0 , 8 ); } result_summary , truncated. Full tool results can be large (a kb_search returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values: const MAX_FIELD_LEN = 70 ; export funct