Observability - A Counter in RAM, an ID in a Header, and a Batch Export
For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps. That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how. The seductive wrong model The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right? So my working theory became: observability is logging plus some fancy logic to analyze the logs. Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other. Descent one: metrics aren't events at all A metric is not a record you write. It's a number sitting in your app's memory . requests_total . increment () // 1, 2, 3... request_duration . record ( 0.23 ) // adds to a histogram Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out. That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time. And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app. Descent two: logs are the familiar part Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched. The only upgrade