标签：#Observability

For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps. That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how. The seductive wrong model The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right? So my working theory became: observability is logging plus some fancy logic to analyze the logs. Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other. Descent one: metrics aren't events at all A metric is not a record you write. It's a number sitting in your app's memory . requests_total . increment () // 1, 2, 3... request_duration . record ( 0.23 ) // adds to a histogram Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out. That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time. And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app. Descent two: logs are the familiar part Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched. The only upgrade

One of the most common assumptions in engineering leadership is that a rising number of reported incidents signals declining system reliability. However, a recent article from Great Circle argues that the opposite is often true: an increase in incident counts may actually indicate that an organization's incident management culture is improving. By Craig Risi

Per-tenant cost visibility changes the design: don't begin with parallel API calls; begin with a durable ledger that ties every classification result and usage record to a tenant, policy version, and source item. For a customer-support catalog backfill, the practical choice is a bounded Node.js worker that reads existing posts and comments, classifies them through a replaceable adapter, checkpoints each result, and exports tenant-scoped JSONL. Short answer: make the ledger the product of the job and the LLM call one restartable step inside it. That ordering matters when support conversations contain messy product descriptions such as “the small blue charger for the old tablet.” The moderation label decides whether the text is safe to reuse; the enrichment labels connect it to a catalog candidate. Operations still need to answer a less glamorous question: which tenant consumed the tokens? Make tenant cost visible before optimizing it Token totals belong beside decisions, not in an unrelated monthly dashboard. Record normalized input and output token counts on every completed row, then aggregate by tenantId , policyVersion , and time window. If the API reports different usage units, preserve the raw usage payload in restricted telemetry and map it explicitly; don't pretend unlike units are interchangeable. Start there. Three signals are enough for the first useful view: Signal Group by Operational question Completed items tenant, policy version Is the backfill moving? Input and output tokens tenant, model Where is consumption occurring? Review and block counts tenant, content kind Did the decision mix shift? Cost in currency should be derived from a versioned rate configuration, not baked into historical rows. Store usage and the model identifier, then apply the applicable rate when producing a report. This keeps a rate change from rewriting what the runtime actually observed. It also lets finance reproduce an invoice-period view while engineering inspects tokens per

An adapter layer becomes strategically useful when several teams need one observability contract but cannot, or should not, standardize on one agent framework. AI SDK, LangChain.js, OpenAI Agents SDK, and direct model clients organize execution differently. One emphasizes generation and streaming, another exposes hierarchical callbacks, another has agent runs and handoffs, and a direct client exposes only provider requests unless the application adds its own spans. Unified tracing should preserve those differences while translating the common lifecycle into one model. Done well, teams can share execution-tree tooling, CI quality gates, privacy policy, and telemetry export without coupling every consumer to every framework. Unify Semantics, Not APIs The frameworks do not need a shared callback interface. They need a shared answer to a smaller set of questions: What is the root operation? Which model, tool, retrieval, decision, and handoff spans occurred? What was each span’s parent? How did it end? Which usage and timing metrics are available? Which facts are unavailable from this integration? The framework adapter converts its native lifecycle into those semantics. Consumers never call framework hooks directly. AI SDK lifecycle ---------┐ LangChain callbacks ------+--> framework adapters --> trace core OpenAI Agents tracing ----+ direct client wrappers ---┘ | +------------+------------+ | | | execution UI CI rules telemetry sinks A Practical Mapping Matrix The exact public APIs change over time, so keep the mapping conceptual and verify it against the supported framework version. Normalized concept AI SDK-style integration LangChain-style integration OpenAI Agents-style integration Direct client Root run Application request or generation Chain, graph, or agent run Agent trace or runner invocation Manual application span Model span Generation or stream lifecycle LLM/chat-model callback Model generation item/span Provider request wrapper Tool span Tool execution lifec

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch "Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step. The minimum trace payload Every tool call in this build logs a structured record — src/trace.ts : export interface TraceStep { trace_id : string ; step_id : number ; tool_name : string ; args_hash : string ; // hashed, never raw args duration_ms : number ; result_summary : string ; model : string ; token_usage : { input : number ; output : number }; } Two details here that look small and aren't: args_hash , not raw args . This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values: export function hashArgs ( args : Record < string , unknown > ): string { return " sha256: " + createHash ( " sha256 " ). update ( JSON . stringify ( args )). digest ( " hex " ). slice ( 0 , 8 ); } result_summary , truncated. Full tool results can be large (a kb_search returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values: const MAX_FIELD_LEN = 70 ; export funct

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

It’s 2:15 AM. Your phone buzzes aggressively. 🚨 You jump out of bed, open your laptop with half-closed eyes, and join an emergency incident response call. Your team’s Slack channel is exploding: ⚠️ [ALERT] Payment API 500 Error Rate > 15% ⚠️ [ALERT] Redis Latency Timeout (>5000ms) ⚠️ [ALERT] Node-04 CPU Saturation (98%) You spend the next 2 hours manually connecting the dots: querying Prometheus metrics, scrolling through endless Loki logs, cross-referencing Tempo traces, and checking recent ArgoCD deployments. Eventually, you uncover the truth: Deployment #218 , pushed right before midnight, introduced a subtle memory leak that triggered GC pressure, spiked CPU, starved the Redis connection pool, and knocked down the Payment API. Sounds familiar? 😅 💥 The Problem: Observability Shows Symptoms , Not Causes Modern observability tools like Grafana, Prometheus, Loki, and Jaeger are fantastic at collecting metrics, logs, and traces. But they suffer from one fundamental design limitation: They tell you WHAT is breaking, but leave you to figure out WHY it broke. When a microservice fails in Kubernetes, it triggers a domino effect ( cascading failure ): Deployment #218 (Memory Leak) │ ▼ Garbage Collection Pressure │ ▼ CPU Saturation (98%) │ ▼ Redis Connection Timeout │ ▼ API Gateway Retry Storm │ ▼ Payment Service Down (HTTP 500) Traditional alerting floods you with alerts for the bottom 4 nodes (the symptoms), leaving SREs and DevOps engineers stuck sifting through noise during high-stakes outages. 💡 Introducing IRCAE: Autonomous Root Cause Engine To solve this, we are building IRCAE (Intelligent Root Cause Analysis Engine) —an open-source, enterprise-grade platform designed to turn raw telemetry into autonomous causal reasoning . Instead of asking SREs to correlate telemetry manually, IRCAE automatically answers: "Why did the system fail?" in less than 10 seconds. 🌟 Key Highlights 🚀 Written in Rust (Axum + Tokio) : Built for high-throughput, near-bare-metal performance wi

Artificial intelligence is rapidly changing how engineering teams respond to production incidents, offering the ability to summarize incident channels, analyze unfamiliar code, suggest remediation steps, generate pull requests, and increasingly assist with diagnosis. By Craig Risi

Point a model at your repositories, ask why something broke, and you get an answer. Coherent, names a mechanism, shows you how it got there. The working is the problem rather than the reassurance. It looks the same whether it ends at a value in a record or at what usually happens, and only one of those is evidence. I ran the same question at four levels of evidence, on two systems, across three models. The question was never find me a bug. It was always this already happened, work out how. The pass that worries me is the one just short of enough evidence. There is enough there to build a real hypothesis, and not enough to tell a real one from a plausible one. Two of its guesses sent me to look at things I had not checked. One was a dead end. The other was a real problem I had not known about, and nothing in either answer told me which was which. A good guess costs the same to chase as a real one. You find out which it was at the end. The four rungs, and what each one removes I stopped thinking about this as adding context. Each rung takes away something the model would otherwise have to guess at, and that is the more useful way to look at it. One. Repository access, broad. Point it at everything and ask. It removes nothing. The answer was structurally sensible, named components that were not involved, and arrived with no more hesitation than the correct one did three rungs later. If your team tried this once, got something confident and wrong, and decided the tooling is not there yet, this is probably where you stopped. Two. Three bounded repositories, plus a written map. Which service talks to which, over what protocol, with what delivery and ordering guarantees. This removes rediscovery. What made it work was not the map. It was telling the model to treat the map as true and not go and check. That bought focus and gave up verification. Maps go stale quietly, and I have removed the step where it might have noticed. Three. Plus traces and logs covering real executio

The owner had already asked for the alert emails to stop. A fix shipped. Then another email landed. Then another. "ong it just ssent me abother email," he said, voice-dictated, unedited. Fifteen minutes later: "go another one." The system was reporting an outage that did not exist. The Transport That Only Ever Failed A 14-PR merge train had just moved every cron producer's alerting off shared email and onto Buzz, a Nostr-relay team chat. One producer per PR, each with its own liveness contract and a bead receipt. It shipped cleanly. But the library backing those producers carried a default that had only one job: fail. AF_BUZZ_CMD = " ${ AF_BUZZ_CMD :- af_default_buzz_post } " af_default_buzz_post returned 1 with "no Buzz transport injected". Every caller that sourced the library (which is every cron producer) exhausted its Buzz retries and fell through to the email floor. The system reported a false Buzz outage while the relay was healthy. It did this 2 to 5 times per hour. Evidence arrived in the logs: 581 dedup markers, a steady stream of "[INTENT ALERT FLOOR: Buzz unreachable]" emails, and sweep.log showing buzz=ok only for the handful of callers invoked through the CLI entrypoint rather than by sourcing the library. That asymmetry was the bug. The CLI had a one-line fixup swapping in the real transport, annotated in a comment as "the library path is unchanged". The library path did not, and the cron producers all take the library path. The fix promoted the real transport to the default for both seams. af_buzz_transport already discovers the installed buzz-notify.sh and already fails closed when it is genuinely missing. The dead CLI fixup was deleted. Fail-closed behavior survives, but now it is conditional on genuine absence rather than on every caller remembering to opt in. Why not migrate callers one at a time? Because the per-caller route leaves the next new producer to rediscover this the same way. Flipping the default fixes the class, not the instance. The

TL;DR If you're a five-person startup shipping a Node.js API and you want a metrics dashboard by Friday, send your telemetry to a managed backend and keep only the instrumentation layer inside your own repo. The alternative — standing up a time-series database, an object store for long-term blocks, and a dashboard service — puts three more components on an on-call rotation that hasn't earned its first SLO yet. Settle the wire format now and treat the backend as a config line you can change later. I own the platform team's roadmap, which in practice means I'm the person who defends the monitoring bill in a budget review and also the person who gets paged when a disk fills at 03:00. Those two jobs pull in opposite directions, and most of the advice online is written by people who only hold one of them. Usually the pager wins the argument. Should a startup run its own metrics stack, or pay for a managed dashboard? Start with capacity, because that's the step everyone skips before signing anything. A moderately instrumented Node.js API — say 40 HTTP routes, two queue workers, default runtime and event-loop metrics, one latency histogram with ten buckets — sits somewhere around 3,000 to 8,000 active series per process. Multiply by replicas. Multiply again by every environment you keep alive, including the staging cluster nobody admits to. You are at 50k active series before a single engineer has written a custom counter, and a self-hosted scraper will chew through that on a 2 GB VM without noticing. It will still be fine at 500k. Past a few million active series you're into sharding, remote storage, and a retention argument with whoever pays for object storage — that's the point where the self-hosted route stops being free and turns into a project with a headcount attached. None of that work is hard. It's just never zero. Dimension Self-hosted stack Managed metrics backend Time to first dashboard 1–3 days under an hour Who owns retention you, plus the storage bill vendor

Wide Events is a Rails gem that puts the production context a coding agent needs onto one OpenTelemetry root span per request or job. In one production search request, the root event showed 30.0 seconds total duration, 446 ms of Postgres time, and 29.4 seconds of outbound HTTP time. That was enough to focus the investigation on an external dependency. The trace then identified a POST that took 28.9 seconds. The trace contained 82 spans and 20,261 bytes of attribute JSON. The root event contained 40 attributes and 1,420 bytes. This is not a token benchmark, but it shows why the root event is a more compact starting point for an agent. Agents can read the code, but not the running system A coding agent starts with an unusual advantage: it can search every model, controller, job, migration, and test in a few seconds. It also starts with a serious blind spot. The repository cannot tell it: which account experienced the problem which build was running which feature-flag variant was active how many queries the request issued whether a semantic-search leg degraded how much an LLM call cost whether the same symptom appears in one tenant or every tenant Those answers often exist somewhere, but “somewhere” might mean a trace waterfall, application logs, a feature-flag service, product analytics, and a database console. Pulling all of that into a context window is expensive and usually requires several joins that were never designed in advance. A wide event changes the starting point. The app accumulates the context it learns while processing one unit of work, then attaches the completed flat map to the OpenTelemetry root span. The span is marked main=true , so every request or job can be queried as one row. request or job -> Rails and domain context accumulate -> child spans contribute dependency counts and timings -> one flat map is flushed onto the root span -> ClickHouse stores one queryable row The trace still exists. Wide Events gives it an application-shaped index. If y

Use a durable idempotency receipt when feature flag retries can reach a rollout toggle endpoint, otherwise reach for a read-only flag evaluation that cannot create duplicate writes. Short answer: the backend must bind one caller-generated key to one operation and commit the receipt beside the state change; a retry should recover that recorded result, not perform the write again. The flag is not the transaction. Record the invariant at the write boundary My architecture decision is to enforce idempotency inside the backend that owns the mutable state. The caller creates an operation key before its first attempt, sends the same key and operation on every retry, and never manufactures a fresh key inside the retry loop. The backend binds that key to a stable digest of the requested change. If the key and digest have already been committed, it returns the stored result. If the key exists with a different digest, it rejects the integration error as a conflict. The state mutation and receipt belong in one transaction, because two separate commits create an interval in which the state says “done” while the receipt still says nothing. I write the invariant this way: one idempotency key identifies one logical operation within a documented scope; one committed operation has one durable result. The defensible claim is effectively-once mutation within that scope, not exactly-once delivery. Clients, queues, proxies, and deployment controllers can all repeat an attempt, so delivery count isn't a useful correctness boundary. There are three failure boundaries I test. A response can disappear after commit, two workers can race on the same key, and the flag decision can change between attempts. The first requires replaying the stored result. The second requires a uniqueness constraint rather than a check-then-insert sequence. The third requires persisting the evaluated decision with the operation; reevaluating a flag during recovery can turn one logical request into two different his

Watching an autonomous agent run through a loop of tasks is like watching a black box try to solve a puzzle in another room. You can see the final result, but the middle part—the reasoning, the failures, and that pivotal moment where it realizes its plan was garbage—is buried in thousands of lines of unstructured logs. If you've ever deployed an agentic workflow only to check back an hour later and find it has been stuck in a high-latency loop of 'I made a mistake... let me try again' for forty minutes, you know the pain. You didn't have failure; you had expensive, silent repetition. The problem with current LLM observability is that we focus too much on the input and output (the traces) and not enough on the internal state transitions of the agent itself. We need to quantify how often an agent is actually self-correcting versus just spinning its wheels. I recently started working with a specific tool designed for this exact visibility gap: the Agent Self-Reflection & Sentiment Scanner . The Observability Gap in Agentic Loops When we talk about 'agents,' we're usually talking about a loop: Observe, Think, Act, Repeat. In a perfect world, the 'Think' step includes self-correction. If an action fails (e.g., a 403 error from an API), the agent should reflect on that failure and adjust its next move. But how do you measure if your agent is actually getting better during a session? How do you distinguish between an agent that is 'Proceeding' with confidence and one that is in a state of constant 'Correction'? You can't just look at the final success/fail status. You need to parse the execution logs for deterministic markers. Why Deterministic Matching Wins Over LLM-Based Analysis The temptation here would be to pipe your agent logs into another, even larger LLM and ask, 'Is this agent struggling?' Don't do that. It’s redundant, it’s slow, and if you're running high-volume loops, the cost will kill your margin. You've already paid for the primary reasoning engine; don't p

The Observability Black Box As autonomous AI agents evolve from isolated chat assistants into multi-agent systems executing multi-step business logic across databases, APIs, and microservices, enterprise platform teams face an acute operational challenge: black-box opacity. When an autonomous agent fails, hallucinates, or executes an out-of-bounds API call, traditional Application Performance Monitoring (APM) tools fall short. Standard HTTP request logging and basic prompt-response captures cannot reconstruct the non-deterministic reasoning loops, tool selection branches, or sub-agent delegations that led to an incident. Furthermore, enterprise auditors, security teams, and regulatory bodies (governed by SOC 2, FedRAMP, and the EU AI Act) now require non-repudiable proof of agent execution. Organizations must be able to answer five fundamental questions for every production run: Which human or non-human identity authorized the agent run? What planner reasoning path or tool routing logic was chosen? Which exact data assets or vector embeddings were retrieved into context? What was the precise execution latency, token cost, and error tax of each intermediate step? Can the complete execution graph be cryptographically reconstructed for compliance review? To resolve this challenge, platform engineering teams must deploy Audit, Observability & Lineage —an architecture anchored in OpenTelemetry (OTel), OWASP Agent Observability Standards, and immutable lineage graphs. Deep-Dive Architecture: OpenTelemetry & Lineage Integration A production-grade Agent Observability stack avoids proprietary vendor lock-in by standardizing on OpenTelemetry (OTel) OTLP trace ingestion and open metadata stores. 1. The Unified OpenTelemetry Span Tree Every agent execution unit — from user intent trigger to final task completion — is encapsulated within a single root trace context ( agent.run ). Sub-tasks, tool calls, and model invocations are recorded as hierarchical child spans: [ Root Trace:

Autonomous agents don't just fail loudly—they fail expensively. A single misconfigured retry loop between an agent and an LLM can generate thousands of redundant tool calls and API requests before anyone notices, turning a minor logic bug into a five-figure cloud bill. PolicyAware is built to be the operational safety net that catches this class of failure before it reaches your finance team's dashboard. 1. The Recursive Agent Crisis Every SRE and platform engineer who has run agentic workloads in production has a version of this story. An agent is wired to call an LLM, interpret the response, and take an action—often invoking another tool, which produces output that gets fed straight back into the same LLM. Under normal conditions this loop terminates in a few steps. Under a bad prompt, a malformed tool response, or a subtle logic error, it doesn't. The agent gets stuck reasoning in circles: it calls a tool, receives an ambiguous or malformed result, decides the task is incomplete, and calls the LLM again to "retry." Each retry consumes tokens, each tool call hits a downstream API, and there is no natural circuit breaker unless one has been explicitly engineered. Within minutes, a single stuck session can produce: Thousands of duplicate or contradictory API calls to internal and third-party services. Sustained LLM token consumption that dwarfs normal daily usage. Cascading load on downstream systems that were never designed for machine-speed request volume. By the time monitoring dashboards catch the anomaly—if they catch it at all—the damage is already done: a runaway bill, a rate-limited API partner, or a compromised production database from thousands of unchecked write attempts. Traditional APM tools tell you a service is under load; they don't tell you an autonomous agent is the one generating that load, or why. This is why the recursive agent crisis is fundamentally a governance problem, not just a monitoring problem. Rate limits and cost alerts fire after the

A follow-up to How Old Is My Data? The failure mode that gets worse when a machine is reading the data In a classic dashboard, stale data is a human problem: someone looks at a number that's six hours old and makes a slightly worse decision. Annoying, rarely catastrophic. Now hand that same data to a retrieval-augmented-generation (RAG) pipeline, or to an autonomous agent. The stakes change. The system doesn't pause to sanity-check the timestamp — it acts. And when the data it acts on is stale, three things are true at once: The answer is confidently wrong. There is no error to fire on — the query succeeded, the model responded, latency was normal. Every other signal on your dashboard is green. That's the worst combination in observability: a real failure that is completely invisible to the signals we currently emit. Where staleness hides in AI systems RAG: index vs. corpus. Your vector index was built from a corpus at some point in time. The corpus keeps changing — documents get added, edited, retracted. If the re-embedding job stalls or falls behind, the index quietly drifts out of date. The retriever still returns plausible chunks; the model still writes a fluent answer. It's just answering from a version of reality that no longer exists. The quantity you care about is the age of the index relative to its source — not the age of either one alone. Feature stores: online–offline skew. The features your model trained on and the features it serves on are supposed to match. When the online store lags the offline pipeline, predictions degrade in a way that looks like model drift but is actually data staleness wearing a costume. Agents: stale shared state. Multi-agent systems coordinate through shared memory, scratchpads, and context. An agent reasoning over state that another agent updated ten steps ago — but which never propagated — makes locally reasonable, globally wrong decisions. This isn't a new or exotic problem: it's exactly the regime that Age of Information t

A useful agent trace is not a list of timestamps. It is a causal tree. When a TypeScript agent retrieves documents in parallel, calls a model, retries a tool, and falls back to cached data, each operation needs a trace ID, its own span ID, and the correct parent span. Without those relationships, completion order is easily mistaken for execution structure. This article builds a small Node.js tracer to demonstrate the core mechanics: immutable async context, parent-child spans, reliable finalization, and a pluggable sink. It is intentionally smaller than a production observability library, but the design avoids several common mistakes found in minimal examples. Completion Order Is Not Causality Imagine three tools running in parallel: 80 ms search_tickets completes 100 ms load_account completes 120 ms search_docs completes Those timestamps describe completion order. The execution tree describes why the operations existed: research_agent └─ parallel_retrieval ├─ search_docs ├─ search_tickets └─ load_account Both views are useful, but only the tree preserves the relationship between the agent decision and its child tools. The normal JavaScript call stack cannot serve as that tree. Async work may resume later, execute concurrently, or outlive the function that scheduled it. Tracing therefore needs an explicit logical context. The Context We Need Each asynchronous branch needs two values: type TraceContext = { traceId : string ; parentSpanId : string | null ; }; When a new span starts, it reads the current context, records parentSpanId , creates its own spanId , and runs child work inside a new context whose parent is that span. In Node.js, AsyncLocalStorage provides the propagation primitive. It carries a value through normal asynchronous resources without adding trace parameters to every application function. Do not mutate one shared context object. Parallel siblings would race to replace the current span. Create a new context value for every nested span instead. Defin

Ten agent sessions ("minds," in this codebase) run continuously on one box, each with its own responsibility — one writes code, one talks to me on Telegram, one watches sensors, one just measures the fleet itself. They coordinate the way a lot of multi-agent systems eventually do: a shared log file, one line per event, [task] / [taking] / [done] . That log is fine for "what happened." It is useless for "what do we owe, and how much did it cost" — the two questions I actually needed answered before I was willing to let the fleet run unattended overnight. The board is not a ledger, but it can feed one The fix wasn't a new coordination protocol. It was noticing that every line on that board is already a transaction if you're willing to look at it that way: board event ledger meaning [task] fix-the-thing a liability opens [taking] pub: fix-the-thing the liability moves to a specific debtor [done] pub: fix-the-thing the liability settles a provider round-trip (one agent turn) a unit of labour is spent So the board gets replayed into three separate double-entry hledger journals, each tracking a different commodity: money — imputed USD (token counts priced through one rate table). promises — commodity PROMISE : an open [task] with no matching [done] is a standing liability, not a line that scrolled off screen. labour — commodity TURN : one provider round-trip, the fungible unit every mind actually spends, regardless of whether it's writing code or answering a sensor. Each journal gets checked two independent ways — hledger check for internal parity, plus a second, independently-written replay of the same board that has to agree with the balance query. A booking bug fails loud, not silently, because two things that should compute the same number just disagreed. Querying "who owes what" stops being a grep and starts being a query: $ mesh-promises --balance standing open obligations (bal liabilities:promises · 1 PROMISE = open, netted): 1 PROMISE liabilities:promises:pub:chat

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up. The problem OpAMP solves A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans. Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed. OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane. What OpAMP actually is OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel