What I learned wiring an AI agent fleet into self-hosted SigNoz

I spent a week trying to answer one question about my own AI agents: when one of them does something stupid in production, how do I prove the fix worked? For normal software the answer is boring. You have monitoring, an incident, a regression test, a staged rollout. For an agent you usually have a trace viewer and a shrug. So I built ArcNet on self-hosted SigNoz for the Agents of SigNoz hackathon, and most of what I learned was about SigNoz internals I could not have guessed from the docs. Here are the parts that cost me real time. The setup The stack is small. Agents run on Agno. An in-process SDK wraps them and does two jobs: OpenTelemetry instrumentation, and guardrails from unplug-ai at four checkpoints (input, retrieved content, tool call, output). Traces go to self-hosted SigNoz over OTLP. A FastAPI server reads back out of SigNoz, and a React UI sits on top. Installing SigNoz was the easiest part, which surprised me. Foundry takes one file: apiVersion : v1alpha1 kind : Installation metadata : name : signoz spec : deployment : flavor : compose mode : docker signoz : spec : image : signoz/signoz:v0.133.0 foundryctl cast -f casting.yaml That brings up SigNoz and its MCP server together and writes a casting.yaml.lock with checksums. I committed the lock file, and re-running foundryctl forge against it later produced a byte-identical file. That is a genuinely nice property for a hackathon judge or a teammate. Lesson 1: check what your instrumentor actually emits This is the one I would tell everyone. I assumed Agno instrumentation would produce OpenTelemetry's gen_ai.* semantic conventions, because that is what the GenAI spec describes. I started sketching dashboard queries against gen_ai.usage.input_tokens before anything was running. Then I turned it on. openinference-instrumentation-agno emits OpenInference conventions, which are a different attribute set. The spans I actually got were shaped like this: agent_j.run └── gpt-5.6-luna.invoke └── search_tickets Eve