Nights Watch: Guarding AI Agents Beyond the Wall

"Night gathers, and now my watch begins." The Night's Watch didn't exist to fight wars nobody saw coming — they existed because someone had to actually stand on the Wall and notice when something crossed it. That's the exact problem I kept running into with AI agents, and it's why I built Nights Watch for the "Agents of SigNoz" hackathon: a runtime resilience layer that catches an agent quietly drifting off its plan, explains why, and recovers — automatically. The problem nobody's watching for Most agent failures aren't dramatic. An agent doesn't crash, it doesn't throw an exception, it doesn't get flagged by a content filter. It just... does something slightly different from what it was asked. Told to "find and book a flight under $400," a subtly-drifted agent might reason its way into a $1,200 upgrade and report back "done" — technically true, catastrophically wrong. Nothing in a normal observability stack notices this, because nothing failed . The agent succeeded at the wrong thing. I wanted a system where SigNoz wasn't just a dashboard you check after something breaks — where it actively fed a decision-making loop while the agent was still running . Architecture, in one rule Everything else in the project falls out of one non-negotiable decision I made on day one: rollback state has to be local and durable, never dependent on an external service being reachable. If your resilience system's own safety net depends on a third-party API being up, you haven't built resilience, you've built a second point of failure. So the split looks like this: Local, critical path (SQLite): the Checkpoint Manager. Every agent step writes a durable checkpoint — plan, budget consumed, completed steps — to disk via Node's built-in node:sqlite . Rollback reads from here, always, no exceptions. SigNoz, decision-support only: the Policy Engine queries SigNoz's Query API for prior-run context before scoring severity, and the Explanation Layer calls SigNoz's MCP server to ground its natura