Three layers of automated fact-checking for an LLM newsroom (and the bugs that forced each one)

Our site, presentofai.com , publishes AI industry analysis daily with no human in the writing loop: agents ingest news and company filings into an event timeline, score them, and synthesize digests and long form articles. This post is about the part nobody plans for on day one: the verification pipeline we had to build after the writing pipeline embarrassed us. If you are shipping LLM-generated content to the public, here is the architecture that stopped the bleeding, and the specific bugs that forced each layer. Layer 1: an article-level critic After every render, a judge model checks the draft against the source events it was built from: wrong attribution, merged or split entities, date errors, dek-vs-body contradictions, load bearing claims resting on a single source, number errors. Any high severity finding triggers exactly one revision pass, grounded only in the source events. Why one pass and not a loop? Because we watched each regeneration fix the flagged error and introduce a new one, always in the hardest to verify detail: a bill's sponsors, two similar bills merged into one, a date that was actually the date reporting confirmed the event rather than the date it happened. Unbounded self-revision does not converge, it wanders. Layer 2: search-verified claim checking The critic can only see the source events. If the error is IN your source data, the critic faithfully reproduces it. So a second stage extracts every load bearing claim (who, what mechanism, when, why, number) with a neutral search query for each, runs a fresh news search per claim, reads two or three independent articles, and rules each claim supported, wrong, contested or unverified. This layer caught an invented attribution that had survived five prior review rounds: the draft credited a named former official with a specific quoted phrase, and the fresh search showed he had co-signed a group letter with different wording. The phrase belonged to someone else. One rule keeps this layer honest: t