389 Tests Passed. NIST Still Caught the Bug.

I gave an AI agent a calculator because I wanted one hard, inspectable point inside a probabilistic workflow. The model could interpret the request and explain the result. The calculator would perform the computation. It seemed like a clean division of labor. Then I changed one multiplication sign into addition. The calculator still passed 389 of the 390 tests in its Rust library harness. The sole failure compared its answer with NIST's certified results for the Longley regression dataset. That bothered me more than a completely broken build would have. I had treated deterministic computation as safer than asking a language model to improvise arithmetic. But deterministic does not mean trustworthy. A program can return the same wrong answer forever. “Source of truth” suddenly felt too comfortable. Before an AI agent delegates authority to a tool, that authority should be challenged—and remain revocable by evidence. The calculator is only the specimen. The larger idea is a way to place inspectable, replayable instruments inside probabilistic systems. The useful boundary is generation versus execution The interesting distinction is not model weights versus a “real CPU.” Model inference also runs on processors, and language models can learn genuine arithmetic procedures. The useful boundary is between generating an answer and executing a defined operation under a tested contract . Research on Program-Aided Language Models (PAL) makes a related split: the language model reads and decomposes a natural-language problem, while a runtime such as a Python interpreter executes the generated program. The model contributes flexible interpretation; the runtime contributes executable semantics. That is the division I want in an agent: At the semantic edge , the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result. At the computational edge , a narrow tool validates inputs, applies specified operations, enforces limits, and ret