JSON Schema Doesn't Prevent AI Hallucinations (And That's Okay)

A few months ago, if you had asked me whether Structured Outputs solved hallucinations, I probably would have said yes. Today, I wouldn't. Not because Structured Outputs don't work-they absolutely do. But because they solve a different problem . After building ShapeCraft, an open-source structured output library supporting OpenAI, Groq, Ollama (GBNF), and Anthropic, I realized something that fundamentally changed how I think about production AI. Structure and correctness are two completely different guarantees. The misconception Let's say you're extracting data from an invoice. The invoice says: Invoice Total: $900 Your model returns: { "invoiceTotal" : 1200 } Let's validate it. Validation Result Valid JSON ✅ Matches JSON Schema ✅ Required field present ✅ Correct data type ✅ Correct answer ❌ The response is structurally perfect. The extracted value is still wrong. JSON Schema didn't fail. It did exactly what it was designed to do. What JSON Schema actually guarantees JSON Schema is responsible for ensuring your application receives data in a predictable format. It guarantees things like: Valid JSON Required fields Correct data types Nested object structure Array validation Enum validation That makes applications dramatically easier to build. But it does not guarantee: The extracted value is correct The answer exists in the source document The model didn't hallucinate Business rules were followed Those are different problems. Structure ≠ Truth I've started thinking about AI validation as two independent layers. Layer 1 - Structural Validation Can my application safely consume this response? Examples: Is it valid JSON? Does it match my schema? Are required fields present? JSON Schema solves this extremely well. Layer 2 - Semantic Validation Can I trust the information? Questions become: Did the model extract the correct value? Can I trace it back to the source? Is there supporting evidence? Would I make a business decision based on this response? This layer is much ha