Part 3: Build the Eval Set Before the Agent Exists

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet. That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo. The eval set: eval/cases.json 21 cases, three buckets: Bucket Count Covers Easy 12 Shipping-status lookups, simple KB questions, a cancelled-order info request, one no-KB-match case that must escalate rather than fabricate Hard 6 Refund eligibility inside/outside the 30-day window, multi-item orders where only one item is refunded, boundary cases just past the window Edge 3 Legal-threat, fraud-flag, and duplicate-ticket patterns — must auto-escalate with zero tool/LLM calls A sample case, checking both the outcome and the trajectory that produced it: { "case_id" : "hard_03" , "bucket" : "hard" , "ticket" : { "ticket_id" : "hard_03" , "subject" : "Wrong size shoes, keep the socks" , "body" : "The running shoes from order ord_1004 are the wrong size. I want a refund for just the shoes, not the socks." , "customer_id" : "cust_002" , "order_id" : "ord_1004" }, "expected_trajectory" : [ "order_lookup" , "refund_eligibility" , "issue_refund" ], "expected_outcome" : "refund_proposed" , "expected_max_steps" : 4 , "policy_checks" : [ "refund amount reflects only the shoe item (~$74), not the full order total" , "issue_refund gated behind human approval" ] } Three things being checked per case, not just "did the answer loo