I have been Vibecoding Evals (works better than I thought)

I’ve been building AI apps with coding agents for a while. Lately, I’ve been experimenting with evals too. The app in this example mostly worked. That was the problem. The bug I built a small support-triage app for a fictional shipment-tracking company. A customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond. A real outage should be escalated. But this ticket was different: “URGENT need key rotation now” The customer was asking how to rotate their own API key before a security review. The app classified it as a security incident and escalated it to a human. That was wrong. The policy said normal key rotation was a self-service how-to request. Nothing crashed. The app returned valid JSON. The fields all contained allowed values. The behavior was still wrong. Why clicking around wasn’t enough I could test a few tickets manually and convince myself the app worked. But after changing the prompt, what would I actually know? Would the outage case still escalate? Would normal how-to questions stay in the normal queue? Would another API-key question behave differently? I didn’t want to change the prompt and simply hope for the best. I wanted a set of cases I could run again. Adding DeepEval with Cursor I installed the DeepEval agent skill: npx skills add confident-ai/deepeval --skill "deepeval" Then I asked Cursor to add evals to the app: This app sometimes treats normal support questions like emergencies and sends them to a human. Add DeepEval so I can test this using the tickets and policy already in the repo. I am new to evals, so use the simplest setup DeepEval already provides, explain what you create, and ask me anything you need. Run the app as it is first and show me what fails. Do not fix it yet. Cursor already had the app, tickets, and policy, so it went straight to creating the baseline. Goldens are the checklist The first useful artifact was a JSON dataset. Each golden contained: the custome