How to Actually Test a Chatbot (Not "The Answer Looked Fine")

Most teams accept a chatbot the same way: they ask it a handful of questions, the answers look reasonable, and the service is signed off. That procedure proves almost nothing. The short answer Evaluating a Persian-language assistant properly needs three things: a fixed set of questions replayed identically every time, an explicit assertion for each one about which source the answer must come from, and response time measured separately from network time. Without all three, any change to the system can break something nobody notices. "The answer looked fine" is not a criterion A retrieval or generation system almost always returns something. If it does not recognise the topic it returns a default; if it recognises the wrong topic it returns an answer that is correct in itself and irrelevant to the question. The second failure is the more deceptive one, because it reads well. In our own evaluation, the question do you offer enterprise training? was answered with the project-timeline text — entirely accurate, entirely unrelated. Anyone reading only the answer sees nothing wrong. The fix is for each test case to assert not "the answer was good" but that the answer contains a specific marker: the path of the page it should point to, a distinctive phrase from the reference text, or the contact number. A fluent irrelevant answer then fails. // Not this — it passes for any non-empty string: assert ( answer . length > 0 ); // This — it fails a fluent answer that came from the wrong place: assert ( answer . includes ( " /chatbot-karaj " )); Build a question set, not a few scattered checks What worked in practice was a list where every entry carries four fields: the question text, the language, a string that must appear in the answer, and a label saying what the case is testing. const CASES = [ { q : " آموزش سازمانی دارید؟ " , lang : " fa " , expect : " /enterprise-ai-training-karaj " , why : " enterprise vs project timeline " }, { q : " چت‌بات چقدر طول می‌کشد؟ " , lang : " fa