A Small, Checkable Test for AI Memory Systems

AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication. AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work. A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately. 1. Begin with one source you understand Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search. Example: The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet. Ask questions whose answers are directly present in the note: When is the review? Who owns the checklist? Has the database change been approved? The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.” 2. Inspect the supplied evidence A plausible answer is not enough. Open the source or evidence shown beside the answer and check: Did the system retrieve the right document? Did it select the relevant passage? Did the answer preserve names, dates, and negation? Can another person repeat the check? This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes. 3. Ask for something that is missing Now ask a question the note cannot answer, such as: Which meeting room is booked? A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models a