Knowledge and Memory Management: Directions 1-3 Finalization Record

We just closed the finalization record for Directions 1 through 3 in our knowledge and memory management subsystem. This covers the core pipeline: ingestion, storage, retrieval, and context integration. Here’s what that actually means for the architecture, why we made specific tradeoffs, and how to use it in your own stack. The project has been iterating on how to decouple knowledge persistence from runtime memory while maintaining a unified query interface. Directions 1-3 form the foundation: a document store, a vector index, and a structured memory buffer that combines both. No more ad hoc caching or reinventing the retrieval loop. Everything lives behind a single KnowledgeGraph interface. Direction 1: Raw Document Ingestion and Storage We settled on a partitioned document store backed by a local SQLite database with a blob column for serialized content. Each document entry stores a UUID, source URI, raw text or bytes, a content hash, and a timestamp. The ingestion pipeline deduplicates by hash and runs through an optional extractor chain (e.g., PDF parser, markdown splitter, code chunker). The design decision is to separate storage from indexing entirely. The store is dumb—it only handles CRUD and metadata queries. This keeps the ingestion path simple and testable. Direction 2: Vector Index with Filtered Search Instead of building our own vector database, we wrapped existing infrastructure—Pinecone and a local FAISS fallback—behind an abstraction layer. The finalization record specifies a mandatory metadata filter set that must be packed into every upsert and query call. Each vector embedding carries a document UUID, chunk index, and a free-form tags map. This enables queries like “retrieve all chunks where module == 'networking' and version >= '2.0' ” without scanning unrelated vectors. The finalization also enforces a max-k retrieval of 50 with a similarity threshold of 0.65. Below that, the system returns an empty set rather than noisy garbage. We decided to p