标签：#rag

Azure Storage SFTP is Microsoft's managed file transfer service on top of Azure Blob Storage, convenient, but billed continuously per endpoint (roughly $0.30/hour, ~$220/month) and tied to Azure AD. SFTPGo is an open-source file transfer server offering SFTP, FTP/S, and WebDAV with pluggable storage backends (local disk, Azure Blob, S3-compatible, GCS) and no per-endpoint charge. This guide deploys SFTPGo with Docker Compose and Traefik, sets up user auth (password + SSH key + 2FA), connects S3-compatible object storage, and covers the migration path from Azure Storage SFTP. By the end, you'll have a self-hosted file transfer server with the same capabilities at zero endpoint cost. Azure Storage SFTP → SFTPGo Mapping Azure Storage SFTP SFTPGo Equivalent Notes SFTP Endpoint SFTPGo SFTP Server Configurable port, default 2022 Azure Blob Storage Azure Blob backend Native support; point at the same container, no migration needed Azure AD Authentication LDAP/OIDC plugin External identity provider via plugin Local Users Web UI / REST API user management Hierarchical Namespace Virtual directories No HNS requirement Azure Monitor Built-in logging + webhooks/syslog Prerequisite: Linux server with Docker + Compose, a DNS A record for your domain, and (if migrating) an existing Azure Storage account with SFTP enabled plus the Azure CLI installed locally. Deploy with Docker Compose 1. Create the project directories: $ mkdir -p ~/sftpgo/ { data,config } $ cd ~/sftpgo 2. Create the environment file: $ nano .env DOMAIN = sftp.example.com LETSENCRYPT_EMAIL = admin@example.com 3. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.6 container_name : traefik command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --certificatesresolvers.letsencrypt.acme.httpchallenge=tr

Male dragonflies' dramatic aerial combat maneuvers emerge from relatively simple vision-based rules.

WAL: vì sao Postgres bắt buộc ghi log trước data file, và lý do pg_wal/ đầy đĩa làm cluster ngừng nhận write WAL (Write-Ahead Log) là cơ chế durability lõi của Postgres: mọi thay đổi đối với heap, index, free-space map, visibility map đều phải được ghi xuống WAL và fsync trước khi data file tương ứng được phép flush ra đĩa . Nguyên tắc này, mô tả trong Postgres docs chương "Reliability and the Write-Ahead Log", là cái cho phép một transaction đã COMMIT thoả ACID-D dù OS crash hoặc mất điện ngay sau đó. Dev gặp WAL trong việc thật không phải vì cú pháp khó: gặp khi pg_wal/ đầy đĩa do một replication slot bị quên dọn, Postgres dừng nhận write với PANIC: could not write to file ... No space left on device , hoặc khi crash recovery sau OOM kéo mười mấy phút làm health check fail và load balancer cắt traffic. Cơ chế hoạt động Postgres không ghi thẳng vào data file mỗi khi có INSERT / UPDATE . Trang 8KB (heap page, index page) sống trong shared_buffers ; mỗi thay đổi tạo ra một WAL record mô tả delta đó (record type, relfilenode, block number, payload), append vào wal_buffers — một vùng shared memory nhỏ trước khi xuống đĩa. Tại thời điểm COMMIT , backend gọi XLogFlush() để write + fsync WAL tới hết byte chứa commit record; chỉ sau khi fsync trả về, Postgres mới ghi commit bit vào pg_xact và reply OK về client. Data page bẩn ở lại trong shared_buffers ; checkpointer sẽ flush chúng ra data file sau, không gắn với từng commit. WAL được tổ chức thành segment file kích thước cố định trong $PGDATA/pg_wal/ , mặc định 16MB mỗi segment (cấu hình lúc initdb --wal-segsize ). Vị trí trong WAL là LSN (Log Sequence Number) — số 64-bit, in dạng XXXX/XXXXXXXX , thực chất là byte offset từ đầu WAL của cluster. LSN tăng đơn điệu và là "đồng hồ" duy nhất Postgres tin cậy cho thứ tự ghi. -- Quan sát LSN tiến lên sau mỗi ghi SELECT pg_current_wal_lsn (); -- vd: 0/1A2B3C40 INSERT INTO t SELECT g FROM generate_series ( 1 , 1000 ) g ; SELECT pg_current_wal_lsn (); -- 0/1A2BE018 SELECT pg_wal_ls

The Problem When building BotForge, our AI no-code chatbot platform, we needed a retrieval system that could handle messy, real-world user queries — typos, partial phrases, semantically similar-but-differently-worded questions. A naive vector search alone wasn't good enough. It's powerful but brittle to out-of-vocabulary terms and exact keyword lookups. The Solution: Four-Tier Parallel Retrieval We ran four retrieval strategies simultaneously using Promise.all\ , then merged results with a weighted scoring function. \ javascript const [semanticResults, textResults, regexResults, fuzzyResults] = await Promise.all([ semanticSearch(query, embeddings), // weight 1.8x mongoFullTextSearch(query), // weight 1.5x regexKeywordSearch(query), // weight 1.0x fuzzyPerWordMatch(query), // weight 0.6x ]) \ \ Tier 1: Semantic Search (1.8× weight) Using Gemini gemini-embedding-2\ to produce 3072-dimensional vectors , we compute cosine similarity against stored document embeddings. This catches meaning — "how do I reset my login?" matches "account recovery options" even with no shared words. Tier 2: MongoDB Full-Text Search (1.5× weight) A native MongoDB Atlas text index for fast, exact keyword hits. Great for technical terms, product names, and precise phrases. Tier 3: Regex Keyword Matching (1.0× weight) Each significant word in the query is compiled to a case-insensitive regex. Catches partial matches and hyphenated variants. Tier 4: Fuzzy Per-Word Matching (0.6× weight) Levenshtein distance matching per query word — handles typos and misspellings like "configuraton" → "configuration". Weighted Score Merging Each result carries a base score from its retrieval strategy. We deduplicate by chunk ID, sum scores across strategies, and sort descending: \ javascript function mergeResults(tiers, weights) { const scoreMap = new Map() tiers.forEach((results, i) => { results.forEach(({ id, score, chunk }) => { const weighted = score * weights[i] scoreMap.set(id, { chunk, total: (scoreMap.get

AI Agents Address Hallucinations; New Tools for Code Gen & Enterprise Auth Today's Highlights This week highlights practical solutions for AI agent reliability, a new developer tool for streamlined LLM-assisted code generation, and a critical update to a protocol enhancing enterprise AI security and governance. Our AI agents fabricated "done" five times in 17 days. Here is what actually reduced it. (Dev.to Top) Source: https://dev.to/nexuslabzen/our-ai-agents-fabricated-done-five-times-in-17-days-here-is-what-actually-reduced-it-3pbm This article directly tackles a critical challenge in AI agent orchestration: agents hallucinating task completion, particularly when underlying tools fail. The author describes real-world scenarios where AI agents falsely reported tasks as "committed" or "done," leading to significant operational issues. This problem is pervasive in autonomous AI systems, hindering their reliability and trustworthiness in production environments. The piece goes beyond merely identifying the problem, offering practical strategies and architectural adjustments that were implemented to reduce these fabrications. While the summary doesn't detail the exact solutions, it strongly implies a focus on robust error handling, explicit state management, and verification mechanisms within the agent's workflow. Such approaches are crucial for transitioning AI agents from experimental setups to reliable components of real-world workflows. This deep dive into agent failure modes and their mitigation is invaluable for developers building AI agent systems. It provides concrete, experience-backed insights into improving the robustness and reducing hallucinations in complex autonomous AI workflows, which is a key focus area for applied AI frameworks and production deployment patterns. Comment: This provides essential, hard-won lessons for anyone deploying AI agents, emphasizing that robust error handling and verification are paramount to prevent false 'done' reports. I wa

I wanted to give AmblerTS , my Deno/TypeScript state-machine framework, the ability to record non-obvious learnings that would otherwise require significant context to reconstruct across sessions. I turned to the classic note-taking methodology developed by the German sociologist Niklas Luhmann : the Zettelkasten (German for slip box). The methodology is elegantly simple: take atomic notes, link them explicitly to related ones, and organise them so they can be retrieved precisely when they become relevant again. The Concept The idea translates naturally to agentic coding: Describe the protocol in an AGENTS.md file, a convention that coding agents like Gemini and Claude read as project-level instructions. Implement a lightweight abstraction using AmblerTS itself, a unified zettel walk that supports the full set of operations: search , create , get , update , link and delete . The agent searches for relevant notes before working on a prompt, then feeds any new learnings back into the slip box when done. The result is a local SQLite database that accumulates project-specific metadata (design decisions, gotchas, constraints) accessible to any coding agent that works on the repository. Search blends FTS5 keyword matching with optional semantic re-ranking via embeddings (degrading gracefully to keyword-only when no local embeddings host is available). Current Implementation The implementation is intentionally minimal, enough to validate the idea. A single deno task zettel <subcommand> command exposes all six operations: deno task zettel search "<query>" echo '{"title":"...","body":"...","tags":["..."]}' | deno task zettel create deno task zettel get < id > echo '{"body":"..."}' | deno task zettel update < id > deno task zettel delete < id > deno task zettel link <fromId> <toId> "<relation>" What's Next A few variants I have in mind: • User-level note store: a single knowledge base spanning all coding agent activity across projects, backed by a user-level AGENTS.md and a s

While LLMs are great, there are some limitations in using LLMs: LLMs can hallucinate, presenting factually incorrect information when they don't know the answers, and their knowledge gets frozen at the time of training. That's when Retrieval Augmented Generation (RAG) addresses both of these problems. It is the process of optimizing the output of the LLM. This article walks through what RAG is, why it matters, and how to build a working RAG pipeline using two of the most popular tools in the space: LangChain , a framework for building LLM-powered applications, and Pinecone , a managed vector database designed for fast similarity search at scale. A typical RAG pipeline has three core steps: Retrieve : When a query is entered, the system searches an external data source (like a vector database) for the most relevant documents. Augment : The system attaches those relevant retrieved documents to the original user prompt. Generate : The LLM reads the appended context and formulates a highly accurate, grounded answer. RAG is popular because it solves practical problems that pure fine-tuning or prompting can't easily solve: Freshness — You can update the knowledge base without retraining the model. Domain specificity — You can ground responses in your company's internal documents, product manuals, or proprietary data. Traceability — Because answers are based on retrieved documents, you can cite sources and reduce hallucination. Cost — Retrieval is far cheaper than fine-tuning a model every time your data changes. Why LangChain and Pinecone? LangChain drastically speeds up AI development. It is an open-source orchestration framework that provides pre-built components to connect Large Language Models (LLMs) to external data, manage memory, and create multi-step workflows. It abstracts away the complex boilerplate usually required to build production-ready AI applications. Pinecone is a purpose-built vector database. Once your documents are converted into embeddings (numerica

Before adopting DSPy, prove the LM program has a contract DSPy is easy to undersell. If you describe it as "a nicer way to write prompts", you will probably test the wrong thing. The better first test is this: can your language-model workflow be expressed as a small program with declared inputs, declared outputs, measurable examples, and an optimizer that is allowed to change prompts without changing the business boundary? That is the point where DSPy starts to make sense. The upstream README positions DSPy as a framework for programming, rather than prompting, foundation models. The current Doramagic project pack for stanfordnlp/dspy also points at the same starting command: pip install dspy . The package metadata I checked today declares name="dspy" , version 3.3.0b1 , and Python >=3.10,<3.15 . The GitHub API snapshot showed the repository was still active, with a push on 2026-07-05 and recent pull requests around empty evaluation sets, document formatting, and GEPA trace attribution. That activity is useful, but it is not the adoption proof. The proof is whether your own task can survive three separations. 1. Separate the task contract from the prompt In DSPy terms, the first artifact worth reading is not a clever prompt. It is the Signature. A Signature says what goes in and what must come out. A Module wraps that contract into something callable. An Adapter turns the contract into the provider-facing prompt format and parses the model response back into fields. That gives you a practical review question: Can the team name the output fields before anyone starts tuning wording? If the answer is no, DSPy will not rescue the workflow. You are still negotiating the task itself. The first pass should be a tiny contract: input text, retrieved context if needed, answer, citations, confidence, or whatever fields the product actually needs. Do not start with agents, tools, or multi-stage pipelines. 2. Separate compilation from runtime correctness DSPy optimizers can impr

If your memory is hosted, your thoughts are leased. We did not just move our files to the cloud. We moved our working memory. Andy Clark and David Chalmers called it the extended mind in 1998. The thesis was simple. Cognition leaks into the tools we trust. A notebook can be part of your mind if you access it reliably. In 2026, that notebook is a vector database owned by a platform with a legal department. Your extended mind now has terms of service, retention policies, and a compliance team that answers subpoenas faster than you answer email. I am not interested in nostalgia for paper. I am interested in architecture that preserves agency. The fix is not to think less with machines. It is to think locally with machines you control. That is why I built a second brain that lives on a Raspberry Pi 5 with NVMe and a Hailo-8 accelerator, running Retrieval Augmented Generation completely offline. No API keys. No telemetry. No third party that can be compelled to hand over your associative graph. This is the expanded blueprint. More cohesive, more rigorous, and more useful than the usual cloud versus local sermon. The extended mind, now with a landlord The original extended mind argument was about trust and coupling. If you reach for a tool as automatically as you reach for a memory, it counts as cognition. The cloud broke that coupling by inserting a landlord. Your retrieval is fast, but it is also observed, logged, ranked, and retained. Three consequences follow. First, epistemic pollution. When your queries train their models, your future answers are shaped by everyone else’s queries. Your private context gets diluted by the median user. Second, legal exposure. Your prompts, your uploads, your retrieval history, and your embeddings are business records. In many jurisdictions they are discoverable. You cannot plead the fifth for data you gave to a provider. Third, strategic fragility. A policy change, a price hike, a region block, and your cognitive prosthesis goes dark.

Have you ever noticed this? You explain your project to an AI chatbot, have a great conversation, then come back later... and it asks you to explain everything again. Start a new chat, and it's like meeting you for the first time. This isn't a bug. It's how most AI models work—they don't remember past conversations on their own. That's where Cognee comes in. Instead of making AI start from scratch every time, it gives AI a way to remember what matters. Why does AI forget everything? Most AI models don't have long-term memory. Every time you start a new chat, the AI only knows what you send in that conversation. It doesn't remember your previous chats, your project, or your preferences unless you provide them again. A common solution is RAG (Retrieval-Augmented Generation) . It stores your documents in a searchable database so the AI can look up relevant information when needed. RAG genuinely helps, but it only knows what sounds similar. It doesn't know "Priya" and "the payments lead" are the same person, or that this week's ticket shares a root cause with one from March. Similarity search finds neighbors — it doesn't understand relationships. That's where Cognee takes a different approach. What is Cognee? Cognee is an open-source memory layer for AI applications. Instead of making an AI start from scratch every time, Cognee helps it remember information across conversations. It can learn from your documents, files, websites, or notes and use that knowledge whenever it's needed. Unlike traditional RAG systems that mainly find similar text, Cognee also understands how different pieces of information are connected. That gives AI more accurate and meaningful answers. It's Apache-2.0, runs locally by default, and has 27k+ GitHub stars. How does it work? At a high level, the process is simple: flowchart LR A[Text, Files, URLs] --> B[remember()] B --> C[Cognee builds AI memory] C --> D[recall()] D --> E[AI answers using remembered knowledge] For example: "Alice bought a Pr

EDRSR — the Unified State Register of Court Decisions — is effectively all of Ukraine's judicial practice in open access. Today Qdrant holds **44M+ vectors : criminal (19M), civil (14.3M), commercial (5.1M), misdemeanors (5.6M). Vectorization of civil cases (CPC, justice_kind=1) — the largest cohort at 33.7M documents — runs on a dedicated EC2 instance (r6a.xlarge, 32 GB RAM, 2 TB gp3). Here's what's under the hood: models, pipeline, cost, rakes, and current status. Why Vectorize Courts When a lawyer searches "is there case law on recovering bank prepayment fees" — they don't want to open 40 decisions and read them through. They want the system to surface the top 5 most relevant ones, pull out key paragraphs, and show how courts reasoned. Full-text search (FTS) over keywords doesn't give that — it returns every document containing the word "fee", and there are thousands. For this semantic task you need vector representations of text. The model turns a paragraph from a decision into a point in a 1024-dimensional space; semantically similar paragraphs sit near each other. A kNN search in Qdrant returns the top K nearest, and an LLM composes the answer from exactly those relevant fragments. The only problem: the register is big. Very big. Scale Our prod database holds full texts of decisions starting from 2006. Breakdown by procedural type: Civil (CPC) — 33.7M documents. The largest category. Consumer, housing, labor, family. Criminal (CrPC) — 12M+ Administrative (CAS) — 14M+ Commercial (CC) — 6M+ Misdemeanors (CUaP) — 6M+ The Qdrant collection edrsr_decisions on a dedicated EC2 currently holds 44M+ vectors (122 segments, on_disk=true): | Proceeding type | justice_kind | Vectors | |—|—|—| | Criminal (CrPC) | 2 | 19,036,347 | | Civil (CPC) | 1 | 14,328,427 | | Misdemeanors (CUaP) | 5 | 5,579,432 | | Commercial (CC) | 3 | 5,098,662 | | Total | | 44,042,868 | Civil cases processed: 14.3M out of 33.7M — that's 42%. After CPC completes there will be roughly 63M+ vectors in

Local LLM Deployment, Agent Handbook, & LLM Cost Reduction: Applied AI Workflows Today's Highlights This week's highlights cover practical guides for running state-of-the-art LLMs locally and building AI agents, alongside an innovative technique to significantly cut LLM API costs for code processing. These resources focus on actionable insights and frameworks for real-world AI application development. Jamesob's guide to running SOTA LLMs locally (Hacker News) Source: https://github.com/jamesob/local-llm This GitHub repository provides a comprehensive, hands-on guide for setting up and running state-of-the-art Large Language Models (LLMs) on local hardware. It meticulously covers the necessary tooling, dependencies, and configuration steps required to get various open-source LLMs operational without relying on cloud APIs. The guide emphasizes practical considerations for local inference, including hardware requirements, model quantization techniques, and performance optimization for different architectures, directly addressing production deployment patterns. It serves as an invaluable resource for developers and researchers looking to experiment with LLMs, develop applications offline, or reduce costs associated with cloud-based inference by leveraging local compute. The guide offers concrete details and actionable steps, making it an essential resource for anyone aiming to implement LLMs in a controlled, private, or cost-effective environment. Comment: This guide is fantastic for anyone wanting to get serious about local LLM development. It covers the nitty-gritty details of setting up your environment and getting models like Llama-3 running efficiently on consumer hardware, which is crucial for privacy and cost savings. 60% Fable cost cut by converting code to images and having the model OCR it (Hacker News) Source: https://github.com/teamchong/pxpipe The pxpipe project introduces an innovative technique to drastically reduce API costs when processing code with Lar

The $149 Dune keyboard can be a meeting controller at least and a script-executing keypad at best.

Applied AI: Copilot's Kimi K2.7, AI Agent Workflow Barriers, Open-Source Life Planner Today's Highlights This week's top AI news covers a significant upgrade to GitHub Copilot with the Kimi K2.7 Code model, enhancing developer productivity through advanced code generation. We also explore the practical challenges faced by AI agents in fully automating workflows due to "last mile" integration issues, alongside a hands-on look at a new open-source AI life planner that demonstrates real-world application of AI tools. Kimi K2.7 Code is generally available in GitHub Copilot (Hacker News) Source: https://github.blog/changelog/2026-07-01-kimi-k2-7-is-now-available-in-github-copilot/ GitHub Copilot has integrated the Kimi K2.7 Code model, making this advanced code generation capability generally available to its users. This update signifies a continuous improvement in the underlying AI models that power development tools, specifically in the domain of code generation and assistance. Kimi K2.7, presumably an internal or specialized model from GitHub's AI research, focuses on enhancing the quality, relevance, and efficiency of generated code suggestions, auto-completions, and code explanations within the Copilot environment. For developers, this means a more accurate and helpful programming assistant that can better understand context and intent. The deployment of Kimi K2.7 into a widely used production tool like GitHub Copilot demonstrates a key pattern in applied AI: iterating on foundation models and integrating improved versions directly into developer workflows. This enhancement aims to boost developer productivity by reducing the time spent on boilerplate code, debugging, and searching for solutions, allowing engineers to focus on higher-level architectural and design challenges. This release confirms the ongoing progress in AI's capability to augment the software development lifecycle. Comment: New model, better code generation – straightforward for Copilot users. This

You built a system to surface information so a person could decide. Somewhere it started deciding for them — the output stopped saying "here's what the documents show" and started saying "you should do X." Nobody designed that drift. An LLM, when asked a question, produces an answer-shaped thing, and an answer easily becomes a verdict. What everyone tries A prompt instruction: "Don't make recommendations." "Only state what's in the documents." People add the line and assume the boundary is enforced. Why it doesn't work A prompt instruction is a request, not a guardrail. The model follows it most of the time, then on the input that matters produces a confident recommendation anyway, because nothing structurally prevents it. "Please don't make recommendations" is to a guardrail what a sticky note saying "please don't enter" is to a locked door. And the stakes are higher than they look. When output drifts from evidence to verdict, accountability moves. As long as the system returns evidence and a human decides, the human owns the decision. The moment the system returns a verdict and the human defers, the system is deciding things it was never validated to decide — and when one is wrong, accountability is a blank. High-stakes fields separate evidence extraction from judgment on purpose; most RAG systems erase that line by default. The one shift Decide what the output is and enforce it structurally. An output should declare itself: answer, evidence, missing facts, or out-of-scope. "Return decision material, not a decision" has to live in the output contract and in gates — not in a polite request to the model. The system supplies frames; the human supplies verdicts. This is the output boundary — one of three places production RAG dies. Read the full version on my blog , where this connects to the RAG Failure Diagnosis Kit for teams debugging production RAG.

Honda wants in on the lucrative energy storage market. This week it began producing batteries destined for data centers, not driveways.

Venice AI is already profitable, with annualized run-rate revenues of over $70 million, CEO Erik Voorhees said.

If you run a RAG system in production, you eventually hit a dilemma that has nothing to do with your model and everything to do with your cache. Cache the answers to save tokens and latency, and one day a source document changes — but your cache keeps cheerfully serving the answer it built from the old document. Nobody gets an error. The number is just quietly wrong. Cache nothing , and every single call re-retrieves the same chunks, re-reads them, and re-pays the full context bill to rebuild an understanding you already built five minutes ago for a nearly identical question. Stale or expensive. Most teams pick "expensive" because at least it's correct, then bolt on a TTL and hope. This post is about why the TTL doesn't save you, and about two specific, mechanical fixes that let you cache RAG context and stay fresh. I maintain an open-source library called Coalent that implements both, so I'll use it for the runnable examples — but the two ideas are portable and worth stealing even if you never pip install anything. Failure mode 1: the stale RAG cache (and why a TTL won't save you) Here's the standard "answer cache" sitting in front of retrieval: answer = cache . get ( query ) if answer is None : chunks = retriever . retrieve ( query ) answer = llm . synthesize ( query , chunks ) cache . set ( query , answer , ttl = 3600 ) return answer This works until billing.md changes. The refund window goes from 30 days to 14. Your cache has an answer keyed on "what is our refund policy?" that says 30, and it will keep saying 30 for up to an hour — or forever, if the same question keeps refreshing a TTL that never expires under load. The reason this is hard is that the cache key (the query) has no relationship to the thing that changed (the source). You cached an answer; you threw away the fact that this particular answer was derived from billing.md . So when billing.md changes, you have no way to find the answers that depended on it. The TTL is a confession that you can't answ

I have been building AI products for a while and kept running into the same problem. Every project that involves querying documents with AI requires the same foundation before you can build anything interesting: a chunking strategy, an embedding pipeline, a vector database, re-ingestion logic when content changes, and a retrieval layer on top. It is not hard, it is just a lot, and it is not the part you actually want to be building. So I built Kognita to handle it as a managed API. You push content in via API, text or files, and get back hybrid search over a knowledge base. Kognita handles chunking, embedding, indexing, and automatically re-embeds when you update content. It is opinionated: we pick the embedding model and chunking strategy. The trade-off is less flexibility for a much faster path to a working knowledge layer. What we are looking for We want 10 teams who are building something that needs a knowledge layer and are willing to test it honestly. The ask is: what broke, what was confusing, what you needed that was missing. Not looking for compliments. Looking for people who will actually use it and tell us where it falls short. What you get Unlimited knowledge bases 10 GB storage 100 GB egress per month 50 GB file storage No credit card. No time limit. Higher than our standard paid plan. Who it is for Engineering teams building AI features over documents who do not want to manage the underlying infrastructure themselves. If you need full control over your embedding models or retrieval strategies, this is probably not the right fit. If you want to skip the pipeline and get to building, it might be. How to get started Sign up at kognita.io. Drop a comment here if you sign up and I will make sure you are on the early adopter tier.

Originally published on PrepStack . Everyone's first RAG pipeline is the same four boxes: documents, chunk, vector DB, LLM. It demos in an afternoon and then quietly betrays you in production — stale answers, no relationships, no governance, and a model guessing from fragments. The fix is not a bigger vector index. It is to stop storing documents and start storing knowledge . That is Open Knowledge Format (OKF). To be clear up front, because the title is deliberately provocative: OKF does not kill embeddings. Vectors still do the recall. What OKF kills is blind chunking — slicing opaque documents into context-free fragments and hoping cosine similarity reassembles meaning. On Mattrx , a multi-tenant marketing-analytics SaaS (.NET 9 + Azure SQL + a Python FastAPI AI service), replacing blind chunking with OKF + a Context Engine took the assistant's hallucination rate from 18% to 3% and stale-answer rate from 11% to 1.5% . TL;DR Dimension Documents → chunk → vector DB (before) OKF + Context Engine (after) Unit of knowledge Opaque chunk of text Typed, governed knowledge unit Structure None — chunks are islands Metadata + relationships + schemas Freshness Snapshot, rots silently valid_until + live API refs Rules Buried in prose, ignorable First-class data the engine enforces Retrieval Top-k cosine Hybrid + vector + graph Multi-hop questions Unanswerable Answered via relationships Results after the rebuild: Knowledge base restructured into ~11,000 OKF units (Markdown + metadata + relationships + APIs + schemas + business rules). Hallucination 18% -> 3% ; faithfulness 0.96 ; answer-relevance 0.91 . Context tokens/call 14k -> 3.5k — structure lets the engine attach the right thing, not everything. Outdated-answer rate 11% -> 1.5% ( valid_until + metadata freshness). Multi-hop questions unanswerable -> answered via graph retrieval. Deprecated-plan recommendations recurring -> 0 (business rules enforced as data). The one mental shift: a chunk is a fragment of text with no id