标签：#rag

TL;DR An agentic RAG pipeline treats retrieval as a tool the AI agent can call, evaluate, and call again rather than as a fixed step. The pipeline can search an internal knowledge base first, then use real-time web search when the available evidence is missing, weak, or outdated. Internal documents and web results should be converted into a shared evidence format before the model generates an answer. A reliable system must preserve URLs, publication dates, document identifiers, and the claims supported by each source. Retrieval quality, web-search precision, citation correctness, latency, cost, and stopping behaviour should all be evaluated. A basic RAG pipeline works well until the answer is not in the knowledge base. Imagine an enterprise copilot that can answer questions about internal product documentation. It performs semantic search against a vector database, retrieves several relevant passages, and passes them to a language model. For questions covered by the indexed documents, the system may work remarkably well. Then a user asks about a release announced yesterday, a recently changed regulation, or how the company’s product compares with a new competitor. The vector database cannot retrieve information it has never indexed. A conventional pipeline may return no answer, but it may also produce a confident response from incomplete or outdated context. Adding a Web Search API helps solve the freshness problem, but it introduces another decision: when should the system trust its internal knowledge, and when should it search the open web? An agentic RAG pipeline places that decision inside the retrieval workflow. What Makes a RAG Pipeline Agentic? A traditional RAG pipeline usually follows a fixed path: transform the question into a search query, retrieve the most similar passages, add those passages to the prompt, and generate an answer. An agentic RAG pipeline allows the model to make decisions between those stages. Retrieval becomes a tool rather than a manda

Why a Local RAG Chatbot for Trading Research Most "AI trading assistant" products are black boxes: your notes, strategy docs, and market notes get shipped to a third-party API, billed per token, and stored who-knows-where. For a retail NIFTY trader or a quant researcher, that is the worst of all worlds — you pay continuously, you leak your edge, and you cannot audit what the model actually read. This guide shows how to build a Retrieval-Augmented Generation (RAG) chatbot that runs 100% locally on an Android phone using Termux + Ollama. It ingests your own research (PDFs, markdown notes, option-chain exports) and answers questions grounded only in that data. No OpenAI key. No Anthropic key. No monthly bill. No data leaving the device. OBSERVED: Running ollama run llama3.2 on a mid-range phone inside Termux is slow but usable for document Q&A (3–8 tokens/sec). On a laptop it is smooth. SOURCE: Local testing on Termux 0.118, Ollama 0.3.x, Android 14. DERIVED: For production research volumes, run Ollama on a spare x64 machine and point Termux at it over LAN. What You Will Build A four-part pipeline: Ingest — load your research docs (markdown, PDF, CSV) into chunks. Embed — turn chunks into vectors with a local embedding model. Store — keep vectors in a local file-based index (no server needed). Answer — retrieve top-k chunks and ask a local LLM to answer strictly from them. The whole thing is ~200 lines of Python. No paid APIs. Prerequisites Android phone with Termux installed (F-Droid version, not Play Store). ~2 GB free storage. Basic Python comfort. pkg update && pkg upgrade -y pkg install python clang ffmpeg -y pip install ollama numpy Install Ollama inside Termux: curl -fsSL https://ollama.com/install.sh | sh NOTE: The official install script targets Linux. On Termux you often need the community build. If the script fails, install the ollama package via a Termux-compatible binary or run Ollama on a LAN machine and use ollama serve remotely. Pull a small model and a

The first time my RAG system gave a confidently wrong answer, I did what everyone does: I blamed the...

I wrote this on X a few weeks ago: I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win. Ask my RAG assistant to verify an Interswitch webhook signature, and it didn't say "not in my knowledge base." It wrote a full authentication flow — real-looking endpoint, real-looking headers — and cited a source URL. The URL wasn't in my corpus. It wasn't anywhere. The model invented a citation for content it also invented, with zero hedging. I'm building StacksNG for the Africa Deep Tech Challenge 2026 — an offline coding assistant scoped to the African fintech stack: Paystack, Flutterwave, Monnify, Termii. Before I submitted, I ran a 20-prompt adversarial batch against my own pipeline. Category A (in-corpus baseline) and D (phrasing brittleness) came back clean. Category B — five prompts asking about payment providers I deliberately never scraped into the corpus, Kuda, PalmPay, Interswitch, Paga, OPay — did not. Three of five ignored a system prompt that already said, in plain language, "if the context doesn't contain enough information, say so." That's the failure mode that zeroes out half the score in a hackathon where accuracy is 50% of the total. My first theory was wrong, and I could prove it My instinct was: this is a retrieval-confidence problem. Set a similarity threshold, refuse to answer below it, done. I checked the actual numbers before writing that fix. Top-1 similarity What happened Correct in-corpus answer 0.718 correct Worst fabrication (Interswitch) 0.712 fully invented, fake citation Correct decline (out-of-domain topic) 0.691 "not in my knowledge base" The worst hallucination had higher retrieval similarity than the cleanest correct decline. There's no threshold that lets the good case

从 Demo 到生产：那些真正让 AI Agent 敢上线的护栏 开场钩子： 你在网上看到的多数「AI Agent」都是 demo。它们之所以上不了生产，原因往往 只有一个 —— 而下面这个开源的小脚手架，专门解决它。 我们已经过了「能调通大模型」就算赢的阶段。现在真正难的是那没人讲的 10%： 是什么阻止 Agent 做出伤害性的事？ 我在微软跑过一套约 25 个 Agent 的生产平台，现在也帮团队把 Agent 从笔记本推进到真实用户面前。两边的体会是一致的。 一个不太舒服的真相：能调 5 个工具的聊天机器人， 不是产品 。周末项目和你敢放到客户面前的 系统之间，差的只有三件事 —— 而且全都是不酷、不性感的工程： 你怎么给输出质量打分 （质量门）。 你怎么决定什么时候必须人签字 （审批门）。 你如何让整套东西模型无关 ，不被某个厂商锁死。 所以我写了一个很小的 harness，把这三件事摆在最显眼的位置。它故意做得很小 —— 一小时能 读完 —— 因为价值不在「框架」，在 模式 本身。 仓库： github.com/zhasun0818/ai-agent-scaffold 1. 质量门：别发布你无法打分的东西 Agent 的输出是「预测」不是「承诺」。上线前它必须过一道 检查 ：是否达到你的标准。脚手架里 这是一个可插拔的 QualityGate ，你可以换成 LLM 裁判或测试套件： # agent_harness/eval.py @dataclass class EvalReport : passed : bool score : float checks : List [ str ] class QualityGate : def grade ( self , proposal : str , context : str = "" ) -> EvalReport : return self . grader ( proposal , context ) 循环在门没过之前拒绝执行： result . report = self . quality . grade ( proposal , f " state= { state } " ) if not result . report . passed : self . approval . log ( " quality-gate " , " blocked " , result . report . __str__ ()) return result 注意它 把拦截记录下来了 。生产里你会想把这些被拦的尝试都进可观测性系统。「这周我们拦下 了 12% 的 Agent 提议」是个真实 KPI —— 它说明门在工作。 2. 审批门：所有人都忘掉的那一步 这才是让企业真正点头说「可以」的东西。当 Agent 想加急订单、取消订阅、或动钱的时候，它应该 停下来问人 。沉默不等于同意。 # agent_harness/approval.py class ApprovalGate : def request ( self , action : str , detail : str ) -> bool : # 生产里：推一条通知到 Teams / Slack / 邮件，然后等待。 decision = input ( f " Approve { action } ? [y/N] " ). strip (). lower () self . audit . append ( AuditEntry ( time . time (), action , " human-reviewer " , decision , detail )) return decision . startswith ( " y " ) 在脚手架里，标记 needs_approval=True 就够了： @tool ( " expedite_order " , " Mark an order as expedited. " , needs_approval = True ) def expedite_order ( order_id : str ) -> str : return f " PO { order_id } : marked expedited " 而且因为有 审计链 ，你永远能回答「谁改的、为什么」—— 这通常是合规团队问的第一个问题。 3. 模型无关的 provider：别跟一个厂商结婚 模型每几周就变，价格也是。你的 Agent 循环不该知道自己在对谁说话： # agent_harness/providers.py class ModelProvider ( Protocol ): def

From Demo to Production: The Guardrails That Make an AI Agent Safe to Ship Hook: Most "AI agents" you see on the internet are demos. Here's the single most common reason they never reach production — and a small, open-source harness that gets past it. We are past the phase where the hard part of building an AI agent was calling the model. The hard part now is the 10% nobody talks about: what stops the agent from doing something harmful? I've seen this from both sides — I built and ran a ~25-agent platform in production at Microsoft, and now I help teams take agent ideas from a notebook to real users. The uncomfortable truth: a chatbox that can call 5 tools is not a product. The difference between a weekend project and a system you can put in front of customers is three things — and they're all boring, non-glamorous engineering: How you grade output quality (the quality gate). How you decide when a human must sign off (the approval gate). How you make the whole thing model-agnostic so you're not locked into one vendor. So I wrote a tiny harness that keeps these front and center. It's intentionally small — small enough to read in an hour — because the value isn't in a framework, it's in the pattern . Repo: github.com/zhasun0818/ai-agent-scaffold 1. The quality gate: don't ship what you can't grade An agent's output is a prediction, not a promise. Before it ships, you need a check that it passes your bar. In the harness this is a pluggable QualityGate — a rule of thumb you swap with an LLM judge or a test suite: # agent_harness/eval.py @dataclass class EvalReport : passed : bool score : float checks : List [ str ] class QualityGate : def grade ( self , proposal : str , context : str = "" ) -> EvalReport : return self . grader ( proposal , context ) The loop refuses to execute if the gate fails: result . report = self . quality . grade ( proposal , f " state= { state } " ) if not result . report . passed : self . approval . log ( " quality-gate " , " blocked " , result

OVHcloud will raise prices from September, with 2026-edition gaming servers up 87 percent and other recent servers 40 to 59 percent. Founder Octave Klaba says memory cost six times more in June than a year earlier, as RAM suppliers shifted capacity toward high-bandwidth memory for AI. AWS, buying years ahead, has repriced one reserved GPU product. By Steef-Jan Wiggers

If your mental model of RAG is "chunk → embed → search → LLM," you're missing about 80% of what actually makes a RAG system production-ready. Here's a practical checklist across all 10 lifecycles I ran into while building one. Full technical breakdown with diagrams is on Hashnode (linked above) — this is the condensed, "what to actually check" version. ✅ Document lifecycle [ ] Can you update a single document without a full re-index? [ ] Do you have a deletion path (not just an addition path)? [ ] Are you deduplicating before you embed? ✅ Embedding lifecycle [ ] Do you know what happens if you switch embedding models? [ ] Are you tracking dimensions and normalization consistently? [ ] Can you re-embed the whole store without downtime? ✅ Retrieval lifecycle [ ] Are you tuning Top-K, or using a default and hoping? [ ] Do you have metadata filtering before similarity search? [ ] Have you tried hybrid (keyword + semantic) search yet? ✅ Inference lifecycle [ ] Do you know your cold-start latency vs. warm inference? [ ] Are you tracking tokens/sec as a real metric, not a vibe? [ ] CPU or GPU — did you choose, or did it choose you? ✅ Prompt lifecycle [ ] Are you compressing context, or dumping everything retrieved? [ ] Do you track input vs. output tokens separately? [ ] Is your system prompt fighting your retrieved context? ✅ Request lifecycle [ ] Can you see latency broken down by stage (embed / retrieve / generate)? [ ] Do you know which stage is your actual bottleneck? ✅ Cache lifecycle [ ] Are you caching query embeddings? [ ] Are you caching full responses for repeated questions? ✅ Evaluation lifecycle [ ] Can you measure retrieval precision/recall? [ ] Do you have a faithfulness or answer-relevance check? [ ] If you "improved" something, can you prove it? ✅ Production lifecycle [ ] Health checks, retries, rate limiting — in place or assumed? [ ] Are secrets actually out of your codebase? [ ] Do you have CI/CD, or are you deploying by hand? ✅ Cloud lifecycle [ ] Do y

I spent two months building Vestibule, an open-source Python framework for the boring layer of RAG ingestion — stable document IDs, a state ledger, error classification, per-vertical governance. The parts every team struggles with once the demo works and production doesn't. Most of the code wasn't typed by me. Four AI agents did the work — one wrote designs, one reviewed them, one implemented, one reviewed the code — all through real GitHub pull requests, with me signing off at every gate. The result: twelve components, three releases, 878 tests. Two moments defined the whole experience. When the process caught what I couldn't The trickiest component provisions vector indexes on first use, safely even when workers race each other. Its design was rejected and revised five times before any code existed. In the first round, the reviewer agent found a genuine race condition: a worker still inside a slow index-creation call (~390 seconds with retries) would look stale (the threshold defaulted to 300 seconds), lose its claim to a waiting worker, and now two workers create the same index. A production race, in the default configuration, spotted by one AI reading another AI's design — before a single line was written. When green tests lied to me After v0.2 shipped, I wrote a quickstart script and ran the pipeline the way a stranger would — for the first time. pip install didn't work. At all. A packaging conflict made the whole framework uninstallable, while 483 tests sat green. An hour of actually using it turned up two more: a default model name that had never once worked against the real SDK, and an import that took down an entire package when an optional dependency was absent. What went wrong wasn't the tests — it was what they measured. They proved the code agreed with itself: same working tree, same mocked seams. Nothing ever checked the world a user lives in: clean machine, real install, real SDK. Passing tests and a working product turn out to be two different claims

What's the difference? An AI agent is a loop-driven system that can decide which tool to call next, keep state across interactions, and adapt its behaviour. An automation is a fixed sequence of steps that runs the same way every time. In this guide you'll build both a plain n8n workflow that sends a prompt to OpenAI and stores the answer, and a full RAG-enabled AI agent that decides when to fetch documents, when to query the LLM, and when to respond. By the end you'll see why most teams over-engineer, and you'll have a production-ready example you can ship tomorrow. Key insight: If your use-case requires conditional tool use, memory, or dynamic goal-setting, you need an AI agent; otherwise a straight automation is cheaper, faster, and easier to maintain. What you need Tool Plan / Price Role n8n (open-source workflow engine) Community edition (self-hosted, free) - see https://n8n.io/pricing for hosted options Orchestrates both automation and agent pipelines OpenAI API (ChatGPT/GPT-4) Pay-as-you-go - see https://openai.com/api/pricing Generates natural-language responses Pinecone (vector store) Free tier or paid plan - see https://www.pinecone.io/pricing Holds document embeddings for RAG Docker (container runtime) Free Runs n8n locally or in CI Git (version control) Free Stores workflow definitions Estimated build time: ~4 hours for a complete agent (including embedding documents) and ~1 hour for the plain automation. Step-by-step build 1. Set up n8n locally # Pull the official n8n Docker image and start it on port 5678 docker run -d --name n8n \ -p 5678:5678 \ -e N8N_BASIC_AUTH_ACTIVE = true \ -e N8N_BASIC_AUTH_USER = admin \ -e N8N_BASIC_AUTH_PASSWORD = secret \ n8nio/n8n What this does: launches a self-hosted n8n instance with basic auth. After a few seconds open http://localhost:5678 and log in with the credentials above. 2. Create the plain automation workflow In the n8n UI, click New Workflow . Add a Webhook node (trigger URL: /automation ). This receives a JSON

If you are building AI-powered test automation, you may eventually run into this question: Should we use RAG or MCP? The question sounds reasonable, but it is slightly misleading. RAG and MCP solve very different problems. In testing, you will probably need both. The Problem With AI-Generated Tests LLMs can already generate Selenium, Cypress, and Playwright tests from natural-language prompts. Ask: Test the login flow with valid credentials. and an AI can produce a reasonable script. But there is a problem. The AI does not automatically know: Your actual business rules Existing test cases Previous defects Test data API behaviour High-risk workflows Team-specific automation standards It knows how testing works , but not necessarily how your product works . That is where RAG becomes useful. What RAG Actually Solves RAG gives the AI access to project-specific information. Instead of working from a generic prompt, the model can retrieve relevant: Requirements Test Cases API Docs Bug History Business Rules Existing Automation Test Data Now consider the same request: Test the checkout flow. Without RAG, the AI may create a fairly standard checkout process. With RAG, it could first learn: Which payment methods are supported Whether guest checkout is allowed Which validations are required Which checkout bugs appeared previously Which scenarios already exist The generated test becomes much more relevant. But there is still a limitation. Knowing what should happen does not mean the AI can actually test it. That Is Where MCP Comes In MCP gives an AI system access to external tools. For browser testing, that could mean allowing an AI agent to use Playwright capabilities to: Open Page ↓ Inspect UI ↓ Enter Data ↓ Click ↓ Observe Result ↓ Validate So the difference is simple: RAG gives the AI context. MCP gives the AI capabilities. Or even shorter: RAG = What does the AI know? MCP = What can the AI do? Why This Matters for Test Automation Imagine an AI receives this instruction: C

Security researchers at Wiz recently examined S3-compatible object storage services across six popular neoclouds, revealing significant security gaps compared to Amazon S3. While S3 has become the de facto standard for object storage, most services lack several of AWS's security protections. By Renato Losio

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The series checkpoint The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. What coverage percentage actually measures Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of b

2.6 Why Retrieval Is Necessary A Large Language Model only knows what was available during its training. If the user asks about something that changed after the model was trained, the model may not know the latest information. Instead of forcing the language model to guess, ACAI first determines whether external information is required. User Prompt │ ▼ Need External Knowledge? │ ┌────┴────┐ │ │ No Yes │ │ ▼ ▼ Continue Search Engine │ ▼ Document Ranking │ ▼ Source Selection │ ▼ Context Builder Internal Workflow The Retrieval Engine performs several sequential operations: Stage 1 — Query Generation The original user prompt is transformed into one or more optimized search queries. Example User Prompt Explain quantum computing. Generated Queries Quantum computing basics Quantum computing architecture Quantum algorithms Quantum hardware Instead of searching only once, multiple optimized queries increase the chance of retrieving relevant information. Stage 2 — Source Collection The retrieval system collects candidate documents. Possible sources include: • Internal Knowledge Base • Company Documentation • Scientific Papers • Technical Documentation • API Documentation • User Files • Vector Database Stage 3 — Ranking Not every document is equally useful. The Ranking Engine assigns a relevance score. Document A Score 96% Document B Score 91% Document C Score 72% Document D Score 44% Only the highest-quality documents move to the next stage. Stage 4 — Filtering The system removes: • Duplicate documents • Low-quality sources • Irrelevant information • Outdated documents The objective is to reduce noise before reasoning begins. 2.7 Context Optimization Layer One of the largest limitations of LLMs is the context window. Suppose the retrieval engine returns 500 pages. The model cannot efficiently process every page. Therefore ACAI introduces a Context Optimizer. Workflow 500 Pages ↓ Ranking ↓ Filtering ↓ Compression ↓ Important Facts ↓ LLM Instead of sending every token, only the

Ever looked at a pile of medicine bottles and wondered, "Is it actually safe to take these together?" Polypharmacy—the simultaneous use of multiple drugs—is a significant challenge in modern healthcare. Misunderstanding Drug-Drug Interactions (DDI) can lead to severe side effects or reduced efficacy. In this tutorial, we are building an AI Pharmacist Assistant , an automated engine that uses Optical Character Recognition (OCR) to scan drug labels and Retrieval-Augmented Generation (RAG) to cross-reference a drug database. By leveraging AI healthcare automation and sophisticated LLM reasoning , we can create a safety net that identifies potential contraindications in seconds. The Architecture 🏗️ The system follows a linear pipeline: capturing raw image data, converting it to structured text, retrieving medical facts from a local SQLite-based knowledge base, and finally, using an LLM to reason about the interactions. graph TD A[Drug Packaging Image] -->|Tesseract OCR| B(Extract Drug Names) B --> C{Search SQLite DB} C -->|Found Interaction Data| D[Context Construction] D --> E[LLM Reasoning Engine] E --> F[Safety Report & Warnings] C -->|Not Found| G[Web Search/LLM General Knowledge] G --> E Prerequisites 🛠️ To follow along, you'll need the following tech stack: Python 3.10+ Tesseract OCR : For extracting text from images. SQLite : To store our curated DrugBank-style interaction data. RAG Pattern : To provide the LLM with ground-truth medical data. OpenAI SDK : For the final reasoning step. Step 1: Extracting Labels with OCR 📸 First, we need to turn those pixels into text. We use pytesseract to handle the OCR process. import pytesseract from PIL import Image def extract_drug_names ( image_path ): # Pre-processing could be added here (grayscale, thresholding) text = pytesseract . image_to_string ( Image . open ( image_path )) # In a real scenario, use an LLM or Regex to pull specific # active ingredients from the raw text print ( f " Detected Text: { text } " ) return t

Companies including Tesla and Base Power are vying for a piece of the rapidly growing market for home batteries. One technology has made it all possible.

OpenAI has cut prices on its frontier models again. If you're running any production workload on the API, your cost assumptions from six months ago are probably stale. The Real Impact of a Pricing Halving A 50% price cut sounds like pure good news, but it changes the calculus on decisions you already made. Projects you shelved because the token costs didn't pencil out deserve a second look. Architectures you built around cheaper, less capable models to save money may now be false economies - the cost gap between "good enough" and "best available" just got smaller. The more interesting shift is for teams running retrieval-augmented generation (RAG) pipelines - systems that pull relevant documents from a database at query time and feed them into the model as context. RAG workflows tend to be token-heavy because every retrieved chunk counts against your input token bill. At the old pricing, teams were aggressively trimming context windows and limiting retrieved chunks to stay within budget. At half the cost, you can retrieve more, keep longer context, and let the model reason over richer information - without changing a line of retrieval logic. Real Example Here's a simplified cost check you can drop into any project that calls the OpenAI API: import openai # Approximate pricing per 1M tokens (check platform.openai.com for current rates) INPUT_COST_PER_1M = 2.50 # update to current figure OUTPUT_COST_PER_1M = 10.00 # update to current figure def estimate_cost ( input_tokens : int , output_tokens : int ) -> float : return ( input_tokens / 1_000_000 * INPUT_COST_PER_1M + output_tokens / 1_000_000 * OUTPUT_COST_PER_1M ) # Example: a RAG call with 3,000 input tokens and 500 output tokens print ( f " Estimated cost per call: $ { estimate_cost ( 3000 , 500 ) : . 5 f } " ) # Run this across your monthly volume to see the real delta Multiply that per-call number by your actual monthly call volume and compare it against what you budgeted. For many teams, the difference will jus

We started building OliverGraph to give teams and their AI agents shared context across GitHub, Slack, docs, and the other places where work happens. At first, we thought the main problem was retrieval. Company knowledge is scattered across GitHub, Slack, docs, tickets, and people so we could connect those systems and get the right context to the agent when needed. But we ran into another problem. Agent runs were becoming another place where important context lived. An agent also gets context directly from the engineer using it. An engineer might tell an agent that the team tried something before, that a customer depends on a certain behavior, or that there's a constraint that isn't documented anywhere else. The agent uses that context while doing the work, but when the run ends, it can disappear with it. The next engineer's agent may see the resulting code without knowing what context was given to the previous agent. This gets messier during outages. Several engineers might be investigating at once with their own agents. One agent rules out a recent deploy while another discovers an issue with a database query. Those findings are now spread across separate agent sessions, and another agent might spend time investigating something that was already ruled out. Humans already deal with this Companies have fragmented context. One engineer remembers an old outage and another engineer remembers that the team already tried an approach but abandoned it. So we ask each other. Who worked on this? Why is this here? Didn’t we try this already? You might not know the answer, but you know John worked on that part of the system. John remembers the PR and the PR points to an incident. People slowly build a mental map of where all that context lives in the company. Agents don’t have that. As everyone starts using more agents, it becomes harder for humans too. My agent may be changing onboarding while your agent is modifying authentication. Another teammate’s agent may have just disc

This is the data appendix for Posts 1–4 . The narrative and takeaways live in the main posts. This page is pure measurement. Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward) Corpus evolution: P1/P2: ~3,700 docs (raw threads + code chunks) P3/P4: 16,315 docs (distilled threads + bursts + code) Quick comparison (same 31-question set) Metric Vector P2 Hybrid P2 Vector P3 Hybrid P3 Hybrid + Rerank (P4) recall@1 0.68 0.61 0.52 0.39 0.87 recall@3 0.84 0.65 0.71 0.65 0.94 recall@10 0.90 0.90 0.81 0.94 0.94 MRR 0.77 0.67 0.63 0.57 0.90 Takeaway: Hybrid alone never beat pure vector on this corpus. Hybrid + LLM rerank is the first clear win. P1 — Naive vector baseline Corpus: 3,000 raw issue threads + 687 code chunks Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine Numbers (22 questions) Metric Score recall@10 1.00 (22/22) recall@3 0.95 recall@1 0.77 (17/22) Main k=1 misses Exact error pastes ( TypeError: Object of type int64... , AttributeError: 'Depends'... ) — ranked 4–5 instead of 1 jsonable_encoder code chunk outranked by issues about the function API key header implementation (code vs similar issues) Paraphrase questions (dependency injection outside routes, custom 404) Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal. Ops notes Ingest wall time ~40 min (GitHub API is the bottleneck) BGE-M3 OOM on Apple Silicon fixed by capping max_seq_length=1024 Python 3.13 + uv editable install issue fixed by pinning 3.12 P2 — Hybrid (vector + FTS + RRF) Corpus: Same size as P1, with better comment pagination and symbol-based code IDs Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers) Numbers Metric Vector FTS Hybrid recall@1 0.68 0.42 0.61 recall@3 0.84 0.48 0.65 recall@10 0.90 0.65 0.90 MRR 0.77 0.47 0.67 Headline: Hybrid is not a strict win over vector-only. Where hybrid helped Exact error pastes (e.g. TypeError: int64 is not JSON serializable ) → moved from rank 5 → 1 Near-d

This English version is an AI translation of my original article on Qiita (in Japanese) . Background I've been experimenting with Zenoh via its Elixir bindings, Zenohex , not for its usual pub/sub use case but for its put / get storage feature. It mostly worked, except every so the state I picked back up was one step behind. Digging into why turned into a fun rabbit hole, so here's the writeup. Reproducing it To keep things simple, strip out the GenServer part entirely and just loop put immediately followed by get on the same key: { :ok , session_id } = Zenohex . Session . open ( config ) Enum . each ( 1 .. 2000 , fn i -> payload = Integer . to_string ( i ) :ok = Zenohex . Session . put ( session_id , key , payload ) { :ok , replies } = Zenohex . Session . get ( session_id , key , 3_000 , consolidation: :latest ) case Enum . find ( replies , & match? (% Zenohex . Sample {}, &1 )) do % Zenohex . Sample { payload: ^ payload } -> :ok % Zenohex . Sample { payload: other } -> IO . puts ( "stale! put #{ payload } but got #{ other } " ) nil -> IO . puts ( "no reply at all" ) end end ) Out of 2000 iterations, a small fraction print stale! — about 78 (3.9%) in one run. The interesting part: querying again immediately afterward almost always returns the correct value (the fastest I measured was a single extra get about 1ms later). So it's not that the value disappears — there's just a small window of lag before the write is actually visible. Why Zenohex.Session.put/4 is a thin Rustler wrapper around zenoh-rust's put . Looking at the NIF implementation : fn session_put ( ... ) -> rustler :: NifResult < rustler :: Atom > { ... publication_builder .apply_opts ( opts ) ? .wait () // <- only waits for the local publish to be queued ... Ok ( rustler :: types :: atom :: ok ()) } .wait() only waits for the local session to finish handing the message off — not for the remote side (the zenohd router backing the storage) to actually receive and apply it. session_get , on the other hand,