标签：#rag

How RAGScope Knows Which Chunks Your LLM Actually Used Your retriever fetched 10 chunks. Your LLM only used 3. RAGScope shows a precision score of 30 out of 100. The question every new user asks: how does it know? There is no OpenTelemetry attribute that says "this chunk was in the context window." RAGScope infers it — and the way it does this is the most consequential piece of engineering in the whole tool. There Is No "In Context" Attribute in OTel The OpenTelemetry semantic conventions for generative AI ( gen_ai.* ) define attributes for model, input/output tokens, and retrieved documents. They do not define anything like gen_ai.chunk.reached_llm or gen_ai.retrieval.used_document_ids . When your RETRIEVER span fires, you get a list of documents. When your LLM span fires, you get a prompt and a completion. The two spans are connected by a parent-child trace relationship — but there is no attribute that maps which retrieved documents appear in which prompt. This gap matters. A reranker might drop 7 of your 10 chunks. Your application code might apply a token budget and truncate 4 more. From the trace alone, you cannot tell. RAGScope needs this information to compute the precision sub-score — the highest-weighted metric at 40% of the overall score. Getting it wrong would make precision meaningless. The Substring Match — How assembleContext Works RAGScope's answer is in src/enrichment/pipeline.ts , in a function called assembleContext : function assembleContext ( chunks : RagChunk [], llmSpans : ParsedSpan []): RagChunk [] { const llmPrompts = llmSpans . map (( s ) => s . prompt ). filter (( p ): p is string => !! p ); if ( llmPrompts . length === 0 ) return chunks ; let position = 0 ; return chunks . map (( chunk ) => { if ( ! chunk . content ) return chunk ; const inContext = llmPrompts . some (( p ) => p . includes ( chunk . content ! )); if ( inContext ) { return { ... chunk , inContext : true , contextPosition : position ++ }; } return { ... chunk , inContext :

AI applications are moving quickly from simple chatbots to systems that can search, reason, recommend, summarize, and act on live business data. For developers, that usually means wiring together databases, embedding models, vector search, rerankers, orchestration logic, and application code. For no-code AI builders, it often means waiting for those integrations to exist before an idea can become a working prototype. The MongoDB extensions for Dify help close that gap. With the new MongoDB Atlas and Voyage AI extensions, Dify builders can visually compose AI workflows and agents that connect directly to MongoDB data, perform semantic retrieval with Atlas Vector Search, improve result quality with Voyage AI embeddings and reranking, and optionally interact with operational documents through controlled database tools. The result is a practical path from idea to working AI application: less custom orchestration code, more reusable building blocks, and a smoother experience for both developers and no-code builders. Why Dify and MongoDB Belong Together Dify provides a visual environment for building AI apps, workflows, and agents. It makes it easy to connect user input, model calls, tools, prompts, and outputs into a working application. MongoDB Atlas provides the data foundation: flexible documents, operational queries, aggregation, full-text search, and vector search in one platform. Together, they create a powerful pattern: Dify orchestrates the AI experience — workflows, agents, prompts, tools, and user interactions. MongoDB Atlas stores and retrieves the data — documents, application records, knowledge sources, and vector embeddings. Voyage AI improves retrieval quality — embeddings for semantic search and reranking for precision. For a no-code builder, this means you can assemble a retrieval-augmented generation workflow visually. For a developer, it means the integration points are packaged as reusable Dify tools rather than one-off glue code. Meet the Extensions

I once submitted an essay with three citations that I hadn't personally verified. The AI had suggested them, and they sounded right. None of them existed. That's not a quirk or a bug — it's exactly how LLMs work. And once you understand why, a technique called RAG starts to make a lot of sense. AI assistants are remarkably good at sounding right. The model isn't lying — it's doing its best with what it knows. The problem is that what it knows has limits, and it doesn't always know where those limits are. Ask one about a recent event, a niche regulation, or anything from a source it's never seen — and it fills the gap anyway. Confidently. That's the gap RAG was built to close. Once you understand how it works, you'll have a much clearer picture of why some AI tools are genuinely reliable and others are just very convincing guessers. Here's what's actually going on. First, What's the Problem? Large language models (LLMs)—the technology powering AI assistants like ChatGPT and Claude—are trained on vast amounts of data from across the internet. That training gives them a remarkable ability to reason, summarize, and generate content. But it also comes with some real limitations: They have a knowledge cutoff. An LLM trained last year doesn't know what happened last month. They can hallucinate. When they don't know something, they don't say "I don't know"—they generate a confident-sounding answer anyway. Wrong facts, fake statistics, invented sources. All delivered with a straight face. They don't know your specific sources. Think of a software engineer asking an AI assistant about their company's internal API documentation, deployment runbooks, or architecture decisions. None of that is in the training data. The model has never seen it — and it will still try to answer. The model isn't lying — it's generating the most plausible answer it can. It just has no way to know when it's wrong. So, what do you do when you need an AI that's accurate, current, and knows your specifi

Most RAG tutorials I found were either "pip install langchain and you're done" or 50-page academic papers. I wanted something in between — a pipeline I could actually explain in an interview, where I understood every line. So I built one from scratch. No LangChain, no LlamaIndex, no frameworks. Just FastAPI, FAISS, sentence-transformers, and an LLM API. Here's what I built, what worked, and what broke. The architecture PDF --> extract text (pypdf) --> chunk (500 char, 50 overlap) --> embed (MiniLM-L6-v2) | v question --> embed --> FAISS top-k search --> build prompt with chunks --> LLM --> answer + sources Five Python files, ~300 lines total: File Responsibility main.py FastAPI app, 3 endpoints, prompt engineering pdf_loader.py PDF text extraction via pypdf rag.py Chunking + embedding store.py FAISS vector store wrapper llm.py Swappable LLM client (Groq / OpenAI / Anthropic) How the upload works When you POST a PDF to /upload , three things happen: 1. Text extraction — pypdf reads each page and returns the raw text. Pages with no extractable text (scanned images) are skipped. 2. Chunking — each page is split into ~500-character chunks with 50 characters of overlap. The overlap prevents losing context at chunk boundaries. CHUNK_SIZE = 500 CHUNK_OVERLAP = 50 def chunk_pages ( pages ): chunks = [] chunk_id = 0 for text , page_num in pages : start = 0 while start < len ( text ): end = min ( start + CHUNK_SIZE , len ( text )) chunk_text = text [ start : end ]. strip () if chunk_text : chunks . append ( Chunk ( chunk_id = chunk_id , text = chunk_text , page = page_num )) chunk_id += 1 if end == len ( text ): break start = end - CHUNK_OVERLAP return chunks 3. Embedding — each chunk is embedded into a 384-dimensional vector using all-MiniLM-L6-v2 . This runs locally on CPU, no API call needed. Vectors are normalized so we can use inner product as cosine similarity. def embed_texts ( texts ): model = get_embed_model () # lazy-loaded singleton vectors = model . encode ( texts

The .txt File as the Soul of a Personal AI — FileRAG Memory Architecture By Dharanidharan J (JD) Full Stack & AI Engineer | Building Jarvix The Problem Nobody Talks About Every chatbot tutorial teaches you the same thing: history = [] history . append ({ " role " : " user " , " content " : message }) And that works — until it doesn't. After 500 turns, your dict has forgotten who the user is. After 1000 turns, you're hitting token limits. After a restart, everything is gone. Redis helps with persistence but still buries early facts under noise. Vector DBs help with retrieval but bloat storage and need infrastructure. What if the memory itself was just a file? The Idea Every conversation a user has gets distilled into a plain .txt file. That file is the brain. On every new query, a hybrid BM25 + semantic RAG retrieves the most relevant chunks from it and injects them as context. users/ └── jd.txt ← the soul file The soul file looks like this: [Turns 1-5] - User's name is JD, software engineer - Building FileRAG, a novel memory architecture - Uses Pop!_OS with Fish shell and NVIDIA GPU [Turns 6-10] - Has a cat named Pixel who distracts during coding - Paused TaskNest due to burnout - Now focused on AgenticMesh Human readable. Editable. Yours. Why This Is Different Most memory systems store messages . FileRAG stores a relationship . System What it stores Dict / Redis Raw message objects Vector DB Embeddings of messages FileRAG Distilled understanding of the user The longer you use it, the more the AI understands you — not because it has more messages, but because it has a better summary of who you are. The Architecture User message ↓ Topic drift check (cosine similarity) ├── Drift detected → distill current buffer immediately └── No drift → continue ↓ Hybrid retrieval (BM25 + ChromaDB) from soul file ↓ Inject context → LLM responds ↓ Append to turn buffer ↓ Every 5 turns → distill → append to soul file → update ChromaDB ↓ Emergency distillation on exit (SIGINT/SIGTERM)

"The crown is a weight that crushes. You'll do things that spell death for all involved."

"The output filter runs after the LLM has already seen the confidential data. By then, three classes of leak can no longer be stopped. The right surface is retrieval. Walking through a real implementation." TL;DR Most RAG-with-RBAC stacks I see in production put the access-control gate at the output stage: an LLM-response post-filter that masks PII or redacts confidential strings. This is defense-in-depth, not the load-bearing layer. By the time the filter runs, the LLM has already received the confidential context, and three classes of leak — creative paraphrasing, inference, cross-turn persistence — can no longer be stopped by string-matching the output. The protective surface that actually carries the weight is retrieval-stage ABAC: documents and graph nodes the user can't read are never traversed, never make it into the prompt, never seen by the model. The output filter still belongs in the stack, but as the second-to-last line, not the first. This post is a walk through why and how, with code references from a working implementation. It was prompted by a 6-turn LinkedIn DM exchange with Ali Afana (Provia founder, dev.to Featured) on injection-fixture schema design, where the framing crystallized. The seductive default You build a RAG system. You have documents at different sensitivity levels — public, internal, confidential. You want the model to answer based on whichever documents the user is allowed to see. The default mental model: "I'll let the model answer freely, and then I'll filter the response on the way out." This is appealing because: The retrieval pipeline stays simple (one query, one vector search, one response) The access control feels surgical (just before the user, just before damage) The PII-mask vocabulary is well-established (Presidio, regex catalogs, named-entity recognition models) So you wire up something like: Python The seductive default def answer(query, user): chunks = retrieve(query, top_k=10) # No ABAC here context = "\n".join(c.text

RAG SOTA, Agent Harnessing, and Langfuse Observability for AI Frameworks Today's Highlights Today's top stories delve into optimizing RAG performance with open-source benchmarks, designing robust AI agent systems, and implementing best practices for LLM observability in production. RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source) (Dev.to Top) Source: https://dev.to/__2ddbae6bb7d/--5cec This article presents a comprehensive benchmark of seven Retrieval-Augmented Generation (RAG) pipelines, culminating in the development and open-sourcing of SEQUOIA, a new RAG system. The author details over 20 hours of compute time spent locally to rigorously test different RAG configurations against real-world tasks, providing valuable insights into their performance characteristics. The technical deep dive includes discussions on various components like chunking strategies, embedding models, vector databases, and re-rankers, along with their impact on retrieval quality and generation coherence. Readers gain an understanding of the trade-offs involved in designing effective RAG systems and the empirical evidence supporting different architectural choices. The release of SEQUOIA as an open-source project means developers can directly implement and experiment with a battle-tested RAG pipeline, offering a tangible starting point for their own projects. Comment: This is an invaluable resource for anyone building RAG. Benchmarking 7 pipelines and open-sourcing a well-performing one provides immediate practical value and a solid foundation for further experimentation. Stop Upgrading the Model. Start Engineering the Harness. (Dev.to Top) Source: https://dev.to/tacoda/stop-upgrading-the-model-start-engineering-the-harness-194 This insightful article argues that instead of solely focusing on larger or "better" base models, teams should invest in "engineering the harness" around their AI agents to improve performance. The author highlights that the supporting architecture—compri

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source) After 20+ hours of compute time on local hardware, I benchmarked 7 RAG configurations against real-world tasks. SEQUOIA (RAPTOR tree + step-back prompting) consistently outperformed alternatives. The Full Pipeline List Method Core Approach No-RAG Direct LLM generation Classical RAG Dense retrieval (BGE-small + FAISS) Hybrid RAG BM25 + Dense + RRF + reranker LightRAG Key-value graph + dense hybrid PageIndex Two-stage hierarchical retrieval GraphRAG Entity graph + dense fallback Agentic RAG Multi-step reasoning pipeline SEQUOIA RAPTOR tree + step-back prompting SEQUOIA Pro Multi-query + rerank + compression Why LightRAG Underperformed The hype suggested graph-based RAG would revolutionize retrieval. On real banking documents and technical manuals: Graph construction is expensive (entity extraction, relationship mapping) Retrieval quality did not justify the overhead Academic benchmarks do not equal production reality Why RAPTOR Works Recursive Abstractive Processing for Tree-Organized Retrieval: Cluster leaf nodes (individual chunks) Summarize upward (hierarchical abstraction) Retrieve at multiple levels (specific details + high-level context) This mirrors how humans organize knowledge. Step-Back Prompting: Free Performance Before retrieving, generalize the query: User asks: "What's the error rate for Q3?" Step-back: "What metrics are tracked quarterly?" Retrieve broader context first, then narrow Result: ~15% improvement in recall. Zero latency cost. SEQUOIA Architecture User Query Step-back Prompting (generalize) RAPTOR Tree Retrieval (multi-level) Context Compression (summarize long contexts) Re-ranking (cross-encoder) Local LLM Generation Local LLM Evaluation I used a local model weaker than GPT-4 for judging. Key finding: relative rankings between methods stayed consistent even with a weaker evaluator. You can prototype and compare approaches without burning API credits on GPT-4 evaluations. Productio

Problem Statement We have a misinformation problem. But more specifically, we have a speed problem. A journalist spots a suspicious claim. They search for sources. Cross-reference databases. Call experts. Write a verdict. Get it edited. Publish, maybe 6 hours later. Maybe 3 days later. Meanwhile, the original claim has been screenshot, reposted, quoted in newsletters, and cited in arguments across five platforms. I wanted to build something that closed that gap. Not a chatbot that guesses. A proper pipeline, one that retrieves real evidence, reasons from it, and tells you why it reached a verdict. That's what Sift is. What is Sift? Sift (Source Inspection & Fact-checking Tool) is an open-source multi-agent AI pipeline that takes any text, extracts every factual claim, retrieves grounded evidence, and returns auditable verdicts — TRUE, FALSE, or UNCERTAIN, with cited sources and full reasoning chains. Paste a news article. A politician's speech. A viral statistic. A WhatsApp forward. Sift breaks it into individual claims and fact-checks each one independently. Why Multi-Agent? The naive approach is to ask an LLM: "Is this claim true?" The problem: LLMs hallucinate. They have knowledge cutoffs. They're confidently wrong in ways that are hard to detect. And critically, they don't show their work. A single LLM call can't reliably handle the full pipeline of: Extracting structured claims from noisy text Retrieving dated, traceable evidence from live sources Reasoning across conflicting evidence without confabulating Adversarially reviewing its own conclusions for overconfidence Finding corrections when something is wrong Each of these is a distinct task that benefits from its own prompt, its own tools, and its own failure modes. That's why I built five separate agents, orchestrated with LangGraph. The 5-Agent Pipeline Agent 1 — Claim Extractor A single paragraph can contain 4-5 distinct factual claims. Generic LLMs miss them or conflate them. This agent uses LLaMA 3.3 70

I built a bot trained on my own X bookmarks and likes. Around 50,000 of them, accumulated over years...